Comparison between 2000 and 2018 on the reporting of statistical significance and clinical relevance in physiotherapy clinical trials in six major physiotherapy journals: a meta-research design

DesignMeta-research.ObjectiveTo compare the prevalence of reporting p values, effect estimates and clinical relevance in physiotherapy randomised controlled trials (RCTs) published in the years 2000 and 2018.MethodsWe performed a meta-research study of physiotherapy RCTs obtained from six major physiotherapy peer-reviewed journals that were published in the years 2000 and 2018. We searched the databases Embase, Medline and PubMed in May 2019, and extracted data on the study characteristics and whether articles reported on statistical significance, effect estimates and confidence intervals for baseline, between-group, and within-group differences, and clinical relevance. Data were presented using descriptive statistics and inferences were made based on proportions. A 20% difference between 2000 and 2018 was regarded as a meaningful difference.ResultsWe found 140 RCTs: 39 were published in 2000 and 101 in 2018. Overall, there was a high prevalence (>90%) of reporting p values for the main (between-group) analysis, with no difference between years. Statistical significance testing was frequently used for evaluating baseline differences, increasing from 28% in 2000 to 61.4% in 2018. The prevalence of reporting effect estimates, CIs and the mention of clinical relevance increased from 2000 to 2018 by 26.6%, 34% and 32.8% respectively. Despite an increase in use in 2018, over 40% of RCTs failed to report effect estimates, CIs and clinical relevance of results.ConclusionThe prevalence of using p values remains high in physiotherapy research. Although the proportion of reporting effect estimates, CIs and clinical relevance is higher in 2018 compared to 2000, many publications still fail to report and interpret study findings in this way.

Breast-cancer detection using blood-based infrared molecular fingerprints

Background Breast cancer screening is currently predominantly based on mammography, tainted with the occurrence of both false positivity and false negativity, urging for innovative strategies, as effective detection of early-stage breast cancer bears the potential to reduce mortality. Here we report the results of a prospective pilot study on breast cancer detection using blood plasma analyzed by Fourier-transform infrared (FTIR) spectroscopy – a rapid, cost-effective technique with minimal sample volume requirements and potential to aid biomedical diagnostics. FTIR has the capacity to probe health phenotypes via the investigation of the full repertoire of molecular species within a sample at once, within a single measurement in a high-throughput manner. In this study, we take advantage of cross-molecular fingerprinting to probe for breast cancer detection. Methods We compare two groups: 26 patients diagnosed with breast cancer to a same-sized group of age-matched healthy, asymptomatic female participants. Training with support-vector machines (SVM), we derive classification models that we test in a repeated 10-fold cross-validation over 10 times. In addition, we investigate spectral information responsible for BC identification using statistical significance testing. Results Our models to detect breast cancer achieve an average overall performance of 0.79 in terms of area under the curve (AUC) of the receiver operating characteristic (ROC). In addition, we uncover a relationship between the effect size of the measured infrared fingerprints and the tumor progression. Conclusion This pilot study provides the foundation for further extending and evaluating blood-based infrared probing approach as a possible cross-molecular fingerprinting modality to tackle breast cancer detection and thus possibly contribute to the future of cancer screening.

1409. Pulmonary Non-tuberculous Mycobacterium Infection (PNTMI) and COVID-19: Characterization of the National COVID Collaborative Cohort (N3C)

Background Establishing whether a low-prevalence clinical condition is a risk factor for COVID-19 infection, or serious adverse outcomes, is difficult due to a limited number of patients, and lack of access to patient’s data by researchers. The National COVID Collaborative Cohort (N3C), a centralized national data resource to study COVID-19, provides access to structured clinical data derived from electronic health records. As of June 2021, N3C contains data on 6,193,738 patients (2,090,138 with COVID-19, 33.7%) from 55 participating sites (Figure 1). We describe the characteristics of patients with PNTMI based on COVID-19 infection status. Figure 1 N3C Basic Demographic Data Methods COVID-19 is defined by positive lab result (PCR, antigen, or antibody) or COVID-19 coding diagnosis, as defined by N3C. PNTMI phenotype was built with N3C Data Enclave concept set tool, and ATLAS (https://atlas.ohdsi.org/). We limited analysis to adults (18 years-old or older). We used de-identified data sets stripped of protected health information (PHI). We used N3C Data Enclave analytical tools for exploratory data analysis, and descriptive statistics. Results We identified five hundred and eighty six individuals from 19 sites fulfilling the PNTMI phenotype (9.46 cases per 100,000 people). After our age limit, 555 individuals were included for analysis (Figure 2). 340 were females (61.3%), 447 of white race (80.5%), and 30 were Hispanic (5.4%). Additional descriptive statistics and statistical significance testing are provided (Table 1). The most common concept were "Non-tuberculous mycobacterial pneumonia", and "Pulmonary Mycobacterium avium complex infection". Four sites accounted for more than 50% of identified patients (Figure 2). We identified 24 individuals with COVID-19 (4.32%), and 44 deaths in this cohort (7.9%). Deaths were unrelated to COVID-19 event. Figure 2. Basic demographic data of pulmonary non-tuberculous Mycobacterium infection phenotype in N3C Figure 3. Concepts and data sources of pulmonary non-tuberculous Mycobacterium infection phenotype in N3C Conclusion In N3C, the PNTMI cohort has a lower proportion of COVID-19 infection than the general population, and it was not a cause of mortality. Further analysis to study impact of comorbidities, and differences in race and geographical location are warranted. N3C is a powerful research platform to study the impact of COVID-19 in special populations with low prevalence, and it can be used to study other populations of interest. Disclosures All Authors: No reported disclosures

Adaptive treatment allocation and selection in multi-arm clinical trials: a Bayesian perspective

Background: Adaptive designs offer added flexibility in the execution of clinical trials, including the possibilities of allocating more patients to the treatments that turned out more successful, and early stopping due to either declared success or futility. Commonly applied adaptive designs, such as group sequential methods, are based on the frequentist paradigm and on ideas from statistical significance testing. Interim checks during the trial will have the effect of inflating the Type 1 error rate, or, if this rate is controlled and kept fixed, lowering the power. Results: The purpose of the paper is to demonstrate the usefulness of the Bayesian approach in the design and in the actual running of randomized clinical trials during Phase II and III. This approach is based on comparing the performance of the different treatment arm in terms of the respective joint posterior probabilities evaluated sequentially from the accruing outcome data, and then taking a control action if such posterior probabilities fall below a pre-specified critical threshold value. Two types of actions are considered: treatment allocation, putting on hold at least temporarily further accrual of patients to a treatment arm (Rule 1), and treatment selection, removing an arm from the trial permanently (Rule 2). The main development in the paper is in terms of binary outcomes, but extensions for handling time-to-event data, including data from vaccine trials, are also discussed. The performance of the proposed methodology is tested in extensive simulation experiments, with numerical results and graphical illustrations documented in a Supplement to the main text. As a companion to this paper, an implementation of the methods is provided in the form of a freely available R package. Conclusion: The proposed methods for trial design provide an attractive alternative to their frequentist counterparts.

How to embrace variation and accept uncertainty in linguistic and psycholinguistic data analysis

The use of statistical inference in linguistics and related areas like psychology typically involves a binary decision: either reject or accept some null hypothesis using statistical significance testing. When statistical power is low, this frequentist data-analytic approach breaks down: null results are uninformative, and effect size estimates associated with significant results are overestimated. Using an example from psycholinguistics, several alternative approaches are demonstrated for reporting inconsistencies between the data and a theoretical prediction. The key here is to focus on committing to a falsifiable prediction, on quantifying uncertainty statistically, and learning to accept the fact that – in almost all practical data analysis situations – we can only draw uncertain conclusions from data, regardless of whether we manage to obtain statistical significance or not. A focus on uncertainty quantification is likely to lead to fewer excessively bold claims that, on closer investigation, may turn out to be not supported by the data.

Cumulative deviation of a subpopulation from the full population

Assessing equity in treatment of a subpopulation often involves assigning numerical “scores” to all individuals in the full population such that similar individuals get similar scores; matching via propensity scores or appropriate covariates is common, for example. Given such scores, individuals with similar scores may or may not attain similar outcomes independent of the individuals’ memberships in the subpopulation. The traditional graphical methods for visualizing inequities are known as “reliability diagrams” or “calibrations plots,” which bin the scores into a partition of all possible values, and for each bin plot both the average outcomes for only individuals in the subpopulation as well as the average outcomes for all individuals; comparing the graph for the subpopulation with that for the full population gives some sense of how the averages for the subpopulation deviate from the averages for the full population. Unfortunately, real data sets contain only finitely many observations, limiting the usable resolution of the bins, and so the conventional methods can obscure important variations due to the binning. Fortunately, plotting cumulative deviation of the subpopulation from the full population as proposed in this paper sidesteps the problematic coarse binning. The cumulative plots encode subpopulation deviation directly as the slopes of secant lines for the graphs. Slope is easy to perceive even when the constant offsets of the secant lines are irrelevant. The cumulative approach avoids binning that smooths over deviations of the subpopulation from the full population. Such cumulative aggregation furnishes both high-resolution graphical methods and simple scalar summary statistics (analogous to those of Kuiper and of Kolmogorov and Smirnov used in statistical significance testing for comparing probability distributions).

Graphical Data Analysis

Limitations in statistical significance testing are an issue of ongoing debate in many academic disciplines. Moving away from a reliance on their use in quantitative research is seen as an important step towards improving the quality of such research. One area that can help here is graphical data analysis, both as an exploratory and explanatory tool. This paper presents an overview of graphical techniques for quantitative data analysis. After outlining the rationale for the use of graphical data analysis, consideration of the appropriate types of graphs to use is provided. A number of useful graphs, created using the R statistical package, are introduced, along with a link to the full code to reproduce the examples. Suggestions are presented for how graphical techniques can help with both the exploration and conformation stages in research. 統計的有意差検定の限界は、多くの学術分野で継続的に議論されている問題である。量的研究における統計的有意差検定への使用依存から脱却することは、研究の質を向上させるための重要なステップであると考えられる。ここで役立つのが、探索的・説明的なツールとしてのグラフデータ分析である。本稿は、定量的なデータ分析のためのグラフ技術の概要を説明する。グラフィカルなデータ分析を使用する理由を説明した後、使用するグラフの適切な種類について検討する。R統計パッケージを用いて作成された便利なグラフの数々を紹介し、例題を再現するための完全なコードへのリンクも掲載する。本稿は、研究における探索と適合の両方の段階で、グラフ技術がどのように役立つかを提案するものである。

Demographic transformation and the influence of eleven selected socio-demographic variables on the face validity of an engineering English for specific purposes approach at a South African university

In English for Specific Purposes (ESP), as accountable education, face validity, which could be an undesirable “negative” or a desirable “positive”, is one of the ways in which we determine the learners’ attitudes toward, and probable consequent motivation for, ESP learning programmes. In this study, we sought to explain the reactions of the 226 first year Engineering student respondents to Likert items by their sociodemographic characteristics. We measured and classified as an undesirable “negative” (if the mean was < 3.50), or a desirable “positive” (if the mean was ≥ 3.50), the face validity generated by eleven sociodemographic characteristics for an ESP approach employed in the design and delivery of a compulsory ESP module (Engineering Communication) at a university in South Africa’s Gauteng Province. We subjected the data to statistical significance testing with the ANOVA suite of inferential statistics to identify statistically significant relationships between the sociodemographic variables and the face validity variables, of which 20 were confirmed, and then measured the variance (influence) in the latter that could be associated with the former. The aggregate influence explainable by the sociodemographic variables was an eta2 of .908 (90.8%), of which High School Type (19.8%) recorded the most, and Engineering Work Experience (1,7%), the least. Whereas, Race was associated with the most “negative” influence, the Black1 and White groups were practically indistinguishable in attaining unequal, but “negative”, scores. When we compared the sample demographic statistics to available institutional and national statistics to check for demographic transformation, the statistics suggested that the research university was transforming demographically at a fast pace, given its history.