scholarly journals Changing the logic of replication: A case from infant studies

2019 ◽  
Author(s):  
Francesco Margoni ◽  
Martin Shepperd
Keyword(s):  
Statistical Power ◽  
Sampling Error ◽  
Meta Analysis ◽  
Statistical Significance ◽  
Small Sample ◽  
Infant Research ◽  
Replication Studies ◽  
Wide Range ◽  
Small Sample Sizes ◽  
Meta Analyses

Infant research is making considerable progresses. However, among infant researchers there is growing concern regarding the widespread habit of undertaking studies that have small sample sizes and employ tests with low statistical power (to detect a wide range of possible effects). For many researchers, issues of confidence may be partially resolved by relying on replications. Here, we bring further evidence that the classical logic of confirmation, according to which the result of a replication study confirms the original finding when it reaches statistical significance, could be usefully abandoned. With real examples taken from the infant literature and Monte Carlo simulations, we show that a very wide range of possible replication results would in a formal statistical sense constitute confirmation as they can be explained simply due to sampling error. Thus, often no useful conclusion can be derived from a single or small number of replication studies. We suggest that, in order to accumulate and generate new knowledge, the dichotomous view of replication as confirmatory/disconfirmatory can be replaced by an approach that emphasizes the estimation of effect sizes via meta-analysis. Moreover, we discuss possible solutions for reducing problems affecting the validity of conclusions drawn from meta-analyses in infant research.

Clinical outcomes of cancer patients with COVID-19: A systematic review and meta-analysis.

2021 ◽  
Vol 39 (15_suppl) ◽  
pp. e18600-e18600
Author(s):  
Maryam Alasfour ◽  
Salman Alawadi ◽  
Malak AlMojel ◽  
Philippos Apolinario Costa ◽  
Priscila Barreto Coelho ◽  
...  
Keyword(s):  
Mortality Rate ◽  
Cancer Patients ◽  
Clinical Outcomes ◽  
Meta Analysis ◽  
Small Sample ◽  
Prediction Intervals ◽  
Invasive Ventilation ◽  
Newcastle Ottawa Scale ◽  
Small Sample Sizes ◽  
Meta Analyses

e18600 Background: Patients with coronavirus disease 2019 (COVID-19) and cancer have worse clinical outcomes compared to those without cancer. Primary studies have examined this population, but most had small sample sizes and conflicting results. Prior meta-analyses exclude most US and European data or only examine mortality. The present meta-analysis evaluates the prevalence of several clinical outcomes in cancer patients with COVID-19, including new emerging data from Europe and the US. Methods: A systematic search of PubMED, medRxiv, JMIR and Embase by two independent investigators included peer-reviewed papers and preprints up to July 8, 2020. The primary outcome was mortality. Other outcomes were ICU and non-ICU admission, mild, moderate and severe complications, ARDS, invasive ventilation, stable, and clinically improved rates. Study quality was assessed through the Newcastle–Ottawa scale. Random effects model was used to derive prevalence rates, their 95% confidence intervals (CI) and 95% prediction intervals (PI). Results: Thirty-four studies (N = 4,371) were included in the analysis. The mortality prevalence rate was 25.2% (95% CI: 21.1–29.7; 95% PI: 9.8-51.1; I 2 = 85.4), with 11.9% ICU admissions (95% CI: 9.2-15.4; 95% PI: 4.3-28.9; I 2= 77.8) and 25.2% clinically stable (95% CI: 21.1-29.7; 95% PI: 9.8-51.1; I 2 = 85.4). Furthermore, 42.5% developed severe complications (95% CI: 30.4-55.7; 95% PI: 8.2-85.9; I 2 = 94.3), with 22.7% developing ARDS (95% CI: 15.4-32.2; 95% PI: 5.8-58.6; I 2 = 82.4), and 11.3% needing invasive ventilation (95% CI: 6.7-18.4; 95% PI: 2.3-41.1; I 2 = 79.8). Post-follow up, 49% clinically improved (95% CI: 35.6-62.6; 95% PI: 9.8-89.4; I 2 = 92.5). All outcomes had large I 2 , suggesting high levels of heterogeneity among studies, and wide PIs indicating high variability within outcomes. Despite this variability, the mortality rate in cancer patients with COVID-19, even at the lower end of the PI (9.8%), is higher than the 2% mortality rate of the non-cancer with COVID-19 population, but not as high as what other meta-analyses conclude, which is around 25%. Conclusions: Patients with cancer who develop COVID-19 have a higher probability of mortality compared to the general population with COVID-19, but possibly not as high as previous studies have shown. A large proportion of them developed severe complications, but a larger proportion recovered. Prevalence of mortality and other outcomes published in prior meta-analyses did not report prediction intervals, which compromises the clinical utilization of such results.

The null-hypothesis significance-test procedure is still warranted

1998 ◽  
Vol 21 (2) ◽  
pp. 228-235 ◽  
Author(s):  
Siu L. Chow
Keyword(s):  
Null Hypothesis ◽  
Statistical Power ◽  
Meta Analysis ◽  
Statistical Significance ◽  
Test Procedure ◽  
Significance Test ◽  
Null Hypothesis Significance Test ◽  
Wide Range ◽  
Statistical Hypotheses ◽  
Alternative Hypotheses

Entertaining diverse assumptions about empirical research, commentators give a wide range of verdicts on the NHSTP defence in Statistical significance. The null-hypothesis significance-test procedure (NHSTP) is defended in a framework in which deductive and inductive rules are deployed in theory corroboration in the spirit of Popper's Conjectures and refutations (1968b). The defensible hypothetico-deductive structure of the framework is used to make explicit the distinctions between (1) substantive and statistical hypotheses, (2) statistical alternative and conceptual alternative hypotheses, and (3) making statistical decisions and drawing theoretical conclusions. These distinctions make it easier to show that (1) H0 can be true, (2) the effect size is irrelevant to theory corroboration, and (3) “strong” hypotheses make no difference to NHSTP. Reservations about statistical power, meta-analysis, and the Bayesian approach are still warranted.

scholarly journals Epigenome-wide meta-analysis of blood DNA methylation and its association with subcortical volumes: findings from the ENIGMA Epigenetics Working Group

2019 ◽  
Author(s):  
Tianye Jia ◽  
Congying Chu ◽  
Yun Liu ◽  
Jenny van Dongen ◽  
Evangelos Papastergios ◽  
...  
Keyword(s):  
Dna Methylation ◽  
Nucleus Accumbens ◽  
Large Scale ◽  
Biomarker Discovery ◽  
Association Studies ◽  
Meta Analysis ◽  
Hippocampal Volume ◽  
Small Sample ◽  
Small Sample Sizes ◽  
Meta Analyses

AbstractDNA methylation, which is modulated by both genetic factors and environmental exposures, may offer a unique opportunity to discover novel biomarkers of disease-related brain phenotypes, even when measured in other tissues than brain, such as blood. A few studies of small sample sizes have revealed associations between blood DNA methylation and neuropsychopathology, however, large-scale epigenome-wide association studies (EWAS) are needed to investigate the utility of DNA methylation profiling as a peripheral marker for the brain. Here, in an analysis of eleven international cohorts, totalling 3337 individuals, we report epigenome-wide meta-analyses of blood DNA methylation with volumes of the hippocampus, thalamus and nucleus accumbens (NAcc)—three subcortical regions selected for their associations with disease and heritability and volumetric variability. Analyses of individual CpGs revealed genome-wide significant associations with hippocampal volume at two loci. No significant associations were found for analyses of thalamus and nucleus accumbens volumes. Cluster-based analyses revealed additional differentially methylated regions (DMRs) associated with hippocampal volume. DNA methylation at these loci affected expression of proximal genes involved in learning and memory, stem cell maintenance and differentiation, fatty acid metabolism and type-2 diabetes. These DNA methylation marks, their interaction with genetic variants and their impact on gene expression offer new insights into the relationship between epigenetic variation and brain structure and may provide the basis for biomarker discovery in neurodegeneration and neuropsychiatric conditions.

scholarly journals Influence of Pilot and Small Trials in Meta-Analyses of Behavioral Interventions: A Meta-epidemiological Study

2020 ◽  
Author(s):  
Michael W. Beets ◽  
R. Glenn Weaver ◽  
John P.A. Ioannidis ◽  
Alexis Jones ◽  
Lauren von Klinggraeff ◽  
...  
Keyword(s):  
Sample Size ◽  
Behavioral Interventions ◽  
Meta Analysis ◽  
Statistical Significance ◽  
Feasibility Studies ◽  
Small Sample ◽  
Effect Sizes ◽  
Absolute Difference ◽  
Meta Analyses ◽  
The Impact

Abstract Background: Pilot/feasibility or studies with small sample sizes may be associated with inflated effects. This study explores the vibration of effect sizes (VoE) in meta-analyses when considering different inclusion criteria based upon sample size or pilot/feasibility status. Methods: Searches were conducted for meta-analyses of behavioral interventions on topics related to the prevention/treatment of childhood obesity from 01-2016 to 10-2019. The computed summary effect sizes (ES) were extracted from each meta-analysis. Individual studies included in the meta-analyses were classified into one of the following four categories: self-identified pilot/feasibility studies or based upon sample size (N≤100, N>100, and N>370 the upper 75th of sample size). The VoE was defined as the absolute difference (ABS) between the re-estimations of summary ES restricted to study classifications compared to the originally reported summary ES. Concordance (kappa) of statistical significance between summary ES was assessed. Fixed and random effects models and meta-regressions were estimated. Three case studies are presented to illustrate the impact of including pilot/feasibility and N≤100 studies on the estimated summary ES.Results: A total of 1,602 effect sizes, representing 145 reported summary ES, were extracted from 48 meta-analyses containing 603 unique studies (avg. 22 avg. meta-analysis, range 2-108) and included 227,217 participants. Pilot/feasibility and N≤100 studies comprised 22% (0-58%) and 21% (0-83%) of studies. Meta-regression indicated the ABS between the re-estimated and original summary ES where summary ES were comprised of ≥40% of N≤100 studies was 0.29. The ABS ES was 0.46 when summary ES comprised of >80% of both pilot/feasibility and N≤100 studies. Where ≤40% of the studies comprising a summary ES had N>370, the ABS ES ranged from 0.20-0.30. Concordance was low when removing both pilot/feasibility and N≤100 studies (kappa=0.53) and restricting analyses only to the largest studies (N>370, kappa=0.35), with 20% and 26% of the originally reported statistically significant ES rendered non-significant. Reanalysis of the three case study meta-analyses resulted in the re-estimated ES rendered either non-significant or half of the originally reported ES. Conclusions: When meta-analyses of behavioral interventions include a substantial proportion of both pilot/feasibility and N≤100 studies, summary ES can be affected markedly and should be interpreted with caution.

Effectiveness of Outdoor Exercise Parks on Health Outcomes in Older Adults—A Mixed-Methods Systematic Review and Meta-Analysis

2020 ◽  
pp. 1-13
Author(s):  
Yoke Leng Ng ◽  
Keith D. Hill ◽  
Pazit Levinger ◽  
Elissa Burton
Keyword(s):  
Physical Activity ◽  
Systematic Review ◽  
Older Adults ◽  
Meta Analysis ◽  
Small Sample ◽  
Activity Levels ◽  
Single Leg Stance ◽  
Small Sample Sizes ◽  
Meta Analyses

The objective of this systematic review was to examine the effectiveness of outdoor exercise park equipment on physical activity levels, physical function, psychosocial outcomes, and quality of life of older adults living in the community and to evaluate the evidence of older adults’ use of outdoor exercise park equipment. A search strategy was conducted from seven databases. Nine articles met the inclusion criteria. The study quality results were varied. Meta-analyses were undertaken for two physical performance tests: 30-s chair stand test and single-leg stance. The meta-analysis results were not statistically significant. It was not possible to conclude whether exercise parks were effective at improving levels of physical activity. The review shows that older adults value the benefits of health and social interaction from the use of exercise parks. Findings should be interpreted with caution due to the small sample sizes and the limited number of studies.

The Identity Status Continuum Revisited

2018 ◽  
Vol 23 (4) ◽  
pp. 289-299 ◽  
Author(s):  
Wim Meeus
Keyword(s):  
Identity Development ◽  
Identity Formation ◽  
Statistical Power ◽  
Developmental Psychology ◽  
Meta Analysis ◽  
Small Sample ◽  
Identity Status ◽  
Identity Statuses ◽  
Identity Maintenance ◽  
Small Sample Sizes

Abstract. The developmental continuum of identity status has been a topic of theoretical debate since the early 1980’s. A recent meta-analysis and recent studies with dual cycle models lead to two conclusions: (1) during adolescence there is systematic identity maturation; (2) there are two continuums of identity status progression. Both continuums show that in general adolescents move from transient identity statuses to identity statuses that mark the relative endpoints of development: from diffusion to closure, and from searching moratorium and moratorium to closure and achievement. This pattern can be framed as development from identity formation to identity maintenance. In Identity Status Interview research using Marcia’s model, not the slightest indication for a continuum of identity development was found. This may be due to the small sample sizes of the various studies leading to small statistical power to detect differences in identity status transitions, as well as developmental inconsistencies in Marcia’s model. Findings from this review are interpreted in terms of life-span developmental psychology.

scholarly journals Low statistical power in biomedical science: a review of three human research domains

2017 ◽  
Vol 4 (2) ◽  
pp. 160254 ◽  
Author(s):  
Estelle Dumas-Mallet ◽  
Katherine S. Button ◽  
Thomas Boraud ◽  
Francois Gonon ◽  
Marcus R. Munafò
Keyword(s):  
Effect Size ◽  
Statistical Power ◽  
Meta Analysis ◽  
Statistical Significance ◽  
Average Power ◽  
Biomedical Science ◽  
Significant Finding ◽  
Biomedical Sciences ◽  
True Effect Size ◽  
Meta Analyses

Studies with low statistical power increase the likelihood that a statistically significant finding represents a false positive result. We conducted a review of meta-analyses of studies investigating the association of biological, environmental or cognitive parameters with neurological, psychiatric and somatic diseases, excluding treatment studies, in order to estimate the average statistical power across these domains. Taking the effect size indicated by a meta-analysis as the best estimate of the likely true effect size, and assuming a threshold for declaring statistical significance of 5%, we found that approximately 50% of studies have statistical power in the 0–10% or 11–20% range, well below the minimum of 80% that is often considered conventional. Studies with low statistical power appear to be common in the biomedical sciences, at least in the specific subject areas captured by our search strategy. However, we also observe evidence that this depends in part on research methodology, with candidate gene studies showing very low average power and studies using cognitive/behavioural measures showing high average power. This warrants further investigation.

scholarly journals Are Difficult-To-Study Populations too Difficult to Study in a Reliable Way?

2020 ◽  
Vol 25 (1) ◽  
pp. 41-50 ◽  
Author(s):  
Florian Lange
Keyword(s):  
Statistical Power ◽  
Scientific Evidence ◽  
Statistical Significance ◽  
Neuropsychological Test ◽  
Small Sample ◽  
Selective Reporting ◽  
Test Procedures ◽  
Psychological Phenomena ◽  
Replication Studies ◽  
Methodological Choices

Abstract. Replication studies, pre-registration, and increases in statistical power will likely improve the reliability of scientific evidence. However, these measures face critical limitations in populations that are inherently difficult to study. Members of difficult-to-study populations (e.g., patients, children, non-human animals) are less accessible to researchers, which typically results in small-sample studies that are infeasible to replicate. Nevertheless, meta-analyses on clinical neuropsychological data suggest that difficult-to-study populations can be studied in a reliable way. These analyses often produce unbiased effect-size estimates despite aggregating across severely underpowered original studies. This finding can be attributed to a neuropsychological research culture involving the non-selective reporting of results from standardized and validated test procedures. Consensus guidelines, test manuals, and psychometric evidence constrain the methodological choices made by neuropsychologists, who regularly report the results from neuropsychological test batteries irrespective of their statistical significance or novelty. Comparable shifts toward more standardization and validation, complete result reports, and between-lab collaborations can allow for a meaningful and reliable study of psychological phenomena in other difficult-to-study populations.

scholarly journals Meta-Analysis of Microdissected Breast Tumors Reveals Genes Regulated in the Stroma but Hidden in Bulk Analysis

2021 ◽  
Author(s):  
Aurora Savino ◽  
Niccolò de Marzo ◽  
Paolo Provero ◽  
Valeria Poli
Keyword(s):  
Statistical Power ◽  
Molecular Mechanisms ◽  
Meta Analysis ◽  
Breast Tumors ◽  
Small Sample ◽  
Bulk Analysis ◽  
Small Sample Sizes ◽  
Microarray Datasets ◽  
User Friendly ◽  
Cellular Components

Background: transcriptome data provide a valuable resource for the study of cancer molecular mechanisms, but technical biases, samples’ heterogeneity and small sample sizes result in poorly reproducible lists of regulated genes. Additionally, the presence of multiple cellular components contributing to cancer development complicate the interpretation of bulk transcriptomic profiles. Methods: we collected 48 microarray datasets of laser capture microdissected breast tumors, and performed a meta-analysis to identify robust lists of genes differentially expressed in these tumors. We created a database with carefully harmonized metadata to be used as a resource for the research community. Results: combining the results of multiple datasets improved the statistical power, and the analysis of stroma and epithelium separately allows identifying genes with different contribution in each compartment. Conclusions: our database can profitably help biomarkers’ discovery and is readily accessible through a user-friendly web interface (https://aurorasavino.shinyapps.io/metalcm/).

Jackson and Schuler (1985) Revisited: A Meta-Analysis of the Relationships Between Role Ambiguity, Role Conflict, and Job Performance

2000 ◽  
Vol 26 (1) ◽  
pp. 155-169 ◽  
Author(s):  
Travis C. Tubre ◽  
Judith M. Collins
Keyword(s):  
Job Performance ◽  
Role Conflict ◽  
Role Ambiguity ◽  
Meta Analysis ◽  
Negative Relationship ◽  
Small Sample ◽  
Reliability Estimates ◽  
And Performance ◽  
Small Sample Sizes ◽  
Meta Analyses

We conducted a meta-analysis of correlations between role ambiguity and job performance and role conflict and job performance. Previous meta-analyses of these role constructs and performance relationships (e.g., Jackson & Schuler, 1985) were limited by small sample sizes and sparse reporting of reliability estimates in primary studies. The present study used a comprehensive database with a larger sample size and a distribution of interrater reliabilities to extend the previous findings. We also tested moderator hypotheses proposed but not conducted by Jackson and Schuler. Results revealed a negative relationship (r52.21) between role ambiguity and job performance with moderating influences due to job type and rating source. A negligible relationship (r52.07) was observed for role conflict and job performance, a finding consistent across job types and rating sources. Conclusions were that role ambiguity ought not to be dismissed as an unimportant variable in the job performance domain.

Sign in / Sign up

Export Citation Format

Share Document