scholarly journals Pooling resources to enhance rigour in psychophysiological research: Insights from open science approaches to meta-analysis

2021 ◽  
Author(s):  
Blair Saunders ◽  
Michael Inzlicht
Keyword(s):  
Statistical Power ◽  
Meta Analysis ◽  
Open Science ◽  
Quantitative Review ◽  
Primary Literature ◽  
Meta Analyses ◽  
Post Hoc ◽  
Traditional Approaches ◽  
Behavioural Sciences ◽  
Sufficient Statistical Power

Recent years have witnessed calls for increased rigour and credibility in the cognitive and behavioural sciences, including psychophysiology. Many procedures exist to increase rigour, and among the most important is the need to increase statistical power. Achieving sufficient statistical power, however, is a considerable challenge for resource intensive methodologies, particularly for between-subjects designs. Meta-analysis is one potential solution; yet, the validity of such quantitative review is limited by potential bias in both the primary literature and in meta-analysis itself. Here, we provide a non-technical overview and evaluation of open science methods that could be adopted to increase the transparency of novel meta-analyses. We also contrast post hoc statistical procedures that can be used to correct for publication bias in the primary literature. We suggest that traditional meta-analyses, as applied in ERP research, are exploratory in nature, providing a range of plausible effect sizes without necessarily having the ability to confirm (or disconfirm) existing hypotheses. To complement traditional approaches, we detail how prospective meta-analyses, combined with multisite collaboration, could be used to conduct statistically powerful, confirmatory ERP research.

scholarly journals P-P39 Perioperative Interventions to Reduce Pancreatic Fistula following Pancreaticoduodenectomy: A Systematic Review and Meta-Analysis

2021 ◽  
Vol 108 (Supplement_9) ◽  
Author(s):  
James Halle-Smith ◽  
Rupaly Pande ◽  
Lewis Hall ◽  
James Hodson ◽  
Keith J Roberts ◽  
...  
Keyword(s):  
Systematic Review ◽  
Pancreatic Fistula ◽  
Statistical Power ◽  
Fundamental Problem ◽  
Meta Analysis ◽  
New Directions ◽  
Randomised Controlled ◽  
Meta Analyses ◽  
Post Hoc ◽  
Pancreatic Duct Drainage

Abstract Background Many studies evaluate interventions to reduce POPF following PD, but often report conflicting results. Previous meta-analyses have generally included non-randomised trials and not considered novel interventions.  Aim To evaluate interventions to reduce postoperative pancreatic fistula (POPF) following pancreatoduodenectomy (PD) with level 1 data. Methods A systematic review and meta-analysis assessed randomised controlled trials (RCTs) evaluating interventions to reduce All-POPF or clinically relevant (CR)-POPF after PD. A post-hoc analysis of negative RCTs assessed whether these had appropriate levels of statistical power. Results Among 22 interventions (n = 7,512 patients, 55 studies), 12 were assessed by multiple studies, and subject to meta-analysis. Of these, external pancreatic duct drainage was the only intervention found to be associated with significantly reduced rates of CR- and all-POPF. In addition, Ulinastatin was associated with significantly reduced rates of CR-POPF, whilst invagination (vs duct to mucosa) pancreatojejunostomy was associated with significantly reduced rates of all-POPF. Review of negative RCTs found the majority to be underpowered, with post-hoc power calculations indicating that interventions would need to reduce the POPF rate to ≤ 1% in order to achieve 80% power in 16/34 (All-POPF) and 19/25 (CR-POPF) studies, respectively.   Conclusions Meta-analysis supports a role for several interventions to reduce POPF after PD, although data is often inconsistent and/or based on small trials. Systematic review identifies other interventions which may benefit from further study. However, underpowered trials appear to be a fundamental problem, inherently more so with CR-POPF. Larger trials, or new directions for research are required to further understanding in this field. 

Power-Enhanced Funnel Plots for Meta-Analysis

2020 ◽  
Vol 228 (1) ◽  
pp. 43-49 ◽  
Author(s):  
Michael Kossmeier ◽  
Ulrich S. Tran ◽  
Martin Voracek
Keyword(s):  
Statistical Power ◽  
Meta Analysis ◽  
Relevant Information ◽  
Selective Reporting ◽  
Graphical Display ◽  
Funnel Plot ◽  
Evidential Value ◽  
Graphical Displays ◽  
Funnel Plots ◽  
Meta Analyses

Abstract. Currently, dedicated graphical displays to depict study-level statistical power in the context of meta-analysis are unavailable. Here, we introduce the sunset (power-enhanced) funnel plot to visualize this relevant information for assessing the credibility, or evidential value, of a set of studies. The sunset funnel plot highlights the statistical power of primary studies to detect an underlying true effect of interest in the well-known funnel display with color-coded power regions and a second power axis. This graphical display allows meta-analysts to incorporate power considerations into classic funnel plot assessments of small-study effects. Nominally significant, but low-powered, studies might be seen as less credible and as more likely being affected by selective reporting. We exemplify the application of the sunset funnel plot with two published meta-analyses from medicine and psychology. Software to create this variation of the funnel plot is provided via a tailored R function. In conclusion, the sunset (power-enhanced) funnel plot is a novel and useful graphical display to critically examine and to present study-level power in the context of meta-analysis.

scholarly journals Partial derivatives meta-analysis: pooled analyses when individual participant data cannot be shared

2016 ◽  
Author(s):  
Hieab HH Adams ◽  
Hadie Adams ◽  
Lenore J Launer ◽  
Sudha Seshadri ◽  
Reinhold Schmidt ◽  
...  
Keyword(s):  
Gold Standard ◽  
Statistical Power ◽  
Large Scale ◽  
Meta Analysis ◽  
Scientific Evidence ◽  
Joint Analysis ◽  
Individual Participant Data ◽  
Partial Derivatives ◽  
Individual Participant ◽  
Post Hoc

Joint analysis of data from multiple studies in collaborative efforts strengthens scientific evidence, with the gold standard approach being the pooling of individual participant data (IPD). However, sharing IPD often has legal, ethical, and logistic constraints for sensitive or high-dimensional data, such as in clinical trials, observational studies, and large-scale omics studies. Therefore, meta-analysis of study-level effect estimates is routinely done, but this compromises on statistical power, accuracy, and flexibility. Here we propose a novel meta-analytical approach, named partial derivatives meta-analysis, that is mathematically equivalent to using IPD, yet only requires the sharing of aggregate data. It not only yields identical results as pooled IPD analyses, but also allows post-hoc adjustments for covariates and stratification without the need for site-specific re-analysis. Thus, in case that IPD cannot be shared, partial derivatives meta-analysis still produces gold standard results, which can be used to better inform guidelines and policies on clinical practice.

scholarly journals Investigation of Risk Of Bias due to Unreported and SelecTively included results in meta-analyses of nutrition research: the ROBUST study protocol

F1000Research ◽  
2019 ◽  
Vol 8 ◽  
pp. 1760 ◽  
Author(s):  
Matthew J. Page ◽  
Lisa Bero ◽  
Cynthia M. Kroeger ◽  
Zhaoli Dai ◽  
Sally McDonald ◽  
...  
Keyword(s):  
Meta Analysis ◽  
Scientific Evidence ◽  
Open Science ◽  
Included Study ◽  
Risk Of Bias ◽  
Dietary Guidelines ◽  
Healthy Population ◽  
Nutrition Research ◽  
Health Related ◽  
Meta Analyses

Background: Dietary guidelines should be informed by systematic reviews (SRs) of the available scientific evidence. However, if the SRs that underpin dietary guidelines are flawed in their design, conduct or reporting, the recommendations contained therein may be misleading or harmful. To date there has been little empirical investigation of bias due to selective inclusion of results, and bias due to missing results, in SRs of food/diet-outcome relationships. Objectives: To explore in SRs with meta-analyses of the association between food/diet and health-related outcomes: (i) whether systematic reviewers selectively included study effect estimates in meta-analyses when multiple effect estimates were available; (ii) what impact selective inclusion of study effect estimates may have on meta-analytic effects, and; (iii) the risk of bias due to missing results (publication bias and selective non-reporting bias) in meta-analyses. Methods: We will systematically search for SRs with meta-analysis of the association between food/diet and health-related outcomes in a generally healthy population, published between January 2018 and June 2019. We will randomly sort titles and abstracts and screen them until we identify 50 eligible SRs. The first reported meta-analysis of a binary or continuous outcome in each SR (the ‘index meta-analysis’) will be evaluated. We will extract from study reports all study effect estimates that were eligible for inclusion in the index meta-analyses (e.g. from multiple instruments and time points) and will quantify and test for evidence of selective inclusion of results. We will also assess the risk of bias due to missing results in the index meta-analyses using a new tool (ROB-ME). Ethics and dissemination: Ethics approval is not required because information will only be extracted from published studies. Dissemination of the results will be through peer-reviewed publications and presentations at conferences. We will make all data collected from this study publicly available via the Open Science Framework.

scholarly journals Investigation of Risk Of Bias due to Unreported and SelecTively included results in meta-analyses of nutrition research: the ROBUST study protocol

F1000Research ◽  
2020 ◽  
Vol 8 ◽  
pp. 1760
Author(s):  
Matthew J. Page ◽  
Lisa Bero ◽  
Cynthia M. Kroeger ◽  
Zhaoli Dai ◽  
Sally McDonald ◽  
...  
Keyword(s):  
Meta Analysis ◽  
Scientific Evidence ◽  
Open Science ◽  
Included Study ◽  
Risk Of Bias ◽  
Dietary Guidelines ◽  
Healthy Population ◽  
Nutrition Research ◽  
Health Related ◽  
Meta Analyses

Background: Dietary guidelines should be informed by systematic reviews (SRs) of the available scientific evidence. However, if the SRs that underpin dietary guidelines are flawed in their design, conduct or reporting, the recommendations contained therein may be misleading or harmful. To date there has been little empirical investigation of bias due to selective inclusion of results, and bias due to missing results, in SRs of food/diet-outcome relationships. Objectives: To explore in SRs with meta-analyses of the association between food/diet and health-related outcomes: (i) whether systematic reviewers selectively included study effect estimates in meta-analyses when multiple effect estimates were available; (ii) what impact selective inclusion of study effect estimates may have on meta-analytic effects, and; (iii) the risk of bias due to missing results (publication bias and selective non-reporting bias) in meta-analyses. Methods: We will systematically search for SRs with meta-analysis of the association between food/diet and health-related outcomes in a generally healthy population, published between January 2018 and June 2019. We will randomly sort titles and abstracts and screen them until we identify 50 eligible SRs. The first reported meta-analysis of a binary or continuous outcome in each SR (the ‘index meta-analysis’) will be evaluated. We will extract from study reports all study effect estimates that were eligible for inclusion in the index meta-analyses (e.g. from multiple instruments and time points) and will quantify and test for evidence of selective inclusion of results. We will also assess the risk of bias due to missing results in the index meta-analyses using a new tool (ROB-ME). Ethics and dissemination: Ethics approval is not required because information will only be extracted from published studies. Dissemination of the results will be through peer-reviewed publications and presentations at conferences. We will make all data collected from this study publicly available via the Open Science Framework.

scholarly journals A Series of Meta-Analytic Tests of the Efficacy of Long-Term Psychoanalytic Psychotherapy

2020 ◽  
Vol 25 (1) ◽  
pp. 51-72 ◽  
Author(s):  
Christian Franz Josef Woll ◽  
Felix D. Schönbrodt
Keyword(s):  
Mental Disorders ◽  
Publication Bias ◽  
Psychiatric Symptoms ◽  
Meta Analysis ◽  
Research Process ◽  
Open Science ◽  
Effect Sizes ◽  
Psychoanalytic Psychotherapy ◽  
Meta Analyses

Abstract. Recent meta-analyses come to conflicting conclusions about the efficacy of long-term psychoanalytic psychotherapy (LTPP). Our first goal was to reproduce the most recent meta-analysis by Leichsenring, Abbass, Luyten, Hilsenroth, and Rabung (2013) who found evidence for the efficacy of LTPP in the treatment of complex mental disorders. Our replicated effect sizes were in general slightly smaller. Second, we conducted an updated meta-analysis of randomized controlled trials comparing LTPP (lasting for at least 1 year and 40 sessions) to other forms of psychotherapy in the treatment of complex mental disorders. We focused on a transparent research process according to open science standards and applied a series of elaborated meta-analytic procedures to test and control for publication bias. Our updated meta-analysis comprising 191 effect sizes from 14 eligible studies revealed small, statistically significant effect sizes at post-treatment for the outcome domains psychiatric symptoms, target problems, social functioning, and overall effectiveness (Hedges’ g ranging between 0.24 and 0.35). The effect size for the domain personality functioning (0.24) was not significant ( p = .08). No signs for publication bias could be detected. In light of a heterogeneous study set and some methodological shortcomings in the primary studies, these results should be interpreted cautiously. In conclusion, LTPP might be superior to other forms of psychotherapy in the treatment of complex mental disorders. Notably, our effect sizes represent the additional gain of LTPP versus other forms of primarily long-term psychotherapy. In this case, large differences in effect sizes are not to be expected.

scholarly journals Quantifying the advantages of conducting a prospective meta-analysis (PMA): a case study of early childhood obesity prevention

Trials ◽  
2021 ◽  
Vol 22 (1) ◽  
Author(s):  
A. L. Seidler ◽  
◽  
K. E. Hunter ◽  
D. Espinoza ◽  
S. Mihrshahi ◽  
...  
Keyword(s):  
Early Childhood ◽  
Childhood Obesity ◽  
External Validity ◽  
Statistical Power ◽  
Meta Analysis ◽  
Intervention Design ◽  
Intervention Effects ◽  
Early Childhood Obesity ◽  
Limited Power ◽  
Meta Analyses

Abstract Background For prospective meta-analyses (PMAs), eligible studies are identified, and the PMA hypotheses, selection criteria, and analysis methods are pre-specified before the results of any of the studies are known. This reduces publication bias and selective outcome reporting and provides a unique opportunity for outcome standardisation/harmonisation. We conducted a world-first PMA of four trials investigating interventions to prevent early childhood obesity. The aims of this study were to quantitatively analyse the effects of prospective planning on variations across trials, outcome harmonisation, and the power to detect intervention effects, and to derive recommendations for future PMA. Methods We examined intervention design, participant characteristics, and outcomes collected across the four trials included in the EPOCH PMA using their registration records, protocol publications, and variable lists. The outcomes that trials planned to collect prior to inclusion in the PMA were compared to the outcomes that trials collected after PMA inclusion. We analysed the proportion of matching outcome definitions across trials, the number of outcomes per trial, and how collaboration increased the statistical power to detect intervention effects. Results The included trials varied in intervention design and participants, this improved external validity and the ability to perform subgroup analyses for the meta-analysis. While individual trials had limited power to detect the main intervention effect (BMI z-score), synthesising data substantially increased statistical power. Prospective planning led to an increase in the number of collected outcome categories (e.g. weight, child’s diet, sleep), and greater outcome harmonisation. Prior to PMA inclusion, only 18% of outcome categories were included in all trials. After PMA inclusion, this increased to 91% of outcome categories. However, while trials mostly collected the same outcome categories after PMA inclusion, some inconsistencies in how the outcomes were measured remained (such as measuring physical activity by hours of outside play versus using an activity monitor). Conclusion Prospective planning led to greater outcome harmonisation and greater power to detect intervention effects, while maintaining acceptable variation in trial designs and populations, which improved external validity. Recommendations for future PMA include more detailed harmonisation of outcome measures and careful pre-specification of analyses to avoid research waste by unnecessary over-collection of data.

scholarly journals Acupuncture for the Relief of Chronic Pain: A Synthesis of Systematic Reviews

Medicina ◽  
2019 ◽  
Vol 56 (1) ◽  
pp. 6
Author(s):  
Carole A. Paley ◽  
Mark I. Johnson
Keyword(s):  
Chronic Pain ◽  
Systematic Reviews ◽  
Statistical Power ◽  
English Language ◽  
Meta Analysis ◽  
Risk Of Bias ◽  
Supporting Evidence ◽  
Pharmacological Interventions ◽  
Eligibility Criteria ◽  
Meta Analyses

Background and Objectives: It is estimated that 28 million people in the UK live with chronic pain. A biopsychosocial approach to chronic pain is recommended which combines pharmacological interventions with behavioural and non-pharmacological treatments. Acupuncture represents one of a number of non-pharmacological interventions for pain. In the current climate of difficult commissioning decisions and constantly changing national guidance, the quest for strong supporting evidence has never been more important. Although hundreds of systematic reviews (SRs) and meta-analyses have been conducted, most have been inconclusive, and this has created uncertainty in clinical policy and practice. There is a need to bring all the evidence together for different pain conditions. The aim of this review is to synthesise SRs of RCTs evaluating the clinical efficacy of acupuncture to alleviate chronic pain and to consider the quality and adequacy of the evidence, including RCT design. Materials and Methods: Electronic databases were searched for English language SRs and meta-analyses on acupuncture for chronic pain. The SRs were scrutinised for methodology, risk of bias and judgement of efficacy. Results: A total of 177 reviews of acupuncture from 1989 to 2019 met our eligibility criteria. The majority of SRs found that RCTs of acupuncture had methodological shortcomings, including inadequate statistical power with a high risk of bias. Heterogeneity between RCTs was such that meta-analysis was often inappropriate. Conclusions: The large quantity of RCTs on acupuncture for chronic pain contained within systematic reviews provide evidence that is conflicting and inconclusive, due in part to recurring methodological shortcomings of RCTs. We suggest that an enriched enrolment with randomised withdrawal design may overcome some of these methodological shortcomings. It is essential that the quality of evidence is improved so that healthcare providers and commissioners can make informed choices on the interventions which can legitimately be provided to patients living with chronic pain.

scholarly journals DOES SIZE MATTER? A CRITICAL REVIEW OF META-ANALYSIS IN AGRONOMY

2019 ◽  
Vol 55 (2) ◽  
pp. 200-229 ◽  
Author(s):  
TIMOTHY J. KRUPNIK ◽  
JENS A. ANDERSSON ◽  
LEONARD RUSINAMHODZI ◽  
MARC CORBEELS ◽  
CAROL SHENNAN ◽  
...  
Keyword(s):  
Statistical Power ◽  
Agricultural Research ◽  
Meta Analysis ◽  
Research Question ◽  
Farming Systems ◽  
Conservation Agriculture ◽  
Research Results ◽  
Analytical Results ◽  
Meta Analyses ◽  
Size Matter

SUMMARYIntended to test broad hypotheses and arrive at unifying conclusions, meta-analysis is the process of extracting, assembling, and analyzing large quantities of data from multiple publications to increase statistical power and uncover explanatory patterns. This paper describes the ways in which meta-analysis has been applied to support claims and counter-claims regarding two topics widely debated in agricultural research, namely organic agriculture (OA) and conservation agriculture (CA). We describe the origins of debate for each topic and assess prominent meta-analyses considering data-selection criteria, research question framing, and the interpretation and extrapolation of meta-analytical results. Meta-analyses of OA and CA are also examined in the context of the political economy of development-oriented agricultural research. Does size matter? We suggest that it does, although somewhat ironically. While meta-analysis aims to pool all relevant studies and generate comprehensive databases from which broad insights can be drawn, our case studies suggest that the organization of many meta-analyses may affect the generalizability and usefulness of research results. The politicized nature of debates over OA and CA also appear to affect the divergent ways in which meta-analytical results may be interpreted and extrapolated in struggles over the legitimacy of both practices. Rather than resolving scientific contestation, these factors appear to contribute to the ongoing debate. Meta-analysis is nonetheless becoming increasingly popular with agricultural researchers attracted by the power for the statistical inference offered by large datasets. This paper consequently offers three suggestions for how scientists and readers of scientific literature can more carefully evaluate meta-analyses. First, the ways in which papers and data are collected should be critically assessed. Second, the justification of research questions, framing of farming systems, and the scales at which research results are extrapolated and discussed should be carefully evaluated. Third, when applied to strongly politicized topics situated in an arena of scientific debate, as is the case with OA and CA, more conservative interpretations of meta-analytical results that recognize the socially and politically embedded nature of agricultural research is are needed.

scholarly journals Low statistical power in biomedical science: a review of three human research domains

2017 ◽  
Vol 4 (2) ◽  
pp. 160254 ◽  
Author(s):  
Estelle Dumas-Mallet ◽  
Katherine S. Button ◽  
Thomas Boraud ◽  
Francois Gonon ◽  
Marcus R. Munafò
Keyword(s):  
Effect Size ◽  
Statistical Power ◽  
Meta Analysis ◽  
Statistical Significance ◽  
Average Power ◽  
Biomedical Science ◽  
Significant Finding ◽  
Biomedical Sciences ◽  
True Effect Size ◽  
Meta Analyses

Studies with low statistical power increase the likelihood that a statistically significant finding represents a false positive result. We conducted a review of meta-analyses of studies investigating the association of biological, environmental or cognitive parameters with neurological, psychiatric and somatic diseases, excluding treatment studies, in order to estimate the average statistical power across these domains. Taking the effect size indicated by a meta-analysis as the best estimate of the likely true effect size, and assuming a threshold for declaring statistical significance of 5%, we found that approximately 50% of studies have statistical power in the 0–10% or 11–20% range, well below the minimum of 80% that is often considered conventional. Studies with low statistical power appear to be common in the biomedical sciences, at least in the specific subject areas captured by our search strategy. However, we also observe evidence that this depends in part on research methodology, with candidate gene studies showing very low average power and studies using cognitive/behavioural measures showing high average power. This warrants further investigation.

Sign in / Sign up

Export Citation Format

Share Document