A Historical Review of Publication Bias

Made In

Publication bias is a well-known threat to the validity of meta-analyses and, more broadly, the reproducibility of scientific findings. When policies and recommendations are predicated on an incomplete evidence-base, it undermines the goals of evidence-based decision-making. Great strides have been made in the last fifty years to understand and address this problem, including calls for mandatory trial registration and the development of statistical methods to detect and correct for publication bias. We offer an historical account of seminal contributions by the evidence synthesis community, with an emphasis on the parallel development of graph-based and selection model approaches. We also draw attention to current innovations and opportunities for future methodological work.

Acting in the Face of Evidentiary Ambiguity, Bias, and Absence Arising from Systematic Reviews in Applied Environmental Science

10.32942/osf.io/4ertu ◽

2021 ◽

Author(s):

Trina Rytwinski ◽

Steven J Cooke ◽

Jessica J Taylor ◽

Dominique Roche ◽

Paul A Smith ◽

...

Keyword(s):

Systematic Review ◽

Decision Making ◽

Systematic Reviews ◽

Environmental Science ◽

Evidence Synthesis ◽

Empirical Studies ◽

Synthesis Method ◽

Evidence Base ◽

Evidence Based ◽

Evidence-based decision-making often depends on some form of a synthesis of previous findings. There is growing recognition that systematic reviews, which incorporate a critical appraisal of evidence, are the gold standard synthesis method in applied environmental science. Yet, on a daily basis, environmental practitioners and decision-makers are forced to act even if the evidence base to guide them is insufficient. For example, it is not uncommon for a systematic review to conclude that an evidence base is large but of low reliability. There are also instances where the evidence base is sparse (e.g., one or two empirical studies on a particular taxa or intervention), and no additional evidence arises from a systematic review. In some cases, the systematic review highlights considerable variability in the outcomes of primary studies, which in turn generates ambiguity (e.g., potentially context specific). When the environmental evidence base is ambiguous, biased, or lacking of new information, practitioners must still make management decisions. Waiting for new, higher validity research to be conducted is often unrealistic as many decisions are urgent. Here, we identify the circumstances that can lead to ambiguity, bias, and the absence of additional evidence arising from systematic reviews and provide practical guidance to resolve or handle these scenarios when encountered. Our perspective attempts to highlight that, with evidence synthesis, there may be a need to balance the spirit of evidence-based decision-making and the practical reality that management and conservation decisions and action is often time sensitive.

Adapting to Learn and Learning to Adapt: Practical Insights from International Development Projects

10.3768/rtipress.2018.pb.0015.1801 ◽

2018 ◽

Cited By ~ 3

Author(s):

Derick W. Brinkerhoff ◽

Sarah Frazer ◽

Lisa McGregor-Mirghani

Keyword(s):

Decision Making ◽

International Development ◽

Adaptive Management ◽

Evidence Base ◽

Evidence Based ◽

Reporting Requirements ◽

Adaptive Programming ◽

Donor Agencies ◽

Program Cycle ◽

Adaptive programming and management principles focused on learning, experimentation, and evidence-based decision making are gaining traction with donor agencies and implementing partners in international development. Adaptation calls for using learning to inform adjustments during project implementation. This requires information gathering methods that promote reflection, learning, and adaption, beyond reporting on pre-specified data. A focus on adaptation changes traditional thinking about program cycle. It both erases the boundaries between design, implementation, and evaluation and reframes thinking to consider the complexity of development problems and nonlinear change pathways.Supportive management structures and processes are crucial for fostering adaptive management. Implementers and donors are experimenting with how procurement, contracting, work planning, and reporting can be modified to foster adaptive programming. Well-designed monitoring, evaluation, and learning systems can go beyond meeting accountability and reporting requirements to produce data and learning for evidence-based decision making and adaptive management. It is important to continue experimenting and learning to integrate adaptive programming and management into the operational policies and practices of donor agencies, country partners, and implementers. We need to devote ongoing effort to build the evidence base for the contributions of adaptive management to achieving international development results.

Dynamic meta-analysis: a method of using global evidence for local decision making

BMC Biology ◽

10.1186/s12915-021-00974-w ◽

2021 ◽

Vol 19 (1) ◽

Author(s):

Gorm E. Shackelford ◽

Philip A. Martin ◽

Amelia S. C. Hood ◽

Alec P. Christie ◽

Elena Kulinskaya ◽

...

Keyword(s):

Decision Making ◽

Evidence Synthesis ◽

Meta Analysis ◽

Global Scale ◽

Classification Systems ◽

Evidence Based ◽

Proof Of Concept ◽

Future Studies ◽

Global Evidence

Abstract Background Meta-analysis is often used to make generalisations across all available evidence at the global scale. But how can these global generalisations be used for evidence-based decision making at the local scale, if the global evidence is not perceived to be relevant to local decisions? We show how an interactive method of meta-analysis—dynamic meta-analysis—can be used to assess the local relevance of global evidence. Results We developed Metadataset (www.metadataset.com) as a proof-of-concept for dynamic meta-analysis. Using Metadataset, we show how evidence can be filtered and weighted, and results can be recalculated, using dynamic methods of subgroup analysis, meta-regression, and recalibration. With an example from agroecology, we show how dynamic meta-analysis could lead to different conclusions for different subsets of the global evidence. Dynamic meta-analysis could also lead to a rebalancing of power and responsibility in evidence synthesis, since evidence users would be able to make decisions that are typically made by systematic reviewers—decisions about which studies to include (e.g. critical appraisal) and how to handle missing or poorly reported data (e.g. sensitivity analysis). Conclusions In this study, we show how dynamic meta-analysis can meet an important challenge in evidence-based decision making—the challenge of using global evidence for local decisions. We suggest that dynamic meta-analysis can be used for subject-wide evidence synthesis in several scientific disciplines, including agroecology and conservation biology. Future studies should develop standardised classification systems for the metadata that are used to filter and weight the evidence. Future studies should also develop standardised software packages, so that researchers can efficiently publish dynamic versions of their meta-analyses and keep them up-to-date as living systematic reviews. Metadataset is a proof-of-concept for this type of software, and it is open source. Future studies should improve the user experience, scale the software architecture, agree on standards for data and metadata storage and processing, and develop protocols for responsible evidence use.

Publication Bias in Science

10.1093/oxfordhb/9780190497620.013.10 ◽

2017 ◽

Author(s):

Andrew W. Brown ◽

Tapan S. Mehta ◽

David B. Allison

Keyword(s):

Decision Making ◽

Publication Bias ◽

Teaching Strategies ◽

Scientific Literature ◽

Systematic Difference ◽

Causal Effects ◽

Evidence Based ◽

Quantities Of Interest

When we rely on science to inform decisions about matters such as the environment, teaching strategies, economics, government, and medicine, evidence-based decision-making can only be as reliable as the totality of the science itself. We must avoid distortions of the scientific literature such as publication bias, which is an expected systematic difference between estimates of associations, causal effects, or other quantities of interest compared to the actual values of those quantities, caused by differences between research that is published and the totality of research conducted. Publication bias occurs when the probability of publishing a result of a study is influenced by the result obtained. It appears to be common and can produce misleading conclusions about interventions, make effects appear greater than they are, lead to irreproducible research, and ultimately undermine the credibility of science in general. Methods to detect publication bias and steps to reduce it are discussed.

Evidence Synthesis and Evidence-Based Decision Making: Related But Distinct Processes

Medical Decision Making ◽

10.1177/0272989x05281156 ◽

2005 ◽

Vol 25 (5) ◽

pp. 487-489 ◽

Cited By ~ 12

Author(s):

Steven M. Teutsch ◽

Marc L. Berger

Keyword(s):

Decision Making ◽

Evidence Synthesis ◽

Evidence Based ◽

Meta-analysis in applied ecology

Biology Letters ◽

10.1098/rsbl.2009.0546 ◽

2009 ◽

Vol 6 (1) ◽

pp. 78-81 ◽

Cited By ~ 116

Author(s):

Gavin Stewart

Keyword(s):

Decision Making ◽

Meta Analysis ◽

Research Synthesis ◽

Statistical Techniques ◽

Evidence Based ◽

Applied Ecology ◽

Space And Time ◽

Meta Analyses ◽

Weighted Combination

This overview examines research synthesis in applied ecology and conservation. Vote counting and pooling unweighted averages are widespread despite the superiority of syntheses based on weighted combination of effects. Such analyses allow exploration of methodological uncertainty in addition to consistency of effects across species, space and time, but exploring heterogeneity remains controversial. Meta-analyses are required to generalize in ecology, and to inform evidence-based decision-making, but the more sophisticated statistical techniques and registers of research used in other disciplines must be employed in ecology to fully realize their benefits.

The Use of Preference Based Measures in Haemophilia: Is the Current Evidence Base Useful for Evidence Based Decision Making?

Value in Health ◽

10.1016/j.jval.2013.08.369 ◽

2013 ◽

Vol 16 (7) ◽

pp. A386-A387

Author(s):

K. Tolley ◽

A. Miners ◽

J. Brazier ◽

L.M. Pericleous ◽

T. Sharma ◽

...

Keyword(s):

Decision Making ◽

Evidence Base ◽

Current Evidence ◽

Evidence Based ◽

Current Evidence Base ◽

The Communication Tool Kit: Encouraging Evidence-based Decision Making

PsycEXTRA Dataset ◽

10.1037/e521582014-044 ◽

2009 ◽

Author(s):

Kristin L. Carman ◽

Sandy Robinson ◽

Lori Agin

Keyword(s):

Decision Making ◽

Evidence Based ◽

Communication Tool ◽

Evidence-based Assessment

The Oxford Handbook of Clinical Psychology ◽

10.1093/oxfordhb/9780195366884.013.0005 ◽

2010 ◽

pp. 76-98

Author(s):

John Hunsley ◽

Eric J. Mash

Keyword(s):

Decision Making ◽

Clinical Assessment ◽

Assessment Process ◽

Evidence Based ◽

The Past ◽

Specific Assessment ◽

The Many ◽

Selection Of ◽

Made In

Evidence-based assessment relies on research and theory to inform the selection of constructs to be assessed for a specific assessment purpose, the methods and measures to be used in the assessment, and the manner in which the assessment process unfolds. An evidence-based approach to clinical assessment necessitates the recognition that, even when evidence-based instruments are used, the assessment process is a decision-making task in which hypotheses must be iteratively formulated and tested. In this chapter, we review (a) the progress that has been made in developing an evidence-based approach to clinical assessment in the past decade and (b) the many challenges that lie ahead if clinical assessment is to be truly evidence-based.