Classifying information-sharing methods

Abstract Background Sparse relative effectiveness evidence is a frequent problem in Health Technology Assessment (HTA). Where evidence directly pertaining to the decision problem is sparse, it may be feasible to expand the evidence-base to include studies that relate to the decision problem only indirectly: for instance, when there is no evidence on a comparator, evidence on other treatments of the same molecular class could be used; similarly, a decision on children may borrow-strength from evidence on adults. Usually, in HTA, such indirect evidence is either included by ignoring any differences (‘lumping’) or not included at all (‘splitting’). However, a range of more sophisticated methods exists, primarily in the biostatistics literature. The objective of this study is to identify and classify the breadth of the available information-sharing methods. Methods Forwards and backwards citation-mining techniques were used on a set of seminal papers on the topic of information-sharing. Papers were included if they specified (network) meta-analytic methods for combining information from distinct populations, interventions, outcomes or study-designs. Results Overall, 89 papers were included. A plethora of evidence synthesis methods have been used for information-sharing. Most papers (n=79) described methods that shared information on relative treatment effects. Amongst these, there was a strong emphasis on methods for information-sharing across multiple outcomes (n=42) and treatments (n=25), with fewer papers focusing on study-designs (n=23) or populations (n=8). We categorise and discuss the methods under four ’core’ relationships of information-sharing: functional, exchangeability-based, prior-based and multivariate relationships, and explain the assumptions made within each of these core approaches. Conclusions This study highlights the range of information-sharing methods available. These methods often impose more moderate assumptions than lumping or splitting. Hence, the degree of information-sharing that they impose could potentially be considered more appropriate. Our identification of four ‘core’ methods of information-sharing allows for an improved understanding of the assumptions underpinning the different methods. Further research is required to understand how the methods differ in terms of the strength of sharing they impose and the implications of this for health care decisions.

Download Full-text

A systematic review and classification of information-sharing methods

10.21203/rs.3.rs-42027/v2 ◽

2021 ◽

Author(s):

Georgios F. Nikolaidis ◽

Beth Woods ◽

Stephen Palmer ◽

Marta O Soares

Keyword(s):

Information Sharing ◽

Decision Problem ◽

Evidence Synthesis ◽

Relative Effectiveness ◽

Indirect Evidence ◽

Evidence Base ◽

Synthesis Methods ◽

Shared Information ◽

Multivariate Relationships ◽

Study Designs

Abstract Background: Sparse relative effectiveness evidence is a frequent problem in Health Technology Assessment (HTA). Where evidence directly pertaining to the decision problem is sparse, it may be feasible to expand the evidence-base to include studies that relate to the decision problem only indirectly: for instance, when there is no evidence on a comparator, evidence on other treatments of the same molecular class could be used; similarly, a decision on children may borrow-strength from evidence on adults. Usually, in HTA, such indirect evidence is either included by ignoring any differences (‘lumping‘) or not included at all (‘splitting‘). However, a range of more sophisticated methods exists, primarily in the biostatistics literature. The objective of this study is identify and classify the breadth of the available information-sharing methods. Methods: Forwards and backwards citation-mining techniques were used on a set of seminal papers on the topic of information-sharing. Papers were included if they specified (network) meta-analytic methods for combining information from distinct populations, interventions, outcomes or study-designs. Results: Overall, 89 papers were included. A plethora of evidence synthesis methods have been used for information-sharing. Most papers (n = 78) described methods that shared information on relative treatment effects. Amongst these, there was a strong emphasis on methods for information-sharing across multiple outcomes (n = 39) and treatments (n = 23), with fewer papers focusing on study-designs (n = 10) or populations (n = 6). We categorise and discuss the methods under four ’core’ relationships of information-sharing: functional, exchangeability-based, prior-based and multivariate relationships, and explain the assumptions made within each of these core approaches. Conclusions: This study highlights the range of information-sharing methods available. These methods often impose more moderate assumptions than lumping or splitting. Hence, the degree of information-sharing that they impose could potentially be considered more appropriate. Our identification of four ‘core‘ methods of information-sharing allows for an improved understanding of the assumptions underpinning the different methods. Further research is required to understand how the methods differ in terms of the strength of sharing they impose and the implications of this for health care decisions.

Download Full-text

Classifying information-sharing methods: a citation-mining systematic review

10.21203/rs.3.rs-42027/v1 ◽

2020 ◽

Author(s):

Georgios Filippos Nikolaidis ◽

Beth Woods ◽

Stephen Palmer ◽

Marta O Soares

Keyword(s):

Information Sharing ◽

Decision Problem ◽

Evidence Synthesis ◽

Synthesis Methods ◽

Health Care Decisions ◽

Shared Information ◽

Multivariate Relationships ◽

Study Designs ◽

Available Information ◽

Frequent Problem

Abstract BackgroundSparse relative eectiveness evidence is a frequent problem in Health Technology Assessment(HTA). Where evidence directly pertaining to the decision problem is sparse, it may be feasible to expand theevidence-base to include studies that relate to the decision problem only indirectly: for instance, when there isno evidence on a comparator, evidence on other treatments of the same molecular class could be used;similarly, a decision on children may borrow-strength from evidence on adults. Usually, in HTA, such indirectevidence is either included by ignoring any dierences (`lumping`) or not included at all (`splitting`). However,a range of more sophisticated methods exists, primarily in the biostatistics literature. The objective of thisstudy is identify and classify the breadth of the available information-sharing methods.MethodsForwards and backwards citation-mining techniques were used on a set of seminal papers on thetopic of information-sharing. Papers were included if they specied (network) meta-analytic methods forcombining information from distinct populations, interventions, outcomes or study-designs.Results: Overall, 89 papers were included. A plethora of evidence synthesis methods have been used forinformation-sharing. Most papers (n = 78) described methods that shared information on relative treatmenteects. Amongst these, there was a strong emphasis on methods for information-sharing across multipleoutcomes (n = 39) and treatments (n = 23), with fewer papers focusing on study-designs (n = 10) orpopulations (n = 6). We categorise and discuss the methods under four 'core' relationships ofinformation-sharing: functional, exchangeability-based, prior-based and multivariate relationships, and explainthe assumptions made within each of these core approaches.ConclusionsThis study highlights the range of information-sharing methods available. These methods oftenimpose more moderate assumptions than lumping or splitting. Hence, the degree of information-sharing thatthey impose could potentially be considered more appropriate. Our identication of four `core` methods ofinformation-sharing allows for an improved understanding of the assumptions underpinning the dierentmethods. Further research is required to understand how the methods dier in terms of the strength of sharingthey impose and the implications of this for health care decisions.

Download Full-text

The challenge of heterogeneous evidence in conservation

10.1101/797639 ◽

2019 ◽

Cited By ~ 2

Author(s):

Alec P. Christie ◽

Tatsuya Amano ◽

Philip A. Martin ◽

Silviu O. Petrovan ◽

Gorm E. Shackelford ◽

...

Keyword(s):

North America ◽

Threatened Species ◽

Western Europe ◽

Evidence Synthesis ◽

Evidence Base ◽

Future Research ◽

Conservation Action ◽

Biodiversity Crisis ◽

Conservation Actions ◽

Study Designs

AbstractConservation efforts to tackle the current biodiversity crisis need to be as efficient and effective as possible. To inform decision-makers of the most effective conservation actions, it is important to identify biases and gaps in the conservation literature to prioritize future evidence generation. We assessed the state of this global literature base using the Conservation Evidence database, a comprehensive collection of quantitative tests of conservation actions (interventions) from the published literature. For amphibians and birds, we investigated the nature of Conservation Evidence spatially and taxonomically, as well as by biome, effectiveness metrics, and study design. Studies were heavily concentrated in Western Europe and North America for birds and particularly amphibians. Studies that used the most robust study designs - Before-After Control-Impact and Randomized Controlled Trials - were also the most geographically restricted. Furthermore, there was no relationship between the number of studies in each 1×1 degree grid cell and the number of species, threatened species or data-deficient species. Taxonomic biases and gaps were apparent for amphibians and birds - some orders were absent from the evidence base and others were poorly represented relative to the proportion of threatened species they contained. Temperate forest and grassland biomes were highly represented, which reinforced observed geographic biases. Various metrics were used to evaluate the effectiveness of a given conservation action, potentially making studies less directly comparable and evidence synthesis more difficult. We also found that the least robust study designs were the most commonly used; studies using robust designs were scarce. Future research should prioritize testing conservation actions on threatened species outside of Western Europe and North America. Standardizing metrics and improving the robustness of study designs used to test conservation actions would also improve the quality of the evidence base for synthesis and decision-making.

Download Full-text

Improving the Utility of Evidence Synthesis for Decision Makers in the Face of Insufficient Evidence

10.23970/ahrqepcwhitepaperimproving ◽

2021 ◽

Author(s):

M. Hassan Murad ◽

Stephanie M. Chang ◽

Celia Fiordalisi ◽

Jennifer S. Lin ◽

Timothy J. Wilt ◽

...

Keyword(s):

Literature Review ◽

Systematic Reviews ◽

Evidence Synthesis ◽

Indirect Evidence ◽

Evidence Base ◽

Healthcare Research ◽

Decision Makers ◽

Healthcare Decision ◽

Insufficient Evidence ◽

Convenience Sample

Background: Healthcare decision makers strive to operate on the best available evidence. The Agency for Healthcare Research and Quality Evidence-based Practice Center (EPC) Program aims to support healthcare decision makers by producing evidence reviews that rate the strength of evidence. However, the evidence base is often sparse or heterogeneous, or otherwise results in a high degree of uncertainty and insufficient evidence ratings. Objective: To identify and suggest strategies to make insufficient ratings in systematic reviews more actionable. Methods: A workgroup comprising EPC Program members convened throughout 2020. We conducted interative discussions considering information from three data sources: a literature review for relevant publications and frameworks, a review of a convenience sample of past systematic reviews conducted by the EPCs, and an audit of methods used in past EPC technical briefs. Results: Several themes emerged across the literature review, review of systematic reviews, and review of technical brief methods. In the purposive sample of 43 systematic reviews, the use of the term “insufficient” covered both instances of no evidence and instances of evidence being present but insufficient to estimate an effect. The results of the literature review and review of the EPC Program systematic reviews illustrated the importance of clearly stating the reasons for insufficient evidence. Results of both the literature review and review of systematic reviews highlighted the factors decision makers consider when making decisions when evidence of benefits or harms is insufficient, such as costs, values, preferences, and equity. We identified five strategies for supplementing systematic review findings when evidence on benefit or harms is expected to be or found to be insufficient, including: reconsidering eligible study designs, summarizing indirect evidence, summarizing contextual and implementation evidence, modelling, and incorporating unpublished health system data. Conclusion: Throughout early scoping, protocol development, review conduct, and review presentation, authors should consider five possible strategies to supplement potential insufficient findings of benefit or harms. When there is no evidence available for a specific outcome, reviewers should use a statement such as “no studies” instead of “insufficient.” The main reasons for insufficient evidence rating should be explicitly described.

Download Full-text

OP380 A Review Of The Methodology Used To Synthesize Continuous And Time-To-Event Outcomes For Clinical And Cost-Effectiveness

International Journal of Technology Assessment in Health Care ◽

10.1017/s0266462320001051 ◽

2020 ◽

Vol 36 (S1) ◽

pp. 7-7

Author(s):

Suzanne Freeman ◽

Alex Sutton ◽

Nicola Cooper

Keyword(s):

Cost Effectiveness ◽

Decision Model ◽

Evidence Synthesis ◽

Meta Analysis ◽

Clinical Effectiveness ◽

Evidence Base ◽

Synthesis Methods ◽

Time To Event ◽

Health Technologies ◽

Treatment Comparison

IntroductionSynthesis of continuous and time-to-event outcomes is often complicated by the use of multiple outcome scales and heterogeneous reporting of outcomes across trials. Simple methods of evidence synthesis for clinical effectiveness can fail to account for these issues and result in a reduction of the evidence base, which can be further reduced at the cost-effectiveness stage as common outcome measures, such as standardized mean differences, cannot easily be incorporated into the economic decision model. Recent methodological advances for synthesizing continuous and time-to-event outcomes aim to include a greater proportion of the available evidence base within a single coherent analysis.MethodsTo assess the statistical methods commonly used in health technology assessment (HTA) and establish whether recent advances in synthesis methods have been adopted in practice, we conducted a review of HTA reports and guidelines published in the United Kingdom (UK) between 1 April 2018 and 31 March 2019 reporting a quantitative meta-analysis (MA), network meta-analysis (NMA) or indirect treatment comparison (ITC) of at least one continuous or time-to-event outcome.ResultsForty-seven articles were considered eligible for this review. Fifty-one percent of eligible articles reported at least one continuous outcome and 55 percent at least one time-to-event outcome. Twenty-nine articles reported NMA or ITC and twenty-seven reported MA of a continuous or time-to-event outcome. Forty articles included a decision model, of which twenty-seven incorporated evidence from a synthesis of a continuous or time-to-event outcome with eleven informed by a single trial (despite synthesis being conducted).ConclusionsUptake of methods to include a greater proportion of the available evidence base within a single coherent analysis in UK HTA reports has been slow. Evaluating health technologies using an evidence-based approach often results in better outcomes for patients. Therefore, HTA analysts and decision modelers must be aware of the expanding literature for synthesis of continuous and time-to-event outcomes and appreciate the limitations of simpler approaches.

Download Full-text

Qualitative Evidence Synthesis: Where Are We at?

International Journal of Qualitative Methods ◽

10.1177/1609406921993276 ◽

2021 ◽

Vol 20 ◽

pp. 160940692199327

Author(s):

Kate Flemming ◽

Jane Noyes

Keyword(s):

Qualitative Research ◽

Systematic Reviews ◽

Evidence Synthesis ◽

Synthesis Methods ◽

Qualitative Evidence Synthesis ◽

Policy Makers ◽

Qualitative Evidence ◽

Health And Social Care ◽

Method Paper ◽

Increasing Demand

Qualitative evidence syntheses (QES) have increased in prominence and profile over the last decade as a discrete set of methodologies to undertake systematic reviews of primary qualitative research in health and social care and in education. The findings from a qualitative evidence synthesis can enable a richer interpretation of a particular phenomenon, set of circumstances, or experiences than single primary qualitative research studies can achieve. Qualitative evidence synthesis methods were developed in response to an increasing demand from health and social professionals, policy makers, guideline developers and educationalists for review evidence that goes beyond “what works” afforded by systematic reviews of effectiveness. The increasing interest in the synthesis of qualitative research has led to methodological developments documented across a plethora of texts and journal articles. This “State of the Method” paper aims to bring together these methodological developments in one place, contextualizing advances in methods with exemplars to support readers in making choices in approach to a synthesis and aid understanding. The paper clarifies what a “qualitative evidence synthesis” is and explores its role, purpose and development. It details the kind of questions a QES can explore, the processes associated with a QES, including the methods for synthesis. The rational and methods for integrating a QES with systematic reviews of effectiveness are also detailed. Finally approaches reporting and recognition of what a “good” or rigorous QES look like are provided.

Download Full-text

OP179 Quantitative Evidence Synthesis Methods For Assessing The Effectiveness Of Treatment Sequences For Clinical And Economic Decision-Making: Methodology Review

International Journal of Technology Assessment in Health Care ◽

10.1017/s0266462321000738 ◽

2021 ◽

Vol 37 (S1) ◽

pp. 4-4

Author(s):

Ruth Lewis ◽

Dyfrig Hughes ◽

Alex Sutton ◽

Clare Wilkinson

Keyword(s):

Decision Making ◽

Chronic Conditions ◽

Evidence Synthesis ◽

Policy Decision ◽

Clinical Effectiveness ◽

Synthesis Methods ◽

Economic Decision Making ◽

Treatment Sequencing ◽

Policy Decision Making ◽

Treatment Sequences

IntroductionThe sequential use of alternative treatments for chronic conditions represents a complex, dynamic intervention pathway; previous treatment and patient characteristics affect both choice and effectiveness of subsequent treatments. Evidence synthesis methods that produce the least biased estimates of treatment-sequencing effects are required to inform reliable clinical and policy decision-making. A comprehensive review was conducted to establish what existing methods are available, outline the assumptions they make, and identify their shortcomings.MethodsThe review encompassed both meta-analytic techniques and decision-analytic modelling, any disease condition, and any type of treatment sequence, but not diagnostic tests, screening, or treatment monitoring. It focused on the estimation of clinical effectiveness and did not consider the impact of treatment sequencing on the estimation of costs or utility values.ResultsThe review included ninety-one studies. Treatment-sequencing is usually dealt with at the decision-modelling stage and is rarely addressed using evidence synthesis methodology for clinical effectiveness. Most meta-analyses are of discrete treatments, sometimes stratified by line of therapy. Prospective sequencing trials are scarce. In their absence, there is no single best way to evaluate treatment sequences, rather there is a range of approaches, each of which has advantages and disadvantages and is influenced by the evidence available and the decision problem. Due to the scarcity of data on sequential treatments, modelling studies generally apply simplifying assumptions to data on discrete treatments. A taxonomy for all possible assumptions was developed, providing a unique resource to aid the critique of decision-analytic models.ConclusionsThe evolution of network meta-analysis in HTA demonstrates that clinical and policy decision-making should account for the multiple treatments available for many chronic conditions. However, treatment-sequencing has yet to be accounted for within clinical evaluations. Economic modelling is often based on the simplifying assumption of treatment independence. This can lead to misrepresentation of the true level of uncertainty, potential bias in estimating the effectiveness and cost effectiveness of treatments and, eventually, the wrong decision.

Download Full-text

EVIDENCE, VALUES, AND DECISION MAKING

International Journal of Technology Assessment in Health Care ◽

10.1017/s0266462314000154 ◽

2014 ◽

Vol 30 (2) ◽

pp. 233-238 ◽

Cited By ~ 11

Author(s):

Michael D. Rawlins

Keyword(s):

Decision Making ◽

Cost Effectiveness ◽

Observational Study ◽

Social Values ◽

Clinical Excellence ◽

Health Technology ◽

Evidence Base ◽

Social Value ◽

Value Judgments ◽

Study Designs

Background: The evidence supporting the use of new, or established, interventions may be derived from either (or both) experimental or observational study designs. Although a rigorous examination of the evidence base for clinical and cost-effectiveness is essential, it is never sufficient, and those undertaking a health technology assessment (HTA) also have to exercise judgments.Methods: The basis for this discussion is largely from the author's experience as chairman of the national Institute for Health and Clinical Excellence (NICE).Results: The judgments necessary for HTA to make are twofold. Scientific judgments relate to the interpretation of the science. Social value judgments are concerned with the ethical principles, preferences, culture, and aspirations of society.Conclusions: How scientific and social value judgments might be most appropriately captured is a challenge for all HTA agencies. Although competent HTA bodies should be able to exercise scientific judgments they have no legitimacy to impose their own social values. These must ultimately be informed by the general public.

Download Full-text

CogTale: An online platform for the evaluation, synthesis and dissemination of evidence from cognitive interventions studies

10.21203/rs.3.rs-39520/v2 ◽

2020 ◽

Author(s):

Julieta Sabates ◽

Sylvie Belleville ◽

Mary Castellani ◽

Tzvi Dwolatsky ◽

Benjamin M. Hampstead ◽

...

Keyword(s):

Systematic Reviews ◽

Intervention Studies ◽

Evidence Synthesis ◽

Research Synthesis ◽

Cognitive Intervention ◽

Synthesis Methods ◽

Data Synthesis ◽

Advanced Search ◽

Health Related ◽

Meta Analyses

Abstract Systematic reviews and meta-analyses are critical in health-related decision making, and are considered the gold standard in research synthesis methods. However, with new trials being regularly published and with the development of increasingly rigorous standards of data synthesis, systematic reviews often require much expertise and long periods of time to be completed. Automation of some of the steps of evidence synthesis productions is a promising improvement in the field, capable of reducing the time and costs associated with the process. This article describes the development and main characteristics of a novel online repository of cognitive intervention studies entitled Cognitive Treatments Article Library and Evaluation (CogTale). The platform is currently in a Beta Release phase, as it is still under development. However, it already contains over 70 studies, and the CogTale team is continuously coding and uploading new studies into the repository. Key features include advanced search options, the capability to generate meta-analyses, and an up-to-date display of relevant published studies.

Download Full-text

Identifying stroke therapeutics from preclinical models: A protocol for a novel application of network meta-analysis

F1000Research ◽

10.12688/f1000research.15869.1 ◽

2019 ◽

Vol 8 ◽

pp. 11 ◽

Cited By ~ 1

Author(s):

Manoj M. Lalu ◽

Dean A. Fergusson ◽

Wei Cheng ◽

Marc T. Avey ◽

Dale Corbett ◽

...

Keyword(s):

Ischemic Stroke ◽

Burden Of Illness ◽

Evidence Synthesis ◽

Meta Analysis ◽

Synthesis Methods ◽

Preclinical Models ◽

Ovid Medline ◽

Scientific Meetings ◽

Ischemia Testing

Introduction: Globally, stroke is the second leading cause of death. Despite the burden of illness and death, few acute interventions are available to patients with ischemic stroke. Over 1,000 potential neuroprotective therapeutics have been evaluated in preclinical models. It is important to use robust evidence synthesis methods to appropriately assess which therapies should be translated to the clinical setting for evaluation in human studies. This protocol details planned methods to conduct a systematic review to identify and appraise eligible studies and to use a network meta-analysis to synthesize available evidence to answer the following questions: in preclinical in vivo models of focal ischemic stroke, what are the relative benefits of competing therapies tested in combination with the gold standard treatment alteplase in (i) reducing cerebral infarction size, and (ii) improving neurobehavioural outcomes? Methods: We will search Ovid Medline and Embase for articles on the effects of combination therapies with alteplase. Controlled comparison studies of preclinical in vivo models of experimentally induced focal ischemia testing the efficacy of therapies with alteplase versus alteplase alone will be identified. Outcomes to be extracted include infarct size (primary outcome) and neurobehavioural measures. Risk of bias and construct validity will be assessed using tools appropriate for preclinical studies. Here we describe steps undertaken to perform preclinical network meta-analysis to synthesise all evidence for each outcome and obtain a comprehensive ranking of all treatments. This will be a novel use of this evidence synthesis approach in stroke medicine to assess pre-clinical therapeutics. Combining all evidence to simultaneously compare mutliple therapuetics tested preclinically may provide a rationale for the clinical translation of therapeutics for patients with ischemic stroke. Dissemination: Review findings will be submitted to a peer-reviewed journal and presented at relevant scientific meetings to promote knowledge transfer. Registration: PROSPERO number to be submitted following peer review.

Download Full-text