Experimental and Non-Experimental Methods in Development Economics: A Porous Dialectic

AbstractThis paper surveys six widely-used non-experimental methods for estimating treatment effects (instrumental variables, regression discontinuity, direct matching, propensity score matching, linear regression and non-parametric methods, and difference-in-differences), and assesses their internal and external validity relative both to each other and to randomized controlled trials. While randomized controlled trials can achieve the highest degree of internal validity when cleanly implemented in the field, the availability of large, nationally representative data sets offers the opportunity for a high degree of external validity using non-experimental methods. We argue that each method has merits in some context and they are complements rather than substitutes.

Download Full-text

Randomized Controlled Trials in Psychiatry. Part II: Their Relationship to Clinical Practice

Australian & New Zealand Journal of Psychiatry ◽

10.1046/j.1440-1614.2003.01176.x ◽

2003 ◽

Vol 37 (3) ◽

pp. 265-269 ◽

Cited By ~ 24

Author(s):

Roger T. Mulder ◽

Chris Frampton ◽

Peter R. Joyce ◽

Richard Porter

Keyword(s):

Clinical Practice ◽

Randomized Controlled Trials ◽

External Validity ◽

Internal Validity ◽

Small Samples ◽

Controlled Trials ◽

Treatment Protocols ◽

Randomized Controlled ◽

Clinical Conditions ◽

Internal And External Validity

Objective: To discuss the extent to which the results of randomized controlled trials (RCTs) in psychiatry can be generalized to clinical practice. Method: Threats to internal and external validity in psychiatric RCTs are reviewed. Results: Threats to internal validity increase the possibility of bias. Psychiatric RCTs have problems with small samples, arbitrary definitions of caseness, disparate definitions of outcome and high spontaneous recovery rates. Particular issues arise in psychotherapy RCTs. Threats to external validity reduce the extent to which the results of a RCT produce a correct basis for generalization to other circumstances. These include high rates of comorbidity and sub syndromal pathology in normal clinical practice, manual-based treatment protocols and varying definitions of successful treatment. Conclusions: Randomized controlled trials remain the most robust design to investigate the effectiveness of treatments. They should be applied to important clinical questions; and carried out, as far as possible, with typical patients in the clinical conditions in which the treatment is likely to be used.

Download Full-text

External Validity and Worksite Wellness: Lessons From China and Illinois

American Journal of Health Promotion ◽

10.1177/0890117120970333 ◽

2020 ◽

Vol 35 (1) ◽

pp. 9-12

Author(s):

Paul E. Terry

Keyword(s):

Best Practices ◽

Randomized Controlled Trials ◽

External Validity ◽

Careful Consideration ◽

Controlled Trials ◽

Worksite Wellness ◽

Randomized Controlled ◽

Negative Findings ◽

New Evidence ◽

Internal And External Validity

This editorial describes recent randomized controlled trials of worksite wellness interventions and argues that fidelity to intervention designs should be contingent on careful consideration of internal and external validity. A China based hypertension management study which achieved impressive outcomes across 60 workplaces using a comprehensive approach is contrasted with the traditional wellness practices employed in other randomized controlled trials conducted in America. Why studies with negative findings receive more media and professional scrutiny than studies with positive findings is discussed. Three reasons are posited for why bad is stronger than good when it comes to capturing attention. Adoption of new evidence is discussed along with what health promotion professionals can do to advance best practices by considering adoption as an ongoing process.

Download Full-text

Tools to Assess the External Validity of Randomized Controlled Trials in Systematic Reviews: a Systematic Review of Measurement Properties

10.21203/rs.3.rs-830181/v1 ◽

2021 ◽

Author(s):

Andres Jung ◽

Julia Balzer ◽

Tobias Braun ◽

Kerstin Luedtke

Keyword(s):

Randomized Controlled Trials ◽

Systematic Reviews ◽

External Validity ◽

Measurement Properties ◽

Controlled Trials ◽

Measurement Instruments ◽

Health Measurement ◽

Quality Of Evidence ◽

Randomized Controlled

Abstract Background: Internal and external validity are the most relevant components when critically appraising randomized controlled trials (RCTs) for systematic reviews. However, there is no gold standard to assess external validity. This might be related to the heterogeneity of terminology as well as to unclear evidence of the measurement properties of available tools. The aim of this review was to identify tools to assess the external validity of RCTs in systematic reviews and to evaluate the quality of evidence regarding their measurement properties.Methods: A two-phase systematic literature search was performed in four databases: MEDLINE via PubMed, Scopus, PsycINFO via OVID, and CINAHL via EBSCO. First, tools to assess the external validity of RCTs were identified. Second, studies aiming to investigate the measurement properties of these tools were selected. The measurement properties of each included tool were appraised using an adapted version of the COnsensus based Standards for the selection of health Measurement INstruments (COSMIN) guidelines.Results: 34 publications reporting on the development or validation of 26 included tools were included. For 62% of the included tools, there was no evidence of any measurement property. For the remaining tools, reliability was assessed most frequently. Reliability was judged as “sufficient” for three tools (very low quality of evidence). Content validity was rated as “sufficient” for one tool (moderate quality of evidence).Conclusions: Based on these results, no available tool can be fully recommended to assess the external validity of RCTs in systematic reviews. Several steps are required to overcome the identified difficulties to either adapt and validate available tools or to develop a new one. There is a need for more research for this purpose.Trial registration: Prospective registration at Open Science Framework (OSF): https://doi.org/10.17605/OSF.IO/PTG4D

Download Full-text

Virginia Is Not Switzerland: Challenging the External Validity of the European Randomized Controlled Trials Comparing Laparoscopic Gastric Bypass and Sleeve Gastrectomy

Journal of the American College of Surgeons ◽

10.1016/j.jamcollsurg.2018.08.206 ◽

2018 ◽

Vol 227 (4) ◽

pp. e76-e77 ◽

Cited By ~ 1

Author(s):

Guilherme S. Mazzini ◽

Jad Khoraki ◽

Matthew G. Brownning ◽

Luke G. Wolfe ◽

Gretchen R. Aquilina ◽

...

Keyword(s):

Gastric Bypass ◽

Sleeve Gastrectomy ◽

Randomized Controlled Trials ◽

Laparoscopic Gastric Bypass ◽

External Validity ◽

Controlled Trials ◽

Randomized Controlled

Download Full-text

Slow-pull compared to suction technique for EUS-guided sampling of pancreatic solid lesions: a meta-analysis of randomized controlled trials

Endoscopy International Open ◽

10.1055/a-1120-8428 ◽

2020 ◽

Vol 08 (05) ◽

pp. E636-E643 ◽

Cited By ~ 1

Author(s):

Gabriele Capurso ◽

Livia Archibugi ◽

Maria Chiara Petrone ◽

Paolo Giorgio Arcidiacono

Keyword(s):

Randomized Controlled Trials ◽

Sensitivity And Specificity ◽

Meta Analysis ◽

Controlled Trials ◽

Blood Contamination ◽

Randomized Controlled ◽

Bibliographic Search ◽

Accuracy Difference ◽

Solid Lesions ◽

High Degree

Abstract Background and study aims Current ESGE guidelines suggest employing the suction (SU) technique for endoscopic ultrasound (EUS)-guided sampling of pancreatic solid lesions. Nonetheless, recent randomized controlled trials (RCT) have reported that the slow-pull (SP) technique has similar diagnostic accuracy with possibly less blood contamination. However, these results are heterogeneous and limited to small cohorts. The aim of this meta-analysis was to compare adequacy, accuracy, sensitivity and specificity of the SU and SP techniques for EUS-guided sampling of solid pancreatic lesions. Methods A computerized bibliographic search was restricted to RCTs. Pooled effects were calculated using a random-effects model and expressed in terms of pooled sensitivity and specificity and OR (95 % CI) for adequacy and accuracy. Results Overall, seven RCTs were included, for a total of 475 patients (163 lesions sampled with SU, 164 with SP and 148 by both). The adequacy was similar (OR = 0.98) without heterogeneity (I2 = 0 %), but a high degree of blood contamination was more common with SU than SP (pooled rate 27.6 % vs 19.7 %). A non-significant superiority of SP in terms of pooled accuracy (OR = 0.82; 95 % CI 0.36–1.85) was recorded, with moderate heterogeneity (I2 = 52.4 %). The SP technique showed a slightly higher pooled sensitivity compared to SU (88.7 % vs 83.4 %), while specificity was similar (97.2 % SP vs 96.9 % SU), with considerable heterogeneity. Conclusion The current meta-analysis reveals non-superiority of SU over SP, while SP results in reduced blood contamination. If the 5 % accuracy difference favouring SP is true, with alfa error = 0.05 and beta = 0.20, a RCT of 982 patients per arm is needed to confirm significance.

Download Full-text

Assessment of generalizability, applicability and predictability (GAP) for evaluating external validity in studies of universal family-based prevention of alcohol misuse in young people: systematic methodological review of randomized controlled trials

Addiction ◽

10.1111/j.1360-0443.2012.03867.x ◽

2012 ◽

Vol 107 (9) ◽

pp. 1570-1579 ◽

Cited By ~ 19

Author(s):

Jose Ramon Fernandez-Hermida ◽

Amador Calafat ◽

Elisardo Becoña ◽

Alexander Tsertsvadze ◽

David R. Foxcroft

Keyword(s):

Young People ◽

Randomized Controlled Trials ◽

External Validity ◽

Alcohol Misuse ◽

Controlled Trials ◽

Methodological Review ◽

Randomized Controlled ◽

Universal Family ◽

Family Based

Download Full-text

External validity of randomized controlled trials in COPD

Respiratory Medicine ◽

10.1016/j.rmed.2006.10.011 ◽

2007 ◽

Vol 101 (6) ◽

pp. 1313-1320 ◽

Cited By ~ 80

Author(s):

Justin Travers ◽

Suzanne Marsh ◽

Brent Caldwell ◽

Mathew Williams ◽

Sarah Aldington ◽

...

Keyword(s):

Randomized Controlled Trials ◽

External Validity ◽

Controlled Trials ◽

Randomized Controlled

Download Full-text

Participant Retention in Randomized, Controlled Trials: The Value of Relational Engagement

International Journal for Human Caring ◽

10.20467/1091-5710.11.4.46 ◽

2007 ◽

Vol 11 (4) ◽

pp. 46-51 ◽

Cited By ~ 1

Author(s):

Elizabeth Andersen

Keyword(s):

Randomized Controlled Trials ◽

Internal Validity ◽

Cost Effective ◽

Small Sample ◽

Controlled Trials ◽

Potential Threat ◽

Retention Strategies ◽

Randomized Controlled ◽

Values And Beliefs ◽

Participant Retention

Although participant retention is critically important for the conduct of randomized controlled trials, relatively little is known about factors that lead to attrition. I argue that commonly used retention strategies are not cost effective in large trials because most participants do not value them. Attrition in experimental research occurs because we rely on conventional retention strategies and we neglect the unspoken values and beliefs of our participants. The most important of these is the value of relational engagement. Relational engagement does pose a potential threat to internal validity in trials with small sample sizes. This threat is discussed. In conclusion, I suggest that large trials should consider supporting relational engagement rather than spending money on conventional retention strategies.

Download Full-text