An Empirical Validation of the Regression Point Displacement Design Using Within-Study Comparison Logic: Emerging Possibilities and Cautions

2022 ◽  
pp. 0193841X2110644
Author(s):  
Joshua Hendrickse ◽  
William H. Yeaton
Keyword(s):  
False Negative ◽  
Internal Validity ◽  
Primary Objective ◽  
Control Groups ◽  
Intervention Effect ◽  
Individual Level ◽  
Fixed Set ◽  
Alpha Level ◽  
Research Questions ◽  
Cluster Level

Background The regression point displacement (RPD) design is a quasi-experiment (QE) that aims to control many threats to internal validity. Though it has existed for several decades, RPD has only recently begun to answer applied research questions in lieu of stronger QEs. Objectives Our primary objective was to implement within-study comparison (WSC) logic to create RPD replicates and to determine conditions under which RPD might provide estimates comparable to those found in validating experiments. Research Design We utilize three randomized controlled trials (two cluster-level, one individual-level), artificially decomposing or creating cluster structures, to create multiple RPDs. We compare results in each RPD treatment group to a fixed set of control groups to gauge the congruence of these repeated RPD realizations with results found in these three RCTs. Results RPD’s performance was uneven. Using multiple criteria, we found that RPDs successfully predicted the direction of the RCT’s intervention effect but inconsistently fell within the .10 SD threshold. A scant 13% of RPD results were statistically significant at either the .05 or .01 alpha-level. RPD results were within the 95% confidence interval of RCTs around half the time, and false negative rates were substantially higher than false positive rates. Conclusions RPD consistently underestimates treatment effects in validating RCTs. We analyze reasons for this insensitivity and offer practical suggestions to improve the chances RPD will correctly identify favorable results. We note that the synthetic, “decomposition of cluster RCTs,” WSC design represents a prototype for evaluating other QEs.

scholarly journals A novel preference-informed complementary trial (PICT) design for clinical trial research influenced by strong patient preferences

Trials ◽  
2021 ◽  
Vol 22 (1) ◽  
Author(s):  
Samina Ali ◽  
◽  
Gareth Hopkin ◽  
Naveen Poonai ◽  
Lawrence Richer ◽  
...  
Keyword(s):  
External Validity ◽  
Treatment Options ◽  
Internal Validity ◽  
Clinical Effectiveness ◽  
Drop Out ◽  
Innovative Design ◽  
Clinical Trial Research ◽  
Research Questions ◽  
Internal And External Validity ◽  
Treatment Population

Abstract Background Patients and their families often have preferences for medical care that relate to wider considerations beyond the clinical effectiveness of the proposed interventions. Traditionally, these preferences have not been adequately considered in research. Research questions where patients and families have strong preferences may not be appropriate for traditional randomized controlled trials (RCTs) due to threats to internal and external validity, as there may be high levels of drop-out and non-adherence or recruitment of a sample that is not representative of the treatment population. Several preference-informed designs have been developed to address problems with traditional RCTs, but these designs have their own limitations and may not be suitable for many research questions where strong preferences and opinions are present. Methods In this paper, we propose a novel and innovative preference-informed complementary trial (PICT) design which addresses key weaknesses with both traditional RCTs and available preference-informed designs. In the PICT design, complementary trials would be operated within a single study, and patients and/or families would be given the opportunity to choose between a trial with all treatment options available and a trial with treatment options that exclude the option which is subject to strong preferences. This approach would allow those with strong preferences to take part in research and would improve external validity through recruiting more representative populations and internal validity. Here we discuss the strengths and limitations of the PICT design and considerations for analysis and present a motivating example for the design based on the use of opioids for pain management for children with musculoskeletal injuries. Conclusions PICTs provide a novel and innovative design for clinical trials with more than two arms, which can address problems with existing preference-informed trial designs and enhance the ability of researchers to reflect shared decision-making in research as well as improving the validity of trials of topics with strong preferences.

Does Sophistication Affect Electoral Outcomes?

2016 ◽  
Vol 53 (2) ◽  
pp. 181-204 ◽  
Author(s):  
Lauri Rapeli
Keyword(s):  
Future Research ◽  
Citizen Competence ◽  
Individual Level ◽  
Electoral Outcomes ◽  
Sociodemographic Determinants ◽  
Political Left ◽  
Electoral Behaviour ◽  
Research Questions ◽  
The Individual ◽  
The Right

It is widely assumed that a representative democracy requires an enlightened citizenry in order to function properly. The competence of citizens has been studied extensively and the sociodemographic determinants of political sophistication are particularly well known. Much less is known about whether and how citizen competence affects electoral behaviour and outcomes. This article reviews the existing literature on these topics. Despite the widespread consensus that, generally speaking, citizen competence matters for electoral outcomes, the review produced a mixed result: some studies suggest that the political left would benefit from a better-informed electorate, while other studies suggest the opposite. Although the majoritarian electoral context is overrepresented in the evidence, the review shows that at the individual level, political knowledge greatly increases a person’s ability to match personal preferences with the right candidate or party in an election. The article also identifies several gaps in existing knowledge, thereby suggesting future research questions.

scholarly journals An Improved Model for Evaluating Change in Randomized Pretest, Posttest, Follow-Up Designs

Methodology ◽  
2012 ◽  
Vol 8 (3) ◽  
pp. 97-103 ◽  
Author(s):  
Constance A. Mara ◽  
Robert A. Cribbie ◽  
David B. Flora ◽  
Cathy LaBrish ◽  
Laura Mills ◽  
...  
Keyword(s):  
Monte Carlo Simulation ◽  
Monte Carlo ◽  
Group Differences ◽  
Primary Research ◽  
Control Groups ◽  
Proposed Model ◽  
Research Questions ◽  
Improved Model ◽  
And Control

Randomized pretest, posttest, follow-up (RPPF) designs are often used for evaluating the effectiveness of an intervention. These designs typically address two primary research questions: (1) Do the treatment and control groups differ in the amount of change from pretest to posttest? and (2) Do the treatment and control groups differ in the amount of change from posttest to follow-up? This study presents a model for answering these questions and compares it to recently proposed models for analyzing RPPF designs due to Mun, von Eye, and White (2009) using Monte Carlo simulation. The proposed model provides increased power over previous models for evaluating group differences in RPPF designs.

Collecting and Analyzing Qualitative Data

2019 ◽  
pp. 213-228 ◽  
Author(s):  
Brent Wolff ◽  
Frank Mahoney ◽  
Anna Leena Lohiniva ◽  
Melissa Corkum
Keyword(s):  
Qualitative Data ◽  
Computer Software ◽  
Internal Validity ◽  
Large Data ◽  
Data Sets ◽  
Qualitative Approaches ◽  
Rapport Building ◽  
Research Questions ◽  
High Internal Validity ◽  
Rigorous Method

Qualitative research provides an adaptable, open-ended, rigorous method to explore local perceptions of an issue. Qualitative approaches are effective at revealing the subjective logic motivating behavior. They are particularly appropriate for research questions that are exploratory in nature or involve issues of meaning rather than magnitude or frequency. Key advantages of qualitative approaches include speed, flexibility and high internal validity resulting from emphasis on rapport building and ability to probe beneath the surface of initial responses. Given the time-intensive nature of qualitative analysis, samples tend to be small and purposively selected to assure every interview counts. Qualitative studies can be done independently or embedded in mixed-method designs. Qualitative data analysis depends on rigorous reading and rereading texts ideally with more than one analyst to confirm interpretations. Computer software is useful for analyzing large data sets but manual coding is often sufficient for rapid assessments in field settings..

Designing Experiments

2015 ◽  
pp. 82-96
Keyword(s):  
Experimental Design ◽  
Internal Validity ◽  
Random Selection ◽  
Experimental Designs ◽  
Control Groups ◽  
Test Measurement ◽  
Post Test ◽  
Quasi Experimental ◽  
And Control

In this chapter, students will learn the process of designing experiments. The classic experimental design is presented first. Following this, three distinct quasi-experimental designs are presented. The benefits and burdens of the classic and quasi-experimental designs are discussed in depth. By the end of this chapter, students will understand concepts related to random selection, generalizability, treatment and control groups, pre- and post-test measurement of the dependent variable, and internal validity.

A Decision-Analytic Approach to Addressing the Evidence About Football and Chronic Traumatic Encephalopathy

2019 ◽  
Vol 40 (04) ◽  
pp. 450-460
Author(s):  
Kevin P. Brand ◽  
Adam M. Finkel
Keyword(s):  
Public Health ◽  
Chronic Traumatic Encephalopathy ◽  
Health Effect ◽  
False Negative ◽  
Weight Of Evidence ◽  
False Negatives ◽  
Head Impacts ◽  
Alternative Hypotheses ◽  
Research Questions ◽  
Logical Fallacies

AbstractDoubts can be raised about almost any assertion that a particular exposure can lead to an increase in a given adverse health effect. Even some of the most well-accepted causal associations in public health, such as that linking cigarette smoking to increased lung cancer risk, have intriguing research questions remaining to be answered. The inquiry whether an exposure causes a disease is never wholly a yes/no question but ought to follow from an appraisal of the weight of evidence supporting the positive conclusion in light of any coherent theories casting doubt on this evidence and the data supporting these. More importantly, such an appraisal cannot be made sensibly without considering the relative consequences to public health and economic welfare of specific actions based on unwarranted credulity (false positives) versus unwarranted skepticism (false negatives). Here we appraise the weight of evidence for the premise that repeated head impacts (RHIs) in professional football can increase the incidence of chronic traumatic encephalopathy (CTE) and, in turn, cause a variety of cognitive and behavioral symptoms. We first dismiss four logical fallacies that should not affect the appraisal of the weight of evidence. We then examine four alternative hypotheses in which RHI is not associated with CTE or symptoms (or both), and we conclude that the chances are small that the RHI→ CTE→ symptoms link is coincidental or artifactual. In particular, we observe that there are many specific interventions for which, even under a skeptical appraisal of the weight of evidence, the costs of a false positive are smaller than the false negative costs of refusing to intervene.

Recidivism Among Justice-Involved Youth: Findings From JJ-TRIALS

2020 ◽  
Vol 47 (9) ◽  
pp. 1059-1078
Author(s):  
Angela A. Robertson ◽  
Zhou Fang ◽  
Doris Weiland ◽  
George Joe ◽  
Sheena Gardner ◽  
...  
Keyword(s):  
Drug Abuse ◽  
Juvenile Justice ◽  
County Level ◽  
Multilevel Analyses ◽  
Recidivism Rates ◽  
Policy Makers ◽  
Individual Level ◽  
Research Questions ◽  
Complex Relationships ◽  
Using Data

Recidivism, and the factors related to it, remains a highly significant concern among juvenile justice researchers, practitioners, and policy makers. Recent studies highlight the need to examine multiple measures of recidivism as well as conduct multilevel analyses of this phenomenon. Using data collected in a National Institute on Drug Abuse (NIDA)-funded Juvenile Justice-Translational Research on Interventions for Adolescents in the Legal System (JJ-TRIALS) cooperative agreement, we examined individual- and site-level factors related to 1-year recidivism among probation youth in 20 sites in five states to answer research questions related to how recidivism rates differ across sites and the relationships between individual-level variables and a county-level concentrated disadvantage measure and recidivism. Our findings of large site differences in recidivism rates, and complex relationships between individual and county-level predictors of recidivism, highlight the need for more nuanced, contextually informed, multilevel approaches in studying recidivism among juveniles.

Internal Validity of Group Experimental Research

1983 ◽  
Vol 4 (3) ◽  
pp. 77-83
Author(s):  
Jean R. Harber
Keyword(s):  
Research Design ◽  
Experimental Research ◽  
Internal Validity ◽  
Experimental Designs ◽  
Causal Relationships ◽  
Control Groups ◽  
Quasi Experimental ◽  
Research Studies

This article stresses the importance of controlling extraneous variables when studying educational problems. Various types of research studies are described. The experimental research design, which is ideally suited to detecting causal relationships if proper controls are used, and quasi-experimental procedures, which are employed when true experimental designs cannot be used, are discussed. Threats to internal validity are presented and hypothetical examples are given to illustrate these threats and the means of controlling them. The importance of utilizing control groups is illustrated.

scholarly journals Learning From Cluster Randomized Trials in Education: An Assessment of the Capacity of Studies to Determine What Works, For Whom, and Under What Conditions

2020 ◽  
Vol 42 (3) ◽  
pp. 354-374
Author(s):  
Jessaca Spybrook ◽  
Qi Zhang ◽  
Ben Kelcey ◽  
Nianbo Dong
Keyword(s):  
Randomized Trials ◽  
Educational Interventions ◽  
School Type ◽  
Cluster Randomized Trials ◽  
Individual Level ◽  
The Past ◽  
Moderator Effects ◽  
What Works ◽  
Cluster Randomized ◽  
Cluster Level

Over the past 15 years, we have seen an increase in the use of cluster randomized trials (CRTs) to test the efficacy of educational interventions. These studies are often designed with the goal of determining whether a program works, or answering the what works question. Recently, the goals of these studies expanded to include for whom and under what conditions an intervention is effective. In this study, we examine the capacity of a set of CRTs to provide rigorous evidence about for whom and under what conditions an intervention is effective. The findings suggest that studies are more likely to be designed with the capacity to detect potentially meaningful individual-level moderator effects, for example, gender, than cluster-level moderator effects, for example, school type.

Difference in Response Effort Across Sample Types: Perception or Reality?

2015 ◽  
Vol 8 (2) ◽  
pp. 202-208 ◽  
Author(s):  
Shan Ran ◽  
Mengqiao Liu ◽  
Lisa A. Marchiondo ◽  
Jason L. Huang
Keyword(s):  
Internal Validity ◽  
Sampling Strategy ◽  
Careful Consideration ◽  
Sampling Strategies ◽  
Response Effort ◽  
Potential Influence ◽  
Organizational Research ◽  
Research And Practice ◽  
Research Findings ◽  
Research Questions

Landers and Behrend (2015) question organizational researchers’ stubborn reliance on sample source to infer the validity of research findings, and they challenge the arbitrary distinctions researchers often make between sample sources. Unconditional favoritism toward particular sampling strategies (e.g., organizational samples) can restrict choices in methodology, which in turn may limit opportunities to answer certain research questions. Landers and Behrend (2015) contend that no sampling strategy is inherently superior (or inferior), and therefore, all types of samples warrant careful consideration before any validity-related conclusions can be made. Despite sound arguments, the focal article focuses its consideration on external validity and deemphasizes the potential influence of sample source on internal validity. Agreeing with the position that no samples are the “gold standard” in organizational research and practice, we focus on insufficient effort responding (IER; Huang, Curran, Keeney, Poposki, & DeShon, 2012) as a threat to internal validity across sample sources.

Sign in / Sign up

Export Citation Format

Share Document