scholarly journals Evaluating the degree of “unhappy randomization” in psychology: Are psychology experiments appropriately arranged to counter internal validity threats?

2017 ◽  
Author(s):  
Chih-Long Yen ◽  
Chung-Ping Cheng
Keyword(s):  
Randomized Controlled Trials ◽  
High Probability ◽  
Internal Validity ◽  
Medium Effect ◽  
Controlled Trials ◽  
Random Assignment ◽  
Causal Inferences ◽  
Randomized Controlled ◽  
Medium Effect Size ◽  
Psychological Experiments

This study sought to evaluate whether the randomized controlled trials currently conducted in psychology are suitable for countering internal validity threats in terms of precluding alternative explanations. Experimental research primarily achieves the goal of drawing causal inferences through random assignment. As such, one of the threats to the internal validity of experiments is when random assignment fails to balance nuisance variables, thereby causing the treatment effect to be confounded by nuisance variables. The authors argue that the successful random assignment is largely decided by the sample size and the confounding effects that the researchers are willing to tolerate. In this study, one thousand and eighteen randomized controlled trials published in three leading psychology journals were investigated in order to estimate the probability that nuisance variables were balanced in these experiments. The results indicated that, even if researchers are willing to tolerate a medium effect size of confounding effects, there is still a high probability that over one-fifth (22.99%) of the psychological experiments investigated could be confounded by nuisance variables, and the situation would be even worse if a stricter criterion was set. Some suggestions are proposed for enhancing the internal validity of future experiments.

Participant Retention in Randomized, Controlled Trials: The Value of Relational Engagement

2007 ◽  
Vol 11 (4) ◽  
pp. 46-51 ◽  
Author(s):  
Elizabeth Andersen
Keyword(s):  
Randomized Controlled Trials ◽  
Internal Validity ◽  
Cost Effective ◽  
Small Sample ◽  
Controlled Trials ◽  
Potential Threat ◽  
Retention Strategies ◽  
Randomized Controlled ◽  
Values And Beliefs ◽  
Participant Retention

Although participant retention is critically important for the conduct of randomized controlled trials, relatively little is known about factors that lead to attrition. I argue that commonly used retention strategies are not cost effective in large trials because most participants do not value them. Attrition in experimental research occurs because we rely on conventional retention strategies and we neglect the unspoken values and beliefs of our participants. The most important of these is the value of relational engagement. Relational engagement does pose a potential threat to internal validity in trials with small sample sizes. This threat is discussed. In conclusion, I suggest that large trials should consider supporting relational engagement rather than spending money on conventional retention strategies.

scholarly journals Issues in the statistical detection of data fabrication and data errors in the scientific literature: simulation study and reanalysis of Carlisle, 2017

2017 ◽  
Author(s):  
Scott W. Piraino
Keyword(s):  
Randomized Controlled Trials ◽  
Randomized Trials ◽  
Screening Tool ◽  
Scientific Literature ◽  
Controlled Trials ◽  
Large Set ◽  
Random Assignment ◽  
Data Errors ◽  
Randomized Controlled ◽  
Statistical Detection

AbstractBackgroundThe detection of fabrication or error within the scientific literature is an important and underappreciated problem. Retraction of scientific articles is rare, but retraction may also be conservative, leaving open the possiblity that many fabricated or erroneous findings remain in the literature as a result of lack of scrutiny. A recently statistical analysis of randomized controlled trials [1] has suggested that the reported statistics form these trials deviate substantially from expectation under truely random assignment, raising the possiblity of fraud or error. It has also been proposed that the method used could be implemented to prospectively screen research, for example by applying the method prior to publication.Methods and FindingsTo assess the properties of the method proposed in [1], I carry out both theoretical and empirical evaluations of the method. Simulations suggest that the method is sensitive to assumptions that could reasonably be violated in real randomized controlled trials. This suggests that deviation for expectation under this method can not be used to measure the extent of fraud or error within the literature, and raises questions about the utlity of the method for propsective screening. Empirically analysis of the results of the method on a large set of randomized trials suggests that important assumptions may plausibly be violated within this sample. Using retraction as a proxy for fraud or serious error, I show that the method faces serious challenges in terms of precision and sensitivity for the purposes of screening, and that the performance of the method as a screening tool may vary across journals and classes of retractions.ConclusionsThe results in [1] should not be interpreted as indicating large amount of fraud or error within the literature. The use of this method for screening of the literature should be undertaken with great caution, and should recognize critical challenges in interpreting the results of this method.

scholarly journals Resource Group Assertive Community Treatment (RACT) as a Tool of Empowerment for Clients with Severe Mental Illness: A Meta-Analysis

2012 ◽  
Vol 8 (1) ◽  
pp. 144-151 ◽  
Author(s):  
Tommy Nordén ◽  
Ulf Malm ◽  
Torsten Norlander
Keyword(s):  
Randomized Controlled Trials ◽  
Assertive Community Treatment ◽  
Meta Analysis ◽  
Well Being ◽  
Community Treatment ◽  
Medium Effect ◽  
Controlled Trials ◽  
Psychiatric Diagnoses ◽  
Randomized Controlled ◽  
Resource Group

The aim of the current meta-analysis was to explore the effectiveness of the method here labeled Resource Group Assertive Community Treatment (RACT) for clients with psychiatric diagnoses as compared to standard care during the period 2001 – 2011. Included in the meta-analysis were 17 studies comprising a total of 2263 clients, 1291 men and 972 women, with a weighted mean age of 45.44 years. The diagnoses of 86 % of the clients were within the psychotic spectrum while 14 % had other psychiatric diagnoses. There were six randomized controlled trials and eleven observational studies. The studies spanned between 12 and 60 months, and 10 of them lasted 24 months. The results indicated a large effect-size for the ”grand total measure” (Cohen´sd= 0.80). The study comprised three outcome variables: Symptoms, Functioning, and Well-being. With regard to Symptoms, a medium effect for both randomized controlled trials and non-randomized studies was found, whereas Functioning showed large effects for both types of design. Concerning Well-being both large and medium effects were evident. The conclusions of the meta-analysis were that the treatment of clients with Resource Group Assertive Community Treatment yields positive effects for clients with psychoses and that the method may be of use for clients within the entire psychiatric spectrum.

Evidence-Based Medicine: Determining Quality of Randomized Controlled Trials

2014 ◽  
Vol 19 (4) ◽  
pp. 3-5
Author(s):  
Melissa Cheng
Keyword(s):  
Randomized Controlled Trials ◽  
Evidence Based Medicine ◽  
Internal Validity ◽  
Cochrane Collaboration ◽  
Dropout Rate ◽  
Controlled Trials ◽  
Evidence Based ◽  
Double Blind ◽  
Randomized Controlled ◽  
Based Medicine

Abstract Evidence-based medicine is based on evidence gathered by randomized controlled trials (RCTs), which are considered the “gold standard” of research studies. Users of the medial literature must be able to read critically and evaluate RCTs. The present article uses the standards of the American College of Occupational and Environmental Medicine (ACOEM) practice guidelines in evaluating RCTs according to 11 criteria; these criteria follow those used by the Cochrane Collaboration in their evidence-based reviews. Well-written articles present their randomization schemes to create comparable groups, and studies must be controlled for co-interventions; in a double-blind trial, the co-interventions would be used equally in both groups, and treatment allocations should be concealed. Readers should ask if the study had acceptable compliance; that is, were patients doing what they were asked, and was the dropout rate acceptable (typically, less than 20%)? RCTs should be analyzed by an intention-to-treat analysis that includes all study subjects who were randomized, not just those who completed the study. Having high internal validity ensures a more accurate study that can be reproduced by others, so readers may ask if results are likely to be affected by observational bias, confounding, or chance variation. Readers can determine external validity by assessing study participants according to inclusion and exclusion criteria and baseline characteristics.

scholarly journals Experimental and Non-Experimental Methods in Development Economics: A Porous Dialectic

2015 ◽  
Vol 6 (1) ◽  
Author(s):  
Rajeev Dehejia
Keyword(s):  
Randomized Controlled Trials ◽  
External Validity ◽  
Internal Validity ◽  
Experimental Methods ◽  
Controlled Trials ◽  
Data Sets ◽  
Randomized Controlled ◽  
Nationally Representative ◽  
High Degree ◽  
Internal And External Validity

AbstractThis paper surveys six widely-used non-experimental methods for estimating treatment effects (instrumental variables, regression discontinuity, direct matching, propensity score matching, linear regression and non-parametric methods, and difference-in-differences), and assesses their internal and external validity relative both to each other and to randomized controlled trials. While randomized controlled trials can achieve the highest degree of internal validity when cleanly implemented in the field, the availability of large, nationally representative data sets offers the opportunity for a high degree of external validity using non-experimental methods. We argue that each method has merits in some context and they are complements rather than substitutes.

scholarly journals Empirical Evidence of an Association Between Internal Validity and Effect Size in Randomized Controlled Trials of Low-Back Pain

Spine ◽  
2009 ◽  
Vol 34 (16) ◽  
pp. 1685-1692 ◽  
Author(s):  
Maurits W. van Tulder ◽  
Marika Suttorp ◽  
Sally Morton ◽  
Lex M. Bouter ◽  
Paul Shekelle
Keyword(s):  
Low Back Pain ◽  
Back Pain ◽  
Randomized Controlled Trials ◽  
Empirical Evidence ◽  
Effect Size ◽  
Internal Validity ◽  
Controlled Trials ◽  
Low Back ◽  
Randomized Controlled

scholarly journals A Review of Existing Treatments for Substance abuse among the Elderly and Recommendations for Future Directions

2013 ◽  
Vol 7 ◽  
pp. SART.S7865 ◽  
Author(s):  
Alexis Kuerbis ◽  
Paul Sacco
Keyword(s):  
Substance Abuse ◽  
Randomized Controlled Trials ◽  
Healthcare System ◽  
Behavioral Therapy ◽  
Specific Treatment ◽  
Internal Validity ◽  
Future Research ◽  
Controlled Trials ◽  
Randomized Controlled ◽  
The Impact

Background With population aging, there is widespread recognition that the healthcare system must be prepared to serve the unique needs of substance using older adults (OA) in the decades ahead. As such, there is an increasingly urgent need to identify efficient and effective substance abuse treatments (SAT) for OA. Despite this need, there remains a surprising dearth of research on treatment for OA. Aims of review This review describes and evaluates studies on SAT applied to and specifically designed for OA over the last 30 years with an emphasis on methodologies used and the knowledge gained. Methods Using three research databases, 25 studies published in the last 30 years which investigated the impact of SAT on OA and met specific selection criteria were reviewed. Results A majority of the studies were methodologically limited in that they were pre-to-post or post-test only studies. Of the randomized controlled trials, many were limited by sample sizes of 15 individuals or less per group, making main effects difficult to detect. Thus, with caution, the literature suggests that among treatment seeking OA, treatment, whether age-specific or mixed-age, generally works yielding rates of abstinence comparable to general populations and younger cohorts. It also appears that with greater treatment exposure (higher dosage), regardless of level of care, OA do better. Finally, based on only two studies, age-specific treatment appears to potentiate treatment effects for OA. Like younger adults, OA appear to have a heterogeneous response to treatments, and preliminary evidence suggests a possibility of treatment matching for OA. Conclusions Expansion of research on SAT for OA is urgently needed for maximum effectiveness and efficiency of the healthcare system serving these individuals. Future research needs to include laboratory and community based randomized controlled trials with high internal validity of previously vetted evidenced-based practices, including Motivational Interviewing, cognitive behavioral therapy, and medications such as naltrexone, to determine the best fit for OA.

scholarly journals Effect of Kegel Exercises on the Management of Female Stress Urinary Incontinence: A Systematic Review of Randomized Controlled Trials

2014 ◽  
Vol 2014 ◽  
pp. 1-10 ◽  
Author(s):  
Seong-Hi Park ◽  
Chang-Bum Kang
Keyword(s):  
Urinary Incontinence ◽  
Stress Urinary Incontinence ◽  
Randomized Controlled Trials ◽  
Meta Analysis ◽  
Internal Validity ◽  
Female Stress Urinary Incontinence ◽  
Controlled Trials ◽  
Inclusion Criteria ◽  
Randomized Controlled ◽  
Kegel Exercises

Objective. The purpose of this study was to evaluate the effect of Kegel exercises on reducing urinary incontinence symptoms in women with stress urinary incontinence. Methods. Randomized controlled trials (RCTs) were conducted on females with stress urinary incontinence who had done Kegel exercises and met inclusion criteria in articles published between 1966 and 2012. The articles from periodicals indexed in KoreaMed, NDSL, Ovid Medline, Embase, Scopus, and other databases were selected, using key terms such as “Kegel” or “pelvic floor exercise.” Cochrane’s risk of bias was applied to assess the internal validity of the RCTs. Eleven selected studies were analyzed by meta-analysis using RevMan 5.1. Results. Eleven trials involving 510 women met the inclusion criteria. All trials contributed data to one or more of the main or secondary outcomes. They indicated that Kegel exercises significantly reduced the urinary incontinence symptoms of female stress urinary incontinence. There was no heterogeneity in the selected studies except the standardized bladder volumes of the pad test. Conclusion. There is some evidence that, for women with stress urinary incontinence, Kegel exercises may help manage urinary incontinence. However, while these results are helpful for understanding how to treat or cure stress urinary incontinence, further research is still required.

Sign in / Sign up

Export Citation Format

Share Document