The methodological quality of 176,620 randomized controlled trials published between 1966 and 2018 reveals a positive trend but also an urgent need for improvement

Many randomized controlled trials (RCTs) are biased and difficult to reproduce due to methodological flaws and poor reporting. There is increasing attention for responsible research practices and implementation of reporting guidelines, but whether these efforts have improved the methodological quality of RCTs (e.g., lower risk of bias) is unknown. We, therefore, mapped risk-of-bias trends over time in RCT publications in relation to journal and author characteristics. Meta-information of 176,620 RCTs published between 1966 and 2018 was extracted. The risk-of-bias probability (random sequence generation, allocation concealment, blinding of patients/personnel, and blinding of outcome assessment) was assessed using a risk-of-bias machine learning tool. This tool was simultaneously validated using 63,327 human risk-of-bias assessments obtained from 17,394 RCTs evaluated in the Cochrane Database of Systematic Reviews (CDSR). Moreover, RCT registration and CONSORT Statement reporting were assessed using automated searches. Publication characteristics included the number of authors, journal impact factor (JIF), and medical discipline. The annual number of published RCTs substantially increased over 4 decades, accompanied by increases in authors (5.2 to 7.8) and institutions (2.9 to 4.8). The risk of bias remained present in most RCTs but decreased over time for allocation concealment (63% to 51%), random sequence generation (57% to 36%), and blinding of outcome assessment (58% to 52%). Trial registration (37% to 47%) and the use of the CONSORT Statement (1% to 20%) also rapidly increased. In journals with a higher impact factor (>10), the risk of bias was consistently lower with higher levels of RCT registration and the use of the CONSORT Statement. Automated risk-of-bias predictions had accuracies above 70% for allocation concealment (70.7%), random sequence generation (72.1%), and blinding of patients/personnel (79.8%), but not for blinding of outcome assessment (62.7%). In conclusion, the likelihood of bias in RCTs has generally decreased over the last decades. This optimistic trend may be driven by increased knowledge augmented by mandatory trial registration and more stringent reporting guidelines and journal requirements. Nevertheless, relatively high probabilities of bias remain, particularly in journals with lower impact factors. This emphasizes that further improvement of RCT registration, conduct, and reporting is still urgently needed.

Download Full-text

Randomized clinical trial quality has improved over time but is still not good enough: an analysis of 176,620 randomized controlled trials published between 1966 and 2018

10.1101/2020.04.22.20072371 ◽

2020 ◽

Author(s):

Christiaan H. Vinkers ◽

Herm J. Lamberink ◽

Joeri K. Tijdink ◽

Pauline Heus ◽

Lex Bouter ◽

...

Keyword(s):

Random Sequence ◽

Allocation Concealment ◽

Risk Of Bias ◽

Trial Registration ◽

Controlled Trials ◽

Sequence Generation ◽

Randomized Controlled ◽

Random Sequence Generation ◽

Over Time ◽

Number Of Authors

AbstractBackgroundMany randomized controlled trials (RCTs) are biased and difficult to reproduce due to methodological flaws and poor reporting. There is increasing attention for responsible research practices including reporting guidelines, but it is unknown whether these efforts have improved RCT quality (i.e. reduced risk of bias). We therefore mapped trends over time in trial publication, trial registration, reporting according to CONSORT, and characteristics of publication and authors.MethodsMeta-information of 176,620 RCTs published between 1966 and 2018 was extracted. Risk of bias probability (four domains: random sequence generation, allocation concealment, blinding of patients/personnel, and blinding of outcome assessment) was assessed using validated risk-of-bias machine learning tools. In addition, trial registration and reporting according to CONSORT were assessed with automated searches. Characteristics were extracted related to publication (number of authors, journal impact factor, medical discipline) and authors (gender and Hirsch-index).FindingsThe annual number of published RCTs substantially increased over four decades, accompanied by increases in the number of authors (5.2 to 7.8), institutions (2.9 to 4.8), female authors (20 to 42%, first authorship; 17 to 29%, last authorship), and Hirsch-indices (10 to 14, first authorship; 16 to 28, last authorship). Risk of bias remained present in most RCTs but decreased over time for the domains allocation concealment (63 to 51%), random sequence generation (57 to 36%), and blinding of outcome assessment (58 to 52%). Trial registration (37 to 47%) and CONSORT (1 to 20%) rapidly increased in the latest period. In journals with higher impact factor (>10), risk of bias was consistently lower, higher levels of trial registration more frequent, and mentioning CONSORT.InterpretationThe likelihood of bias in RCTs has generally decreased over the last decades. This may be driven by increased knowledge and improved education, augmented by mandatory trial registration, and more stringent reporting guidelines and journal requirements. Nevertheless, relatively high probabilities of bias remain, particularly in journals with lower impact factors. This emphasizes that further improvement of RCT registration, conduct, and reporting is still urgently needed.FundingThis study was funded by The Netherlands Organisation for Health Research and Development (445001002).

Download Full-text

Judgments of risk of bias associated with random sequence generation in trials included in Cochrane systematic reviews are frequently erroneous

10.1101/366674 ◽

2018 ◽

Cited By ~ 2

Author(s):

Ognjen Barcot ◽

Matija Boric ◽

Tina Poklepovic Pericic ◽

Marija Cavar ◽

Svjetlana Dosenovic ◽

...

Keyword(s):

Systematic Reviews ◽

Random Number ◽

Random Sequence ◽

Computer Software ◽

Risk Of Bias ◽

Controlled Trials ◽

Sequence Generation ◽

Randomized Controlled ◽

Random Sequence Generation ◽

Random Number Table

AbstractBackgroundPurpose of this study was to analyze adequacy of judgments about risk of bias (RoB) for random sequence generation in Cochrane systematic reviews (CSRs) of randomized controlled trials (RCTs).MethodsInformation was extracted from RoB tables of CSRs using automated data scraping. We categorized all comments provided as supports for judgments for RoB related to randomization. We analyzed number and type of various supporting comments and assessed adequacy of RoB judgment for randomization in line with recommendations from the Cochrane Handbook.ResultsWe analyzed 10527 RCTs that were included in 729 CSRs. For 5682 RCTs randomization was not described; for the others it was indicated randomization was done using computer/software/internet (N=2886), random number table (N=888), mechanic method (N=366), or it was incomplete/inappropriate (N=303).Overall, 1194/10125 trials (12%) had erroneous RoB judgment about randomization. The highest proportion of errors was found for trials with high RoB (28%), followed by those with low (19%), or unclear (3%). Therefore, one in eight judgments for the analyzed domain in CSRs was erroneous, and one in three if the judgment was “high risk”.ConclusionCochrane systematic reviews cannot be necessarily trusted when it comes to judgments for risk of bias related to randomized sequence generation.

Download Full-text

Effectiveness, quality and process evaluation of interventions to prevent childhood obesity.

Proceedings of The Nutrition Society ◽

10.1017/s0029665120002426 ◽

2020 ◽

Vol 79 (OCE2) ◽

Cited By ~ 1

Author(s):

Miguel Seral-Cortes ◽

Pilar De Miguel-Etayo ◽

Paola Zapata ◽

Luis Moreno-Aznar

Keyword(s):

Childhood Obesity ◽

Process Evaluation ◽

Systematic Reviews ◽

Random Sequence ◽

Allocation Concealment ◽

Risk Of Bias ◽

Global Public Health ◽

Program Outcomes ◽

Sequence Generation ◽

Random Sequence Generation

AbstractChildhood obesity is one of the most serious global public health challenges of the 21stcentury. Health and scientific organizations demand early interventions, although their complexity generates difficulties in their implementation. In systematic reviews, quality assessments are intended to limit mislead reporting and it is important they are conducted in a manner that minimizes bias. Process evaluation (PE) is used to monitor and document program implementation and can aid in understanding the relationship between specific program elements and program outcomes. Failure to deliver an intervention as intended can lead to unclear conclusions about the effectiveness of the intervention. The aim of the present study is to evaluate the effectiveness of the studies based on the process evaluation and the quality on its interventions. We performed a systematic review of randomized control trials aiming to prevent childhood obesity. The Cochrane Handbook for Systematic Reviews of Interventions tool was used to assess and report methodological risk of bias. Each item was judged as being at high (HR), low (LR) or unclear (UN) risk of bias as per criteria. Key domains random sequence generation and allocation concealment were also assessed. Effectiveness was estimated when the study showed changes in body composition as the main outcome (BMI z-score or waist circumference). Process evaluation was evaluated if at least one process evaluation indicator was identified as being adequately implemented. From the 41 studies, 26 showed any degree of effectiveness and 15 were not effective. Almost half of studies scored high risk of bias: 20/41 (Effective 13/26; non effective 7/15). Moreover, 16/41 studies reported to have unclear risk of bias (effective 10/26; and non-effective 6/15). Only 5/41 articles reported to have low risk (effective 3/26; non effective 2/15). Regarding random sequence generation, there was no difference according to the quality of the studies (effective: LR 25/26, UN 1/26 AND HR 0/26; non effective: LR 15/15, UN 0/15 AND HR 0/15). Concerning the allocation concealment domain, there were no differences found either (effective: LR 11/26, UN 10/26 AND HR 5/26; non effective: LR 7/15, UN 5/15 AND HR 3/15). PE was used in 7 papers (effective 3/7 and non-effective 4/7). Quality seems to have slightly more influence in the non-effective studies (LR 13.3%) than in the effective studies (LR 11.5%). The non-effective studies showed the highest proportion of performing PE. There seems to be a relationship between the quality and PE performance.

Download Full-text

Is Scientific Medical Literature Related to Endometriosis Treatment Evidence-Based? A Systematic Review on Methodological Quality of Randomized Clinical Trials

Medicina ◽

10.3390/medicina55070372 ◽

2019 ◽

Vol 55 (7) ◽

pp. 372

Author(s):

Roxana-Denisa Capraş ◽

Andrada Elena Urda-Cîmpean ◽

Sorana D. Bolboacă

Keyword(s):

Randomized Clinical Trials ◽

Random Sequence ◽

Scientific Evidence ◽

Allocation Concealment ◽

Sequence Generation ◽

Hierarchy Of Evidence ◽

Random Sequence Generation ◽

Level 1 ◽

Double Blinded

Background and objectives: Informed decision-making requires the ability to identify and integrate high-quality scientific evidence in daily practice. We aimed to assess whether randomized controlled trials (RCTs) on endometriosis therapy follow methodological criteria corresponding to the RCTs’ specific level in the hierarchy of evidence in such details to allow the reproduction and replication of the study. Materials and Methods: Using the keywords “therapy” and “endometriosis” and “efficacy” three bibliographic databases were searched for English written scientific articles published from 1 January 2008 to 3 March 2018. Only the randomized clinical trials (RCTs) were evaluated in terms of whether they provided the appropriate level of scientific evidence, equivalent to level 1, degree 1b in the hierarchy of evidence. A list of criteria to ensure study replication and reproduction, considering CONSORT guideline and MECIR standards, was developed and used to evaluate RCTs’ methodological soundness, and scores were granted. Three types of bias, namely selection bias (random sequence generation and allocation concealment), detection bias (blinding of outcome assessment), and attrition bias (incomplete outcome data) were also evaluated. Results: We found 387 articles on endometriosis therapy, of which 38 were RCTs: 30 double-blinded RCTs and 8 open-label RCTs. No article achieved the maximum score according to the evaluated methodological criteria. Even though 73.3% of the double-blinded RCTs had clear title, abstract, introduction, and objectives, only 13.3% provided precise information regarding experimental design and randomization, and also showed a low risk of bias. The blinding method was poorly reported in 43.3% of the double-blinded RCTs, while allocation concealment and random sequence generation were inadequate in 33.3% of them. Conclusions: None of the evaluated RCTs met all the methodological criteria, none had only a low risk of bias and provided sufficient details on methods and randomization to allow for the reproduction and replication of the study. Consequently, the appropriate level of scientific evidence (level 1, degree 1b) could not be granted. On endometriosis therapy, this study evaluated the quality of reporting in RCTs and not the quality of how the studies were performed.

Download Full-text

Risk of bias assessment of randomised controlled trials referenced in the 2015 American Heart Association guidelines update for cardiopulmonary resuscitation and emergency cardiovascular care: a cross-sectional review

BMJ Open ◽

10.1136/bmjopen-2018-023725 ◽

2019 ◽

Vol 9 (5) ◽

pp. e023725 ◽

Cited By ~ 2

Author(s):

Yongil Cho ◽

Changsun Kim ◽

Bossng Kang

Keyword(s):

Random Sequence ◽

Heart Association ◽

Allocation Concealment ◽

Risk Of Bias ◽

Cross Sectional ◽

Sequence Generation ◽

Random Sequence Generation ◽

Randomised Controlled ◽

Cardiovascular Care ◽

Unclear Risk

ObjectivesTo identify the risk of bias of randomised controlled trials (RCTs) referenced in the 2015 American Heart Association (AHA) guidelines update for cardiopulmonary resuscitation (CPR) and emergency cardiovascular care (ECC).DesignA cross-sectional review.SettingAll RCTs cited as references in the 2015 AHA guidelines update for CPR and ECC were extracted. After excluding non-human trials, studies that analysed existing RCTs, and RCTs published in a letter format, two reviewers assessed the risk of bias among RCTs included in this study.Outcome measuresThe Cochrane Collaboration’s tool for assessing the risk of bias in six domains (random sequence generation, allocation concealment, blinding of participants and personnel, blinding of outcome assessment, incomplete outcome data and selective reporting) was used.ResultsTwo hundred seventy-three RCTs were selected for the analyses. Of these RCTs, 78.8% had a high risk of bias for blinding of participants and personnel, mostly (87.7%) non-drug trials. In drug trials, the proportion of trials with a low risk of bias for blinding of participants and personnel was 73.0%. The proportion of RCTs with an unclear risk of bias were higher for random sequence generation (38.5%) and allocation concealment (34.1%) than in other domains. Unclear risk of bias proportions was 65.4% for random sequence generation and 57.7% for allocation concealment before the introduction of Consolidated Standards of Reporting Trials (CONSORT) but decreased to 31.3% and 32.2% after the 2010 CONSORT update, respectively.ConclusionsThe proportion of RCTs with an unclear risk of bias was still high for random sequence generation and allocation concealment in the 2015 AHA guidelines for CPR and ECC. The risk of bias should be considered when interpreting and applying the CPR guidelines. Authors should plan and report their research using CONSORT guidelines and the Cochrane Collaboration’s tool to reduce the risk of bias.

Download Full-text

Deficiencies of effectiveness of intervention studies in veterinary medicine: a cross-sectional survey of ten leading veterinary and medical journals

PeerJ ◽

10.7717/peerj.1649 ◽

2016 ◽

Vol 4 ◽

pp. e1649 ◽

Cited By ~ 11

Author(s):

Nicola Di Girolamo ◽

Reint Meursinge Reynders

Keyword(s):

Random Sequence ◽

Allocation Concealment ◽

Estimation Methods ◽

Random Allocation ◽

Cross Sectional Survey ◽

Cross Sectional ◽

Medical Journals ◽

Sequence Generation ◽

Random Sequence Generation

The validity of studies that assess the effectiveness of an intervention (EoI) depends on variables such as the type of study design, the quality of their methodology, and the participants enrolled. Five leading veterinary journals and 5 leading human medical journals were hand-searched for EoI studies for the year 2013. We assessed (1) the prevalence of randomized controlled trials (RCTs) among EoI studies, (2) the type of participants enrolled, and (3) the methodological quality of the selected studies. Of 1707 eligible articles, 590 were EoI articles and 435 RCTs. Random allocation to the intervention was performed in 52% (114/219; 95%CI:45.2–58.8%) of veterinary EoI articles, against 87% (321/371; 82.5–89.7%) of human EoI articles (adjusted OR:9.2; 3.4–24.8). Veterinary RCTs were smaller (median: 26 animals versus 465 humans) and less likely to enroll real patients, compared with human RCTs (OR:331; 45–2441). Only 2% of the veterinary RCTs, versus 77% of the human RCTs, reported power calculations, primary outcomes, random sequence generation, allocation concealment and estimation methods. Currently, internal and external validity of veterinary EoI studies is limited compared to human medical ones. To address these issues, veterinary interventional research needs to improve its methodology, increase the number of published RCTs and enroll real clinical patients.

Download Full-text

A qualitative assessment of randomized controlled trials in otolaryngology

The Journal of Laryngology & Otology ◽

10.1017/s0022215100140770 ◽

1998 ◽

Vol 112 (5) ◽

pp. 460-463 ◽

Cited By ~ 30

Author(s):

K. W. Ah-See ◽

N. C. Molony

Keyword(s):

Randomized Controlled Trials ◽

Consort Statement ◽

Qualitative Assessment ◽

Controlled Trials ◽

Randomized Controlled ◽

Future Assessment ◽

The Consort Statement ◽

The Mean ◽

Better Than

AbstractIn 1996 the CONSORT statement made recommendations on the strict reporting of randomized controlled trials (RCT). This will facilitate the future assessment of such trials and will highlight those trials that have been performed suboptimally and whose results may be biased.We have devised a scoring system, based on CONSORT, to assess RCT quality and by reading each original paper in full we have now assessed the quality of trials published from 1966 to 1995.The mean score for trials identified was 7.3 out of a maximum 12 points. No one journal was significantly better than the others. Trials in rhinology are reported better than head and neck oncology trials (mean scores 7.6 and 6.5 respectively). The past 30 years has not seen an improvement in the quality of the trials.The reporting of RCTs in the ENT literature is poor. CONSORT guidelines now exist and trialists are encouraged to adopt them when conducting future clinical trials.

Download Full-text

Risk of bias judgments for random sequence generation in Cochrane systematic reviews were frequently not in line with Cochrane Handbook

BMC Medical Research Methodology ◽

10.1186/s12874-019-0804-y ◽

2019 ◽

Vol 19 (1) ◽

Cited By ~ 10

Author(s):

Ognjen Barcot ◽

Matija Boric ◽

Tina Poklepovic Pericic ◽

Marija Cavar ◽

Svjetlana Dosenovic ◽

...

Keyword(s):

Systematic Reviews ◽

Random Sequence ◽

Risk Of Bias ◽

Sequence Generation ◽

Random Sequence Generation

Download Full-text

Impact of risk of bias on the magnitude of treatment effects of randomized controlled trials in implant dentistry

10.21203/rs.3.rs-41578/v1 ◽

2020 ◽

Author(s):

Ruolin Ding ◽

Wenxin Lu ◽

Jianru Yi ◽

Liang Zhang ◽

Zhihe Zhao

Keyword(s):

Randomized Controlled Trials ◽

Treatment Effects ◽

Permutation Test ◽

Random Sequence ◽

Allocation Concealment ◽

Risk Of Bias ◽

Selective Reporting ◽

Controlled Trials ◽

Randomized Controlled ◽

Implant Dentistry

Abstract Background: Risk of bias (RoB) could influence the magnitude of treatment effects of randomized controlled trials (RCTs). This study aims to investigate the potential influence of RoB on treatment effects estimates in RCTs in implant dentistry. Methods: The RCTs published in five leading oral implant journals during the recent five years were electronically searched. The RoB was assessed using the Cochrane Collaboration RoB tool. The meta-regression analysis and Monte Carlo permutation test were performed to identify the association between RoB and the magnitude of treatment effects.Results: A considerable amount of studies have high RoB in blinding of participants and personnel, and unclear RoB in allocation concealment and selective reporting. The treatment effects were exaggerated by flaws in allocation concealment for binary outcomes and by deficiencies in random sequence generation and selective reporting for continuous outcomes.Conclusion: RoB frequently exists in RCTs recently published in implant dentistry, which may lead to the exaggeration of treatment effects. Better study design, implementation, and reporting are required for clinical trials in implant dentistry to ensure more reliable evidence.

Download Full-text