scholarly journals Assessing the magnitude of reporting bias in a sample of recent systematic reviews - A comparison of a Prospero and Cochrane Sample

2021 ◽  
Author(s):  
Maximilian Siebert
Keyword(s):  
Systematic Reviews ◽  
Primary Outcome ◽  
Reporting Bias ◽  
Cross Sectional ◽  
Cochrane Database

Objective: To explore differences in published reviews and their respective protocols in a sample of 97 non-Cochrane systematic reviews (non-CSRs) and 97 Cochrane systematic reviews (CSRs) in terms of PICOS elements and to which extent they were reported.Study Design and Setting: We searched PubMed and Cochrane database to identify non-CSRs and CSRs that were published in 2018. Afterwards we searched for their corresponding Cochrane or PROSPERO protocol. The reviews were compared to their protocols. The primary outcome was the change from protocol to review in terms of PICOS elements.Results: More than the half of each sample (54.6% of CSRs & 67% of non-CSRs) (ARR 12.4% [12% ; 12.8%]) presented changes in PICOS elements. A total of 227 changes were identified, distributed as 108 (47.5%) in CSRs and 119 (52.5%) in non-CSRs. For both samples an approximate third of total changes corresponded to changes related to primary outcomes. Marked differences were found between samples with regards to declaration of changes. Only 4.2% of changes in PICOS items have been declared in non-CSRs compared to 57.4% in CSRs (ARR 53.2% [52.9% ; 53.5%]).Conclusion: CSRs showed better results than non-CSRs in terms of reporting changes and showed higher quality. Nevertheless, reporting of changes from protocol should be advanced and requires improvement in general. Limitation of this study lies in its cross-sectional nature.

Evaluation of the completeness of intervention reporting in Cochrane surgical systematic reviews using the TIDieR-SR checklist: a cross-sectional study

2020 ◽  
pp. bmjebm-2020-111417
Author(s):  
Matt Vassar ◽  
Matthew J Page ◽  
James Glasbey ◽  
Craig Cooper ◽  
Austin Jorski ◽  
...  
Keyword(s):  
Systematic Reviews ◽  
Primary Outcome ◽  
Cross Sectional Study ◽  
Cross Sectional ◽  
Surgical Interventions ◽  
Cochrane Database ◽  
Research Findings ◽  
Intervention Reporting ◽  
Intervention Description ◽  
Complete Reporting

IntroductionComplete reporting of systematic reviews of interventions is essential to the interpretation of research findings and the reproducibility of research results. The Template for Intervention Description and Replication (TIDieR) checklist—and the version specific to systematic reviews (TIDieR-SR)—was created to provide authors and researchers an evidence-based guide for reporting trial and systematic review interventions. In this study, we apply TIDieR-SR to Cochrane systematic reviews of surgical interventions.MethodsWe searched the Cochrane Database for relevant systematic reviews. Two investigators applied inclusion/exclusion criteria to all titles/abstracts and full texts. These same investigators extracted all data in duplicate while masked to the other’s data. The primary outcome was adherence to TIDieR-SR items.ResultsTwo hundred and thirty-eight systematic reviews were included. Overall, included SRs adhered to a median of 6 (IQR 5–7) out of eight TIDieR-SR items. The item with the lowest adherence was item 7 (share intervention materials, 1/238 (0.4%).DiscussionOur results are encouraging, but the generalisability of our findings is compromised by the inclusion of only Cochrane systematic reviews. Future reporting of intervention materials is likely to improve the application of effective surgical interventions in the clinical practice.

scholarly journals Do oncology researchers adhere to reproducible and transparent principles? A cross-sectional survey of published oncology literature

BMJ Open ◽  
2019 ◽  
Vol 9 (12) ◽  
pp. e033962 ◽  
Author(s):  
Corbin Walters ◽  
Zachery J Harter ◽  
Cole Wayant ◽  
Nam Vo ◽  
Michael Warren ◽  
...  
Keyword(s):  
Random Sample ◽  
Systematic Reviews ◽  
Primary Outcome ◽  
Reproducible Research ◽  
Cross Sectional Survey ◽  
Research Practices ◽  
Cross Sectional ◽  
Raw Data ◽  
Open Access Journals ◽  
Meta Analyses

ObjectivesAs much as 50%–90% of research is estimated to be irreproducible, costing upwards of $28 billion in USA alone. Reproducible research practices are essential to improving the reproducibility and transparency of biomedical research, such as including preregistering studies, publishing a protocol, making research data and metadata publicly available, and publishing in open access journals. Here we report an investigation of key reproducible or transparent research practices in the published oncology literature.DesignWe performed a cross-sectional analysis of a random sample of 300 oncology publications published from 2014 to 2018. We extracted key reproducibility and transparency characteristics in a duplicative fashion by blinded investigators using a pilot tested Google Form.Primary outcome measuresThe primary outcome of this investigation is the frequency of key reproducible or transparent research practices followed in published biomedical and clinical oncology literature.ResultsOf the 300 publications randomly sampled, 296 were analysed for reproducibility characteristics. Of these 296 publications, 194 contained empirical data that could be analysed for reproducible and transparent research practices. Raw data were available for nine studies (4.6%). Five publications (2.6%) provided a protocol. Despite our sample including 15 clinical trials and 7 systematic reviews/meta-analyses, only 7 included a preregistration statement. Less than 25% (65/194) of publications provided an author conflict of interest statement.ConclusionWe found that key reproducibility and transparency characteristics were absent from a random sample of published oncology publications. We recommend required preregistration for all eligible trials and systematic reviews, published protocols for all manuscripts, and deposition of raw data and metadata in public repositories.

scholarly journals The methodological quality of COVID-19 systematic reviews is low, except for Cochrane reviews: a meta-epidemiological study

2020 ◽  
Author(s):  
Yuki KATAOKA ◽  
Shiho Oide ◽  
Takashi Ariie ◽  
Yasushi Tsujimoto ◽  
Toshi A. Furukawa
Keyword(s):  
Epidemiological Study ◽  
Systematic Reviews ◽  
Methodological Quality ◽  
Mean Difference ◽  
Measurement Tool ◽  
Cross Sectional ◽  
Cochrane Reviews ◽  
Cochrane Database ◽  
The Mean

Objectives: The objective of this study was to investigate the methodological quality of COVID-19 systematic reviews (SRs) indexed in medRxiv and PubMed, compared with Cochrane COVID Reviews. Study Design and Setting: This is a cross-sectional meta-epidemiological study. We searched medRxiv, PubMed, and Cochrane Database of Systematic Reviews for SRs of COVID-19. We evaluated the methodological quality using A MeaSurement Tool to Assess systematic Reviews (AMSTAR) checklists. The maximum AMSTAR score is 11, and minimum is 0. Higher score means better quality. Results: We included 9 Cochrane reviews as well as randomly selected 100 non-Cochrane reviews in medRxiv and PubMed. Compared with Cochrane reviews (mean 9.33, standard deviation 1.32), the mean AMSTAR scores of the articles in medRxiv were lower (mean difference -2.85, 95%confidence intervals (CI): -0.96 to -4.74) and those in PubMed was also lower (mean difference -3.28, 95% CI: -1.40 to -5.15), with no difference between the latter two. Conclusions: It should be noted that AMSTAR is not a perfect tool of assessing quality SRs other than intervention. Readers should pay attention to the potentially low methodological quality of COVID-19 SRs in both PubMed and medRxiv but less so in Cochrane COVID reviews.

scholarly journals The role of pragmatism in explaining heterogeneity in meta-analyses of randomised trials: a protocol for a cross-sectional methodological review

BMJ Open ◽  
2017 ◽  
Vol 7 (9) ◽  
pp. e017887 ◽  
Author(s):  
Theresa Aves ◽  
Katherine S Allan ◽  
Daeria Lawson ◽  
Robby Nieuwlaat ◽  
Joseph Beyene ◽  
...  
Keyword(s):  
Systematic Review ◽  
Systematic Reviews ◽  
Primary Outcome ◽  
Cochrane Library ◽  
Pool Data ◽  
Cross Sectional ◽  
Rater Reliability ◽  
Methodological Review ◽  
Meta Analyses

IntroductionThere has been increasing interest in pragmatic trials methodology. As a result, tools such as the Pragmatic-Explanatory Continuum Indicator Summary-2 (PRECIS-2) are being used prospectively to help researchers design randomised controlled trials (RCTs) within the pragmatic-explanatory continuum. There may be value in applying the PRECIS-2 tool retrospectively in a systematic review setting as it could provide important information about how to pool data based on the degree of pragmatism.ObjectivesTo investigate the role of pragmatism as a source of heterogeneity in systematic reviews by (1) identifying systematic reviews with meta-analyses of RCTs that have moderate to high heterogeneity, (2) applying PRECIS-2 to RCTs of systematic reviews, (3) evaluating the inter-rater reliability of PRECIS-2, (4) determining how much of this heterogeneity may be explained by pragmatism.MethodsA cross-sectional methodological review will be conducted on systematic reviews of RCTs published in the Cochrane Library from 1 January 2014 to 1 January 2017. Included systematic reviews will have a minimum of 10 RCTs in the meta-analysis of the primary outcome and moderate to substantial heterogeneity (I2≥50%). Of the eligible systematic reviews, a random selection of 10 will be included for quantitative evaluation. In each systematic review, RCTs will be scored using the PRECIS-2 tool, in duplicate. Agreement between raters will be measured using the intraclass correlation coefficient. Subgroup analyses and meta-regression will be used to evaluate how much variability in the primary outcome may be due to pragmatism.DisseminationThis review will be among the first to evaluate the PRECIS-2 tool in a systematic review setting. Results from this research will provide inter-rater reliability information about PRECIS-2 and may be used to provide methodological guidance when dealing with pragmatism in systematic reviews and subgroup considerations. On completion, this review will be submitted to a peer-reviewed journal for publication.

scholarly journals Outcome reporting bias in Cochrane systematic reviews: a cross-sectional analysis

BMJ Open ◽  
2020 ◽  
Vol 10 (3) ◽  
pp. e032497
Author(s):  
Kieran Shah ◽  
Gregory Egan ◽  
Lawrence (Nichoe) Huan ◽  
Jamie Kirkham ◽  
Emma Reid ◽  
...  
Keyword(s):  
Systematic Review ◽  
High Risk ◽  
Systematic Reviews ◽  
Cochrane Review ◽  
Reporting Bias ◽  
Outcome Reporting Bias ◽  
Cross Sectional ◽  
Knowledge Of Results ◽  
Outcome Reporting ◽  
Cochrane Reviews

BackgroundDiscrepancies in outcome reporting (DOR) between protocol and published studies include inclusions of new outcomes, omission of prespecified outcomes, upgrade and downgrade of secondary and primary outcomes, and changes in definitions of prespecified outcomes. DOR can result in outcome reporting bias (ORB) when changes in outcomes occur after knowledge of results. This has potential to overestimate treatment effects and underestimate harms. This can also occur at the level of systematic reviews when changes in outcomes occur after knowledge of results of included studies. The prevalence of DOR and ORB in systematic reviews is unknown in systematic reviews published post-2007.ObjectiveTo estimate the prevalence of DOR and risk of ORB in all Cochrane reviews between the years 2007 and 2014.MethodsA stratified random sampling approach was applied to collect a representative sample of Cochrane systematic reviews from each Cochrane review group. DOR was assessed by matching outcomes in each systematic review with their respective protocol. When DOR occurred, reviews were further assessed if there was a risk of ORB (unclear, low or high risk). We classified DOR as a high risk for ORB if the discrepancy occurred after knowledge of results in the systematic review.Results150 of 350 (43%) review and protocol pairings contained DOR. When reviews were further scrutinised, 23% (35 of 150) of reviews with DOR contained a high risk of ORB, with changes being made after knowledge of results from individual trials.ConclusionsIn our study, we identified just under a half of Cochrane reviews with at least one DOR. Of these, a fifth were at high risk of ORB. The presence of DOR and ORB in Cochrane reviews is of great concern; however, a solution is relatively simple. Authors are encouraged to be transparent where outcomes change and to describe the legitimacy of changing outcomes in order to prevent suspicion of bias.

scholarly journals Spin in the reporting, interpretation, and extrapolation of adverse effects of orthodontic interventions: protocol for a cross-sectional study of systematic reviews

2019 ◽  
Vol 4 (1) ◽  
Author(s):  
Pauline A. J. Steegmans ◽  
Nicola Di Girolamo ◽  
Reint A. Meursinge Reynders
Keyword(s):  
Adverse Effects ◽  
Systematic Reviews ◽  
Linear Models ◽  
Data Extraction ◽  
Policy Implications ◽  
Eligibility Criteria ◽  
Cross Sectional ◽  
Cochrane Reviews ◽  
Cochrane Database ◽  
Study Results

Abstract Background Titles and abstracts are the most read sections of biomedical papers. It is therefore important that abstracts transparently report both the beneficial and adverse effects of health care interventions and do not mislead the reader. Misleading reporting, interpretation, or extrapolation of study results is called “spin”. In this study, we will assess whether adverse effects of orthodontic interventions were reported or considered in the abstracts of both Cochrane and non-Cochrane reviews and whether spin was identified and what type of spin. Methods Eligibility criteria were defined for the type of study designs, participants, interventions, outcomes, and settings. We will include systematic reviews of clinical orthodontic interventions published in the five leading orthodontic journals and in the Cochrane Database. Empty reviews will be excluded. We will manually search eligible reviews published between 1 August 2009 and 31 July 2019. Data collection forms were developed a priori. All study selection and data extraction procedures will be conducted by two reviewers independently. Our main outcomes will be the prevalence of reported or considered adverse effects of orthodontic interventions in the abstract of systematic reviews and the prevalence of “spin” related to these adverse effects. We will also record the prevalence of three subtypes of spin, i.e., misleading reporting, misleading interpretation, and misleading extrapolation-related spin. All statistics will be calculated for the following groups: (1) all journals individually, (2) all journals together, and (3) the five leading orthodontic journals and the Cochrane Database of Systematic Reviews separately. Generalized linear models will be developed to compare the various groups. Discussion We expect that our results will raise the awareness of the importance of reporting and considering of adverse effects and the presence of the phenomenon of spin related to these effects in abstracts of systematic reviews of orthodontic interventions. This is important, because an incomplete and inadequate reporting, interpretation, or extrapolation of findings on adverse effects in abstracts of systematic reviews can mislead readers and could lead to inadequate clinical practice. Our findings could result in policy implications for making judgments about the acceptance for publication of systematic reviews of orthodontic interventions.

Patient Tolerability of Bowel Preparation is Associated with Polyp Detection Rate During Colonoscopy

2014 ◽  
Vol 23 (2) ◽  
pp. 135-140 ◽  
Author(s):  
Edward W. Holt ◽  
Kidist K. Yimam ◽  
Hanley Ma ◽  
Richard E. Shaw ◽  
Richard A. Sundberg ◽  
...  
Keyword(s):  
Bowel Preparation ◽  
Primary Outcome ◽  
Secondary Outcome ◽  
Polyp Detection ◽  
Cross Sectional ◽  
Quality Outcomes ◽  
Negative Experience ◽  
Polyp Detection Rate ◽  
Patient Reported ◽  
Bowel Prep

Background & Aims: A number of factors have been identified that influence the yield of screeningcolonoscopy. The perceived tolerability of bowel preparation has not been studied as a predictor of quality outcomes in colonoscopy. We aimed to characterize the association between patient-perceived tolerability of bowel preparation and polyp detection during colonoscopy.Methods: We performed a cross-sectional cohort study of 413 consecutive adult patients presenting foroutpatient colonoscopy at two outpatient endoscopy centers at our institution. We developed a standardized questionnaire to assess the patient's experience with bowel preparation. Bowel preparation quality was measured using the validated Ottawa scale and colonoscopic findings were recorded for each patient. The primary outcome was polyp detection and the secondary outcome was the quality of bowel preparation.Results: Patient-reported clarity of effluent during bowel preparation correlated poorly with Ottawa score during colonoscopy, k=0.15. Female gender was an independent risk factor for a poorly tolerated bowel prep (OR 3.93, 95% CI 2.30 - 6.72, p<0.001). Report of a poorly tolerated bowel prep was independently associated with the primary outcome, polyp detection (OR 0.39, 95% CI 0.18 - 0.84, p=0.02) and also with the secondary outcome, lower quality bowel preparation (OR 2.39, 95% CI 1.17 - 4.9, p=0.02).Conclusions: A patient-perceived negative experience with bowel preparation independently predicted both a lower quality bowel preparation and a lower rate of polyp of detection. Assessment of the tolerability of bowel preparation before colonoscopy may be a clinically useful predictor of quality outcomes during colonoscopy.

scholarly journals Misconceptions about COVID-19 among older Rohingya (forcefully displaced Myanmar nationals) adults in Bangladesh: findings from a cross-sectional study

BMJ Open ◽  
2021 ◽  
Vol 11 (5) ◽  
pp. e050427
Author(s):  
Sabuj Kanti Mistry ◽  
A R M Mehrab Ali ◽  
Uday Narayan Yadav ◽  
Saruna Ghimire ◽  
Md. Belal Hossain ◽  
...  
Keyword(s):  
Primary Outcome ◽  
Health Workers ◽  
Multiple Linear Regression Model ◽  
Cross Sectional Study ◽  
Secondary Outcome ◽  
Cross Sectional ◽  
Factors Associated ◽  
Cumulative Score ◽  
Low Health Literacy ◽  
Communication Frequency

ObjectivesDue to low health literacy and adverse situation in the camps, there are possibilities of misconceptions related to COVID-19 among the older Rohingya (forcefully displaced Myanmar nationals or FDMNs) adults in Bangladesh. The present research aimed to assess the level of misconceptions and the factors associated with it among the older FDMNs in Bangladesh.DesignCross-sectional.SettingA selected Rohingya camp situated in Cox’s Bazar, a southeastern district of Bangladesh.ParticipantsInformation was collected from 416 conveniently selected FDMNs who were aged 60 years and above.Primary and secondary outcome measuresThe primary outcome was misconceptions related to the spread, prevention and treatment of COVID-19. Information on 14 different locally relevant misconceptions was gathered, each was scored as one, and obtained a cumulative score, ranging from 0 to 14, with a higher score indicating a higher level of misconceptions. A multiple linear regression model explored the factors associated with misconceptions.ResultsThe participants had an average of five misconceptions. The most prevalent misconceptions were: everyone should wear personal protective equipment when outside (84.6%) and its prevention by nutritious food (62.5%) and drinking water (59.3%). Other notable misconceptions included the spread of COVID-19 through mosquito bites (42%) and its transmissions only to the non/less religious person (31.4%). In regression analyses, memory or concentration problems, communication frequency with social networks, pre-existing conditions and receiving information from health workers were significantly associated with higher COVID-19 misconceptions. These misconceptions were less likely among those overwhelmed by COVID-19, having COVID-19 diagnosed friends or family members and receiving information from friends and family.ConclusionsOverall, we found that misconceptions were prevalent among the older FDMNs in Bangladesh. The associations have important implications for programmes to prevent and manage COVID-19 in these settings. Health workers need to be adequately trained to provide clear communication and counter misconceptions.

scholarly journals Evaluating the relationship between citation set size, team size and screening methods used in systematic reviews: a cross-sectional study

2021 ◽  
Vol 21 (1) ◽  
Author(s):  
Katie O’Hearn ◽  
Cameron MacDonald ◽  
Anne Tsampalieros ◽  
Leo Kadota ◽  
Ryan Sandarage ◽  
...  
Keyword(s):  
Systematic Reviews ◽  
Full Text ◽  
Gold Standard ◽  
Data Extraction ◽  
Statistical Testing ◽  
Computer Assisted ◽  
Screening Methods ◽  
Team Size ◽  
Cross Sectional ◽  
Set Size

Abstract Background Standard practice for conducting systematic reviews (SRs) is time consuming and involves the study team screening hundreds or thousands of citations. As the volume of medical literature grows, the citation set sizes and corresponding screening efforts increase. While larger team size and alternate screening methods have the potential to reduce workload and decrease SR completion times, it is unknown whether investigators adapt team size or methods in response to citation set sizes. Using a cross-sectional design, we sought to understand how citation set size impacts (1) the total number of authors or individuals contributing to screening and (2) screening methods. Methods MEDLINE was searched in April 2019 for SRs on any health topic. A total of 1880 unique publications were identified and sorted into five citation set size categories (after deduplication): < 1,000, 1,001–2,500, 2,501–5,000, 5,001–10,000, and > 10,000. A random sample of 259 SRs were selected (~ 50 per category) for data extraction and analysis. Results With the exception of the pairwise t test comparing the under 1000 and over 10,000 categories (median 5 vs. 6, p = 0.049) no statistically significant relationship was evident between author number and citation set size. While visual inspection was suggestive, statistical testing did not consistently identify a relationship between citation set size and number of screeners (title-abstract, full text) or data extractors. However, logistic regression identified investigators were significantly more likely to deviate from gold-standard screening methods (i.e. independent duplicate screening) with larger citation sets. For every doubling of citation size, the odds of using gold-standard screening decreased by 15 and 20% at title-abstract and full text review, respectively. Finally, few SRs reported using crowdsourcing (n = 2) or computer-assisted screening (n = 1). Conclusions Large citation set sizes present a challenge to SR teams, especially when faced with time-sensitive health policy questions. Our study suggests that with increasing citation set size, authors are less likely to adhere to gold-standard screening methods. It is possible that adjunct screening methods, such as crowdsourcing (large team) and computer-assisted technologies, may provide a viable solution for authors to complete their SRs in a timely manner.

Sign in / Sign up

Export Citation Format

Share Document