scholarly journals Validity of observational evidence on putative risk and protective factors: appraisal of 3744 meta-analyses on 57 topics

BMC Medicine ◽  
2021 ◽  
Vol 19 (1) ◽  
Author(s):  
Perrine Janiaud ◽  
Arnav Agarwal ◽  
Ioanna Tzoulaki ◽  
Evropi Theodoratou ◽  
Konstantinos K. Tsilidis ◽  
...  
Keyword(s):  
Protective Factors ◽  
Observational Studies ◽  
Prediction Interval ◽  
Statistical Significance ◽  
Median Number ◽  
Small Study ◽  
Study Heterogeneity ◽  
Large Heterogeneity ◽  
Meta Analyses ◽  
No Gold Standard

Abstract Background The validity of observational studies and their meta-analyses is contested. Here, we aimed to appraise thousands of meta-analyses of observational studies using a pre-specified set of quantitative criteria that assess the significance, amount, consistency, and bias of the evidence. We also aimed to compare results from meta-analyses of observational studies against meta-analyses of randomized controlled trials (RCTs) and Mendelian randomization (MR) studies. Methods We retrieved from PubMed (last update, November 19, 2020) umbrella reviews including meta-analyses of observational studies assessing putative risk or protective factors, regardless of the nature of the exposure and health outcome. We extracted information on 7 quantitative criteria that reflect the level of statistical support, the amount of data, the consistency across different studies, and hints pointing to potential bias. These criteria were level of statistical significance (pre-categorized according to 10−6, 0.001, and 0.05 p-value thresholds), sample size, statistical significance for the largest study, 95% prediction intervals, between-study heterogeneity, and the results of tests for small study effects and for excess significance. Results 3744 associations (in 57 umbrella reviews) assessed by a median number of 7 (interquartile range 4 to 11) observational studies were eligible. Most associations were statistically significant at P < 0.05 (61.1%, 2289/3744). Only 2.6% of associations had P < 10−6, ≥1000 cases (or ≥20,000 participants for continuous factors), P < 0.05 in the largest study, 95% prediction interval excluding the null, and no large between-study heterogeneity, small study effects, or excess significance. Across the 57 topics, large heterogeneity was observed in the proportion of associations fulfilling various quantitative criteria. The quantitative criteria were mostly independent from one another. Across 62 associations assessed in both RCTs and in observational studies, 37.1% had effect estimates in opposite directions and 43.5% had effect estimates differing beyond chance in the two designs. Across 94 comparisons assessed in both MR and observational studies, such discrepancies occurred in 30.8% and 54.7%, respectively. Conclusions Acknowledging that no gold-standard exists to judge whether an observational association is genuine, statistically significant results are common in observational studies, but they are rarely convincing or corroborated by randomized evidence.

scholarly journals Non-pharmacological treatments for irritable bowel syndrome: study protocol of an umbrella review of systematic review and meta-analyses

BMJ Open ◽  
2019 ◽  
Vol 9 (3) ◽  
pp. e027778 ◽  
Author(s):  
Song Jin ◽  
Yi-Fan Li ◽  
Di Qin ◽  
Dan-Qing Luo ◽  
Hong Guo ◽  
...  
Keyword(s):  
Irritable Bowel Syndrome ◽  
Meta Analysis ◽  
Small Study ◽  
Cochrane Central Register ◽  
Pharmacological Treatments ◽  
Umbrella Review ◽  
Study Heterogeneity ◽  
Ethical Approval ◽  
Irritable Bowel ◽  
Meta Analyses

IntroductionNon-pharmacological treatments are used in the management of irritable bowel syndrome, and their effectiveness has been evaluated in multiple meta-analyses. The robustness of the results in the meta-analyses was not evaluated. We aimed to assess whether there is evidence of diverse biases in the meta-analyses and to identify the treatments without evidence of risk of bias.Methods and analysisWe will search MEDLINE, EMBASE, Cochrane Central Register of Controlled Trials, Web of Science and CINAHL Plus for meta-analyses that evaluate the effectiveness of non-pharmacological treatments. The time of publication will be limited from inception to December 2018. The credibility of the meta-analyses will be evaluated by assessing between-study heterogeneity, small-study effect and excess significance bias. The between-study heterogeneity will be assessed using the Cochrane’s Q test, and the extent of the heterogeneity will be classified using the I2statistics. The existence of a small-study effect in a meta-analysis will be evaluated using the funnel plot method and confirmed by Egger’s test. Excess significance bias will be evaluated by comparing the expected number of clinical studies with positive findings with the observed number.Ethics and disseminationNo formal ethical approval is required since we will use publicly available data. We will disseminate the findings of the umbrella review through publication in a peer-reviewed journal and conference presentations.PROSPERO registration numberCRD42018111516.

scholarly journals Interventions for gestational diabetes prevention: An umbrella review of meta-analyses of RCTs

2019 ◽  
Vol 29 (Supplement_4) ◽  
Author(s):  
K Giannakou ◽  
E Constantinou ◽  
D Lamnisos
Keyword(s):  
Gestational Diabetes ◽  
Random Effects ◽  
Fixed Effects ◽  
Small Study ◽  
Cochrane Library ◽  
Umbrella Review ◽  
Metabolic Conditions ◽  
Large Heterogeneity ◽  
Meta Analyses ◽  
Prenatal Exercise

Abstract Background Gestational diabetes mellitus (GDM) is a common pregnancy complication, defined as glucose intolerance with onset or first recognition during pregnancy, in women without diabetes history during pregnancy. It is a major cause for adverse maternal and fetal outcomes and affects around 15% of all pregnancies. GDM is considered to be a risk factor for future metabolic conditions and cardiovascular disease. Methods An umbrella review was performed to identify systematic reviews and meta-analyses of randomized controlled trials for GDM prevention in order to summarize evidence and evaluate the validity of the interventions. We searched PubMed, the Cochrane library and ISI Web of Science from inception to March 2019. For each meta-analysis we estimated the summary effect size by random-effects and fixed-effects models, the 95% confidence interval, the 95% prediction interval, the between-study heterogeneity expressed by I2, evidence of small-study effects and evidence of excess significance bias. Results Six-teen eligible papers were identified providing data on 39 associations. Eighteen (46%) associations had nominally statistically significant findings at P &lt; 0.05, while only one (3%) remain significant at P &lt; 10-6 under the random-effects model. Eight (21%) associations had large or very large heterogeneity. Evidence for small-study effects and excess significance bias was found in 5 (13%) and 2 (5%) associations, respectively. Only one intervention presented robust evidence for a convincing association: prenatal exercise compared with no exercise (RR 0.62, 95% CI: 0.52-0.75). It was supported by &gt; 500 cases, 95% prediction intervals excluding the null, no large heterogeneity, small-study effects, or excess of significance. Conclusions Prenatal exercise shows the strongest consistent evidence. The findings from our study highlight the importance of patient education about lifestyle modifications to reduce risk of GDM. Key messages Prenatal exercise could lower the risk of GDM based on solid epidemiologic evidence. Obesity, pregnancy hypertension and other metabolic conditions could be prevented as well.

scholarly journals Neutrophil counts and cancer prognosis: an umbrella review of systematic reviews and meta-analyses of observational studies

2018 ◽  
Author(s):  
Meghan A. Cupp ◽  
Margarita Cariolou ◽  
Ioanna Tzoulaki ◽  
Evangelou Evangelos ◽  
Antonio J. Berlanga-Taylor
Keyword(s):  
Systematic Reviews ◽  
Strong Evidence ◽  
Observational Studies ◽  
Cancer Prognosis ◽  
Small Study ◽  
List Type ◽  
Cancer Outcomes ◽  
Umbrella Review ◽  
Eligibility Criteria ◽  
Meta Analyses

ABSTRACTOBJECTIVETo evaluate the strength and validity of evidence on the association between the neutrophil to lymphocyte ratio (NLR) or tumour-associated neutrophils (TAN) and cancer prognosis.DESIGNUmbrella review of systematic reviews and meta-analyses of observational studies.DATA SOURCESMedline, EMBASE, and Cochrane Database of Systematic Reviews.ELIGIBILITY CRITERIASystematic reviews or meta-analyses of observational studies evaluating the association between NLR or TAN and specific cancer outcomes related to disease progression or survival.DATA SYNTHESISThe available evidence was graded as strong, highly suggestive, suggestive, or weak through the application of pre-set grading criteria. For each included meta-analysis, the grading criteria considered the significance of the random effects estimate, the significance of the largest included study, the number of studies and individuals included, the heterogeneity between included studies, the 95% prediction intervals, presence of small study effects, excess significance and credibility ceilings.RESULTS239 meta-analyses investigating the association between NLR or TAN and cancer outcomes were identified from 57 published studies meeting the eligibility criteria, with 81 meta-analyses from 36 studies meeting the criteria for inclusion. No meta-analyses found a hazard ratio (HR) in the opposite direction of effect (HR<1). When assessed for significance and bias related to heterogeneity and small study effects, only three (4%) associations between NLR and outcomes in gastrointestinal and nasopharyngeal cancers were supported by strong evidence.CONCLUSIONDespite many publications exploring the association between NLR and cancer prognosis, the evidence is limited by significant heterogeneity and small study effects. There is a lack of evidence on the association between TAN and cancer prognosis, with all nine associations identified arising from the same study. Further research is required to provide strong evidence for associations between both TAN and NLR and poor cancer prognosis.REGISTRATIONThis umbrella review is registered on PROSPERO (CRD42017069131)FUNDINGMedical Research CouncilCOPYRIGHTOpen access article under terms of CC BYSHORT TITLENeutrophils and cancer prognosis: an umbrella reviewKEY RESULTWhen assessed for significance and bias related to heterogeneity and small study effects, only three (4%) associations between NLR and overall survival and progression-free survival in gastrointestinal and nasopharyngeal cancers were supported by strong evidence.WHAT THIS PAPER ADDSWHAT IS ALREADY KNOWN ON THE TOPICNeutrophil counts have been linked to the progression of cancer due to their tumourigenic role in the cancer microenvironment.Numerous meta-analyses and individual studies have explored the association between neutrophil counts and cancer outcomes for a variety of cancer sites, leading to a large body of evidence with variable strength and validity.Uncertainty exists around the association between neutrophils and cancer outcomes, depending on the site, outcome and treatments considered.WHAT THIS STUDY ADDSAll meta-analyses included in this review indicated an association between high neutrophil counts and poor cancer prognosis.There is strong evidence supporting the association between the neutrophil to lymphocyte ratio and poor cancer prognosis in some respiratory and gastrointestinal cancers.Further research is required to strengthen the existing body of evidence, particularly for the association between tumour-associated neutrophils and cancer outcomes.

scholarly journals Preeclampsia and long-term health outcomes: preliminary results from an umbrella review

2020 ◽  
Vol 30 (Supplement_5) ◽  
Author(s):  
K Giannakou ◽  
T Pittara ◽  
D Lamnisos
Keyword(s):  
Health Outcomes ◽  
Cerebrovascular Disease ◽  
Random Effects ◽  
Fixed Effects ◽  
Small Study ◽  
Umbrella Review ◽  
Increased Risk ◽  
Large Heterogeneity ◽  
Meta Analyses

Abstract Background Preeclampsia (PE) is a common pregnancy disease, linked with several complications during and after pregnancy. Although its pathogenesis is not completely understood, evidence suggests that PE has long-term health effects of both the mother and the offspring. Methods An umbrella review was performed to identify systematic reviews and meta-analyses on the association of PE with long-term health outcomes in women and infants born to pregnancies complicated with PE. We searched PubMed, EMBASE, Scopus and ISI Web of Science from inception to March 2019. For each meta-analysis we estimated the summary effect size by random-effects and fixed-effects models, the 95% confidence interval, the 95% prediction interval (PI), the between-study heterogeneity, evidence of small-study effects and evidence of excess significance bias. To evaluate the credibility of the published evidence we preferred state-of-the-art approaches of predefined criteria. Results Twenty-two eligible meta-analyses were identified providing data on 71 associations. Forty-nine (69%) associations had nominally statistically significant findings at P &lt; 0.05, while nineteen (27%) remain significant at P &lt; 10-6 under the random-effects model. Forty (56%) associations had large or very large heterogeneity. Evidence for small-study effects and excess significance bias was found in 4 (6%) and 7 (10%) associations, respectively. Only five associations presented robust evidence: cerebrovascular disease (cohort studies), cerebrovascular disease (overall), cardiac disease, cardiovascular mortality and attention deficit hyperactivity disorder (ADHD). These were supported by &gt; 1000 cases, 95% PI excluding the null, no large heterogeneity, small-study effects, or excess of significance. Conclusions Our findings suggest an increased risk for women to develop cerebrovascular, cardiovascular and cardiac diseases after a pregnancy complicated by PE, while offspring exposed to PE are at higher risk for ADHD. Key messages Our findings highlight the importance of promoting ongoing patient assessment practices for prevention and early diagnosis of these comorbidities. More research is required for the elevated risks of the reported outcomes to improve maternal health.

scholarly journals Systematic Differences in Effect Estimates Between Observational Studies and Randomized Control Trials in Meta-Analyses Combining Both Study Designs in Nephrology

2020 ◽  
Author(s):  
Miho Kimachi ◽  
Akira Onishi ◽  
Aran Tajika ◽  
Kimihiko Kimachi ◽  
Toshi Furukawa
Keyword(s):  
Systematic Reviews ◽  
Observational Studies ◽  
Statistical Significance ◽  
Randomized Control Trials ◽  
Causal Inferences ◽  
Randomized Controlled ◽  
The Past ◽  
Randomized Control ◽  
Meta Analyses ◽  
Study Designs

Abstract The limited availability of randomized controlled trials (RCTs) in nephrology undermines causal inferences in meta-analyses. Systematic reviews of observational studies have grown more common under such circumstances. We conducted systematic reviews of all comparative observational studies in nephrology from 2006 to 2016 to assess the trends in the past decade. We then focused on the meta-analyses combining observational studies and RCTs to evaluate the systematic differences in effect estimates between study designs using two statistical methods: by estimating the ratio of odds ratios (ROR) of the pooled OR obtained from observational studies versus those from RCTs and by examining the discrepancies in their statistical significance. The number of systematic reviews of observational studies in nephrology had grown by 11.7-fold in the past decade. Among 56 records combining observational studies and RCTs, ROR suggested that the estimates between study designs agreed well (ROR: 1.05, 95% confidence interval: 0.90-1.23). However, almost half of the reviews led to discrepant interpretations in terms of statistical significance. In conclusion, the findings based on ROR might encourage researchers to justify the inclusion of observational studies in meta-analyses. However, caution is needed as the interpretations based on statistical significance were less concordant than those based on ROR.

scholarly journals Susceptibility variants for rheumatoid arthritis in the TRAF1-C5 and 6q23 loci: a meta-analysis

2009 ◽  
Vol 69 (3) ◽  
pp. 561-566 ◽  
Author(s):  
Nikolaos A Patsopoulos ◽  
John P A Ioannidis
Keyword(s):  
Rheumatoid Arthritis ◽  
Association Studies ◽  
Meta Analysis ◽  
Statistical Significance ◽  
Genome Wide Association Studies ◽  
Gene Environment ◽  
Genome Wide ◽  
Study Heterogeneity ◽  
Meta Analyses ◽  
Citrullinated Peptide

BackgroundGenome-wide association studies have proposed susceptibility variants for rheumatoid arthritis in the TRAF1-C5 locus and 6q23 region. Furthermore, additional independent studies have investigated the same or highly linked polymorphisms in the same regions.ObjectiveTo carry out a meta-analysis of the available evidence for the association of polymorphisms in the TRAF1-C5 locus and 6q23 region with rheumatoid arthritis.MethodsData were synthesised for four polymorphisms: rs3761847 (n=13 datasets) and rs2900180 (n=9 datasets) in the TRAF1-C5 locus, and rs10499194 (n=5 datasets) and rs6920220 (n=7 datasets) in the 6q23 region. Meta-analyses for subgroups defined by anti-cyclic citrullinated peptide (anti-CCP) and rheumatoid factor (RF) status were also performed.ResultsThe polymorphism rs6920220 reached genome-wide statistical significance with p=7.9×10−17 and an allelic odds ratio of 1.24 (95% CI 1.18 to 1.30) and no between-study heterogeneity (I2=0%). The risk was significantly stronger in patients with anti-CCP antibodies and in patients with RF. The other three variants showed large between-study heterogeneity across datasets (I2 range 74–82%); rs10499194 was nominally statistically significant after exclusion of the discovery data. Two variants had genome-wide statistical significance in subgroups defined by the presence of RF (rs3761847 and rs6920220) or anti-CCP (rs6920220).ConclusionsGenetic markers in the 6q23 region and TRAF1-C5 are associated with rheumatoid arthritis, in particular with positive anti-CCP and RF profile. With the exception of rs6920220, which shows highly consistent results, other proposed markers have high between-study heterogeneity that may reflect unrecognised phenotypic or genetic variability (eg, gene environment interactions) within rheumatoid arthritis. Furthermore, these markers may not be the true causative loci but rather be in linkage disequilibrium with the true ones.

scholarly journals Systematic differences in effect estimates between observational studies and randomized control trials in meta-analyses in nephrology

2021 ◽  
Vol 11 (1) ◽  
Author(s):  
Miho Kimachi ◽  
Akira Onishi ◽  
Aran Tajika ◽  
Kimihiko Kimachi ◽  
Toshi A. Furukawa
Keyword(s):  
Systematic Reviews ◽  
Observational Studies ◽  
Statistical Significance ◽  
Randomized Control Trials ◽  
Causal Inferences ◽  
Randomized Controlled ◽  
The Past ◽  
Randomized Control ◽  
Meta Analyses ◽  
Study Designs

AbstractThe limited availability of randomized controlled trials (RCTs) in nephrology undermines causal inferences in meta-analyses. Systematic reviews of observational studies have grown more common under such circumstances. We conducted systematic reviews of all comparative observational studies in nephrology from 2006 to 2016 to assess the trends in the past decade. We then focused on the meta-analyses combining observational studies and RCTs to evaluate the systematic differences in effect estimates between study designs using two statistical methods: by estimating the ratio of odds ratios (ROR) of the pooled OR obtained from observational studies versus those from RCTs and by examining the discrepancies in their statistical significance. The number of systematic reviews of observational studies in nephrology had grown by 11.7-fold in the past decade. Among 56 records combining observational studies and RCTs, ROR suggested that the estimates between study designs agreed well (ROR 1.05, 95% confidence interval 0.90–1.23). However, almost half of the reviews led to discrepant interpretations in terms of statistical significance. In conclusion, the findings based on ROR might encourage researchers to justify the inclusion of observational studies in meta-analyses. However, caution is needed, as the interpretations based on statistical significance were less concordant than those based on ROR.

Addendum to the Acknowledgements: Stepwise-Hierarchical Pooled Analysis for Synergistic Interpretation of Meta-analyses Involving Randomized and Observational Studies: Methodology Development (Preprint)

2021 ◽  
Author(s):  
In-Soo Shin ◽  
Chai Hong Rim
Keyword(s):  
Effect Size ◽  
Observational Studies ◽  
Statistical Significance ◽  
Pooled Analysis ◽  
Analysis Method ◽  
Methodology Development ◽  
True Effect ◽  
Randomized Studies ◽  
Logical Method ◽  
Meta Analyses

UNSTRUCTURED The necessity of including observational studies in meta-analyses has been discussed in the literature, but a synergistic analysis method for combining randomized and observational studies has not been reported. Observational studies differ in validity depending on the degree of the confounders’ influence. Combining interpretations may be challenging, especially if the statistical directions are similar but the magnitude of the pooled results are different between randomized and observational studies (the ”gray zone”). To overcome these hindrances, in this study, we aim to introduce a logical method for clinical interpretation of randomized and observational studies. We designed a stepwise-hierarchical pooled analysis method to analyze both distribution trends and individual pooled results by dividing the included studies into at least three stages (eg, all studies, balanced studies, and randomized studies). According to the model, the validity of a hypothesis is mostly based on the pooled results of randomized studies (the highest stage). Ascending patterns in which effect size and statistical significance increase gradually with stage strengthen the validity of the hypothesis; in this case, the effect size of the observational studies is lower than that of the true effect (eg, because of the uncontrolled effect of negative confounders). Descending patterns in which decreasing effect size and statistical significance gradually weaken the validity of the hypothesis suggest that the effect size and statistical significance of the observational studies is larger than the true effect (eg, because of researchers’ bias). We recommend using the stepwise-hierarchical pooled analysis approach for meta-analyses involving randomized and observational studies.

scholarly journals Stepwise-hierarchical pooled analysis: a synergistic interpretation of meta-analysis including randomized and observational studies: A methodologic study (Preprint)

2021 ◽  
Author(s):  
In-Soo Shin ◽  
Chai Hong Rim
Keyword(s):  
Effect Size ◽  
Observational Studies ◽  
Meta Analysis ◽  
Statistical Significance ◽  
Pooled Analysis ◽  
Grey Zone ◽  
True Effect ◽  
Logical Method ◽  
Three Stages ◽  
Meta Analyses

BACKGROUND The necessity of meta-analyses including observational studies has been discussed in the literature, but a synergistic analysis method combining randomised and observational studies has not been reported. OBJECTIVE This study introduces a logical method for clinical interpretation. METHODS Observational studies differ in validity depending on the degree of the confounders’ influence. Combining interpretations might be challenging, especially if the statistical directions are similar but the magnitude of the pooled results are different, between randomised and observational studies (grey zone). We designed a stepwise-hierarchical pooled analysis, a method of analysing distribution trends as well as individual pooled results by dividing included studies into at least three stages (e.g. all studies, balanced studies, and randomised studies), to overcome such hindrances. RESULTS According to the model, the validity of hypothesis are mostly based on the pooled results of randomised studies (the highest stage). In addition, ascending patterns where effect size and statistical significance increase gradually with stage, strengthen the validity of the hypothesis; in this case, the effect size of observational studies is lower than that of the true effect (e.g. because of uncontrolled effect of negative confounders). Descending patterns where decreasing effect size and statistical significance gradually weaken the validity of the hypothesis suggest that the effect size and statistical significance of observational studies is larger than the true effect (e.g. because of researchers’ bias). These are described in more detail in the main text as four descriptive patterns. CONCLUSIONS We recommend using the stepwise-hierarchical pooled analysis for meta-analyses involving randomised and observational studies. CLINICALTRIAL NA

Sign in / Sign up

Export Citation Format

Share Document