The Importance of Drawing Meaningful Conclusions from Data: A Review of the Literature with Meta-Analytic Inquiry

Using correct statistical concepts is an important component when conducting quantitative research. Ideas such as power, effect size, and confidence intervals need to be addressed appropriately every time a research study is initiated. The intent of this review of the literature is to reacquaint faculty, practitioners, and graduate students with scholarly information pertaining to these important concepts to facilitate improved implementation of quantitative research designs. Practical cases are interwoven within the review to furnish examples of concept importance, and a meta-analysis of concept usage found in articles published in the NASPA Journal is provided as a measure for implications.

Download Full-text

Abstract 3789: Nessun Dorma : Have the Risks of Rosiglitazone been Exaggerated?

Circulation ◽

10.1161/circ.116.suppl_16.ii_861-c ◽

2007 ◽

Vol 116 (suppl_16) ◽

Author(s):

George A Diamond ◽

Sanjay Kaul

Keyword(s):

Confidence Intervals ◽

Effect Size ◽

Fixed Effects ◽

Meta Analysis ◽

Cardiovascular Death ◽

Sensitivity Analyses ◽

Odds Ratios ◽

True Effect Size ◽

Index Study ◽

The Stability

Background A highly publicized meta-analysis of 42 clinical trials comprising 27,844 diabetics ignited a firestorm of controversy by charging that treatment with rosiglitazone was associated with a “…worrisome…” 43% greater risk of myocardial infarction ( p =0.03) and a 64% greater risk of cardiovascular death ( p =0.06). Objective The investigators excluded 4 trials from the infarction analysis and 19 trials from the mortality analysis in which no events were observed. We sought to determine if these exclusions biased the results. Methods We compared the index study to a Bayesian meta-analysis of the entire 42 trials (using odds ratio as the measure of effect size) and to fixed-effects and random-effects analyses with and without a continuity correction that adjusts for values of zero. Results The odds ratios and confidence intervals for the analyses are summarized in the Table . Odds ratios for infarction ranged from 1.43 to 1.22 and for death from 1.64 to 1.13. Corrected models resulted in substantially smaller odds ratios and narrower confidence intervals than did uncorrected models. Although corrected risks remain elevated, none are statistically significant (*p<0.05). Conclusions Given the fragility of the effect sizes and confidence intervals, the charge that roziglitazone increases the risk of adverse events is not supported by these additional analyses. The exaggerated values observed in the index study are likely the result of excluding the zero-event trials from analysis. Continuity adjustments mitigate this error and provide more consistent and reliable assessments of true effect size. Transparent sensitivity analyses should therefore be performed over a realistic range of the operative assumptions to verify the stability of such assessments especially when outcome events are rare. Given the relatively wide confidence intervals, additional data will be required to adjudicate these inconclusive results.

Download Full-text

Meta-analysis of the impacts of water management on aquatic communities

Canadian Journal of Fisheries and Aquatic Sciences ◽

10.1139/f07-175 ◽

2008 ◽

Vol 65 (3) ◽

pp. 437-447 ◽

Cited By ~ 74

Author(s):

Tim J Haxton ◽

C Scott Findlay

Keyword(s):

Water Management ◽

Confidence Intervals ◽

Effect Size ◽

Meta Analysis ◽

Ecological Impacts ◽

Aquatic Communities ◽

Flow Modification ◽

Meta Analyses ◽

Fail Safe ◽

Habitat Generalists

Systematic meta-analyses were conducted on the ecological impacts of water management, including effects of (i) dewatering on macroinvertebrates, (ii) a hypolimnetic release on downstream aquatic fish and macro invertebrate communities, and (iii) flow modification on fluvial and habitat generalists. Our meta-analysis indicates, in general, that (i) macroinvertebrate abundance is lower in zones or areas that have been dewatered as a result of water fluctuations or low flows (overall effect size, –1.64; 95% confidence intervals (CIs), –2.51, –0.77), (ii) hypolimnetic draws are associated with reduced abundance of aquatic (fish and macroinvertebrates) communities (overall effect size, –0.84; 95% CIs, –1.38, –0.33) and macroinvertebrates (overall effect size, –0.73; 95% CIs, –1.24, –0.22) downstream of a dam, and (iii) altered flows are associated with reduced abundance of fluvial specialists (–0.42; 95% CIs, –0.81, –0.02) but not habitat generalists (overall effect size, –0.14; 95% CIs, –0.61, 0.32). Publication bias is evident in several of the meta-analyses; however, multiple experiments from a single study may be contributing to this bias. Fail-safe Ns suggest that many (>100) studies showing positive or no effects of water management on the selected endpoints would be required to qualitatively change the results of the meta-analysis, which in turn suggests that the conclusions are reasonably robust.

Download Full-text

Confidence intervals for the overall effect size in random-effects meta-analysis.

Psychological Methods ◽

10.1037/1082-989x.13.1.31 ◽

2008 ◽

Vol 13 (1) ◽

pp. 31-48 ◽

Cited By ~ 104

Author(s):

Julio Sánchez-Meca ◽

Fulgencio Marín-Martínez

Keyword(s):

Confidence Intervals ◽

Random Effects ◽

Effect Size ◽

Meta Analysis

Download Full-text

Predictive physiological anticipatory activity preceding seemingly unpredictable stimuli: An update of Mossbridge et al’s meta-analysis

F1000Research ◽

10.12688/f1000research.14330.2 ◽

2018 ◽

Vol 7 ◽

pp. 407 ◽

Cited By ~ 4

Author(s):

Michael Duggan ◽

Patrizio Tressoldi

Keyword(s):

Confidence Intervals ◽

Publication Bias ◽

Effect Size ◽

Statistical Estimation ◽

Meta Analysis ◽

Weighted Mean ◽

Weighted Effect Size ◽

Credible Intervals ◽

Weighted Effect ◽

Anticipatory Activity

Background: This is an update of the Mossbridge et al’s meta-analysis related to the physiological anticipation preceding seemingly unpredictable stimuli which overall effect size was 0.21; 95% Confidence Intervals: 0.13 - 0.29 Methods: Nineteen new peer and non-peer reviewed studies completed from January 2008 to June 2018 were retrieved describing a total of 27 experiments and 36 associated effect sizes. Results: The overall weighted effect size, estimated with a frequentist multilevel random model, was: 0.28; 95% Confidence Intervals: 0.18-0.38; the overall weighted effect size, estimated with a multilevel Bayesian model, was: 0.28; 95% Credible Intervals: 0.18-0.38. The weighted mean estimate of the effect size of peer reviewed studies was higher than that of non-peer reviewed studies, but with overlapped confidence intervals: Peer reviewed: 0.36; 95% Confidence Intervals: 0.26-0.47; Non-Peer reviewed: 0.22; 95% Confidence Intervals: 0.05-0.39. Similarly, the weighted mean estimate of the effect size of Preregistered studies was higher than that of Non-Preregistered studies: Preregistered: 0.31; 95% Confidence Intervals: 0.18-0.45; No-Preregistered: 0.24; 95% Confidence Intervals: 0.08-0.41. The statistical estimation of the publication bias by using the Copas selection model suggest that the main findings are not contaminated by publication bias. Conclusions: In summary, with this update, the main findings reported in Mossbridge et al’s meta-analysis, are confirmed.

Download Full-text

Meta-analysis of school leadership effects on student achievement in USA and Turkey

Cypriot Journal of Educational Sciences ◽

10.18844/cjes.v13i4.3539 ◽

2018 ◽

Vol 13 (4) ◽

pp. 590-603

Author(s):

Sengul Uysal ◽

Yılmaz Sarıer

Keyword(s):

Student Achievement ◽

Instructional Leadership ◽

School Leadership ◽

Effect Size ◽

Best Practice ◽

Research Study ◽

Meta Analysis ◽

The Usa ◽

Positive Effect ◽

Modest Level

This research study aims to investigate the effects of school leadership on student achievement in USA and Turkey. The method of meta-analysis is used to calculate the effect size of school leadership on student achievement. Thirty-nine research studies were included in this study. However, several publications included in the research have examined more than one leadership approach. Due to this reason, the dataset used in 39 publications was determined as 68 in total. The results of the analyses performed with a random-effects model revealed that school leadership has a small but positive effect on student achievement in general. However, in Turkey, the effect size is at a modest level, while it is small in the USA. Considering the findings of the study, it can be proposed that professional development opportunities should be supported for the best practice of instructional leadership behaviours. Keywords: Achievement, meta-analysis, school leadership, Turkey, USA

Download Full-text

Introducere în metaanaliză. Metaanaliza corelațiilor

Psihologia Resurselor Umane ◽

10.24837/pru.v6i2.370 ◽

2020 ◽

Vol 6 (2) ◽

pp. 112-127

Author(s):

Laurențiu Maricuțoiu

Keyword(s):

Statistical Method ◽

Confidence Intervals ◽

Effect Size ◽

Meta Analysis ◽

Relevant Information ◽

Effect Sizes ◽

The Mean

The present paper discusses the fundamental principles of meta-analysis, as a statistical method for summarising results of correlational studies. We approach fundamental issues such as: the finality of meta-analysis and the problems associated with study artefacts. The paper also contains recommendations for: selecting the studies for meta-analysis, identifying the relevant information within these studies, computing mean effect sizes, confidence intervals and heterogeneity indexes of the mean effect size. Finally, we present indications for reporting meta-analysis results.

Download Full-text

Ultrasound Markers for Complex Gastroschisis: A Systematic Review and Meta-Analysis

Journal of Clinical Medicine ◽

10.3390/jcm10225215 ◽

2021 ◽

Vol 10 (22) ◽

pp. 5215

Author(s):

Rui Gilberto Ferreira ◽

Carolina Rodrigues Mendonça ◽

Carolina Leão de Moraes ◽

Fernanda Sardinha de Abreu Tacon ◽

Lelia Luanne Gonçalves Ramos ◽

...

Keyword(s):

Systematic Review ◽

Confidence Intervals ◽

Gestational Age ◽

Meta Analysis ◽

Prenatal Ultrasound ◽

Prevalence Rates ◽

Review Of The Literature ◽

Estimate Prevalence ◽

Risk Ratios ◽

Bowel Dilatation

Although gastroschisis is often diagnosed by prenatal ultrasound, there is still a gap in the literature about which prenatal ultrasound markers can predict complex gastroschisis. This systematic review and meta-analysis aimed to investigate the ultrasound markers that characterize complex gastroschisis. A systematic review of the literature was conducted according to the guidelines of PRISMA. The protocol was registered (PROSPERO ID CRD42020211685). Meta-analysis was displayed graphically on Forest plots, which estimate prevalence rates and risk ratios, with 95% confidence intervals, using STATA version 15.0. The combined prevalence of intestinal complications in fetuses with complex gastroschisis was 27.0%, with a higher prevalence of atresia (about 48%), followed by necrosis (about 25%). The prevalence of deaths in newborns with complex gastroschisis was 15.0%. The predictive ultrasound markers for complex gastroschisis were intraabdominal bowel dilatation (IABD) (RR 3.01, 95% CI 2.22 to 4.07; I2 = 15.7%), extra-abdominal bowel dilatation (EABD) (RR 1.55, 95% CI 1.01 to 2.39; I2 = 77.1%), and polyhydramnios (RR 3.81, 95% CI 2.09 to 6.95; I2 = 0.0%). This review identified that IABD, EABD, and polyhydramnios were considered predictive ultrasound markers for complex gastroschisis. However, evidence regarding gestational age at the time of diagnosis is needed.

Download Full-text

Statistical Primer for Athletic Trainers: Using Confidence Intervals and Effect Sizes to Evaluate Clinical Meaningfulness

Journal of Athletic Training ◽

10.4085/1062-6050-51.12.14 ◽

2016 ◽

Vol 51 (12) ◽

pp. 1045-1048 ◽

Cited By ~ 8

Author(s):

Monica Lininger ◽

Bryan L. Riemann

Keyword(s):

Confidence Intervals ◽

Effect Size ◽

Research Study ◽

Statistical Significance ◽

Effect Sizes ◽

Population Parameter ◽

Significant Difference ◽

Sample Data ◽

The Difference ◽

Clinical Meaningfulness

Objective: To describe confidence intervals (CIs) and effect sizes and provide practical examples to assist clinicians in assessing clinical meaningfulness. Background: As discussed in our first article in 2015, which addressed the difference between statistical significance and clinical meaningfulness, evaluating the clinical meaningfulness of a research study remains a challenge to many readers. In this paper, we will build on this topic by examining CIs and effect sizes. Description: A CI is a range estimated from sample data (the data we collect) that is likely to include the population parameter (value) of interest. Conceptually, this constitutes the lower and upper limits of the sample data, which would likely include, for example, the mean from the unknown population. An effect size is the magnitude of difference between 2 means. When a statistically significant difference exists between 2 means, effect size is used to describe how large or small that difference actually is. Confidence intervals and effect sizes enhance the practical interpretation of research results. Recommendations: Along with statistical significance, the CI and effect size can assist practitioners in better understanding the clinical meaningfulness of a research study.

Download Full-text

Relationship between Cooperative learning method and Students’ Mathematics Learning Achievement: A Meta-Analysis Correlation

Al-Jabar Jurnal Pendidikan Matematika ◽

10.24042/ajpm.v11i1.6620 ◽

2020 ◽

Vol 11 (1) ◽

pp. 145-158

Author(s):

Dafid Slamet Setiana ◽

La Ili ◽

Muhammad Irfan Rumasoreng ◽

Anggit Prabowo

Keyword(s):

Cooperative Learning ◽

Effect Size ◽

Quantitative Research ◽

Meta Analysis ◽

Mathematics Learning ◽

Research Approach ◽

Learning Method ◽

Important Lesson ◽

Learning Achievement ◽

And Mathematics

An appropriate learning method can improve students’ learning achievement. The cooperative learning method encourages students to improve their mathematics learning achievement. Mathematics learning achievement is one indicator of achieving learning objectives. Mathematics is an important lesson to be learned and must be mastered by students. The purpose of this research was to analyze the effect size of the relationship between Cooperative learning method and mathematics learning achievement through meta-analysis quantitative research approach. In this research, the cooperative learning method serves as an independent variable, and mathematics learning achievement serves as the dependent variable. The data was obtained from the online database search results on Google Scholar in 2010-2020. The sample used was 16 research publications that have met the specified criteria. The data analysis technique used was the meta-analysis quantitative approach with correlation meta-analysis. The results showed that there was a positive and significant relationship (level of 5%) between the cooperative learning method and students’ mathematics learning achievement. The average effect size value was 0.15 in the range of 0.04 to 0.27 which was included in the low category. These results have proven the consistency of the findings of previous research.

Download Full-text

Predictive physiological anticipation preceding seemingly unpredictable stimuli: an update of Mossbridge’s et al. meta-analysis

10.31234/osf.io/5u48e ◽

2018 ◽

Author(s):

Michael Duggan ◽

Patrizio Tressoldi

Keyword(s):

Confidence Intervals ◽

Effect Size ◽

Bayesian Model ◽

Statistical Estimation ◽

Meta Analysis ◽

Random Model ◽

Weighted Mean ◽

Weighted Effect Size ◽

Credible Intervals ◽

Weighted Effect

Background: This is an update of the Mossbridge et al’s meta-analysis related to the physiological anticipation preceding seemingly unpredictable stimuli which overall effect size was 0.21; 95% Confidence Intervals: 0.13 - 0.29Methods: Nineteen new peer and non-peer reviewed studies completed from January 2008 to June 2018 were retrieved describing a total of 27 experiments and 36 associated effect sizes.Results: The overall weighted effect size, estimated with a frequentist multilevel random model, was: 0.28; 95% Confidence Intervals: 0.18-0.38; the overall weighted effect size, estimated with a multilevel Bayesian model, was: 0.28; 95% Credible Intervals: 0.18-0.38. The weighted mean estimate of the effect size of peer reviewed studies was higher than that of non peer reviewed studies, but with overlapped confidence intervals: Peer reviewed: 0.36; 95% Confidence Intervals: 0.26-0.47; Non peer reviewed: 0.22; 95% Confidence Intervals: 0.05-0.39. Similarly, the weighted mean estimate of the effect size of Preregistered studies was higher than that of Non-Preregistered studies: Preregistered: 0.31; 95% Confidence Intervals: 0.18-0.45; No-Preregistered: 0.24; 95% Confidence Intervals: 0.08-0.41.The statistical estimation of the publication bias by using the Copas selection model suggest that the main findings are not contaminated by publication bias.Conclusions: In summary, with this update, the main findings reported in Mossbridge et al’s meta-analysis, are confirmed.

Download Full-text