Temporal Trends in Effect Sizes: Causes, Detection, and Implications

The general aim of meta-analysis is to combine scientific evidence scattered through a number of individual studies addressing the same topic. Evidence, however, is not static and tends to evolve over time due to changes in research methods, changes in the characteristics of the subjects being studied, for example. A number of recent studies in ecology and evolution have shown that temporal trends in effect sizes are common and often quite dramatic in these fields. Temporal changes in effect sizes could jeopardize the stability of conclusions (i.e., the conclusions of meta-analyses on the same topic conducted in different years might differ). For these reasons, this chapter is devoted to temporal changes in effect sizes. It first summarizes the findings of studies that examined temporal changes in the magnitude and direction of effect sizes in ecology, evolutionary biology, medicine, and the social sciences. It then discusses their possible causes, methods of detection, and implications for the interpretation of the results of the meta-analysis.

Download Full-text

Meta-Analyses as a Multi-Level Model

Organizational Research Methods ◽

10.1177/1094428119857471 ◽

2019 ◽

pp. 109442811985747

Author(s):

Janaki Gooty ◽

George C. Banks ◽

Andrew C. Loignon ◽

Scott Tonidandel ◽

Courtney E. Williams

Keyword(s):

Educational Psychology ◽

Meta Analysis ◽

Effect Sizes ◽

Independence Assumption ◽

The Social ◽

Level Model ◽

Multi Level ◽

Meta Analyses ◽

User Friendly ◽

Organizational Sciences

Meta-analyses are well known and widely implemented in almost every domain of research in management as well as the social, medical, and behavioral sciences. While this technique is useful for determining validity coefficients (i.e., effect sizes), meta-analyses are predicated on the assumption of independence of primary effect sizes, which might be routinely violated in the organizational sciences. Here, we discuss the implications of violating the independence assumption and demonstrate how meta-analysis could be cast as a multilevel, variance known (Vknown) model to account for such dependency in primary studies’ effect sizes. We illustrate such techniques for meta-analytic data via the HLM 7.0 software as it remains the most widely used multilevel analyses software in management. In so doing, we draw on examples in educational psychology (where such techniques were first developed), organizational sciences, and a Monte Carlo simulation (Appendix). We conclude with a discussion of implications, caveats, and future extensions. Our Appendix details features of a newly developed application that is free (based on R), user-friendly, and provides an alternative to the HLM program.

Download Full-text

Graphical Presentation of Results

Handbook of Meta-analysis in Ecology and Evolution ◽

10.23943/princeton/9780691137285.003.0021 ◽

2013 ◽

Author(s):

Christopher J. Lortie ◽

Joseph Lau ◽

Marc J. Lajeunesse

Keyword(s):

Evolutionary Biology ◽

Meta Analysis ◽

Statistical Technique ◽

The Social ◽

Error Bar ◽

Funnel Plots ◽

Meta Analyses ◽

Main Effects ◽

The Relationship ◽

Graphical Presentation

Visualizations of data are one of the most compelling means to effectively communicate ideas in science. Graphs present data in a visual form enabling the reader to read values, identify patterns, assess the outcome of a statistical technique, or analyze relationships within or between variables. Effective visualizations of meta-analyses have been discussed extensively in the evidence-based medical literature and to a lesser extent in ecology and evolutionary biology. The two most common meta-analysis plots are derived from the social sciences and include (1) modified error bar plots called forest plots used to summarize and compare weighted mean effects, and (2) meta-regression plots (scatterplots with significant fit lines) used to show the relationship between main effects and covariates. This chapter describes these two standard meta-analysis plots and provides sample graphics to illustrate usage. Details are also included for the use of simple histograms and funnel plots.

Download Full-text

Meta-Analysis of the Effect of Red on Perceived Attractiveness

Evolutionary Psychology ◽

10.1177/1474704918802412 ◽

2018 ◽

Vol 16 (4) ◽

pp. 147470491880241 ◽

Cited By ~ 8

Author(s):

Gabrielle K. Lehmann ◽

Andrew J. Elliot ◽

Robert J. Calin-Jageman

Keyword(s):

Meta Analysis ◽

Effect Sizes ◽

Future Research ◽

Meta Analyses ◽

Heterosexual Attraction ◽

Substantial Heterogeneity ◽

Over Time ◽

Perceived Attractiveness

We conducted meta-analyses of studies that test the red-romance hypothesis, which is that the color red enhances heterosexual attraction in romantic contexts. For men rating women, we found a small, statistically significant effect ( d = 0.26 [0.12, 0.40], p = .0004, N = 2,961), with substantial heterogeneity, Q(44) = 172.5, pQ < .0001, I2 = 89% [82, 94], and equivocal results regarding the possibility of upward bias in the estimate. For women rating men, we found a very small effect ( d = 0.13 [0.01, 0.25], p = .03, N = 2,739), with substantial heterogeneity, Q(35) = 73.0, pQ = .0002, I2 = 53% [33, 80], and evidence of upward bias in the estimate. Moderator analyses suggest effect sizes may have declined over time (both genders), may be largest when an original shade of red is used (men only), and may be smaller in preregistered studies (women only). We present contrasting interpretations and suggestions for future research.

Download Full-text

Methodological Rigor and Temporal Trends of Cardiovascular Medicine Meta‐Analyses in Highest‐Impact Journals

Journal of the American Heart Association ◽

10.1161/jaha.121.021367 ◽

2021 ◽

Vol 10 (18) ◽

Author(s):

Dhruv Mahtta ◽

Ahmed Altibi ◽

Mohamed M. Gad ◽

Amjad Samara ◽

Amr F. Barakat ◽

...

Keyword(s):

Systematic Reviews ◽

Observational Studies ◽

European Heart Journal ◽

Meta Analysis ◽

Temporal Trends ◽

Cardiovascular Medicine ◽

Cardiovascular Research ◽

Methodological Rigor ◽

Meta Analyses ◽

Over Time

Background Well‐conducted meta‐analyses are considered to be at the top of the evidence‐based hierarchy pyramid, with an expansion of these publications within the cardiovascular research arena. There are limited data evaluating the trends and quality of such publications. The objective of this study was to evaluate the methodological rigor and temporal trends of cardiovascular medicine‐related meta‐analyses published in the highest impact journals. Methods and Results Using the Medline database, we retrieved cardiovascular medicine‐related systematic reviews and meta‐analyses published in The New England Journal of Medicine, The Lancet, Journal of the American Medical Association, The British Medical Journal, Annals of Internal Medicine, Circulation, European Heart Journal, and Journal of American College of Cardiology between January 1, 2012 and December 31, 2018. Among 6406 original investigations published during the study period, meta‐analyses represented 422 (6.6%) articles, with an annual decline in the proportion of published meta‐analyses (8.7% in 2012 versus 4.6% in 2018, P trend =0.002). A substantial number of studies failed to incorporate elements of Preferred Reporting Items for Systematic Reviews and Meta‐Analyses or Meta‐Analysis of Observational Studies in Epidemiology guidelines (51.9%) and only a minority of studies (10.4%) were registered in PROSPERO (International Prospective Register of Systematic Reviews). Fewer manuscripts failed to incorporate the Preferred Reporting Items for Systematic Reviews and Meta‐Analyses or Meta‐Analysis of Observational Studies in Epidemiology elements over time (60.2% in 2012 versus 40.0% in 2018, P trend <0.001) whereas the number of meta‐analyses registered at PROSPERO has increased (2.4% in 2013 versus 17.5% in 2018, P trend <0.001). Conclusions The proportion of cardiovascular medicine‐related meta‐analyses published in the highest impact journals has declined over time. Although there is an increasing trend in compliance with quality‐based guidelines, the overall compliance remains low.

Download Full-text

The Orchard Plot: Cultivating a Forest Plot for Use in Ecology, Evolution and Beyond

10.32942/osf.io/epqa7 ◽

2019 ◽

Author(s):

Shinichi Nakagawa ◽

Malgorzata Lagisz ◽

Rose E O'Dea ◽

Joanna Rutkowska ◽

Yefeng Yang ◽

...

Keyword(s):

Meta Analysis ◽

R Package ◽

Effect Sizes ◽

Forest Plot ◽

Point Estimates ◽

Aggregate Effect ◽

The Individual ◽

Meta Analyses ◽

Heterogeneous Effect ◽

Intuitive Interpretation

‘Classic’ forest plots show the effect sizes from individual studies and the aggregate effect from a meta-analysis. However, in ecology and evolution meta-analyses routinely contain over 100 effect sizes, making the classic forest plot of limited use. We surveyed 102 meta-analyses in ecology and evolution, finding that only 11% use the classic forest plot. Instead, most used a ‘forest-like plot’, showing point estimates (with 95% confidence intervals; CIs) from a series of subgroups or categories in a meta-regression. We propose a modification of the forest-like plot, which we name the ‘orchard plot’. Orchard plots, in addition to showing overall mean effects and CIs from meta-analyses/regressions, also includes 95% prediction intervals (PIs), and the individual effect sizes scaled by their precision. The PI allows the user and reader to see the range in which an effect size from a future study may be expected to fall. The PI, therefore, provides an intuitive interpretation of any heterogeneity in the data. Supplementing the PI, the inclusion of underlying effect sizes also allows the user to see any influential or outlying effect sizes. We showcase the orchard plot with example datasets from ecology and evolution, using the R package, orchard, including several functions for visualizing meta-analytic data using forest-plot derivatives. We consider the orchard plot as a variant on the classic forest plot, cultivated to the needs of meta-analysts in ecology and evolution. Hopefully, the orchard plot will prove fruitful for visualizing large collections of heterogeneous effect sizes regardless of the field of study.

Download Full-text

Identification of and Correction for Publication Bias: Comment

10.31222/osf.io/dh87m ◽

2019 ◽

Author(s):

Amanda Kvarven ◽

Eirik Strømland ◽

Magnus Johannesson

Keyword(s):

Publication Bias ◽

False Positive ◽

Large Scale ◽

Meta Analysis ◽

False Positive Rate ◽

Effect Sizes ◽

Replication Studies ◽

Moderate Reduction ◽

Positive Rate ◽

Meta Analyses

Andrews & Kasy (2019) propose an approach for adjusting effect sizes in meta-analysis for publication bias. We use the Andrews-Kasy estimator to adjust the result of 15 meta-analyses and compare the adjusted results to 15 large-scale multiple labs replication studies estimating the same effects. The pre-registered replications provide precisely estimated effect sizes, which do not suffer from publication bias. The Andrews-Kasy approach leads to a moderate reduction of the inflated effect sizes in the meta-analyses. However, the approach still overestimates effect sizes by a factor of about two or more and has an estimated false positive rate of between 57% and 100%.

Download Full-text

A246 THE GLOBAL INCIDENCE OF ACUTE PANCREATITIS IS INCREASING OVER TIME: A SYSTEMATIC REVIEW AND META-ANALYSIS

Journal of the Canadian Association of Gastroenterology ◽

10.1093/jcag/gwab002.244 ◽

2021 ◽

Vol 4 (Supplement_1) ◽

pp. 304-306

Author(s):

J Iannuzzi ◽

J H Leong ◽

J Quan ◽

J A King ◽

J W Windsor ◽

...

Keyword(s):

Acute Pancreatitis ◽

Data Extraction ◽

Meta Analysis ◽

Temporal Trends ◽

Western World ◽

Common Disease ◽

Biliary Disease ◽

Comprehensive Overview ◽

Annual Percent Change ◽

Over Time

Abstract Background Acute pancreatitis is a common disease with significant associated morbidity and mortality. Historically, acute pancreatitis has been considered a disease with multiple etiologies and risk factors but is driven by alcohol and biliary disease. Multiple studies have shown that the incidence of acute pancreatitis is increasing globally among both adults and children. Aims The purpose of this study was to assess temporal trends in incidence of acute pancreatitis globally. Methods We performed a systematic literature search to identify population-based studies reporting the annual incidence of acute pancreatitis. Abstracts were independently assessed in duplicate to identify applicable papers for full-text review and data extraction. Joinpoint temporal trend analyses were performed to calculate the average annual percent change (AAPC) with 95% confidence intervals (CI). The AAPCs were pooled in a meta-analysis to capture the overall and regional trends in acute pancreatitis incidence over time. Temporal data were summarized in a static map and an interactive, web-based map to illustrate global differences. Results Forty-five studies reported the temporal incidence of acute pancreatitis (static map provided, online interactive map: https://kaplan-acute-pancreatitis-ucalgary.hub.arcgis.com/). The incidence of acute pancreatitis has increased from 1961 to 2016 (AAPC = 2.89%; 95% CI: 2.26, 3.52; n=41). Increasing incidence was observed in North America (AAPC = 2.71%; 95% CI: 1.93, 3.50; n=10) and Europe (AAPC = 2.79%; 95% CI: 1.95, 3.63; n=24). The incidence of acute pancreatitis was stable in Asia (AAPC = −0.28%; 95% CI: −5.03, 4.47; n=2). Conclusions This meta-analysis provides a comprehensive overview of the global incidence of acute pancreatitis over the last five decades and demonstrates a steadily rising incidence over time in most countries of the Western world. More studies are needed to better define the changing incidence of acute pancreatitis in Asia, Africa and Latin America. Funding Agencies None

Download Full-text

Effects of Sport-Based Interventions on Children’s Executive Function: A Systematic Review and Meta-Analysis

Brain Sciences ◽

10.3390/brainsci11060755 ◽

2021 ◽

Vol 11 (6) ◽

pp. 755

Author(s):

Falonn Contreras-Osorio ◽

Christian Campos-Jara ◽

Cristian Martínez-Salazar ◽

Luis Chirosa-Ríos ◽

Darío Martínez-García

Keyword(s):

Systematic Review ◽

Working Memory ◽

Executive Function ◽

Inhibitory Control ◽

Cognitive Flexibility ◽

Meta Analysis ◽

Scientific Evidence ◽

Healthy Children ◽

Meta Analyses ◽

The Impact

One of the most studied aspects of children’s cognitive development is that of the development of the executive function, and research has shown that physical activity has been demonstrated as a key factor in its enhancement. This meta-analysis aims to assess the impact of specific sports interventions on the executive function of children and teenagers. A systematic review was carried out on 1 November 2020 to search for published scientific evidence that analysed different sports programs that possibly affected executive function in students. Longitudinal studies, which assessed the effects of sports interventions on subjects between 6 and 18 years old, were identified through a systematic search of the four principal electronic databases: Web of Science, PubMed, Scopus, and EBSCO. A total of eight studies, with 424 subjects overall, met the inclusion criteria and were classified based on one or more of the following categories: working memory, inhibitory control, and cognitive flexibility. The random-effects model for meta-analyses was performed with RevMan version 5.3 to facilitate the analysis of the studies. Large effect sizes were found in all categories: working memory (ES −1.25; 95% CI −1.70; −0.79; p < 0.0001); inhibitory control (ES −1.30; 95% CI −1.98; −0.63; p < 0.00001); and cognitive flexibility (ES −1.52; 95% CI −2.20; −0.83; p < 0.00001). Our analysis concluded that healthy children and teenagers should be encouraged to practice sports in order to improve their executive function at every stage of their development.

Download Full-text

Fractal Dimension and Retinal Pathology: A Meta-Analysis

Applied Sciences ◽

10.3390/app11052376 ◽

2021 ◽

Vol 11 (5) ◽

pp. 2376

Author(s):

Sam Yu ◽

Vasudevan Lakshminarayanan

Keyword(s):

Fractal Dimension ◽

Fractal Analysis ◽

Meta Analysis ◽

Scientific Evidence ◽

Retinal Disease ◽

Clinical Value ◽

Disease Category ◽

Normal Individuals ◽

Retinal Pathology ◽

Meta Analyses

Due to the fractal nature of retinal blood vessels, the retinal fractal dimension is a natural parameter for researchers to explore and has garnered interest as a potential diagnostic tool. This review aims to summarize the current scientific evidence regarding the relationship between fractal dimension and retinal pathology and thus assess the clinical value of retinal fractal dimension. Following the PRISMA guidelines, a literature search for research articles was conducted in several internet databases (EMBASE, MEDLINE, Web of Science, Scopus). This led to a result of 28 studies included in the final review, which were analyzed via meta-analysis to determine whether the fractal dimension changes significantly in retinal disease versus normal individuals. From the meta-analysis, summary effect sizes and 95% confidence intervals were derived for each disease category. The results for diabetic retinopathy and myopia suggest decreased retinal fractal dimension for those pathologies with the association for other diseases such as diabetes mellitus, hypertension, and glaucoma remaining uncertain. Due to heterogeneity in imaging/fractal analysis setups used between studies, it is recommended that standardized retinal fractal analysis procedures be implemented in order to facilitate future meta-analyses.

Download Full-text

A protocol for the systematic review and meta-analysis of thigmotactic behaviour in the open field test in rodent models associated with persistent pain

BMJ Open Science ◽

10.1136/bmjos-2020-100135 ◽

2021 ◽

Vol 5 (1) ◽

pp. e100135

Author(s):

Xue Ying Zhang ◽

Jan Vollert ◽

Emily S Sena ◽

Andrew SC Rice ◽

Nadia Soliman

Keyword(s):

Systematic Review ◽

Outcome Measure ◽

Meta Analysis ◽

Persistent Pain ◽

Effect Sizes ◽

Bias Assessment ◽

Animal Pain ◽

Trim And Fill ◽

Meta Analyses ◽

The Impact

ObjectiveThigmotaxis is an innate predator avoidance behaviour of rodents and is enhanced when animals are under stress. It is characterised by the preference of a rodent to seek shelter, rather than expose itself to the aversive open area. The behaviour has been proposed to be a measurable construct that can address the impact of pain on rodent behaviour. This systematic review will assess whether thigmotaxis can be influenced by experimental persistent pain and attenuated by pharmacological interventions in rodents.Search strategyWe will conduct search on three electronic databases to identify studies in which thigmotaxis was used as an outcome measure contextualised to a rodent model associated with persistent pain. All studies published until the date of the search will be considered.Screening and annotationTwo independent reviewers will screen studies based on the order of (1) titles and abstracts, and (2) full texts.Data management and reportingFor meta-analysis, we will extract thigmotactic behavioural data and calculate effect sizes. Effect sizes will be combined using a random-effects model. We will assess heterogeneity and identify sources of heterogeneity. A risk-of-bias assessment will be conducted to evaluate study quality. Publication bias will be assessed using funnel plots, Egger’s regression and trim-and-fill analysis. We will also extract stimulus-evoked limb withdrawal data to assess its correlation with thigmotaxis in the same animals. The evidence obtained will provide a comprehensive understanding of the strengths and limitations of using thigmotactic outcome measure in animal pain research so that future experimental designs can be optimised. We will follow the Preferred Reporting Items for Systematic Reviews and Meta-Analyses reporting guidelines and disseminate the review findings through publication and conference presentation.

Download Full-text