Recommendations for Reviewing Meta-Analyses in Organizational Research

This article encourages transparency in the reporting of meta-analytic procedures. Specifically, we highlight aspects of meta-analytic search, coding, data presentation, and data analysis where published meta-analyses often fall short in presenting sufficient information to allow replication. We identify opportunities where reviewers can request additional information or analyses that will enhance transparent reporting practices and facilitate the evaluation of quality in meta-analytic reporting. We focus on concerns specific to (or prevalent in) meta-analyses conducted in organizational research. In doing so, we reference a number of existing and emerging techniques, highlighting their contribution to meta-analysis while emphasizing key information reviewers may request. Our focus is primarily on meta-analyses, but secondary uses of meta-analytic data are also considered. We conclude by providing a checklist for reviewers in an effort to facilitate the review process as it pertains to the goals of transparency and replicability.

Download Full-text

Transparency and Reproducibility of Meta-Analyses in Psychology: A Meta-Review

Perspectives on Psychological Science ◽

10.1177/1745691620906416 ◽

2020 ◽

Vol 15 (4) ◽

pp. 1026-1041 ◽

Cited By ~ 2

Author(s):

Joshua R. Polanin ◽

Emily A. Hennessy ◽

Sho Tsuji

Keyword(s):

Effect Size ◽

Best Practice ◽

Meta Analysis ◽

Eligibility Criteria ◽

Reporting Practices ◽

Transparent Reporting ◽

Meta Review ◽

Meta Analyses ◽

High Degree ◽

Specific Search

Systematic review and meta-analysis are possible as viable research techniques only through transparent reporting of primary research; thus, one might expect meta-analysts to demonstrate best practice in their reporting of results and have a high degree of transparency leading to reproducibility of their work. This assumption has yet to be fully tested in the psychological sciences. We therefore aimed to assess the transparency and reproducibility of psychological meta-analyses. We conducted a meta-review by sampling 150 studies from Psychological Bulletin to extract information about each review’s transparent and reproducible reporting practices. The results revealed that authors reported on average 55% of criteria and that transparent reporting practices increased over the three decades studied ( b = 1.09, SE = 0.24, t = 4.519, p < .001). Review authors consistently reported eligibility criteria, effect-size information, and synthesis techniques. Review authors, however, on average, did not report specific search results, screening and extraction procedures, and most importantly, effect-size and moderator information from each individual study. Far fewer studies provided statistical code required for complete analytical replication. We argue that the field of psychology and research synthesis in general should require review authors to report these elements in a transparent and reproducible manner.

Download Full-text

Does testosterone affect lateralization of brain and behaviour? A meta-analysis in humans and other animal species

Philosophical Transactions of the Royal Society B Biological Sciences ◽

10.1098/rstb.2008.0282 ◽

2008 ◽

Vol 364 (1519) ◽

pp. 929-942 ◽

Cited By ~ 69

Author(s):

Kristina A Pfannkuche ◽

Anke Bouma ◽

Ton G.G Groothuis

Keyword(s):

Sex Differences ◽

Animal Species ◽

Meta Analysis ◽

Mammalian Species ◽

Experimental Studies ◽

Fundamental Principle ◽

Sufficient Information ◽

Visuospatial Cognition ◽

Organizational Effects ◽

Meta Analyses

Lateralization of brain and behaviour has been the topic of research for many years in neuropsychology, but the factors guiding its development remain elusive. Based on sex differences in human lateralization, four hypotheses have been postulated that suggest a role for androgens, specifically testosterone. With the discovery that lateralization is a fundamental principle in the organization of brain and behaviour among vertebrates, it has now become possible to experimentally test such hypotheses in animal models. The use of different taxa, humans, other mammalian species and birds (with oestradiol and not testosterone involved in sexual differentiation in birds) facilitates to differentiate between the hypotheses. We used meta-analyses for analysing papers that provided sufficient information, and a semi-quantitative approach based on all relevant studies that we extracted from the literature. We tested the predictions of these hypotheses regarding strength and direction of lateralization for motor output, language and visuospatial cognition in these three taxa. We tested for sex differences and early organizational effects of testosterone (both correlative and experimental studies). We found sex differences in the direction of lateralization for non-human mammals (motor biases similar to humans) and in direction and strength in birds (visual cognitive tasks). However, the prediction that prenatal testosterone exposure affects the direction of lateralization was not supported for humans. In birds and non-human mammals, opposite trends were found, with the effect in non-human mammals being opposite to the expectation based on sex differences. None of the four hypotheses was sufficiently supported and more studies, testing a wider array of functions in different taxa while reporting the data more completely are needed.

Download Full-text

Not restricted to selection research: Accounting for indirect range restriction in organizational research

10.31234/osf.io/xnu5z ◽

2019 ◽

Author(s):

Jeffrey Alan Dahlke ◽

Brenton M. Wiernik

Keyword(s):

Large Scale ◽

Meta Analysis ◽

Predictor Variables ◽

Sampling Variance ◽

Range Restriction ◽

Organizational Research ◽

New Methods ◽

Selection Research ◽

Meta Analyses ◽

Statistical Artifact

Range restriction is a common problem in organizational research and is an important statistical artifact to correct for in meta-analysis. Historically, researchers have had to rely on range-restriction correc-tions that only make use of range-restriction information for one variable, but it is not uncommon for researchers to have such information for both variables in a correlation (e.g., when studying the cor-relation between two predictor variables). However, existing meta-analytic methods incorporating these corrections overlook their unique implications for estimating the sampling variance of corrected correlations and for accurately assigning weights to studies in individual-correction meta-analyses. We introduce new methods for computing individual-correction and artifact-distribution meta-analyses us-ing the bivariate indirect range-restriction (BVIRR; “Case V”) correction and describe improved meth-ods for applying BVIRR corrections that substantially reduce bias in parameter estimation. We illustrate the effectiveness of these methods in a large-scale simulation and in meta-analyses of expatriate data. We provide R code to implement the methods described in this article; more comprehensive and robust functions for applying these methods are available in the psychmeta package for R (Dahlke & Wiernik, 2018, 2017/2019).

Download Full-text

What components of smoking cessation care during pregnancy are implemented by health providers? A systematic review and meta-analysis

BMJ Open ◽

10.1136/bmjopen-2018-026037 ◽

2019 ◽

Vol 9 (8) ◽

pp. e026037 ◽

Cited By ~ 2

Author(s):

Gillian Sandra Gould ◽

Laura Twyman ◽

Leah Stevenson ◽

Gabrielle R Gribbin ◽

Billie Bonevski ◽

...

Keyword(s):

Systematic Review ◽

Smoking Cessation ◽

Data Extraction ◽

Meta Analysis ◽

Self Report ◽

Health Providers ◽

Eligibility Criteria ◽

Reporting Practices ◽

Registration Number ◽

Meta Analyses

BackgroundPregnancy is an opportunity for health providers to support women to stop smoking.ObjectivesIdentify the pooled prevalence for health providers in providing components of smoking cessation care to women who smoke during pregnancy.DesignA systematic review synthesising original articles that reported on (1) prevalence of health providers’ performing the 5As (‘Ask’, ‘Advise’, ‘Assess’, ‘Assist’, ‘Arrange’), prescribing nicotine replacement therapy (NRT) and (2) factors associated with smoking cessation care.Data sourcesMEDLINE, EMBASE, CINAHL and PsycINFO databases searched using ‘smoking’, ‘pregnancy’ and ‘health provider practices’.Eligibility criteria for selecting studiesStudies included any design except interventions (self-report, audit, observed consultations and women’s reports), in English, with no date restriction, up to June 2017.ParticipantsHealth providers of any profession.Data extraction, appraisal and analysisData were extracted, then appraised with the Hawker tool. Meta-analyses pooled percentages for performing each of the 5As and prescribing NRT, using, for example, ‘often/always’ and ‘always/all’. Meta-regressions were performed of 5As for ‘often/always’.ResultsOf 3933 papers, 54 were included (n=29 225 participants): 33 for meta-analysis. Health providers included general practitioners, obstetricians, midwives and others from 10 countries. Pooled percentages of studies reporting practices ‘often/always’ were: ‘Ask’ (n=9) 91.6% (95% CI 88.2% to 95%); ‘Advise’ (n=7) 90% (95% CI 72.5% to 99.3%), ‘Assess’ (n=3) 79.2% (95% CI 76.5% to 81.8%), ‘Assist (cessation support)’ (n=5) 59.1% (95% CI 56% to 62.2%), ‘Arrange (referral)’ (n=6) 33.3% (95% CI 20.4% to 46.2%) and ‘prescribing NRT’ (n=6) 25.4% (95% CI 12.8% to 38%). Heterogeneity (I2) was 95.9%–99.1%. Meta-regressions for ‘Arrange’ were significant for year (p=0.013) and country (p=0.037).ConclusionsHealth providers ‘Ask’, ‘Advise’ and ‘Assess’ most pregnant women about smoking. ‘Assist’, ‘Arrange’ and ‘prescribing NRT’ are reported at lower rates: strategies to improve these should be considered.PROSPERO registration numberCRD42015029989.

Download Full-text

An Empirical Review of Research and Reporting Practices in Psychological Meta-Analyses

Review of General Psychology ◽

10.1177/1089268020918844 ◽

2020 ◽

Vol 24 (3) ◽

pp. 195-209

Author(s):

Richard E. Hohn ◽

Kathleen L. Slaney ◽

Donna Tafreshi

Keyword(s):

Meta Analysis ◽

Reporting Standards ◽

Exclusion Criteria ◽

Research Practices ◽

Reporting Practices ◽

Reporting Decisions ◽

Research Findings ◽

State Of Research ◽

Meta Analyses ◽

Made In

As meta-analytic studies have come to occupy a sizable contingent of published work in the psychological sciences, clarity in the research and reporting practices of such work is crucial to the interpretability and reproducibility of research findings. The present study examines the state of research and reporting practices within a random sample of 384 published psychological meta-analyses across several important dimensions (e.g., search methods, exclusion criteria, statistical techniques). In addition, we surveyed the first authors of the meta-analyses in our sample to ask them directly about the research practices employed and reporting decisions made in their studies, including the assessments and procedures they conducted and the guidelines or materials they relied on. Upon cross-validating the first author responses with what was reported in their published meta-analyses, we identified numerous potential gaps in reporting and research practices. In addition to providing a survey of recent reporting practices, our findings suggest that (a) there are several research practices conducted by meta-analysts that are ultimately not reported; (b) some aspects of meta-analysis research appear to be conducted at disappointingly low rates; and (c) the adoption of the reporting standards, including the Meta-Analytic Reporting Standards (MARS), has been slow to nonexistent within psychological meta-analytic research.

Download Full-text

The Effectiveness of Telemedicine Solutions for the Management of Type 2 Diabetes: A Systematic Review, Meta-Analysis, and Meta-Regression

Journal of Diabetes Science and Technology ◽

10.1177/19322968211064633 ◽

2021 ◽

pp. 193229682110646

Author(s):

Stine Hangaard ◽

Sisse H. Laursen ◽

Jonas D. Andersen ◽

Thomas Kronborg ◽

Peter Vestergaard ◽

...

Keyword(s):

Systematic Review ◽

Type 2 Diabetes ◽

Systematic Reviews ◽

Meta Analysis ◽

Cochrane Library ◽

Controlled Trials ◽

Sufficient Information ◽

Meta Regression ◽

Meta Analyses

Background: Previous systematic reviews have aimed to clarify the effect of telemedicine on diabetes. However, such reviews often have a narrow focus, which calls for a more comprehensive systematic review within the field. Hence, the objective of the present systematic review, meta-analysis, and meta-regression is to evaluate the effectiveness of telemedicine solutions versus any comparator without the use of telemedicine on diabetes-related outcomes among adult patients with type 2 diabetes (T2D). Methods: This review was conducted in accordance with the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines. We considered telemedicine randomized controlled trials (RCT) including adults (≥18 years) diagnosed with T2D. Change in glycated hemoglobin (HbA1c, %) was the primary outcome. PubMed, EMBASE, and the Cochrane Library Central Register of Controlled Trials (CENTRAL) were searched on October 14, 2020. An overall treatment effect was estimated using a meta-analysis performed on the pool of included studies based on the mean difference (MD). The revised Cochrane risk-of-bias tool was applied and the certainty of evidence was graded using the Grading of Recommendations, Assessment, Development and Evaluations (GRADE) approach. Results: The final sample of papers included a total of 246, of which 168 had sufficient information to calculate the effect of HbA1c%. The results favored telemedicine, with an MD of −0.415% (95% confidence interval [CI] = −0.482% to −0.348%). The heterogeneity was great (I2 = 93.05%). A monitoring component gave rise to the higher effects of telemedicine. Conclusions: In conclusion, telemedicine may serve as a valuable supplement to usual care for patients with T2D. The inclusion of a telemonitoring component seems to increase the effect of telemedicine.

Download Full-text

The accuracy of saliva versus nasopharyngeal and/or oropharyngeal samples for the detection of SARS-CoV-2 in children. A rapid systematic review and meta-analysis

10.1101/2021.06.21.21259284 ◽

2021 ◽

Author(s):

Sharonjit Kaur Dhillon ◽

Petra Schelstraete ◽

Laura Cornelissen ◽

Yves Lafort ◽

Jesper Bonde ◽

...

Keyword(s):

Meta Analysis ◽

High Viral Load ◽

Reference Standard ◽

Rt Pcr ◽

Comparative Performance ◽

Additional Information ◽

Asymptomatic Children ◽

Comparable Performance ◽

Polymerase Chain ◽

Meta Analyses

Background The comparative performance of saliva and nasopharyngeal samples for the detection of severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) infection by reverse transcriptase polymerase chain reaction (RT-PCR) in children remains unclear. As schools reopen around the world, there is an interest in the use of saliva samples for detection of SARS-CoV-2 in children to circumvent barriers with nasopharyngeal sampling. We systematically reviewed the literature to understand the performance of saliva sampling using RT-PCR on naso- and/or oropharyngeal swabs as the reference standard. Methods Articles from PubMed/MEDLINE and Living Evidence were accessed until 28th April 2021. A search method without restriction to children population was applied and during the review phase, if a study included patients <18 years old, authors were contacted to provide additional information on the subset of children. Studies were eligible if they reported on matched saliva and naso- and/or oropharyngeal samples, taken from the same patient on the same day. Studies using other respiratory samples such as sputum samples were excluded. Each paired patient sample had to be tested on the same RT-PCR platform. Results Ten studies were included, comprising 1486 matched saliva and on naso- and/or oropharyngeal pairs from children aged 0 to 18 years old. The pooled absolute sensitivity and specificity of saliva sampling using RT-PCR on nasopharyngeal samples as the reference standard was 84.5% (95% CI; 78.0%-90.3%) and 99.5% (95% CI; 98.2%-100.0%). Comparable performance of saliva to nasopharyngeal samples was shown in both symptomatic and asymptomatic children. Stratified analyses of various covariates showed no significant differences. Discussion Our pooled accuracy estimates of RT-PCR SARS-CoV-2 testing on saliva in children did not seem to be different from meta-analyses of studies that enrolled mainly adults. Saliva could potentially be considered an alternative sampling method for screening in children and to pick up those with high viral load.

Download Full-text

A Topical and Methodological Systematic Review of Meta-Analyses Published in the Educational Measurement Literature

10.35542/osf.io/ygds4 ◽

2019 ◽

Author(s):

Joseph Rios ◽

Samuel Ihlenfeldt ◽

Michael Dosedel ◽

Amy Riegelman

Keyword(s):

Systematic Review ◽

Statistical Power ◽

Meta Analysis ◽

Educational Measurement ◽

Test Accommodations ◽

Evidence Based Practices ◽

Reporting Practices ◽

Operational Testing ◽

Format Effects ◽

Meta Analyses

This systematic review investigated the topics studied and reporting practices of published meta-analyses in educational measurement. Our findings indicated that meta-analysis is not a highly utilized methodological tool in educational measurement, as on average, less than one meta-analysis has been published per year over the last 30 years (28 meta-analyses were published between 1986 to 2016). Within the field, researchers have utilized meta-analysis to study three primary subject areas: test format effects, test accommodations, and predictive validity of operational testing programs. In regard to reporting practices, authors often failed to provide descriptive details of both their search strategy and sample characteristics limiting reproducibility and generalizability of findings, respectively. Furthermore, diagnostic analyses of outliers, publication bias, and statistical power were not provided for the majority of studies, putting into question the validity of inferences made from the meta-analyses sampled. The lack of transparent and replicable practices of meta-analyses in educational measurement are concerns for generating credible research syntheses that can assist the field in improving evidence-based practices. Recommendations are provided for improving training and editorial standards of meta-analytic research.

Download Full-text

Statistical reporting of metabolomics data: experience from a high-throughput NMR platform and epidemiological applications

Metabolomics ◽

10.1007/s11306-019-1626-y ◽

2019 ◽

Vol 16 (1) ◽

Cited By ~ 2

Author(s):

Stefan Mutter ◽

Carrie Worden ◽

Kara Paxton ◽

Ville-Petteri Mäkinen

Keyword(s):

Literature Review ◽

High Throughput ◽

Meta Analysis ◽

Effect Sizes ◽

Metabolomics Data ◽

P Values ◽

Reporting Practices ◽

Numeric Formats ◽

Meta Analyses

Abstract Introduction Meta-analysis is the cornerstone of robust biomedical evidence. Objectives We investigated whether statistical reporting practices facilitate metabolomics meta-analyses. Methods A literature review of 44 studies that used a comparable platform. Results Non-numeric formats were used in 31 studies. In half of the studies, less than a third of all measures were reported. Unadjusted P-values were missing from 12 studies and exact P-values from 9 studies. Conclusion Reporting practices can be improved. We recommend (i) publishing all results as numbers, (ii) reporting effect sizes of all measured metabolites and (iii) always reporting unadjusted exact P-values.

Download Full-text

Sorafenib in the treatment of patients with hepatocellular carcinoma (HCC) and microvascular infiltration: a systematic review and meta-analysis

Journal of International Medical Research ◽

10.1177/0300060520946872 ◽

2020 ◽

Vol 48 (8) ◽

pp. 030006052094687

Author(s):

Wang Gu ◽

Zhong Tong

Keyword(s):

Hepatocellular Carcinoma ◽

Meta Analysis ◽

Cancer Recurrence ◽

Microvascular Invasion ◽

Cochrane Library ◽

Sufficient Information ◽

Sorafenib Treatment ◽

Rate Versus ◽

Meta Analyses ◽

The Impact

Objective Microvascular invasion is shown to be an independent risk factor for liver cancer recurrence. Timely treatment may reduce the recurrence rate and prolong total survival time. The aim of this study was to investigate the effectiveness of sorafenib in treating patients with hepatocellular carcinoma (HCC) and microvascular invasion Methods A comprehensive literature search was conducted in PubMed, EMBASE, MEDLINE, web of science and Cochrane Library databases for articles published up to December 2019. Two researchers independently reviewed and cross-checked independent reports with sufficient information. A meta-analysis was conducted to assess the impact of sorafenib on mortality in patients with HCC and microvascular involvement. Results Four studies were included in the qualitative and quantitative analyses, comprising 955 cancer events and 505 cancer deaths. Meta-analyses showed that sorafenib treatment was associated with an improved survival rate versus no sorafenib treatment in patients with HCC and microvascular invasion (relative risk 1.369, 95% confidence interval 1.193, 1.570). Conclusions Sorafenib treatment may improve survival in patients with HCC and microvascular invasion. However, due to the potential for residual confounding, the results should be interpreted with caution.

Download Full-text