scholarly journals The meta-analyses of deliberate practice underestimate the effect size because they neglect the core characteristic of individualization—an analysis and empirical evidence

2021 ◽  
Author(s):  
Tobias Debatin ◽  
Manuel D. S. Hopp ◽  
Wilma Vialle ◽  
Albert Ziegler
Keyword(s):  
Empirical Evidence ◽  
Effect Size ◽  
Explanatory Power ◽  
Deliberate Practice ◽  
High Level ◽  
Meta Analyses ◽  
Open Question ◽  
Chess Players ◽  
Predictive Strength

AbstractInfluential meta-analyses have concluded that only a small to medium proportion of variance in performance can be explained by deliberate practice. We argue that the authors have neglected the most important characteristic of deliberate practice: individualization of practice. Many of the analyzed effect sizes derived from measures that did not assess individualized practice and, therefore, should not have been included in meta-analyses of deliberate practice. We present empirical evidence which suggests that the level of individualization and quality of practice (indicated by didactic educational capital) substantially influences the predictive strength of practice measures. In our study of 178 chess players, we found that at a high level of individualization and quality of practice, the effect size of structured practice was more than three times higher than that found at the average level. Our theoretical analysis, along with empirical results, support the claim that the explanatory power of deliberate practice has been considerably underestimated in the meta-analyses. The question of how important deliberate practice is for individual differences in performance remains an open question.

scholarly journals Predictability of rotational tooth movement with orthodontic aligners comparing software-based and achieved data: A systematic review and meta-analysis of observational studies

2021 ◽  
pp. 146531252110272
Author(s):  
Despina Koletsi ◽  
Anna Iliadi ◽  
Theodore Eliades
Keyword(s):  
Effect Size ◽  
Tooth Movement ◽  
Data Extraction ◽  
Meta Analysis ◽  
Risk Of Bias ◽  
Patient Treatment ◽  
Qualitative Synthesis ◽  
Meta Analyses ◽  
Enamel Reduction

Objective: To evaluate all available evidence on the prediction of rotational tooth movements with aligners. Data sources: Seven databases of published and unpublished literature were searched up to 4 August 2020 for eligible studies. Data selection: Studies were deemed eligible if they included evaluation of rotational tooth movement with any type of aligner, through the comparison of software-based and actually achieved data after patient treatment. Data extraction and data synthesis: Data extraction was done independently and in duplicate and risk of bias assessment was performed with the use of the QUADAS-2 tool. Random effects meta-analyses with effect sizes and their 95% confidence intervals (CIs) were performed and the quality of the evidence was assessed through GRADE. Results: Seven articles were included in the qualitative synthesis, of which three contributed to meta-analyses. Overall results revealed a non-accurate prediction of the outcome for the software-based data, irrespective of the use of attachments or interproximal enamel reduction (IPR). Maxillary canines demonstrated the lowest percentage accuracy for rotational tooth movement (three studies: effect size = 47.9%; 95% CI = 27.2–69.5; P < 0.001), although high levels of heterogeneity were identified (I2: 86.9%; P < 0.001). Contrary, mandibular incisors presented the highest percentage accuracy for predicted rotational movement (two studies: effect size = 70.7%; 95% CI = 58.9–82.5; P < 0.001; I2: 0.0%; P = 0.48). Risk of bias was unclear to low overall, while quality of the evidence ranged from low to moderate. Conclusion: Allowing for all identified caveats, prediction of rotational tooth movements with aligner treatment does not appear accurate, especially for canines. Careful selection of patients and malocclusions for aligner treatment decisions remain challenging.

Developing Capability for a Continuous Improvement Culture in Petroleum Development Oman PDO

2021 ◽  
Author(s):  
Wayne Bryant ◽  
Zaaima Ghafri ◽  
Ahmed Salmi ◽  
Zeyana Yazeedi ◽  
Christopher Bell
Keyword(s):  
Continuous Improvement ◽  
Deliberate Practice ◽  
Development Program ◽  
Step Change ◽  
Full Time ◽  
Business Sector ◽  
High Potentials ◽  
High Level ◽  
Petroleum Development

Abstract Objectives/Scope To highlight at a high level the successful method of building Continuous Improvement (CI) competence Pan Petroleum Development Oman (PDO) and the different pathways used to engage everyone at every level of the organization circa. 10,000 people as part of PDO transformation to a Continuous Improvement organization Methods, Procedures, Process PDO is developing a CI culture through 2 pathways, CI Professional & CI Generalist. CI Professionals, limited to 50 high potentials, spend 3 years full time developing CI competence to a level they can coach autonomously in the business. CI Generalists build CI skills though training and deliberate practice becoming Lean Practitioners, Lean Managers and Practical Problem Solving (PPS) practitioners & facilitators. Results, Observations, Conclusions A CI function has been created to own the development of staff, standards, and deployment. Having quarterly competence development reviews for CI Professionals, to discuss and monitor progress and establish goals for the next quarter have resulted in a step change in speed and quality of development. We have had 100+ CI Coaches through the development program as of July 2021, either returning to the business to germinate CI in their new teams or developing further to become a career CI specialist. Developing the CI competences for CI Generalist has seen an explosion in ideas circa 10,000 with 500 teams practicing CI Fundamentals, 200+ qualified PPS facilitators & 250+ Lean managers coaching in the business. Benefits to date include circa 150K M3 production gain and many millions saved from the budget. Observation is we are developing autonomous teams able to solve problems for themselves resulting in an engaged workforce:- Improving today for a better tomorrow. Sponsorship from the top is the imperative ingredient for success. Engagement of the workforce from the bottom up, staff doing the work, creates the momentum required to make a CI transformation successful. Novel, Additive Information Holistically the approach used can be transferred to any other business and although tailored for PDO is business/sector is agnostic. The coaches who have graduated through our internal certification can and are applying Continuous Improvement in our business and externally in other organizations. Looking to the future we can partner with universities to have Continuous Improvement as part of the curriculum either as modules or a whole syllabus.

scholarly journals Meta-Analysis of Sensorimotor Gating Deficits in Patients With Schizophrenia Evaluated by Prepulse Inhibition Test

2020 ◽  
Vol 46 (6) ◽  
pp. 1482-1497
Author(s):  
Rodrigo San-Martin ◽  
Leonardo Andrade Castro ◽  
Paulo Rossi Menezes ◽  
Francisco José Fraga ◽  
Priscyla Waleska Simões ◽  
...  
Keyword(s):  
Prepulse Inhibition ◽  
Random Effects ◽  
Effect Size ◽  
Meta Analysis ◽  
Sensorimotor Gating ◽  
Control Group ◽  
Random Effects Models ◽  
Meta Regression Analysis ◽  
High Level ◽  
Meta Analyses

Abstracts Prepulse inhibition (PPI) of startle is an operational measure of sensorimotor gating that is often impaired in patients with schizophrenia. Despite the large number of studies, there is considerable variation in PPI outcomes reported. We conducted a systematic review and meta-analysis investigating PPI impairment in patients with schizophrenia compared with healthy control subjects, and examined possible explanations for the variation in results between studies. Major databases were screened for observational studies comparing healthy subjects and patients with schizophrenia for the prepulse and pulse intervals of 60 and 120 ms as primary outcomes, ie, PPI-60 and PPI-120. Standardized mean difference (SMD) and 95% confidence intervals (CI) were extracted and pooled using random effects models. We then estimated the mean effect size of these measures with random effects meta-analyses and evaluated potential PPI heterogeneity moderators, using sensitivity analysis and meta-regressions. Sixty-seven primary studies were identified, with 3685 healthy and 4290 patients with schizophrenia. The schizophrenia group showed reduction in sensorimotor gating for both PPI-60 (SMD = −0.50, 95% CI = [−0.61, −0.39]) and PPI-120 (SMD = −0.44, 95% CI = [−0.54, −0.33]). The sensitivity and meta-regression analysis showed that sample size, gender proportion, imbalance for gender, source of control group, and study continent were sources of heterogeneity (P &lt; .05) for both PPI-60 and PPI-120 outcomes. Our findings confirm a global sensorimotor gating deficit in schizophrenia patients, with overall moderate effect size for PPI-60 and PPI-120. Methodological consistency should decrease the high level of heterogeneity of PPI results between studies.

scholarly journals Effects of sodium bicarbonate supplementation on exercise performance: an umbrella review

2021 ◽  
Vol 18 (1) ◽  
Author(s):  
Jozo Grgic ◽  
Ivana Grgic ◽  
Juan Del Coso ◽  
Brad J. Schoenfeld ◽  
Zeljko Pedisic
Keyword(s):  
Sodium Bicarbonate ◽  
Effect Size ◽  
Exercise Performance ◽  
Methodological Quality ◽  
Muscle Endurance ◽  
Mean Power ◽  
Rowing Performance ◽  
Ergogenic Effects ◽  
Meta Analyses

Abstract Background We aimed to perform an umbrella review of meta-analyses examining the effects of sodium bicarbonate supplementation on exercise performance. Methods We systematically searched for meta-analyses that examined the effects of sodium bicarbonate supplementation on exercise performance. The methodological quality of the included reviews was evaluated using the Assessing the Methodological Quality of Systematic Reviews 2 (AMSTAR 2) checklist. Grading of Recommendations Assessment, Development, and Evaluation (GRADE) framework for downgrading the certainty in evidence was used, which included assessments of risk of bias, inconsistency, indirectness, imprecision, and publication bias. Results Eight reviews of moderate and high methodological quality met inclusion criteria. Using the GRADE framework, evidence for the ergogenic effects of sodium bicarbonate supplementation on peak and mean power in the Wingate test and Yo-Yo test performance was classified as being of moderate quality. The evidence for these outcomes did not receive a point on the indirectness GRADE item, as “serious indirectness” was detected. Low-quality evidence was found for the ergogenic effect of sodium bicarbonate supplementation on endurance events lasting ∼45 s to 8 min, muscle endurance, and 2000-m rowing performance. Evidence for these outcomes was classified as low quality, given that risk of bias, indirectness, and publication bias were assessed as “unclear”, “serious”, and “strongly suspected”, respectively. The ergogenic effects ranged from trivial (pooled effect size: 0.09) to large (pooled effect size: 1.26). Still, for most outcomes, sodium bicarbonate elicited comparable ergogenic effects. For example, sodium bicarbonate produced similar effects on performance in endurance events lasting ∼45 s to 8 min, muscle endurance tests, and Yo-Yo test (pooled effect size range: 0.36 to 0.40). No significant differences between the effects of sodium bicarbonate and placebo were found for general mean power, muscle strength, and repeated-sprint ability. Conclusion Based on meta-analyses of moderate to high quality, it can be concluded that sodium bicarbonate supplementation acutely enhances peak anaerobic power, anaerobic capacity, performance in endurance events lasting ∼45 s to 8 min, muscle endurance, 2000-m rowing performance, and high-intensity intermittent running. More research is needed among women to improve the generalizability of findings.

scholarly journals The puzzling relationship between multi-lab replications and meta-analyses of the rest of the literature

2020 ◽  
Author(s):  
Molly Lewis ◽  
Maya B Mathur ◽  
Tyler VanderWeele ◽  
Michael C. Frank
Keyword(s):  
Empirical Evidence ◽  
Effect Size ◽  
Meta Analysis ◽  
Scientific Research ◽  
Psychological Phenomena ◽  
Analytic Estimate ◽  
Meta Analyses ◽  
The Relationship ◽  
Size Estimates

What is the best way to estimate the size of important effects? Should we aggregate across disparate findings using statistical meta-analysis, or instead run large, multi-lab replications (MLR)? A recent paper by Kvarven, Strømland, and Johannesson (2020) compared effect size estimates derived from these two different methods for 15 different psychological phenomena. The authors report that, for the same phenomenon, the meta-analytic estimate tends to be about three times larger than the MLR estimate. These results pose an important puzzle: What is the relationship between these two estimates? Kvarven et al. suggest that their results undermine the value of meta-analysis. In contrast, we argue that both meta-analysis and MLR are informative, and that the discrepancy between estimates obtained via the two methods is in fact still unexplained. Informed by re-analyses of Kvarven et al.’s data and by other empirical evidence, we discuss possible sources of this discrepancy and argue that understanding the relationship between estimates obtained from these two methods is an important puzzle for future meta-scientific research.

The Effects of Match Congestion on Physical Performance in Football Referees

2021 ◽  
Author(s):  
Víctor Moreno-Perez ◽  
Javier Courel-Ibáñez ◽  
Juan Del Coso ◽  
Javier Sánchez-Sánchez
Keyword(s):  
Heart Rate ◽  
Effect Size ◽  
Perceived Exertion ◽  
Rating Of Perceived Exertion ◽  
Professional Football ◽  
Internal Load ◽  
Running Activity ◽  
Recovery Strategies ◽  
High Level

AbstractWe examined the changes in performance during congested (two matches within a 7-day interval) and non-congested (one match within≥7-day interval) fixtures in 17 elite football (soccer) referees during 181 official matches. External demands comprised 20 GPS-based metrics. Internal load was assessed by heart rate and rating of perceived exertion. Compared to non-congested fixtures, referees decreased their running distance at 21–24 km·h−1 (p=0.027, effect size [ES]=0.41) and > 24 km·h−1 (p=0.037, ES=0.28), the number of sprints (p=0.012, ES=0.29), and distance sprinting (p=0.022, ES=0.29) in congested matches. Most play metrics were lower in congested versus non-congested fixtures with low-to-moderate ES. During the 2nd half of non-congested fixtures, referees covered larger distances at low-speed running (p=0.025, ES=0.47). Match congestion due to officiating two matches less than a week apart caused a notable decrease in match running activity in professional football referees, especially at above 21 km·h−1. These data reiterate the need for specific conditioning and post-match recovery strategies in high-level referees to ensure optimal judgment performance favouring the quality of the competition. Governing bodies should take these outcomes into account when designating referees for a match.

Electroconvulsive Therapy for Depression: A Review of the Quality of ECT versus Sham ECT Trials and Meta-Analyses

2019 ◽  
Vol 21 (2) ◽  
pp. 64-103 ◽  
Author(s):  
John Read ◽  
Irving Kirsch ◽  
Laura McGrath
Keyword(s):  
Electroconvulsive Therapy ◽  
Effect Size ◽  
Memory Loss ◽  
Severe Depression ◽  
Double Blind ◽  
End Of Treatment ◽  
Significant Difference ◽  
Small Effect Size ◽  
Meta Analyses

BackgroundElectroconvulsive therapy (ECT) is still being administered to approximately a million people annually. There have been no ECT versus simulated ECT (SECT) studies since 1985. The five meta-analyses of ECT versus SECT studies all claim that ECT is more effective than SECT for its primary target, severe depression. This review assesses the quality of those meta-analyses and of the 11 studies on which they are based.MethodsThe meta-analyses were evaluated primarily in terms of whether they considered the quality of the studies they included, but also in terms of whether they addressed efficacy beyond end of treatment. The methodological rigor of the 11 studies included by one or more of the meta-analyses was assessed using a 24-point Quality scale developed for this review.ResultsThe five meta-analyses include between 1 and 7 of the 11 studies. The meta-analyses pay little or no attention to the multiple limitations of the studies they include. The 11 studies have a mean Quality score of 12.3 out of 24. Eight scored 13 or less. Only four studies describe their processes of randomization and testing the blinding. None convincingly demonstrate that they are double-blind. Five selectively report their findings. Only four report any ratings by patients. None assess Quality of Life. The studies are small, involving an average of 37 people. Four of the 11 found ECT significantly superior to SECT at the end of treatment, five found no significant difference and two found mixed results (including one where the psychiatrists reported a difference but patients did not). Only two higher Quality studies report follow-up data, one produced a near-zero effect size (.065) in the direction of ECT, and the other a small effect size (.299) in favor of SECT.ConclusionsThe quality of most SECT–ECT studies is so poor that the meta-analyses were wrong to conclude anything about efficacy, either during or beyond the treatment period. There is no evidence that ECT is effective for its target demographic—older women, or its target diagnostic group—severely depressed people, or for suicidal people, people who have unsuccessfully tried other treatments first, involuntary patients, or children and adolescents. Given the high risk of permanent memory loss and the small mortality risk, this longstanding failure to determine whether or not ECT works means that its use should be immediately suspended until a series of well designed, randomized, placebo-controlled studies have investigated whether there really are any significant benefits against which the proven significant risks can be weighed.

scholarly journals A Methodological Quality Evaluation of Meta-Analyses on Nursing Home Research: Overview and Suggestions for Future Directions

2022 ◽  
Vol 19 (1) ◽  
pp. 505
Author(s):  
In-Soo Shin ◽  
Juh-Hyun Shin ◽  
Dong-Eun Jang ◽  
Jiyeon Lee
Keyword(s):  
Nursing Home ◽  
Effect Size ◽  
Quality Evaluation ◽  
Research Field ◽  
Future Directions ◽  
Quality Reporting ◽  
Randomized Control ◽  
Statistical Issues ◽  
Meta Analyses

(1) Background: The nursing home (NH) research field lacks quality reporting about meta-analyses (MAs), and most gradings of MA evidence are biased on analyzing the effectiveness of independent variables in randomized control trials. (2) Objectives: This study aimed to perform a critical methodological review of MAs in the NH research field. (3) Methods: We searched the articles from four databases (PubMed, MEDLINE, CINAHL, and PsycINFO) until 15th January 2021. We reviewed a total of 41 published review articles in the NH research field. (4) Results: The studies primarily fell into the following categories: medicine (17/41), nursing (7/41), and psychiatry or psychology (6/41); 36.6% of the reviewed studies did not use any validated MA guidelines. The lowest correctly reported PRISMA 2000 guideline item was protocol and registration (14.6%), and more than 50% of articles did not report risk of bias. Moreover, 78.0% of studies did not describe missing reports of effect size formula. (5) Discussion: NH researchers must follow appropriate and updated guidelines for their MAs in order to provide validated reviews, as well as consider statistical issues such as the complexity of interventions, proper grouping, and scientific effect-size calculations to improve the quality of their study. Future quality review studies should investigate more diverse studies.

scholarly journals Addressing Publication Bias in Meta-Analysis

2020 ◽  
Vol 228 (1) ◽  
pp. 50-61 ◽  
Author(s):  
Sho Tsuji ◽  
Alejandrina Cristia ◽  
Michael C. Frank ◽  
Christina Bergmann
Keyword(s):  
Publication Bias ◽  
Effect Size ◽  
Developmental Psychology ◽  
Meta Analysis ◽  
Research Synthesis ◽  
Search Strategies ◽  
Lessons Learned ◽  
Synthesis Tool ◽  
Meta Analyses ◽  
Open Question

Abstract. Meta-analyses are an indispensable research synthesis tool for characterizing bodies of literature and advancing theories. One important open question concerns the inclusion of unpublished data into meta-analyses. Finding such studies can be effortful, but their exclusion potentially leads to consequential biases like overestimation of a literature’s mean effect. We address two questions about unpublished data using MetaLab, a collection of community-augmented meta-analyses focused on developmental psychology. First, we assess to what extent MetaLab datasets include gray literature, and by what search strategies they are unearthed. We find that an average of 11% of datapoints are from unpublished literature; standard search strategies like database searches, complemented with individualized approaches like including authors’ own data, contribute the majority of this literature. Second, we analyze the effect of including versus excluding unpublished literature on estimates of effect size and publication bias, and find this decision does not affect outcomes. We discuss lessons learned and implications.

scholarly journals A study of meta-analyses reporting quality in the large and expanding literature of educational technology

2021 ◽  
pp. 100-115
Author(s):  
Rana M. Tamim ◽  
Evgueni Borokhovski ◽  
Robert M. Bernard ◽  
Richard F. Schmid ◽  
Philip C. Abrami ◽  
...  
Keyword(s):  
Educational Technology ◽  
Effect Size ◽  
Meta Analysis ◽  
Reporting Quality ◽  
Poor Quality ◽  
Empirical Literature ◽  
Publication Date ◽  
Average Effect ◽  
Meta Analyses

As the empirical literature in educational technology continues to grow, meta-analyses are increasingly being used to synthesise research to inform practice. However, not all meta-analyses are equal. To examine their evolution over the past 30 years, this study systematically analysed the quality of 52 meta-analyses (1988–2017) on educational technology. Methodological and reporting quality is defined here as the completeness of the descriptive and methodological reporting features of meta-analyses. The study employed the Meta-Analysis Methodological Reporting Quality Guide (MMRQG), an instrument designed to assess 22 areas of reporting quality in meta-analyses. Overall, MMRQG scores were negatively related to average effect size (i.e., the higher the quality, the lower the effect size). Owing to the presence of poor-quality syntheses, the contribution of educational technologies to learning has been overestimated, potentially misleading researchers and practitioners. Nine MMRQG items discriminated between higher and lower average effect sizes. A publication date analysis revealed that older reviews (1988–2009) scored significantly lower on the MMRQG than more recent reviews (2010–2017). Although the increase in quality bodes well for the educational technology literature, many recent meta-analyses still show only moderate levels of quality. Identifying and using only best evidence-based research is thus imperative to avoid bias.   Implications for practice or policy: Educational technology practitioners should make use of meta-analytical findings that systematically synthesise primary research. Academics, policymakers and practitioners should consider the methodological quality of meta-analyses as they vary in reliability. Academics, policymakers and practitioners could avoid misleading bias in research evidence by using the MMRQG to evaluate the quality of meta-analyses. Meta-analyses with lower MMRQG scores should be considered with caution as they seem to overestimate the effect of educational technology on learning.

Sign in / Sign up

Export Citation Format

Share Document