scholarly journals Strengthening the foundation of educational psychology by integrating construct validation into open science reform

2021 ◽  
Author(s):  
Jessica Kay Flake
Keyword(s):  
Educational Psychology ◽  
Large Scale ◽  
Construct Validation ◽  
Open Science ◽  
Measurement Development ◽  
Reform Movement ◽  
Research Culture ◽  
Replication Studies ◽  
Replication Research ◽  
Access To Data

An increased focus on transparency and replication in science has stimulated reform in research practices and dissemination. As a result, the research culture is changing: the use of preregistration is on the rise, access to data and materials is increasing, and large-scale replication studies are more common. In this paper, I discuss two problems the methodological reform movement is now ready to tackle given the progress thus far and how educational psychology is particularly well suited to contribute. The first problem is that there is a lack of transparency and rigor in measurement development and use. The second problem is caused by the first; replication research is difficult and potentially futile as long as the first problem persists. I describe how to expand transparent practices into measure use and how construct validation can be implemented to bolster the validity of replication studies.

Addressing a Crisis of Generalizability with Large-Scale Construct Validation

2021 ◽  
Author(s):  
Jessica Kay Flake ◽  
Mairead Shaw ◽  
Raymond Luong
Keyword(s):  
Construct Validity ◽  
Large Scale ◽  
Construct Validation ◽  
Replication Studies ◽  
Descriptive Research ◽  
Psychological Science ◽  
Replication Research ◽  
New Ideas ◽  
Post Hoc ◽  
Validation Research

Yarkoni describes a grim state of psychological science in which the gross misspecification of our models and specificity of our operationalizations produce claims with generality so narrow that no one would be interested in them. We consider this a generalizability of construct validity issue and discuss how construct validation research should precede large-scale replication research. We provide ideas for a path forward by suggesting psychologists take a few steps back. By retooling large-scale replication studies, psychologists can execute the descriptive research needed to assess the generalizability of constructs. We provide examples of reusing large-scale replication data to conduct construct validation research post hoc. We also discuss proof of concept research that is on-going at the Psychological Science Accelerator. Big team psychology makes large-scale construct validity and generalizability research feasible and worthwhile. We assert that no one needs to quit the field, in fact, there is plenty of work to do. The optimistic interpretation is that if psychologists focus less on generating new ideas and more on organizing, synthesizing, measuring, and assessing constructs from existing ideas, we can keep busy for at least 100 years.

scholarly journals How to Teach Open Science Principles in the Undergraduate Curriculum—The Hagen Cumulative Science Project

2019 ◽  
Vol 19 (1) ◽  
pp. 91-106 ◽  
Author(s):  
Marc Jekel ◽  
Susann Fiedler ◽  
Ramona Allstadt Torras ◽  
Dorothee Mischkowski ◽  
Angela Rachael Dorrough ◽  
...  
Keyword(s):  
Large Scale ◽  
Meta Analysis ◽  
Undergraduate Curriculum ◽  
Research Process ◽  
Open Science ◽  
Science Project ◽  
Replication Studies ◽  
Community Benefits ◽  
Science Instructors ◽  
Thesis Work

The Hagen Cumulative Science Project is a large-scale replication project based on students’ thesis work. In the project, we aim to (a) teach students to conduct the entire research process for conducting a replication according to open science standards and (b) contribute to cumulative science by increasing the number of direct replications. We describe the procedural steps of the project from choosing suitable replication studies to guiding students through the process of conducting a replication, and processing results in a meta-analysis. Based on the experience of more than 80 replications, we summarize how such a project can be implemented. We present practical solutions that have been shown to be successful as well as discuss typical obstacles and how they can be solved. We argue that replication projects are beneficial for all groups involved: Students benefit by being guided through a highly structured protocol and making actual contributions to science. Instructors benefit by using time resources effectively for cumulative science and fulfilling teaching obligations in a meaningful way. The scientific community benefits from the resulting greater number of replications and teaching state-of-the-art methodology. We encourage the use of student thesis-based replication projects for thesis work in academic bachelor and master curricula.

scholarly journals Advancing the Cognitive Science of Religion through Replication and Open Science

2021 ◽  
Author(s):  
Suzanne Hoogeveen ◽  
Michiel van Elk
Keyword(s):  
Qualitative Research ◽  
Cognitive Science ◽  
Developmental Psychology ◽  
Open Science ◽  
Cross Cultural ◽  
Cognitive Science Of Religion ◽  
Science Practices ◽  
Replication Studies ◽  
Replication Research ◽  
Science Of Religion

The Cognitive Science of Religion (CSR) is a relatively young but prolific field that has offered compelling insights into religious minds and practices. However, many empirical findings within this field are still preliminary and their reliability remains to be determined. In this paper, we first argue that it is crucial to critically evaluate the CSR literature and adopt open science practices and replication research in particular to move the field forward. Second, we highlight the outcomes of previous replications and make suggestions for future replication studies in the CSR, with a particular focus on neuroscience, developmental psychology, and qualitative research. Finally, we provide a ‘replication script’ with advice on how to select, conduct, and organize replication research. Our approach is illustrated with a ‘glimpse behind the scenes’ of the recently launched Cross-Cultural Religious Replication Project, in the hope of inspiring scholars of religion to embrace open science and replication in their own research.

scholarly journals The effects of a temporal framing manipulation on environmentalism: A replication and extension

PLoS ONE ◽  
2021 ◽  
Vol 16 (2) ◽  
pp. e0246058
Author(s):  
Samantha K. Stanley ◽  
Anna Klas ◽  
Edward J. R. Clarke ◽  
Iain Walker
Keyword(s):  
Climate Change ◽  
Political Ideology ◽  
Large Scale ◽  
Environmental Changes ◽  
Open Science ◽  
Political Orientation ◽  
Temporal Framing ◽  
Replication Studies ◽  
The Past ◽  
State Of The Environment

Recent research promotes comparing the current state of the environment with the past (and not the future) to increase the pro-environmental attitudes of those on the political right. We aimed to replicate this temporal framing effect and extend on research in this area by testing the potential drivers of the effect. Across two large-scale replication studies, we found limited evidence that past comparisons (relative to future comparisons) increase pro-environmentalism among those with a more conservative political ideology, thus precluding a full investigation into the mediators of the effect. Where the effect was present, it was not consistent across studies. In Study One, conservatives reported greater certainty that climate change was real after viewing past comparisons, as the environmental changes were perceived as more certain. However, in Study Two, the temporal framing condition interacted with political orientation to instead undermine the certainty about climate change among political liberals in the past-focused condition. Together, these studies present the first evidence of backfire from temporal frames, and do not support the efficacy of past comparisons for increasing conservatives’ environmentalism. We echo recent calls for open science principles, including preregistration and efforts to replicate existing work, and suggest the replication of other methods of inducing temporal comparisons.

Holding replication studies to mainstream standards of evidence

2018 ◽  
Vol 41 ◽  
Author(s):  
Duane T. Wegener ◽  
Leandre R. Fabrigar
Keyword(s):  
Replication Studies ◽  
Replication Research ◽  
Standards Of Evidence ◽  
Do So

AbstractReplications can make theoretical contributions, but are unlikely to do so if their findings are open to multiple interpretations (especially violations of psychometric invariance). Thus, just as studies demonstrating novel effects are often expected to empirically evaluate competing explanations, replications should be held to similar standards. Unfortunately, this is rarely done, thereby undermining the value of replication research.

Identification of and Correction for Publication Bias: Comment

2019 ◽  
Author(s):  
Amanda Kvarven ◽  
Eirik Strømland ◽  
Magnus Johannesson
Keyword(s):  
Publication Bias ◽  
False Positive ◽  
Large Scale ◽  
Meta Analysis ◽  
False Positive Rate ◽  
Effect Sizes ◽  
Replication Studies ◽  
Moderate Reduction ◽  
Positive Rate ◽  
Meta Analyses

Andrews & Kasy (2019) propose an approach for adjusting effect sizes in meta-analysis for publication bias. We use the Andrews-Kasy estimator to adjust the result of 15 meta-analyses and compare the adjusted results to 15 large-scale multiple labs replication studies estimating the same effects. The pre-registered replications provide precisely estimated effect sizes, which do not suffer from publication bias. The Andrews-Kasy approach leads to a moderate reduction of the inflated effect sizes in the meta-analyses. However, the approach still overestimates effect sizes by a factor of about two or more and has an estimated false positive rate of between 57% and 100%.

scholarly journals Genetic and Transcriptomic Biomarkers in Neurodegenerative Diseases: Current Situation and the Road Ahead

Cells ◽  
2021 ◽  
Vol 10 (5) ◽  
pp. 1030
Author(s):  
Julie Lake ◽  
Catherine S. Storm ◽  
Mary B. Makarious ◽  
Sara Bandres-Ciga
Keyword(s):  
Neurodegenerative Diseases ◽  
Large Scale ◽  
Disease Diagnosis ◽  
Open Science ◽  
Future Directions ◽  
The Road ◽  
Current State ◽  
Molecular Complexity ◽  
Science Framework ◽  
Lateral Sclerosis

Neurodegenerative diseases are etiologically and clinically heterogeneous conditions, often reflecting a spectrum of disease rather than well-defined disorders. The underlying molecular complexity of these diseases has made the discovery and validation of useful biomarkers challenging. The search of characteristic genetic and transcriptomic indicators for preclinical disease diagnosis, prognosis, or subtyping is an area of ongoing effort and interest. The next generation of biomarker studies holds promise by implementing meaningful longitudinal and multi-modal approaches in large scale biobank and healthcare system scale datasets. This work will only be possible in an open science framework. This review summarizes the current state of genetic and transcriptomic biomarkers in Parkinson’s disease, Alzheimer’s disease, and amyotrophic lateral sclerosis, providing a comprehensive landscape of recent literature and future directions.

A Systematic Review of Replicative Self-Determination Intervention Studies

2019 ◽  
Vol 54 (1) ◽  
pp. 29-39
Author(s):  
John L. Luckner ◽  
Rashida Banerjee ◽  
Sara Movahedazarhouligh ◽  
Kaitlyn Millen
Keyword(s):  
Systematic Review ◽  
Intervention Studies ◽  
Self Determination ◽  
Future Research ◽  
Replication Studies ◽  
Research Designs ◽  
Replication Research ◽  
Quality Of Research

Current federal legislation emphasizes the use of programs, interventions, strategies, and activities that have been demonstrated through research to be effective. One way to increase the quantity and quality of research that guides practice is to conduct replication research. The purpose of this study was to undertake a systematic review of the replication research focused on self-determination conducted between 2007 and 2017. Using methods used by Cook and colleagues, we identified 80 intervention studies on topics related to self-determination, of which 31 were coded as replications. Intervention study trends, rate of replication studies, percentage of agreements between findings of original and replication studies, amount of author overlap, and types of research designs used are reported along with recommendations for future research.

Sign in / Sign up

Export Citation Format

Share Document