scholarly journals Making the Black Box Transparent: A Template and Tutorial for Registration of Studies Using Experience-Sampling Methods

2021 ◽  
Vol 4 (1) ◽  
pp. 251524592092468
Author(s):  
Olivia J. Kirtley ◽  
Ginette Lafit ◽  
Robin Achterhof ◽  
Anu P. Hiekkaranta ◽  
Inez Myin-Germeys
Keyword(s):  
Degrees Of Freedom ◽  
Experience Sampling ◽  
Open Science ◽  
Data Sets ◽  
Science Practices ◽  
Methodological Rigor ◽  
Assessment Techniques ◽  
Ecological Momentary ◽  
Momentary Assessment ◽  
Experience Sampling Methods

A growing interest in understanding complex and dynamic psychological processes as they occur in everyday life has led to an increase in studies using ambulatory assessment techniques, including the experience-sampling method (ESM) and ecological momentary assessment. These methods, however, tend to involve numerous forking paths and researcher degrees of freedom, even beyond those typically encountered with other research methodologies. Although a number of researchers working with ESM techniques are actively engaged in efforts to increase the methodological rigor and transparency of research that uses them, currently there is little routine implementation of open-science practices in ESM research. In this article, we discuss the ways in which ESM research is especially vulnerable to threats to transparency, reproducibility, and replicability. We propose that greater use of study registration, a cornerstone of open science, may address some of these threats to the transparency of ESM research. Registration of ESM research is not without challenges, including model selection, accounting for potential model-convergence issues, and the use of preexisting data sets. As these may prove to be significant barriers for ESM researchers, we also discuss ways of overcoming these challenges and of documenting them in a registration. A further challenge is that current general preregistration templates do not adequately capture the unique features of ESM. We present a registration template for ESM research and also discuss registration of studies using preexisting data.

scholarly journals Making the black box transparent: A template and tutorial for (pre-)registration of studies using Experience Sampling Methods (ESM)

2019 ◽  
Author(s):  
Olivia J Kirtley ◽  
Ginette Lafit ◽  
Robin Achterhof ◽  
Anu Pauliina Hiekkaranta ◽  
Inez Myin-Germeys
Keyword(s):  
Degrees Of Freedom ◽  
Sampling Method ◽  
Experience Sampling ◽  
Open Science ◽  
Science Practices ◽  
Methodological Rigor ◽  
Assessment Techniques ◽  
Ecological Momentary ◽  
Momentary Assessment ◽  
Experience Sampling Methods

A growing interest in understanding complex and dynamic psychological processes as they occur in everyday life has led to an increase in studies using Ambulatory Assessment techniques, including the Experience Sampling Method (ESM) and Ecological Momentary Assessment (EMA). There are, however, numerous “forking paths” and researcher degrees of freedom, even beyond those typically encountered with other research methodologies. Whilst a number of researchers working with ESM techniques are actively engaged in efforts to increase the methodological rigor and transparency of such research, currently, there is little routine implementation of open science practices in ESM research. In the current paper, we discuss the ways in which ESM research is especially vulnerable to threats to transparency, reproducibility and replicability. We propose that greater use of (pre-)registration, a cornerstone of open science, may address some of these threats to the transparency of ESM research. (Pre-)registration of ESM research is not without challenges, including model selection, accounting for potential model convergence issues and the use of pre-existing datasets. As these may prove to be significant barriers to (pre-)registration for ESM researchers, we also discuss ways of overcoming these challenges and of documenting them in a (pre-)registration. A further challenge is that current general templates do not adequately capture the unique features of ESM. Here we present a (pre-)registration template for ESM research, adapted from the original Pre-Registration Challenge (Mellor et al., 2019) and pre-registration of pre-existing data (van den Akker et al., 2020) templates, and provide examples of how to complete this.

scholarly journals Researcher Degrees of Freedom in the Psychology of Religion

2018 ◽  
Author(s):  
Sarah Jane Charles ◽  
James Edward Bartlett ◽  
Kyle J. Messick ◽  
Thomas Joseph Coleman ◽  
Alex Uzdavines
Keyword(s):  
Systematic Review ◽  
Degrees Of Freedom ◽  
Psychology Of Religion ◽  
Open Science ◽  
Science Practices ◽  
Replication Crisis ◽  
The Way

There is a push in psychology toward more transparent practices, stemming partially as a response to the replication crisis. We argue that the psychology of religion should help lead the way toward these new, more transparent practices to ensure a robust and dynamic subfield. One of the major issues that proponents of Open Science practices hope to address is researcher degrees of freedom (RDF). We pre-registered and conducted a systematic review of the 2017 issues from three psychology of religion journals. We aimed to identify the extent to which the psychology of religion has embraced Open Science practices and the role of RDF within the subfield. We found that many of the methodologies that help to increase transparency, such as pre-registration, have yet to be adopted by those in the subfield. In light of these findings, we present recommendations for addressing the issue of transparency in the psychology of religion and outline how to move toward these new Open Science practices.

scholarly journals Preregistration specificity & adherence: A review of preregistered gambling studies & cross-disciplinary comparison

2021 ◽  
Author(s):  
Robert Heirene ◽  
Debi LaPlante ◽  
Eric R. Louderback ◽  
Brittany Keen ◽  
Marjan Bakker ◽  
...  
Keyword(s):  
Degrees Of Freedom ◽  
Open Data ◽  
Open Science ◽  
Eligibility Criteria ◽  
Science Practices ◽  
Methodological Choices ◽  
The Mean ◽  
The Many ◽  
Practical Recommendations ◽  
Current Practices

Study preregistration is one of several “open science” practices (e.g., open data, preprints) that researchers use to improve the transparency and rigour of their research. As more researchers adopt preregistration as a regular research practice, examining the nature and content of preregistrations can help identify strengths and weaknesses of current practices. The value of preregistration, in part, relates to the specificity of the study plan and the extent to which investigators adhere to this plan. We identified 53 preregistrations from the gambling studies field meeting our predefined eligibility criteria and scored their level of specificity using a 23-item protocol developed to measure the extent to which a clear and exhaustive preregistration plan restricts various researcher degrees of freedom (RDoF; i.e., the many methodological choices available to researchers when collecting and analysing data, and when reporting their findings). We also scored studies on a 32-item protocol that measured adherence to the preregistered plan in the study manuscript. We found that gambling preregistrations had low specificity levels on most RDoF. However, a comparison with a sample of cross-disciplinary preregistrations (N = 52; Bakker et al., 2020) indicated that gambling preregistrations scored higher on 12 (of 29) items. Thirteen (65%) of the 20 associated published articles or preprints deviated from the protocol without declaring as much (the mean number of undeclared deviations per article was 2.25, SD = 2.34). Overall, while we found improvements in specificity and adherence over time (2017-2020), our findings suggest the purported benefits of preregistration—including increasing transparency and reducing RDoF—are not fully achieved by current practices. Using our findings, we provide 10 practical recommendations that can be used to support and refine preregistration practices.

Support for Open Science Practices in Emotion Science: A Survey Study

2020 ◽  
Author(s):  
Olmo Van den Akker ◽  
Laura Danielle Scherer ◽  
Jelte M. Wicherts ◽  
Sander Koole
Keyword(s):  
Open Data ◽  
Open Science ◽  
High Impact ◽  
Survey Study ◽  
Science Practices ◽  
Replication Studies ◽  
Positive Attitudes ◽  
Methodological Rigor ◽  
Negative Findings ◽  
Larger Sample

So-called “open science practices” seek to improve research transparency and methodological rigor. What do emotion researchers think about these practices? To address this question, we surveyed active emotion researchers (N= 144) in October 2019 about their attitudes toward several open science practices. Overall, the majority of emotion researchers had positive attitudes toward open science practices and expressed a willingness to engage in such practices. Emotion researchers on average believed that replicability would improve by publishing more negative findings, by requiring open data and materials, and by conducting studies with larger sample sizes. Direct replications, multi-lab studies, and preregistration were all seen as beneficial to the replicability of emotion research. Emotion researchers believed that more direct replications would be conducted if replication studies would receive increased funding, more citations, and easier publication in high impact journals. Emotion researchers believed that preregistration would be stimulated by providing researchers with more information about its benefits and more guidance on its effective application. Overall, these findings point to considerable momentum with regard to open science among emotion researchers. This momentum may be leveraged to achieve a more robust emotion science.

scholarly journals Effects of intense assessment on statistical power in randomized controlled trials: Informed simulation study on depression

2019 ◽  
Author(s):  
Raphael Schuster ◽  
Manuela Schreyer ◽  
Tim Kaiser ◽  
Thomas Berger ◽  
Jan Philipp Klein ◽  
...  
Keyword(s):  
Randomized Controlled Trials ◽  
Simulation Study ◽  
Ecological Momentary Assessment ◽  
Statistical Power ◽  
Controlled Trials ◽  
Data Sets ◽  
Disease Symptoms ◽  
Randomized Controlled ◽  
Ecological Momentary ◽  
Momentary Assessment

Clinical trials are mainly based on single point assessments of psychopathology. At the same time, automatized repeated assessments based on short scales are an increasing practice to account for daily fluctuations in disease symptoms (e.g. ecological momentary assessment, or time series-based analyses). This study investigated the impact of Intense Pre-Post-Assessment (IPA) on statistical power in randomized controlled trials (RCTs).A simulation study, based on three scenarios and several empirical data sets, estimated the expected power gains of two- or fivefold pre-post-measurements of fluctuating disease symptoms. For each condition, patient data sets of various effect sizes were generated, and AN(C)OVAs were applied to the sample size of interest (N=50 – N=200).Power increases ranged from 6% to 92%, with higher gains in more underpowered scenarios. ANCOVA with baseline as covariate profited from a more precise estimation of the baseline covariate, resulting in additional gains in statistical power. Ecological momentary assessment-like data sources resulted in highest absolute statistical power and outperformed traditional point assessments if fivefold IPA was applied. For example, ANCOVA of automatized PHQ-9 questionnaire data resulted in absolute power of 55 (for N=200 and d=0.3). Fivefold IPA, however, resulted in power of 88.9 to detect a similar effect.IPA integrates short EMA-based assessments into RCT-based research designs. Sensitivity and efficiency of current RCTs could be improved by implementing a low number of automatized repeated assessments. Therefore, the merits of the suggested approach should be tested across various areas of clinical research (e.g. in neuroscience, or drug and psychotherapy research).

scholarly journals Ambulatory Assessment in Psychopathology Research: A Review of Recommended Reporting Guidelines and Current Practices

2019 ◽  
Author(s):  
Timothy J Trull
Keyword(s):  
Best Practices ◽  
Sample Selection ◽  
Ambulatory Assessment ◽  
Methodological Issues ◽  
Psychological Science ◽  
Psychological Medicine ◽  
Ecological Momentary ◽  
Open Question ◽  
Momentary Assessment ◽  
Experience Sampling Methods

The use of ambulatory assessment (AA; Trull & Ebner-Priemer, 2013) in psychopathology research, whichincludes experience-sampling methods (ESM) as well as ecological momentary assessment (EMA), hasincreased dramatically over the last several decades. Previously, methodological and reporting guidelineshave been presented to outline best practices and provide input on methodological issues and decisionsthat are faced when planning and conducting AA studies (e.g., Bolger & Laurenceau, 2013; Mehl & Conner,2012; Stone & Shiffman, 2002). However, despite the publication of these important resources andguidelines, it remains an open question as to how much uniformity or consistency is evident in the designand reporting of AA studies of psychopathology. To address this, we review the reported practices ofpublished studies using AA in major psychopathology journals (Journal of Abnormal Psychology,Psychological Medicine, Clinical Psychological Science) over the last 7 years (2012-2018). Our reviewhighlights: (1) sample selection and size; (2) sampling design; (3) selection and reporting of measures; (4)devices used and software; (5) compliance; (6) participant training, monitoring and remuneration; and (7)data management and analysis. We conclude with recommendations for reporting the features of futureAA studies in psychopathology.

The Importance of Electronic Diary Studies for Occupational Therapy Research

2006 ◽  
Vol 26 (2) ◽  
pp. 80-83 ◽  
Author(s):  
Liisa Holsti ◽  
Ronald G. Barr
Keyword(s):  
Occupational Therapy ◽  
Occupational Performance ◽  
Electronic Diary ◽  
Diary Research ◽  
Diary Studies ◽  
Health Related ◽  
Assessment Techniques ◽  
Pediatric Populations ◽  
Ecological Momentary ◽  
Momentary Assessment

Diary research methods are being used to assess many health-related conditions in both adult and pediatric populations. However, electronic diary data collection, ecological momentary assessment techniques, and multilevel statistical modeling are little used methodologies in occupational therapy research. It is important to add these methodologies to occupational therapy research; they offer prospective, flexible, and timely methods of evaluating the effects of conditions and diseases on occupational performance in many populations and across a variety of settings.

Sign in / Sign up

Export Citation Format

Share Document