scholarly journals Powerful sequential designs using Bayesian estimation: A power analysis tutorial using brms, the tidyverse, and furrr

2021 ◽  
Author(s):  
James Elsey
Keyword(s):  
Data Collection ◽  
Bayesian Estimation ◽  
Sequential Designs ◽  
Stopping Criteria ◽  
General Process ◽  
Misleading Evidence ◽  
Optional Stopping ◽  
Time Requirements ◽  
Sequential Analyses ◽  
Power Analyses

Producing compelling and trustworthy results relies upon performing well-powered studies with low rates of misleading evidence. Yet, resources are limited, and maximum sample sizes required to achieve acceptable power in typical fixed N designs may be disconcerting. ‘Sequential’, ‘optional stopping’, or ‘interim’ designs – in which results may be checked at interim periods and a decision made as to whether to continue data collection or not – provide one means by which researchers may be able to achieve high power and low false positive rates with less of a resource burden. Sequential analyses have received considerable attention from both frequentist and Bayesian hypothesis testing approaches, but fewer approachable resources are available for those wishing to use Bayesian estimation. In this tutorial, we cover a general process for performing power analyses of fixed and sequential designs using Bayesian estimation – simulating data, performing regressions in parallel to reduce time requirements, choosing different stopping criteria and data collection sequences, and calculating observed power and rates of misleading evidence. We conclude with a discussion of some limitations and possible extensions of the presented approach.

scholarly journals A Tutorial on Bayes Factor Design Analysis Using an Informed Prior

2017 ◽  
Author(s):  
Angelika Stefan ◽  
Quentin Frederik Gronau ◽  
Felix D. Schönbrodt ◽  
Eric-Jan Wagenmakers
Keyword(s):  
Bayes Factor ◽  
Design Analysis ◽  
Target Strength ◽  
Sequential Designs ◽  
Web Based ◽  
Strength Of Evidence ◽  
Misleading Evidence ◽  
Research Designs ◽  
Factor Design ◽  
User Friendly

Well-designed experiments are likely to yield compelling evidence with efficient sample sizes. Bayes Factor Design Analysis (BFDA) is a recently developed methodology that allows researchers to balance the informativeness and efficiency of their experiment (Schönbrodt & Wagenmakers, 2017). With BFDA, researchers can control the rate of misleading evidence but, in addition, they can plan for a target strength of evidence. BFDA can be applied to fixed-N and sequential designs. In this tutorial paper, we provide a tutorial-style introduction to BFDA and generalize the method to informed prior distributions. We also present a user-friendly web-based BFDA application that allows researchers to conduct BFDAs with ease. Two practical examples highlight how researchers can use a BFDA to plan for informative and efficient research designs.

scholarly journals Does preregistration improve the credibility of research findings?

2020 ◽  
Author(s):  
Mark Rubin
Keyword(s):  
Data Collection ◽  
Multiple Testing ◽  
Research Data ◽  
Public Access ◽  
Research Report ◽  
Selective Reporting ◽  
Optional Stopping ◽  
Research Findings ◽  
Analytical Approaches ◽  
Published Research

Preregistration entails researchers registering their planned research hypotheses, methods, and analyses in a time-stamped document before they undertake their data collection and analyses. This document is then made available with the published research report to allow readers to identify discrepancies between what the researchers originally planned to do and what they actually ended up doing. This historical transparency is supposed to facilitate judgments about the credibility of the research findings. The present article provides a critical review of 17 of the reasons behind this argument. The article covers issues such as HARKing, multiple testing, p-hacking, forking paths, optional stopping, researchers’ biases, selective reporting, test severity, publication bias, and replication rates. It is concluded that preregistration’s historical transparency does not facilitate judgments about the credibility of research findings when researchers provide contemporary transparency in the form of (a) clear rationales for current hypotheses and analytical approaches, (b) public access to research data, materials, and code, and (c) demonstrations of the robustness of research conclusions to alternative interpretations and analytical approaches.

scholarly journals Variability reduction in Manaus` beer process production

2019 ◽  
Vol 7 (11) ◽  
pp. 1418-1434
Author(s):  
Jonas Gomes da Silva ◽  
Amanda Ramos da Costa
Keyword(s):  
Data Collection ◽  
Production Process ◽  
Consumer Preference ◽  
Quality Standards ◽  
Time Production ◽  
Data Collection And Analysis ◽  
Before And After ◽  
Time Requirements ◽  
Beer Production ◽  
Manufacturing Time

Defined as the favorite drink to celebrate good times, beer has been making the drink market one of the most competitive in the Brazilian industry. Given this scenario, brewery industries need to maintain quality standards to gain consumer preference. In the company under study, located in Manaus Industrial Pole, it was found that in the beer production process, the brewhouse stage was not satisfying the brewery wort manufacturing time requirements, which is why it became the focus of this study. This paper aims to investigate and standardize a method to reduce the variability of beer time production in the brewhouse area. The data were collected from the monitoring of the wort production process, raising each time of the equipment of that stage, both before and after the application of the method. After data collection and analysis, it was concluded that it is possible to significantly reduce the variability in the wort production process by treating critical brewhouse equipment.

scholarly journals Bayes Factor Design Analysis: Planning for compelling evidence

2016 ◽  
Author(s):  
Felix D. Schönbrodt ◽  
Eric-Jan Wagenmakers
Keyword(s):  
Data Collection ◽  
Sample Size ◽  
Bayes Factor ◽  
Design Analysis ◽  
Maximum Efficiency ◽  
Misleading Evidence ◽  
Current State ◽  
Weak Evidence ◽  
Factor Design ◽  
Design Analyses

A sizeable literature exists on the use of frequentist power analysis in the null-hypothesis significance testing (NHST) paradigm to facilitate the design of informative experiments. In contrast, there is almost no literature that discusses the design of experiments when Bayes factors (BFs) are used as a measure of evidence. Here we explore Bayes Factor Design Analysis (BFDA) as a useful tool to design studies for maximum efficiency and informativeness. We elaborate on three possible BF designs, (a) a fixed-n design, (b) an open-ended Sequential Bayes Factor (SBF) design, where researchers can test after each participant and can stop data collection whenever there is strong evidence for either H1 or H0, and (c) a modified SBF design that defines a maximal sample size where data collection is stopped regardless of the current state of evidence. We demonstrate how the properties of each design (i.e., expected strength of evi- dence, expected sample size, expected probability of misleading evidence, expected probability of weak evidence) can be evaluated using Monte Carlo simulations and equip researchers with the necessary information to compute their own Bayesian design analyses.

scholarly journals Should I test more babies? Solutions for transparent data peeking

2018 ◽  
Author(s):  
Esther Schott ◽  
Mijke Rhemtulla ◽  
Krista Byers-Heinlein
Keyword(s):  
Data Collection ◽  
Preliminary Analysis ◽  
Significance Test ◽  
False Positives ◽  
Negative Consequences ◽  
Infant Research ◽  
Transparent Reporting ◽  
Optional Stopping ◽  
Data Points ◽  
Great Pressure

Research with infants is often slow and time-consuming, so infant researchers face great pressure to use the available participants in an efficient way. One strategy that researchers sometimes use to optimize efficiency is data peeking (or “optional stopping”), that is, doing a preliminary analysis (whether a formal significance test or informal eyeballing) of collected data. Data peeking helps researchers decide whether to abandon or tweak a study, decide that a sample is complete, or decide to continue adding data points. Unfortunately, data peeking can have negative consequences such as increased rates of false positives (wrongly concluding that an effect is present when it is not). We argue that, with simple corrections, the benefits of data peeking can be harnessed to use participants more efficiently. We review two corrections that can be transparently reported: one can be applied at the beginning of a study to lay out a plan for data peeking, and a second can be applied after data collection has already started. These corrections are easy to implement in the current framework of infancy research. The use of these corrections, together with transparent reporting, can increase the replicability of infant research.

scholarly journals Transparent manuscript reporting practices in behavioral medicine research: An audit of publications in 2018 and 2008

2020 ◽  
Author(s):  
Megan McVay ◽  
Kellie Cooper ◽  
Montserrat Carrera Seoane ◽  
Marissa Donahue ◽  
Laura Danielle Scherer
Keyword(s):  
Data Collection ◽  
Behavioral Medicine ◽  
Impact Factor ◽  
Qualitative Data ◽  
Medicine Research ◽  
Reporting Practices ◽  
Transparent Reporting ◽  
Secondary Outcomes ◽  
Secondary 51 ◽  
Power Analyses

Objectives: Concerns about rigor and replicability have led to reforms to increase science transparency. We aimed to document the use of transparent reporting practices in behavioral medicine journals in 2018 in order to inform future efforts to improve reporting practices. We also aimed to compare 2018 reporting practices to 2008. Methods: We examined a randomly selected portion of articles published in 2018 and 2008 by the four behavioral medicine journals with the highest impact factor. We excluded manuscripts that were reviews, presented qualitative data, or were purely descriptive. We coded whether articles were clear in their presentation of analyses as being primary or secondary; whether studies were registered/pre-registered; whether they used “exploratory” or a related term to describe analyses/aims; and whether they reported power analyses. Results: We identified and coded 162 manuscripts published in 2018 (87% observational and 12% experimental). Among 2018 studies, 16% were explicit in describing outcomes as primary or secondary, 51% appeared to be reports of secondary outcomes but did not use the term “secondary,” and 33% were unclear. Registration/pre-registration occurred in 14% of studies; 77.3% of registered/pre-registered studies did not report registration timing in relation to data collection, and 91% did not report which analyses were pre-registered. “Exploratory” or a related term was used to describe an aim or analysis in 31% of studies. Power analyses were reported in 8% of studies. Compared to studies from 2008 (n=120), studies published in 2008 were less likely to clearly report whether outcomes presented were primary or secondary and less likely to have been registered/pre-registered. Conclusions: Behavioral medicine stakeholders should consider strategies to increase clarity of reporting of key analysis details.

scholarly journals Does preregistration improve the credibility of research findings?

2020 ◽  
Author(s):  
Mark Rubin
Keyword(s):  
Data Collection ◽  
Multiple Testing ◽  
Research Data ◽  
Public Access ◽  
Research Report ◽  
Selective Reporting ◽  
Optional Stopping ◽  
Research Findings ◽  
Analytical Approaches ◽  
Published Research

Preregistration entails researchers registering their planned research hypotheses, methods, and analyses in a time-stamped document before they undertake their data collection and analyses. This document is then made available with the published research report to allow readers to identify discrepancies between what the researchers originally planned to do and what they actually ended up doing. This historical transparency is supposed to facilitate judgments about the credibility of the research findings. The present article provides a critical review of 17 of the reasons behind this argument. The article covers issues such as HARKing, multiple testing, p-hacking, forking paths, optional stopping, researchers’ biases, selective reporting, test severity, publication bias, and replication rates. It is concluded that preregistration’s historical transparency does not facilitate judgments about the credibility of research findings when researchers provide contemporary transparency in the form of (a) clear rationales for current hypotheses and analytical approaches, (b) public access to research data, materials, and code, and (c) demonstrations of the robustness of research conclusions to alternative interpretations and analytical approaches.

scholarly journals Performing High-Powered Studies Efficiently With Sequential Analyses

2017 ◽  
Author(s):  
Daniel Lakens
Keyword(s):  
Data Collection ◽  
Effect Size ◽  
Statistical Power ◽  
Large Scale ◽  
Analysis Data ◽  
Bayesian Approaches ◽  
Type 1 Error ◽  
Experimenter Bias ◽  
Practical Challenge ◽  
Sequential Analyses

Running studies with high statistical power, while effect size estimates in psychology are often inaccurate, leads to a practical challenge when designing an experiment. This challenge can be addressed by performing sequential analyses while the data collection is still in progress. At an interim analysis, data collection can be stopped whenever the results are convincing enough to conclude an effect is present, more data can be collected, or the study can be terminated whenever it is extremely unlikely the predicted effect will be observed if data collection would be continued. Such interim analyses can be performed while controlling the Type 1 error rate. Sequential analyses can greatly improve the efficiency with which data is collected. Additional flexibility is provided by adaptive designs where sample sizes are increased based on the observed effect size. The need for pre-registration, ways to prevent experimenter bias, and a comparison between Bayesian approaches and NHST are discussed. Sequential analyses, which are widely used in large scale medical trials, provide an efficient way to perform high-powered informative experiments. I hope this introduction will provide a practical primer that allows researchers to incorporate sequential analyses in their research.

scholarly journals A fully automated, transparent, reproducible, and blind protocol for sequential analyses

2018 ◽  
Author(s):  
Brice Beffara Bret ◽  
Amélie Beffara Bret ◽  
Ladislas Nalborczyk
Keyword(s):  
Data Analysis ◽  
Data Collection ◽  
Statistical Power ◽  
Open Science ◽  
Previous Experience ◽  
Major Obstacle ◽  
Sequential Data ◽  
Data Analyses ◽  
Science Framework ◽  
Sequential Analyses

Despite many cultural, methodological and technical improvements, one of the major obstacle to results reproducibility remains the pervasive low statistical power. In response to this problem, a lot of attention has recently been drawn to sequential analyses. This type of procedure has been shown to be more efficient (to require less observations and therefore less resources) than classical fixed-N procedures. However, these procedures are submitted to both intrapersonal and interpersonal biases during data collection and data analysis. In this tutorial, we explain how automation can be used to prevent these biases. We show how to synchronise open and free experiment software programs with the Open Science Framework and how to automate sequential data analyses in R. This tutorial is intended to researchers with beginner experience with R but no previous experience with sequential analyses is required.

Sign in / Sign up

Export Citation Format

Share Document