scholarly journals Questionable Research Practices in Single-Case Experimental Designs: Examples and Possible Solutions

2021 ◽  
Author(s):  
Matt Tincani ◽  
Jason C Travers
Keyword(s):  
Single Case ◽  
The Body ◽  
Research Practices ◽  
Questionable Research Practices ◽  
Participant Selection ◽  
Independent Variable ◽  
Meta Analyses ◽  
Research Choices ◽  
File Drawer ◽  
File Drawer Effect

Questionable research practices (QRPs) are a variety of research choices that introduce bias into the body of scientific literature. Researchers have documented widespread presence of QRPs across disciplines and promoted practices aimed at preventing them. More recently, Single-Case Experimental Design (SCED) researchers have explored how QRPs could manifest in SCED research. In the chapter, we describe QRPs in participant selection, independent variable selection, procedural fidelity documentation, graphical depictions of behavior, and effect size measures and statistics. We also discuss QRPs in relation to the file drawer effect, publication bias, and meta-analyses of SCED research. We provide recommendations for researchers and the research community to promote practices for preventing QRPs in SCED.

scholarly journals Adding Standards and Procedures to the Prevention Services Clearinghouse Handbook on the Transparency, Openness, and Reproducibility of Research on Programs and Services to Enhance Support for Children and Families and Prevent Foster Care Placements

2021 ◽  
Author(s):  
Sean Grant ◽  
Evan Mayo-Wilson ◽  
Lauren Supplee
Keyword(s):  
Systematic Reviews ◽  
Reproducible Research ◽  
Research Practices ◽  
Questionable Research Practices ◽  
Assessment Development ◽  
Prevention Services ◽  
Department Of Health ◽  
Policies And Procedures ◽  
Bodies Of Evidence ◽  
Meta Analyses

The credibility of Prevention Services Clearinghouse designations of programs and services as “promising,” “supported,” and “well supported” are threatened by the prevalence of questionable research practices (e.g., selective non-reporting of results) in the bodies of evidence that the Clearinghouse reviews. Internationally accepted standards for reporting and interpreting the results of systematic reviews of evidence—including the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) and Grading of Recommendations Assessment, Development and Evaluation (GRADE)—recommend that reviews take steps to mitigate bias associated with these questionable research practices. Moreover, Department of Health and Human Services (HHS) policies require that contractors and grantees engage in transparent, open, and reproducible research. We propose that the Clearinghouse adopt standards to mitigate the effects of these questionable research practices, which would be consistent with international guidelines and with complementary HHS policies and procedures.

scholarly journals Assessing robustness against potential publication bias in coordinate based fMRI meta-analyses using the Fail-Safe N

2017 ◽  
Author(s):  
Freya Acar ◽  
Ruth Seurinck ◽  
Simon B. Eickhoff ◽  
Beatrijs Moerkerke
Keyword(s):  
Publication Bias ◽  
Meta Analysis ◽  
Popular Approach ◽  
Analytical Results ◽  
Research Findings ◽  
Meta Analyses ◽  
Fail Safe ◽  
File Drawer ◽  
Cluster Level ◽  
File Drawer Effect

AbstractThe importance of integrating research findings is incontrovertible and coordinate based meta-analyses have become a popular approach to combine results of fMRI studies when only peaks of activation are reported. Similar to classical meta-analyses, coordinate based meta-analyses may be subject to different forms of publication bias which impacts results and possibly invalidates findings. We develop a tool that assesses the robustness to potential publication bias on cluster level. We investigate the possible influence of the file-drawer effect, where studies that do not report certain results fail to get published, by determining the number of noise studies that can be added to an existing fMRI meta-analysis before the results are no longer statistically significant. In this paper we illustrate this tool through an example and test the effect of several parameters through extensive simulations. We provide an algorithm for which code is freely available to generate noise studies and enables users to determine the robustness of meta-analytical results.

scholarly journals Adding Procedures and Standards to the HomVEE Review Handbook on the Transparency, Openness, and Reproducibility of Research on Home Visiting Models

2021 ◽  
Author(s):  
Sean Grant ◽  
Evan Mayo-Wilson ◽  
Lauren Supplee
Keyword(s):  
Home Visiting ◽  
Research Practices ◽  
Questionable Research Practices ◽  
Assessment Development ◽  
Department Of Health ◽  
Policies And Procedures ◽  
Childhood Home ◽  
Meta Analyses ◽  
Delivery Models ◽  
Early Childhood Home Visiting

Questionable research practices threaten the credibility of HomVEE designations for “evidence-based early childhood home visiting service delivery models”. Internationally accepted standards for reporting and interpreting the results of reviews—including the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) and Grading of Recommendations Assessment, Development and Evaluation (GRADE)—recommend that reviews take steps to mitigate bias associated with selective non-reporting of results and other questionable research practices. Moreover, Department of Health and Human Services (HHS) policies require that contractors and grantees engage in transparent and open sciences practices. We propose that HomVEE adopt standards to mitigate the effects of questionable research practices, which would be consistent with international guidelines and with complementary HHS policies and procedures.

scholarly journals A defense of meta-analytic thinking in behavioral science

2018 ◽  
Author(s):  
Caleb Z. Marshall
Keyword(s):  
Behavioral Science ◽  
Type I ◽  
Analytic Thinking ◽  
Type I Errors ◽  
Research Practices ◽  
Questionable Research Practices ◽  
Meta Analyses

We discuss how questionable research practices in behavioral science (such as p-hacking) effect meta-analyses. Moreover, we argue that abandoning meta-analytic techniques is an overreaction to concerns of type I errors.

scholarly journals Estimating the prevalence of missing experiments in a neuroimaging meta-analysis

2017 ◽  
Author(s):  
Pantelis Samartsidis ◽  
Silvia Montagna ◽  
Angela R. Laird ◽  
Peter T. Fox ◽  
Timothy D. Johnson ◽  
...  
Keyword(s):  
Count Data ◽  
Meta Analysis ◽  
Human Connectome Project ◽  
Meta Analyses ◽  
File Drawer ◽  
Coordinate Data ◽  
File Drawer Effect ◽  
Modelling Approach ◽  
File Drawer Problem

AbstractCoordinate-based meta-analyses (CBMA) allow researchers to combine the results from multiple fMRI experiments with the goal of obtaining results that are more likely to generalise. However, the interpretation of CBMA findings can be impaired by the file drawer problem, a type of publications bias that refers to experiments that are carried out but are not published. Using foci per contrast count data from the BrainMap database, we propose a zero-truncated modelling approach that allows us to estimate the prevalence of non-significant experiments. We validate our method with simulations and real coordinate data generated from the Human Connectome Project. Application of our method to the data from BrainMap provides evidence for the existence of a file drawer effect, with the rate of missing experiments estimated as at least 6 per 100 reported.

How to Detect Publication Bias in Psychological Research

2019 ◽  
Vol 227 (4) ◽  
pp. 261-279 ◽  
Author(s):  
Frank Renkewitz ◽  
Melanie Keiner
Keyword(s):  
Publication Bias ◽  
Effect Size ◽  
Statistical Power ◽  
Type I Error ◽  
Psychological Research ◽  
Type I ◽  
True Effect Size ◽  
Questionable Research Practices ◽  
True Effect ◽  
Meta Analyses

Abstract. Publication biases and questionable research practices are assumed to be two of the main causes of low replication rates. Both of these problems lead to severely inflated effect size estimates in meta-analyses. Methodologists have proposed a number of statistical tools to detect such bias in meta-analytic results. We present an evaluation of the performance of six of these tools. To assess the Type I error rate and the statistical power of these methods, we simulated a large variety of literatures that differed with regard to true effect size, heterogeneity, number of available primary studies, and sample sizes of these primary studies; furthermore, simulated studies were subjected to different degrees of publication bias. Our results show that across all simulated conditions, no method consistently outperformed the others. Additionally, all methods performed poorly when true effect sizes were heterogeneous or primary studies had a small chance of being published, irrespective of their results. This suggests that in many actual meta-analyses in psychology, bias will remain undiscovered no matter which detection method is used.

Towards guaranteed data-integrity: A method of preventing Questionable Research Practices

2018 ◽  
Author(s):  
Dick Bierman ◽  
Jacob Jolij
Keyword(s):  
Real Time ◽  
Open Data ◽  
Data Integrity ◽  
Drop Out ◽  
Research Practices ◽  
Major Aspect ◽  
Questionable Research Practices ◽  
Blockchain Technology ◽  
Integrity System ◽  
Secure Server

We have tested the feasibility of a method to prevent the occurrence of so-called Questionable Research Practices (QRP). A part from embedded pre-registration the major aspect of the system is real-time uploading of data on a secure server. We outline the method, discuss the drop-out treatment and compare it to the Born-open data method, and report on our preliminary experiences. We also discuss the extension of the data-integrity system from secure server to use of blockchain technology.

Sign in / Sign up

Export Citation Format

Share Document