Software for designing rigorous and replicable preclinical research: The Experimental Design Accelerator

Within preclinical research, attention has focused on experimental design and how current practices can lead to poor reproducibility. There are numerous decision points when designing experiments. Ethically, when working with animals we need to conduct a harm–benefit analysis to ensure the animal use is justified for the scientific gain. Experiments should be robust, not use more or fewer animals than necessary, and truly add to the knowledge base of science. Using case studies to explore these decision points, we consider how individual experiments can be designed in several different ways. We use the Experimental Design Assistant (EDA) graphical summary of each experiment to visualise the design differences and then consider the strengths and weaknesses of each design. Through this format, we explore key and topical experimental design issues such as pseudo-replication, blocking, covariates, sex bias, inference space, standardisation fallacy and factorial designs. There are numerous articles discussing these critical issues in the literature, but here we bring together these topics and explore them using real-world examples allowing the implications of the choice of design to be considered. Fundamentally, there is no perfect experiment; choices must be made which will have an impact on the conclusions that can be drawn. We need to understand the limitations of an experiment’s design and when we report the experiments, we need to share the caveats that inherently exist.

Download Full-text

Experimental replications in animal trials

Laboratory Animals ◽

10.1177/0023677220907617 ◽

2020 ◽

pp. 002367722090761 ◽

Cited By ~ 1

Author(s):

Florian Frommlet ◽

Georg Heinze

Keyword(s):

Statistical Analysis ◽

Experimental Design ◽

Animal Models ◽

Study Design ◽

Recent Discussion ◽

Preclinical Research ◽

Animal Trials ◽

Scientific Results

The recent discussion on the reproducibility of scientific results is particularly relevant for preclinical research with animal models. Within certain areas of preclinical research, there exists the tradition of repeating an experiment at least twice to demonstrate replicability. If the results of the first two experiments do not agree, then the experiment might be repeated a third time. Sometimes data of one representative experiment are shown; sometimes data from different experiments are pooled. However, there are hardly any guidelines about how to plan for such an experimental design or how to report the results obtained. This article provides a thorough statistical analysis of pre-planned experimental replications as they are currently often applied in practice and gives some recommendations about how to improve on study design and statistical analysis.

Download Full-text

Valid statistical approaches for clustered data: A Monte Carlo simulation study

10.1101/2020.11.27.400945 ◽

2020 ◽

Author(s):

Kristen A. McLaurin ◽

Amanda J. Fairchild ◽

Dexin Shi ◽

Rosemarie M. Booze ◽

Charles F. Mactutus

Keyword(s):

Monte Carlo Simulation ◽

Monte Carlo ◽

Experimental Design ◽

Simulation Study ◽

Fixed Effects ◽

Interactive Effects ◽

Multilevel Data ◽

Preclinical Research ◽

Culture Dish ◽

Monte Carlo Simulation Study

AbstractThe translation of preclinical studies to human applications is associated with a high failure rate, which may be exacerbated by limited training in experimental design and statistical analysis. Nested experimental designs, which occur when data have a multilevel structure (e.g., in vitro: cells within a culture dish; in vivo: rats within a litter), often violate the independent observation assumption underlying many traditional statistical techniques. Although previous studies have empirically evaluated the analytic challenges associated with multilevel data, existing work has not focused on key parameters and design components typically observed in preclinical research. To address this knowledge gap, a Monte Carlo simulation study was conducted to systematically assess the effects of inappropriately modeling multilevel data via a fixed effects ANOVA in studies with sparse observations, no between group comparison within a single cluster, and interactive effects. Simulation results revealed a dramatic increase in the probability of type 1 error and relative bias of the standard error as the number of level-1 (e.g., cells; rats) units per cell increased in the fixed effects ANOVA; these effects were largely attenuated when the nesting was appropriately accounted for via a random effects ANOVA. Thus, failure to account for a nested experimental design may lead to reproducibility challenges and inaccurate conclusions. Appropriately accounting for multilevel data, however, may enhance statistical reliability, thereby leading to improvements in translatability. Valid analytic strategies are provided for a variety of design scenarios.

Download Full-text

Translational Intracerebral Hemorrhage Research: Has Current Neuroprotection Research ARRIVEd at a Standard for Experimental Design and Reporting?

Translational Stroke Research ◽

10.1007/s12975-020-00824-x ◽

2020 ◽

Vol 11 (6) ◽

pp. 1203-1213 ◽

Cited By ~ 2

Author(s):

Lane J. Liddle ◽

Shivani Ralhan ◽

Daniel L. Ward ◽

Frederick Colbourne

Keyword(s):

Experimental Design ◽

Intracerebral Hemorrhage ◽

Animal Studies ◽

Effect Sizes ◽

Healthy Male ◽

Preclinical Research ◽

Exclusion Criteria ◽

Current Review ◽

Pubmed Database ◽

Design Characteristics

Abstract One major aim of preclinical intracerebral hemorrhage (ICH) research is to develop and test potential neuroprotectants. Published guidelines for experimental design and reporting stress the importance of clearly and completely reporting results and methodological details to ensure reproducibility and maximize information availability. The current review has two objectives: first, to characterize current ICH neuroprotection research and, second, to analyze aspects of translational design in preclinical ICH studies. Translational design is the adoption and reporting of experimental design characteristics that are thought to be clinically relevant and critical to reproducibility in animal studies (e.g., conducting and reporting experiments according to the STAIR and ARRIVE guidelines, respectively). Given that ICH has no current neuroprotective treatments and an ongoing reproducibility crisis in preclinical research, translational design should be considered by investigators. We conducted a systematic review of ICH research from 2015 to 2019 using the PubMed database. Our search returned 281 published manuscripts studying putative neuroprotectants in animal models. Contemporary ICH research predominantly uses young, healthy male rodents. The collagenase model is the most commonly used. Reporting of group sizes, blinding, and randomization are almost unanimous, but group size calculations, mortality and exclusion criteria, and animal model characteristics are infrequently reported. Overall, current ICH neuroprotection research somewhat aligns with experimental design and reporting guidelines. However, there are areas for improvement. Because failure to consider translational design is associated with inflation of effect sizes (and possibly hindered reproducibility), we suggest that researchers, editors, and publishers collaboratively consider enhanced adherence to published guidelines.

Download Full-text

Transparency in the reporting of in vivo pre-clinical pain research: The relevance and implications of the ARRIVE (Animal Research: Reporting In Vivo Experiments) guidelines

Scandinavian Journal of Pain ◽

10.1016/j.sjpain.2013.02.002 ◽

2013 ◽

Vol 4 (2) ◽

pp. 58-62 ◽

Cited By ~ 25

Author(s):

Andrew S.C. Rice ◽

Rosemary Morland ◽

Wenlong Huang ◽

Gillian L. Currie ◽

Emily S. Sena ◽

...

Keyword(s):

Experimental Design ◽

Meta Analysis ◽

Bias Reduction ◽

Animal Research ◽

Preclinical Research ◽

Pain Research ◽

Research Reporting ◽

In Vivo Experiments ◽

New Findings

AbstractClear reporting of research is crucial to the scientific process. Poorly designed and reported studies are damaging not only to the efforts of individual researchers, but also to science as a whole. Standardised reporting methods, such as those already established for reporting randomised clinical trials, have led to improved study design and facilitated the processes of clinical systematic review and meta-analysis.Such standards were lacking in the pre-clinical field until the development of the ARRIVE (Animal Research: Reporting In Vivo Experiments) guidelines. These were prompted following a survey which highlighted a widespread lack of robust and consistent reporting of pre-clinical in vivo research, with reports frequently omitting basic information required for study replication and quality assessment.The resulting twenty item checklist in ARRIVE covers all aspects of experimental design with particular emphasis on bias reduction and methodological transparency. Influential publishers and research funders have already adopted ARRIVE. Further dissemination and acknowledgement of the importance of these guidelines is vital to their widespread implementation.Conclusions and implicationsWide implementation of the ARRIVE guidelines for reporting of in vivo preclinical research, especially pain research, are essential for a much needed increased transparency and quality in publishing such research. ARRIVE will also positively influence improvements in experimental design and quality, assist the conduct of accurate replication studies of important new findings and facilitate meta-analyses of preclinical research.

Download Full-text

Identifying suboptimalities with factorial model comparison

Behavioral and Brain Sciences ◽

10.1017/s0140525x18001541 ◽

2018 ◽

Vol 41 ◽

Cited By ~ 2

Author(s):

Wei Ji Ma

Keyword(s):

Experimental Design ◽

Model Comparison ◽

Process Models ◽

Multiple Forms ◽

Factorial Experimental Design ◽

Factorial Model ◽

Multiple Components ◽

The Many

AbstractGiven the many types of suboptimality in perception, I ask how one should test for multiple forms of suboptimality at the same time – or, more generally, how one should compare process models that can differ in any or all of the multiple components. In analogy to factorial experimental design, I advocate for factorial model comparison.

Download Full-text

Combining integrated systems-biology approaches with intervention-based experimental design provides a higher-resolution path forward for microbiome research

Behavioral and Brain Sciences ◽

10.1017/s0140525x18002911 ◽

2019 ◽

Vol 42 ◽

Author(s):

J. Alfredo Blakeley-Ruiz ◽

Carlee S. McClintock ◽

Ralph Lydic ◽

Helen A. Baghdoyan ◽

James J. Choo ◽

...

Keyword(s):

Systems Biology ◽

High Resolution ◽

Experimental Design ◽

Integrated Systems ◽

Microbiome Research ◽

Constructive Criticism

Abstract The Hooks et al. review of microbiota-gut-brain (MGB) literature provides a constructive criticism of the general approaches encompassing MGB research. This commentary extends their review by: (a) highlighting capabilities of advanced systems-biology “-omics” techniques for microbiome research and (b) recommending that combining these high-resolution techniques with intervention-based experimental design may be the path forward for future MGB research.

Download Full-text

Statistics and parallaxes

International Astronomical Union Colloquium ◽

10.1017/s0252921100073875 ◽

1978 ◽

Vol 48 ◽

pp. 7-29

Author(s):

T. E. Lutz

Keyword(s):

Experimental Design ◽

Statistical Methods ◽

Review Paper ◽

Systematic Errors ◽

Past Experience ◽

Random Errors ◽

Trigonometric Parallaxes ◽

Systematic And Random Errors

This review paper deals with the use of statistical methods to evaluate systematic and random errors associated with trigonometric parallaxes. First, systematic errors which arise when using trigonometric parallaxes to calibrate luminosity systems are discussed. Next, determination of the external errors of parallax measurement are reviewed. Observatory corrections are discussed. Schilt’s point, that as the causes of these systematic differences between observatories are not known the computed corrections can not be applied appropriately, is emphasized. However, modern parallax work is sufficiently accurate that it is necessary to determine observatory corrections if full use is to be made of the potential precision of the data. To this end, it is suggested that a prior experimental design is required. Past experience has shown that accidental overlap of observing programs will not suffice to determine observatory corrections which are meaningful.

Download Full-text

Using Video Modeling Intervention and Speech Generating Devices To Teach Requesting Behaviors to Persons With Autism

Perspectives on Augmentative and Alternative Communication ◽

10.1044/aac20.4.109 ◽

2011 ◽

Vol 20 (4) ◽

pp. 109-113

Author(s):

Karen Copple ◽

Rajinder Koul ◽

Devender Banda ◽

Ellen Frye

Keyword(s):

Developmental Disabilities ◽

Experimental Design ◽

Communication Skills ◽

Observational Learning ◽

Video Modeling ◽

Single Subject ◽

Design Studies ◽

Communicative Behaviors ◽

Speech Generating Device ◽

Treatment Package

Abstract One of the instructional techniques reported in the literature to teach communication skills to persons with autism is video modeling (VM). VM is a form of observational learning that involves watching and imitating the desired target behavior(s) exhibited by the person on the videotape. VM has been used to teach a variety of social and communicative behaviors to persons with developmental disabilities such as autism. In this paper, we describe the VM technique and summarize the results of two single-subject experimental design studies that investigated the acquisition of spontaneous requesting skills using a speech generating device (SGD) by persons with autism following a VM intervention. The results of these two studies indicate that a VM treatment package that includes a SGD as one of its components can be effective in facilitating communication in individuals with autism who have little or no functional speech.

Download Full-text

Nepotism in the Hiring of Leaders

Swiss Journal of Psychology ◽

10.1024/1421-0185/a000143 ◽

2014 ◽

Vol 73 (4) ◽

pp. 243-248 ◽

Cited By ~ 2

Author(s):

Annick Darioly ◽

Ronald E. Riggio

Keyword(s):

Experimental Design ◽

Perceived Competence ◽

Negative Evaluation ◽

Leadership Position ◽

Family Ties ◽

Hiring Decision ◽

Job Application ◽

Career Progress

This study examines how applicants who are relatives of the company’s executives are perceived when they are being considered for a leadership position. In a 2 (Family ties: with vs. without) × 2 (Applicant qualifications: well-qualified vs. underqualified) experimental design, 165 Swiss employees read the applicant’s job application and evaluated the hiring decision, the perceived competence, and the perceived career progress of the target employee. This research showed that even a well-qualified potential employee received a more negative evaluation if the candidate had family ties to the company. Despite their negative evaluation of potential nepotistic hires, the participants nevertheless believed that family ties would boost the career progress of an underqualified applicant. Limitations and implications are discussed.

Download Full-text