scholarly journals Falsifiability Is Not Optional

2017 ◽  
Author(s):  
Etienne P. LeBel ◽  
Derek Michael Berger ◽  
Lorne Campbell ◽  
Timothy Loving
Keyword(s):  
Statistical Power ◽  
A Priori ◽  
Open Science ◽  
Research Quality ◽  
Original Research ◽  
Fine Grained ◽  
Psychological Phenomena ◽  
Trade Offs ◽  
Cumulative Knowledge ◽  
Improved Model

Finkel, Eastwick, and Reis (2016; FER2016) argued the post-2011 methodological reform movement has focused narrowly on replicability, neglecting other essential goals of research. We agree multiple scientific goals are essential, but argue, however, a more fine-grained language, conceptualization, and approach to replication is needed to accomplish these goals. Replication is the general empirical mechanism for testing and falsifying theory. Sufficiently methodologically similar replications, also known as direct replications, test the basic existence of phenomena and ensure cumulative progress is possible a priori. In contrast, increasingly methodologically dissimilar replications, also known as conceptual replications, test the relevance of auxiliary hypotheses (e.g., manipulation and measurement issues, contextual factors) required to productively investigate validity and generalizability. Without prioritizing replicability, a field is not empirically falsifiable. We also disagree with FER2016’s position that “bigger samples are generally better, but … that very large samples could have the downside of commandeering resources that would have been better invested in other studies” (abstract). We identify problematic assumptions involved in FER2016’s modifications of our original research-economic model, and present an improved model that quantifies when (and whether) it is reasonable to worry that increasing statistical power will engender potential trade-offs. Sufficiently-powering studies (i.e., >80%) maximizes both research efficiency and confidence in the literature (research quality). Given we are in agreement with FER2016 on all key open science points, we are eager to start seeing the accelerated rate of cumulative knowledge development of social psychological phenomena such a sufficiently transparent, powered, and falsifiable approach will generate.

scholarly journals Using simulation and resampling to improve the statistical power and reproducibility of psychological research

2019 ◽  
Author(s):  
Roger W. Strong ◽  
George Alvarez
Keyword(s):  
Statistical Power ◽  
Synthetic Data ◽  
Simulation Method ◽  
Psychological Research ◽  
Open Science ◽  
Original Research ◽  
Bootstrap Resampling ◽  
Questionable Research Practices ◽  
Pilot Data ◽  
Experimental Findings

The replication crisis has brought about an increased focus on improving the reproducibility of psychological research (Open Science Collaboration, 2015). Although some failed replications reflect false-positives in original research findings, many are likely the result of low statistical power, which can cause failed replications even when an effect is real, no questionable research practices are used, and an experiment’s methodology is repeated perfectly. The present paper describes a simulation method (bootstrap resampling) that can be used in combination with pilot data or synthetic data to produce highly powered experimental designs. Unlike other commonly used power analysis approaches (e.g., G*Power), bootstrap resampling can be adapted to any experimental design to account for various factors that influence statistical power, including sample size, number of trials per condition, and participant exclusion criteria. Ignoring some of these factors (e.g., by using G*Power) can overestimate the power of a study or replication, increasing the likelihood that your findings will not replicate. By demonstrating how these factors influence the consistency of experimental findings, this paper provides examples of how simulation can be used to improve statistical power and reproducibility. Further, we provide a MATLAB toolbox that can be used to implement these simulation-based methods on existing pilot data (https://harvard-visionlab.github.io/power-sim).

Replication of the Superstition and Performance Study by

2014 ◽  
Vol 45 (3) ◽  
pp. 239-245 ◽  
Author(s):  
Robert J. Calin-Jageman ◽  
Tracy L. Caldwell
Keyword(s):  
Task Difficulty ◽  
Statistical Power ◽  
Meta Analysis ◽  
A Priori ◽  
Significant Heterogeneity ◽  
Performance Study ◽  
Improve Performance ◽  
Research Designs ◽  
Series Of Experiments ◽  
And Performance

A recent series of experiments suggests that fostering superstitions can substantially improve performance on a variety of motor and cognitive tasks ( Damisch, Stoberock, & Mussweiler, 2010 ). We conducted two high-powered and precise replications of one of these experiments, examining if telling participants they had a lucky golf ball could improve their performance on a 10-shot golf task relative to controls. We found that the effect of superstition on performance is elusive: Participants told they had a lucky ball performed almost identically to controls. Our failure to replicate the target study was not due to lack of impact, lack of statistical power, differences in task difficulty, nor differences in participant belief in luck. A meta-analysis indicates significant heterogeneity in the effect of superstition on performance. This could be due to an unknown moderator, but no effect was observed among the studies with the strongest research designs (e.g., high power, a priori sampling plan).

Are Public Bureaucracies Supposed to Be High Reliability Organizations?

2020 ◽  
Vol 1 (1) ◽  
Author(s):  
Wolfgang Seibel
Keyword(s):  
Child Protection ◽  
Organization Theory ◽  
Goal Attainment ◽  
High Reliability ◽  
Human Action ◽  
High Reliability Organizations ◽  
Fine Grained ◽  
Trade Offs ◽  
Public Bureaucracy ◽  
Account Standard

This article addresses the question of to what extent conventional theories of high reliability organizations and normal accidents theory are applicable to public bureaucracy. Empirical evidence suggests precisely this. Relevant cases are, for instance, collapsing buildings and bridges due to insufficient supervision of engineering by the relevant authorities, infants dying at the hands of their own parents due to misperceptions and neglect on the part of child protection agencies, uninterrupted serial killings due to a lack of coordination among police services, or improper planning and risk assessment in the preparation of mass events such as soccer games or street parades. The basic argument is that conceptualizing distinct and differentiated causal mechanisms is useful for developing more fine-grained variants of both normal accident theory and high reliability organization theory that take into account standard pathologies of public bureaucracies and inevitable trade-offs connected to their political embeddedness in democratic and rule-of-law-based systems to which belong the tensions between responsiveness and responsibility and between goal attainment and system maintenance. This, the article argues, makes it possible to identify distinct points of intervention at which permissive conditions with the potential to trigger risk-generating human action can be neutralized while the threshold that separates risk-generating human action from actual disaster can be raised to a level that makes disastrous outcomes less probable.

scholarly journals Brainwide functional networks associated with anatomically- and functionally-defined hippocampal subfields using ultrahigh-resolution fMRI

2021 ◽  
Vol 11 (1) ◽  
Author(s):  
Wei-Tang Chang ◽  
Stephanie K. Langella ◽  
Yichuan Tang ◽  
Sahar Ahmad ◽  
Han Zhang ◽  
...  
Keyword(s):  
Resting State ◽  
Functional Organization ◽  
A Priori ◽  
Longitudinal Axis ◽  
Functional Networks ◽  
Resting State Functional Connectivity ◽  
Underlying Assumption ◽  
Isotropic Resolution ◽  
Hippocampal Subfields ◽  
Fine Grained

AbstractThe hippocampus is critical for learning and memory and may be separated into anatomically-defined hippocampal subfields (aHPSFs). Hippocampal functional networks, particularly during resting state, are generally analyzed using aHPSFs as seed regions, with the underlying assumption that the function within a subfield is homogeneous, yet heterogeneous between subfields. However, several prior studies have observed similar resting-state functional connectivity (FC) profiles between aHPSFs. Alternatively, data-driven approaches investigate hippocampal functional organization without a priori assumptions. However, insufficient spatial resolution may result in a number of caveats concerning the reliability of the results. Hence, we developed a functional Magnetic Resonance Imaging (fMRI) sequence on a 7 T MR scanner achieving 0.94 mm isotropic resolution with a TR of 2 s and brain-wide coverage to (1) investigate the functional organization within hippocampus at rest, and (2) compare the brain-wide FC associated with fine-grained aHPSFs and functionally-defined hippocampal subfields (fHPSFs). This study showed that fHPSFs were arranged along the longitudinal axis that were not comparable to the lamellar structures of aHPSFs. For brain-wide FC, the fHPSFs rather than aHPSFs revealed that a number of fHPSFs connected specifically with some of the functional networks. Different functional networks also showed preferential connections with different portions of hippocampal subfields.

A Call to Develop Standards for Those Delivering ‘Research Practice’ Training

2021 ◽  
Vol 03 ◽  
Author(s):  
Danny Kingsley
Keyword(s):  
Information Science ◽  
Task Force ◽  
Core Competencies ◽  
Scholarly Communication ◽  
Academic Research ◽  
Open Science ◽  
Early Career ◽  
Research Quality ◽  
Research Practice ◽  
The North

The nature of the research endeavour is changing rapidly and requires a wide set of skills beyond the research focus. The delivery of aspects of researcher training ‘beyond the bench’ is met by different sections of an institution, including the research office, the media office and the library. In Australia researcher training in open access, research data management and other aspects of open science is primarily offered by librarians. But what training do librarians receive in scholarly communication within their librarianship degrees? For a degree to be offered in librarianship and information science, it must be accredited by the Australian Library and Information Association (ALIA), with a curriculum that is based on ALIA’s lists of skills and attributes. However, these lists do not contain any reference to key open research terms and are almost mutually exclusive with core competencies in scholarly communication as identified by the North American Serials Interest Group and an international Joint Task Force. Over the past decade teaching by academics in universities has been professionalised with courses and qualifications. Those responsible for researcher training within universities and the material that is being offered should also meet an agreed accreditation. This paper is arguing that there is a clear need to develop parallel standards around ‘research practice’ training for PhD students and Early Career Researchers, and those delivering this training should be able to demonstrate their skills against these standards. Models to begin developing accreditation standards are starting to emerge, with the recent launch of the Centre for Academic Research Quality and Improvement in the UK. There are multiple organisations, both grassroots and long-established that would be able to contribute to this project.

Adult judgments and fine-grained analysis of infant facial expressions: Testing the validity of a priori coding formulas.

1992 ◽  
Vol 28 (6) ◽  
pp. 1115-1131 ◽  
Author(s):  
Harriet Oster ◽  
Douglas Hegley ◽  
Linda Nagel
Keyword(s):  
Facial Expressions ◽  
A Priori ◽  
Fine Grained

scholarly journals Statistical Primer for Athletic Trainers: Understanding the Role of Statistical Power in Comparative Athletic Training Research

2018 ◽  
Vol 53 (7) ◽  
pp. 716-719
Author(s):  
Monica R. Lininger ◽  
Bryan L. Riemann
Keyword(s):  
Sample Size ◽  
Athletic Training ◽  
Treatment Effect ◽  
Statistical Power ◽  
Athletic Trainers ◽  
A Priori ◽  
Significant Treatment ◽  
Training Research ◽  
Α Level

Objective: To describe the concept of statistical power as related to comparative interventions and how various factors, including sample size, affect statistical power.Background: Having a sufficiently sized sample for a study is necessary for an investigation to demonstrate that an effective treatment is statistically superior. Many researchers fail to conduct and report a priori sample-size estimates, which then makes it difficult to interpret nonsignificant results and causes the clinician to question the planning of the research design.Description: Statistical power is the probability of statistically detecting a treatment effect when one truly exists. The α level, a measure of differences between groups, the variability of the data, and the sample size all affect statistical power.Recommendations: Authors should conduct and provide the results of a priori sample-size estimations in the literature. This will assist clinicians in determining whether the lack of a statistically significant treatment effect is due to an underpowered study or to a treatment's actually having no effect.

scholarly journals Integrating open science in the teaching of cognitive research methods

2020 ◽  
Author(s):  
Ralph S. Redden ◽  
Colin R McCormick
Keyword(s):  
Research Methods ◽  
Undergraduate Research ◽  
Scientific Practice ◽  
A Priori ◽  
Daily Practice ◽  
Open Science ◽  
Methods Course ◽  
Replication Experiment ◽  
Successful Replication ◽  
Oral Presentations

Openness, transparency, and reproducibility are widely accepted as fundamental aspects of scientific practice. However, a growing body of evidence suggests that these features are not readily adopted in the daily practice of most scientists. The Centre for Open Science has been championing efforts for systemic change in the scientific process, with newly adopted practices such as preregistration and open sharing of data and experimental materials. In an effort to inculcate these practices early in training, we have integrated several key components of open science practice into an undergraduate research methods course in the cognitive sciences. Students were divided into four research teams, each with the goal of carrying out a replication experiment related to the study of attention; specifically, temporal orienting, alertness, prior entry, and the attentional blink. Teams completed a preregistration exercise, and importantly, were encouraged to consider a priori the criteria for a successful replication. They were also required to collect and analyze data, prepare manuscripts, and disseminate their findings in poster symposia and oral presentations. All project materials can be found at https://osf.io/gxkfq/. Critical appraisal of the goals and implementation of the course are discussed.

scholarly journals An iterative and automated computational pipeline for untargeted strain-level identification using MS/MS spectra from pathogenic samples

2019 ◽  
Author(s):  
Mathias Kuhring ◽  
Joerg Doellinger ◽  
Andreas Nitsche ◽  
Thilo Muth ◽  
Bernhard Y. Renard
Keyword(s):  
Statistical Power ◽  
Sequence Data ◽  
A Priori ◽  
Search Space ◽  
Strain Level ◽  
Reference Sequence ◽  
Viral Origin ◽  
Identification Of Species ◽  
Taxonomic Assignments

AbstractUntargeted accurate strain-level classification of a priori unidentified organisms using tandem mass spectrometry is a challenging task. Reference databases often lack taxonomic depth, limiting peptide assignments to the species level. However, the extension with detailed strain information increases runtime and decreases statistical power. In addition, larger databases contain a higher number of similar proteomes.We present TaxIt, an iterative workflow to address the increasing search space required for MS/MS-based strain-level classification of samples with unknown taxonomic origin. TaxIt first applies reference sequence data for initial identification of species candidates, followed by automated acquisition of relevant strain sequences for low level classification. Furthermore, proteome similarities resulting in ambiguous taxonomic assignments are addressed with an abundance weighting strategy to improve candidate confidence.We apply our iterative workflow on several samples of bacterial and viral origin. In comparison to non-iterative approaches using unique peptides or advanced abundance correction, TaxIt identifies microbial strains correctly in all examples presented (with one tie), thereby demonstrating the potential for untargeted and deeper taxonomic classification. TaxIt makes extensive use of public, unrestricted and continuously growing sequence resources such as the NCBI databases and is available under open-source license at https://gitlab.com/rki_bioinformatics.

scholarly journals A Multi-faceted Mess: A Review of Statistical Power Analysis in Psychology Journal Articles

2019 ◽  
Author(s):  
Rob Cribbie ◽  
Nataly Beribisky ◽  
Udi Alter
Keyword(s):  
Sample Size ◽  
Effect Size ◽  
Power Analysis ◽  
Statistical Power ◽  
Type I Error ◽  
A Priori ◽  
Type I ◽  
Specific Level ◽  
Maximum Sample Size ◽  
Power Analyses

Many bodies recommend that a sample planning procedure, such as traditional NHST a priori power analysis, is conducted during the planning stages of a study. Power analysis allows the researcher to estimate how many participants are required in order to detect a minimally meaningful effect size at a specific level of power and Type I error rate. However, there are several drawbacks to the procedure that render it “a mess.” Specifically, the identification of the minimally meaningful effect size is often difficult but unavoidable for conducting the procedure properly, the procedure is not precision oriented, and does not guide the researcher to collect as many participants as feasibly possible. In this study, we explore how these three theoretical issues are reflected in applied psychological research in order to better understand whether these issues are concerns in practice. To investigate how power analysis is currently used, this study reviewed the reporting of 443 power analyses in high impact psychology journals in 2016 and 2017. It was found that researchers rarely use the minimally meaningful effect size as a rationale for the chosen effect in a power analysis. Further, precision-based approaches and collecting the maximum sample size feasible are almost never used in tandem with power analyses. In light of these findings, we offer that researchers should focus on tools beyond traditional power analysis when sample planning, such as collecting the maximum sample size feasible.

Sign in / Sign up

Export Citation Format

Share Document