Recommendations in pre-registrations and internal review board proposals promote formal power analyses but do not increase sample size

In this preregistered study, we investigated whether the statistical power of a study is higher when researchers are asked to make a formal power analysis before collecting data. We compared the sample size descriptions from two sources: (i) a sample of pre-registrations created according to the guidelines for the Center for Open Science Preregistration Challenge (PCRs) and a sample of institutional review board (IRB) proposals from Tilburg School of Behavior and Social Sciences, which both include a recommendation to do a formal power analysis, and (ii) a sample of pre-registrations created according to the guidelines for Open Science Framework Standard Pre-Data Collection Registrations (SPRs) in which no guidance on sample size planning is given. We found that PCRs and IRBs (72%) more often included sample size decisions based on power analyses than the SPRs (45%). However, this did not result in larger planned sample sizes. The determined sample size of the PCRs and IRB proposals (Md = 90.50) was not higher than the determined sample size of the SPRs (Md = 126.00; W = 3389.5, p = 0.936). Typically, power analyses in the registrations were conducted with G*power, assuming a medium effect size, α = .05 and a power of .80. Only 20% of the power analyses contained enough information to fully reproduce the results and only 62% of these power analyses pertained to the main hypothesis test in the pre-registration. Therefore, we see ample room for improvements in the quality of the registrations and we offer several recommendations to do so.

Download Full-text

Is Vascularized Composite Allograft Transplantation Experimental or an Accepted Surgical Procedure: Results from a National Survey

Journal of Reconstructive Microsurgery ◽

10.1055/s-0039-1701029 ◽

2020 ◽

Vol 36 (04) ◽

pp. 276-280

Author(s):

Salih Colakoglu ◽

Seth Tebockhorst ◽

David Woodbridge Mathes

Keyword(s):

Clinical Investigation ◽

The United States ◽

Representative Sampling ◽

Electronic Survey ◽

Internal Review ◽

Vascularized Composite Allograft ◽

American Society ◽

Question Survey ◽

Internal Review Board

Abstract Background More than 85 patients have received over 100 hand/arm transplants and more than 35 patients have received full or partial face transplants at institutions around the world. Given over two decades of experience in the field and in the light of successful outcomes with up to 17 years follow up time, should we still consider vascularized composite allograft (VCA) as a research/clinical investigation? We present the results of a nationwide electronic survey whose intent was to gather institutional bias with regard to this question. Methods An 11 question survey that was developed by VCA advisory committee of American Society of Transplantation was sent to all identified Internal Review Board chairs or directors in the United States. Results We received a total of 54 responses (25.3%) to the survey. The majority (78%) of responses came from either the chairperson, director, or someone who is administratively responsible for an IRB. Conclusion Though certainly not an exhaustive investigation into each institution's preference, we present a representative sampling. The results of which favor VCA as an accepted clinical procedure given the appropriate setting. Further research is needed to fully ascertain practices at each individual institution.

Download Full-text

A Multi-faceted Mess: A Review of Statistical Power Analysis in Psychology Journal Articles

10.31234/osf.io/3bdfu ◽

2019 ◽

Cited By ~ 2

Author(s):

Rob Cribbie ◽

Nataly Beribisky ◽

Udi Alter

Keyword(s):

Sample Size ◽

Effect Size ◽

Power Analysis ◽

Statistical Power ◽

Type I Error ◽

A Priori ◽

Type I ◽

Specific Level ◽

Maximum Sample Size ◽

Power Analyses

Many bodies recommend that a sample planning procedure, such as traditional NHST a priori power analysis, is conducted during the planning stages of a study. Power analysis allows the researcher to estimate how many participants are required in order to detect a minimally meaningful effect size at a specific level of power and Type I error rate. However, there are several drawbacks to the procedure that render it “a mess.” Specifically, the identification of the minimally meaningful effect size is often difficult but unavoidable for conducting the procedure properly, the procedure is not precision oriented, and does not guide the researcher to collect as many participants as feasibly possible. In this study, we explore how these three theoretical issues are reflected in applied psychological research in order to better understand whether these issues are concerns in practice. To investigate how power analysis is currently used, this study reviewed the reporting of 443 power analyses in high impact psychology journals in 2016 and 2017. It was found that researchers rarely use the minimally meaningful effect size as a rationale for the chosen effect in a power analysis. Further, precision-based approaches and collecting the maximum sample size feasible are almost never used in tandem with power analyses. In light of these findings, we offer that researchers should focus on tools beyond traditional power analysis when sample planning, such as collecting the maximum sample size feasible.

Download Full-text

Statistical Power in Psychiatric Research

Australian & New Zealand Journal of Psychiatry ◽

10.3109/00048678609161331 ◽

1986 ◽

Vol 20 (2) ◽

pp. 189-200 ◽

Cited By ~ 19

Author(s):

Kevin D. Bird ◽

Wayne Hall

Keyword(s):

Sample Size ◽

Error Rate ◽

Statistical Power ◽

Error Rates ◽

Psychiatric Research ◽

Type 1 Error ◽

Type 2 Error ◽

Power Analyses

Statistical power is neglected in much psychiatric research, with the consequence that many studies do not provide a reasonable chance of detecting differences between groups if they exist in the population. This paper attempts to improve current practice by providing an introduction to the essential quantities required for performing a power analysis (sample size, effect size, type 1 and type 2 error rates). We provide simplified tables for estimating the sample size required to detect a specified size of effect with a type 1 error rate of α and a type 2 error rate of β, and for estimating the power provided by a given sample size for detecting a specified size of effect with a type 1 error rate of α. We show how to modify these tables to perform power analyses for multiple comparisons in univariate and some multivariate designs. Power analyses for each of these types of design are illustrated by examples.

Download Full-text

When power analyses based on pilot data are biased: Inaccurate effect size estimators and follow-up bias

10.31234/osf.io/b7z4q ◽

2017 ◽

Cited By ~ 2

Author(s):

Daniel Lakens ◽

Casper J Albers

Keyword(s):

Sample Size ◽

Effect Size ◽

A Priori ◽

Main Study ◽

Pilot Studies ◽

Pilot Data ◽

Size Estimate ◽

Size Estimates ◽

Power Analyses

When designing a study, the planned sample size is often based on power analyses. One way to choose an effect size for power analyses is by relying on pilot data. A-priori power analyses are only accurate when the effect size estimate is accurate. In this paper we highlight two sources of bias when performing a-priori power analyses for between-subject designs based on pilot data. First, we examine how the choice of the effect size index (η2, ω2 and ε2) affects the sample size and power of the main study. Based on our observations, we recommend against the use of η2 in a-priori power analyses. Second, we examine how the maximum sample size researchers are willing to collect in a main study (e.g. due to time or financial constraints) leads to overestimated effect size estimates in the studies that are performed. Determining the required sample size exclusively based on the effect size estimates from pilot data, and following up on pilot studies only when the sample size estimate for the main study is considered feasible, creates what we term follow-up bias. We explain how follow-up bias leads to underpowered main studies.Our simulations show that designing main studies based on effect sizes estimated from small pilot studies does not yield desired levels of power due to accuracy bias and follow-up bias, even when publication bias is not an issue. We urge researchers to consider alternative approaches to determining the sample size of their studies, and discuss several options.

Download Full-text

Spotlight

Community Resilience ◽

10.1093/oso/9780197559383.003.0020 ◽

2021 ◽

pp. 206-207

Keyword(s):

Decision Making ◽

New York ◽

New York City ◽

York City ◽

Representative Sample ◽

Digital Age ◽

Internal Review ◽

Using Data ◽

Interdisciplinary Study ◽

Internal Review Board

The HUMAN Project was initiated in 2014 by the Kavli Foundation in partnership with New York University’s Institute for the Interdisciplinary Study of Decision Making. 1 Its goal was to collect vast amounts of data from a representative sample of 10,000 New York City residents in 4,000 households over 20 years. Lacking both internal review board approval and sustainable funding, the HUMAN Project was suspended in 2018. Nonetheless, the ambitious scope of the study and what it revealed about the possibilities for collecting and using data in the digital age are intriguing. It is possible that this type of model could eventually be revived, perhaps with additional privacy protections built in....

Download Full-text

Internal Review Board

Encyclopedia of Public Health ◽

10.1007/978-1-4020-5614-7_1847 ◽

2008 ◽

pp. 802-802

Keyword(s):

Internal Review ◽

Internal Review Board

Download Full-text

Consistency in systematic sampling

Advances in Applied Probability ◽

10.2307/1428160 ◽

1996 ◽

Vol 28 (4) ◽

pp. 982-992 ◽

Cited By ~ 3

Author(s):

X. Gual Arnau ◽

L. M. Cruz-Orive

Keyword(s):

Sample Size ◽

Surface Area ◽

Strong Consistency ◽

Sufficient Conditions ◽

Estimation Accuracy ◽

Systematic Sampling ◽

Geometrical Object ◽

Important Prerequisite ◽

Systematic Pattern ◽

Increase Sample Size

In design-based stereology, fixed parameters (such as volume, surface area, curve length, feature number, connectivity) of a non-random geometrical object are estimated by intersecting the object with randomly located and oriented geometrical probes (e.g. test slabs, planes, lines, points). Estimation accuracy may in principle be increased by increasing the number of probes, which are usually laid in a systematic pattern. An important prerequisite to increase accuracy, however, is that the relevant estimators are unbiased and consistent. The purpose of this paper is therefore to give sufficient conditions for the unbiasedness and strong consistency of design-based stereological estimators obtained by systematic sampling. Relevant mechanisms to increase sample size, compatible with stereological practice, are considered.

Download Full-text

Consistency in systematic sampling

Advances in Applied Probability ◽

10.1017/s0001867800027518 ◽

1996 ◽

Vol 28 (04) ◽

pp. 982-992 ◽

Cited By ~ 2

Author(s):

X. Gual Arnau ◽

L. M. Cruz-Orive

Keyword(s):

Sample Size ◽

Surface Area ◽

Strong Consistency ◽

Sufficient Conditions ◽

Estimation Accuracy ◽

Systematic Sampling ◽

Geometrical Object ◽

Important Prerequisite ◽

Systematic Pattern ◽

Increase Sample Size

In design-based stereology, fixed parameters (such as volume, surface area, curve length, feature number, connectivity) of a non-random geometrical object are estimated by intersecting the object with randomly located and oriented geometrical probes (e.g. test slabs, planes, lines, points). Estimation accuracy may in principle be increased by increasing the number of probes, which are usually laid in a systematic pattern. An important prerequisite to increase accuracy, however, is that the relevant estimators are unbiased and consistent. The purpose of this paper is therefore to give sufficient conditions for the unbiasedness and strong consistency of design-based stereological estimators obtained by systematic sampling. Relevant mechanisms to increase sample size, compatible with stereological practice, are considered.

Download Full-text

Capture-induced physiological stress and postrelease mortality for Southern bluefin tuna (Thunnus maccoyii) from a recreational fishery

Canadian Journal of Fisheries and Aquatic Sciences ◽

10.1139/cjfas-2015-0516 ◽

2016 ◽

Vol 73 (10) ◽

pp. 1547-1556 ◽

Cited By ~ 9

Author(s):

Sean R. Tracey ◽

Klaas Hartmann ◽

Melanie Leef ◽

Jaime McAllister

Keyword(s):

Blood Plasma ◽

Sample Size ◽

Physiological Stress ◽

Bluefin Tuna ◽

Physical Damage ◽

Biochemical Indicators ◽

Southern Bluefin Tuna ◽

Increase Sample Size ◽

Significant Factors ◽

In Blood Plasma

Southern bluefin tuna (SBT; Thunnus maccoyii) are a popular component of the recreational large pelagic game fishery in Australia. The fishery is managed using individual fisher catch limits. Fifty-nine pop-up archival transmitting (PAT) tags were attached to individual SBT to estimate postrelease survival (PRS) rates. Fish caught on lures configured with J-hooks (n = 44) and those caught on circle hooks (n = 8) had similar PRS rates and were combined to increase sample size, revealing a PRS estimate of 83.0% (95% CI: 75.9%–90.7%, n = 54). The PRS estimate of fish caught on lures with treble hooks was much lower, 60% (95% CI: 20%–100%, n = 5). By sampling blood from 233 fish, including 56 of the PAT-tagged individuals, we show that angling duration is related to an elevation of lactate, cortisol, and osmolarity in blood plasma, indicative of increased physiological stress. Physical damage related to hooking location, angling duration, biochemical indicators of physiological stress, and handling duration were not identified as significant factors leading to postrelease mortality. The results quantify a previously unaccounted source of mortality for SBT.

Download Full-text