scholarly journals Determining sample size for progression criteria for pragmatic pilot RCTs: the hypothesis test strikes back!

2021 ◽  
Vol 7 (1) ◽  
Author(s):  
M. Lewis ◽  
K. Bromley ◽  
C. J. Sutton ◽  
G. McCray ◽  
H. L. Myers ◽  
...  
Keyword(s):  
Hypothesis Testing ◽  
Sample Size ◽  
Clinical Outcomes ◽  
Treatment Fidelity ◽  
Outcome Evaluation ◽  
Main Trial ◽  
Process Outcomes ◽  
Pilot Trials ◽  
Green Zone ◽  
Acceptable Outcome

Abstract Background The current CONSORT guidelines for reporting pilot trials do not recommend hypothesis testing of clinical outcomes on the basis that a pilot trial is under-powered to detect such differences and this is the aim of the main trial. It states that primary evaluation should focus on descriptive analysis of feasibility/process outcomes (e.g. recruitment, adherence, treatment fidelity). Whilst the argument for not testing clinical outcomes is justifiable, the same does not necessarily apply to feasibility/process outcomes, where differences may be large and detectable with small samples. Moreover, there remains much ambiguity around sample size for pilot trials. Methods Many pilot trials adopt a ‘traffic light’ system for evaluating progression to the main trial determined by a set of criteria set up a priori. We construct a hypothesis testing approach for binary feasibility outcomes focused around this system that tests against being in the RED zone (unacceptable outcome) based on an expectation of being in the GREEN zone (acceptable outcome) and choose the sample size to give high power to reject being in the RED zone if the GREEN zone holds true. Pilot point estimates falling in the RED zone will be statistically non-significant and in the GREEN zone will be significant; the AMBER zone designates potentially acceptable outcome and statistical tests may be significant or non-significant. Results For example, in relation to treatment fidelity, if we assume the upper boundary of the RED zone is 50% and the lower boundary of the GREEN zone is 75% (designating unacceptable and acceptable treatment fidelity, respectively), the sample size required for analysis given 90% power and one-sided 5% alpha would be around n = 34 (intervention group alone). Observed treatment fidelity in the range of 0–17 participants (0–50%) will fall into the RED zone and be statistically non-significant, 18–25 (51–74%) fall into AMBER and may or may not be significant and 26–34 (75–100%) fall into GREEN and will be significant indicating acceptable fidelity. Discussion In general, several key process outcomes are assessed for progression to a main trial; a composite approach would require appraising the rules of progression across all these outcomes. This methodology provides a formal framework for hypothesis testing and sample size indication around process outcome evaluation for pilot RCTs.

scholarly journals Determining sample size for progression criteria for pragmatic pilot RCTs: The Hypothesis test Strikes Back!

2020 ◽  
Author(s):  
Martyn Lewis ◽  
Kieran Bromley ◽  
Christopher J Sutton ◽  
Gareth McCray ◽  
Helen Lucy Myers ◽  
...  
Keyword(s):  
Hypothesis Testing ◽  
Sample Size ◽  
Clinical Outcomes ◽  
Treatment Fidelity ◽  
Outcome Evaluation ◽  
Main Trial ◽  
Process Outcomes ◽  
Pilot Trials ◽  
Green Zone ◽  
Acceptable Outcome

Abstract BackgroundThe current CONSORT guidelines for reporting pilot trials do not recommend hypothesis testing of clinical outcomes on the basis that a pilot trial is under-powered to detect such differences and this is the aim of the main trial. It states that primary evaluation should focus on descriptive analysis of feasibility/process outcomes (e.g. recruitment, adherence, treatment fidelity). Whilst the argument for not testing clinical outcomes is justifiable, the same does not necessarily apply to feasibility/process outcomes, where differences may be large and detectable with small samples. Moreover, there remains much ambiguity around sample size for pilot trials. MethodsMany pilot trials adopt a ‘traffic light’ system for evaluating progression to the main trial determined by a set of criteria set up a priori. We construct a hypothesis-testing approach for binary feasibility outcomes focused around this system that tests against being in the RED zone (unacceptable outcome) based on an expectation of being in the GREEN zone (acceptable outcome) and choose the sample size to give high power to reject being in the RED zone if the GREEN zone holds true. Pilot point estimates falling in the RED zone will be statistically non-significant and in the GREEN zone will be significant; the AMBER zone designates potentially acceptable outcome and statistical tests may be significant or non-significant.ResultsFor example, in relation to treatment fidelity, if we assume the upper boundary of the RED zone is 50% and the lower boundary of the GREEN zone is 75% (designating unacceptable and acceptable treatment fidelity, respectively), the sample size required for analysis given 90% power and one-sided 5% alpha would be around n=35 (intervention group alone). Observed treatment fidelity in the range of 0-17 participants (0-50%) will fall into the RED zone and be statistically non-significant; 18-26 (51-74%) fall into AMBER and may or may not be significant; 27-35 (75-100%) fall into GREEN and will be significant indicating acceptable fidelity.DiscussionIn general, several key process outcomes are assessed for progression to a main trial; a composite approach would require appraising the rules of progression across all these outcomes. This methodology provides a formal framework for hypothesis-testing and sample size indication around process outcome evaluation for pilot RCTs.

scholarly journals Determining sample size for progression criteria for pragmatic pilot RCTs: The Hypothesis test Strikes Back!

2021 ◽  
Author(s):  
Martyn Lewis ◽  
Kieran Bromley ◽  
Christopher J Sutton ◽  
Gareth McCray ◽  
Helen Lucy Myers ◽  
...  
Keyword(s):  
Hypothesis Testing ◽  
Sample Size ◽  
Clinical Outcomes ◽  
Treatment Fidelity ◽  
Outcome Evaluation ◽  
Main Trial ◽  
Process Outcomes ◽  
Pilot Trials ◽  
Green Zone ◽  
Acceptable Outcome

Abstract Background The current CONSORT guidelines for reporting pilot trials do not recommend hypothesis testing of clinical outcomes on the basis that a pilot trial is under-powered to detect such differences and this is the aim of the main trial. It states that primary evaluation should focus on descriptive analysis of feasibility/process outcomes (e.g. recruitment, adherence, treatment fidelity). Whilst the argument for not testing clinical outcomes is justifiable, the same does not necessarily apply to feasibility/process outcomes, where differences may be large and detectable with small samples. Moreover, there remains much ambiguity around sample size for pilot trials. Methods Many pilot trials adopt a ‘traffic light’ system for evaluating progression to the main trial determined by a set of criteria set up a priori. We construct a hypothesis-testing approach for binary feasibility outcomes focused around this system that tests against being in the RED zone (unacceptable outcome) based on an expectation of being in the GREEN zone (acceptable outcome) and choose the sample size to give high power to reject being in the RED zone if the GREEN zone holds true. Pilot point estimates falling in the RED zone will be statistically non-significant and in the GREEN zone will be significant; the AMBER zone designates potentially acceptable outcome and statistical tests may be significant or non-significant.Results For example, in relation to treatment fidelity, if we assume the upper boundary of the RED zone is 50% and the lower boundary of the GREEN zone is 75% (designating unacceptable and acceptable treatment fidelity, respectively), the sample size required for analysis given 90% power and one-sided 5% alpha would be around n=34 (intervention group alone). Observed treatment fidelity in the range of 0-17 participants (0-50%) will fall into the RED zone and be statistically non-significant; 18-25 (51-74%) fall into AMBER and may or may not be significant; 26-34 (75-100%) fall into GREEN and will be significant indicating acceptable fidelity.Discussion In general, several key process outcomes are assessed for progression to a main trial; a composite approach would require appraising the rules of progression across all these outcomes. This methodology provides a formal framework for hypothesis-testing and sample size indication around process outcome evaluation for pilot RCTs.

scholarly journals Determining sample size for progression criteria for pragmatic pilot RCTs: The Hypothesis test Strikes Back!

2020 ◽  
Author(s):  
Martyn Lewis ◽  
Kieran Bromley ◽  
Christopher J Sutton ◽  
Gareth McCray ◽  
Helen Lucy Myers ◽  
...  
Keyword(s):  
Hypothesis Testing ◽  
Sample Size ◽  
Clinical Outcomes ◽  
Treatment Fidelity ◽  
Outcome Evaluation ◽  
Main Trial ◽  
Process Outcomes ◽  
Pilot Trials ◽  
Green Zone ◽  
Acceptable Outcome

Abstract Background The current CONSORT guidelines for reporting pilot trials do not recommend hypothesis testing of clinical outcomes on the basis that a pilot trial is under-powered to detect such differences and this is the aim of the main trial. It states that primary evaluation should focus on descriptive analysis of feasibility/process outcomes (e.g. recruitment, adherence, treatment fidelity). Whilst the argument for not testing clinical outcomes is justifiable, the same does not necessarily apply to feasibility/process outcomes, where differences may be large and detectable with small samples. Moreover, there remains much ambiguity around sample size for pilot trials. Methods Many pilot trials adopt a ‘traffic light’ system for evaluating progression to the main trial determined by a set of criteria set up a priori. We construct a hypothesis-testing approach focused around this system that tests against being in the RED zone (unacceptable outcome) based on an expectation of being in the GREEN zone (acceptable outcome) and choose the sample size to give high power to reject being in the RED zone if the GREEN zone holds true. Pilot point estimates falling in the RED zone will be statistically non-significant and in the GREEN zone will be significant; the AMBER zone designates potentially acceptable outcome and statistical tests may be significant or non-significant.Results For example, in relation to treatment fidelity, if we assume the upper boundary of the RED zone is 50% and the lower boundary of the GREEN zone is 75% (designating unacceptable and acceptable treatment fidelity, respectively), the sample size required for analysis given 90% power and one-sided 5% alpha would be around n=30 (intervention group alone). Observed treatment fidelity in the range of 0-15 participants (0-50%) will fall into the RED zone and be statistically non-significant; 16-22 (51-74%) fall into AMBER and may or may not be significant; 23-30 (75-100%) fall into GREEN and will be significant indicating acceptable fidelity.Discussion In general, several key process outcomes are assessed for progression to a main trial; a composite approach would require appraising the rules of progression across all these outcomes. This methodology provides a formal framework for hypothesis-testing and sample size indication around process outcome evaluation for pilot RCTs.

scholarly journals Quality of reporting of pilot and feasibility cluster randomised trials: a systematic review

BMJ Open ◽  
2017 ◽  
Vol 7 (11) ◽  
pp. e016970 ◽  
Author(s):  
Claire L Chan ◽  
Clémence Leyrat ◽  
Sandra M Eldridge
Keyword(s):  
Hypothesis Testing ◽  
Sample Size ◽  
Randomised Trials ◽  
Quality Of Reporting ◽  
Cluster Randomised Trials ◽  
Cluster Randomised ◽  
Pilot Trials ◽  
Formal Hypothesis Testing ◽  
Cluster Design

ObjectivesTo systematically review the quality of reporting of pilot and feasibility of cluster randomised trials (CRTs). In particular, to assess (1) the number of pilot CRTs conducted between 1 January 2011 and 31 December 2014, (2) whether objectives and methods are appropriate and (3) reporting quality.MethodsWe searched PubMed (2011–2014) for CRTs with ‘pilot’ or ‘feasibility’ in the title or abstract; that were assessing some element of feasibility and showing evidence the study was in preparation for a main effectiveness/efficacy trial. Quality assessment criteria were based on the Consolidated Standards of Reporting Trials (CONSORT) extensions for pilot trials and CRTs.ResultsEighteen pilot CRTs were identified. Forty-four per cent did not have feasibility as their primary objective, and many (50%) performed formal hypothesis testing for effectiveness/efficacy despite being underpowered. Most (83%) included ‘pilot’ or ‘feasibility’ in the title, and discussed implications for progression from the pilot to the future definitive trial (89%), but fewer reported reasons for the randomised pilot trial (39%), sample size rationale (44%) or progression criteria (17%). Most defined the cluster (100%), and number of clusters randomised (94%), but few reported how the cluster design affected sample size (17%), whether consent was sought from clusters (11%), or who enrolled clusters (17%).ConclusionsThat only 18 pilot CRTs were identified necessitates increased awareness of the importance of conducting and publishing pilot CRTs and improved reporting. Pilot CRTs should primarily be assessing feasibility, avoiding formal hypothesis testing for effectiveness/efficacy and reporting reasons for the pilot, sample size rationale and progression criteria, as well as enrolment of clusters, and how the cluster design affects design aspects. We recommend adherence to the CONSORT extensions for pilot trials and CRTs.

scholarly journals The reporting of progression criteria in protocols of pilot trials designed to assess the feasibility of main trials is insufficient: a meta-epidemiological study

2019 ◽  
Vol 5 (1) ◽  
Author(s):  
Lawrence Mbuagbaw ◽  
Sarah Daisy Kosa ◽  
Daeria O. Lawson ◽  
Rosa Stalteri ◽  
Oluwatobi R. Olaiya ◽  
...  
Keyword(s):  
Sample Size ◽  
Primary Outcome ◽  
Linear Models ◽  
Feasibility Studies ◽  
Main Trial ◽  
Generalised Linear Models ◽  
Definitive Trial ◽  
Research Protocols ◽  
Pilot Trials ◽  
Pubmed Search

Abstract Introduction Pilot and feasibility trials are conducted to determine feasibility or to collect information that would inform the design of a larger definitive trial. Clear progression criteria are required to determine if a definitive or main trial is feasible and how it should be designed. We sought to determine how often progression criteria are reported and the associated factors. Methods We conducted a methodological review of protocols for pilot randomised trials published in three journals that publish research protocols (BMJ Open, Trials, Pilot and Feasibility Studies), using a PubMed search (2013–2017). We extracted bibliometric information including the country in which the study was conducted, source of funding, type of intervention, use of a primary feasibility outcome, sample size reporting, and justification. We used generalised linear models to determine the factors associated with reporting progression criteria. Results Our search retrieved 276 articles, of which 49 were not eligible. We included 227 articles. Overall, 45/227 (19.8%; 95% confidence interval [CI] 14.8–25.6) reported progression criteria. Protocols published in more recent years were significantly associated with higher odds of reporting progression criteria (adjusted odds ratio [aOR] 1.40; 95% CI 1.03–1.92; p = 0.034). Pilot trials from Europe (aOR 0.19; 95% CI 0.08–0.48; p < 0.001) and the rest of the world (aOR 0.05; 95% CI 0.01–0.18; p < 0.003) compared to North America were significantly associated with lower odds of reporting progression criteria. Journal, source of funding, sample size, intervention type, and having a primary outcome related to feasibility were not significantly associated with reporting progression criteria. Conclusion Progression criteria are not often explicitly stated in protocols of pilot trials leaving room for varied interpretation of findings. The development of formal guidance for progression criteria in protocols of pilot trials is warranted.

SELFIE TOURISM DAN ELECTRONIC WORD OF MOUTH SEBAGAI ANTESEDEN RE-VISIT INTENTION

2019 ◽  
Vol 4 (3) ◽  
pp. 526
Author(s):  
Okki Trinanda ◽  
Astri Yuza Sari
Keyword(s):  
Social Media ◽  
Hypothesis Testing ◽  
Sample Size ◽  
Human Behavior ◽  
Word Of Mouth ◽  
Electronic Word Of Mouth ◽  
Tourist Attractions ◽  
Tourism Management ◽  
West Sumatra ◽  
First Time

<p><em>Research linking selfie behavior and tourism management is very rarely implemented. Selfie behavior is more researched as part of psychology that studies human behavior. This study aims to find out (1) the influence of Selfie Tourism on Electronic Word of Mouth, (2) the influence of Selfie Tourism on Re-Visit Intention, and (3) the influence of Electronic Word of Mouth on Re-Visit Intention. This study uses estimates based on the number of parameters obtained by the sample size of 452 respondents with accidental sampling. Respondents who were included in this study were foreign tourists and domestic tourists who visited the tourism sites in West Sumatra for the first time. While hypothesis testing uses SEM. In this study all relationships between variables were found to be positive and significant. The implication of this study is that tourism managers not only pay attention to aspects of service such as hospitality, cleanliness and so on, but also provide attractive tourist attractions to be photographed and distributed to social media.</em></p><p><em><br /></em></p><p><em>Penelitian yang menghubungkan perilaku selfie dan manajemen pariwisata sangat jarang dilaksanakan. Perilaku selfie lebih banyak diteliti sebagai bagian dari psikologi yang mempelajari perilaku manusia. Penelitian ini bertujuan untuk mengetahui (1) pengaruh Selfie Tourism terhadap Electronic Word of Mouth, (2) pengaruh Selfie Tourism terhadap Re-Visit Intention, dan (3) pengaruh Electronic Word of Mouth pada Re-Visit Intention. Penelitian ini menggunakan jumlah parameter yang diperoleh dengan ukuran sampel 452 responden dengan accidental sampling. Responden yang dikunjungi oleh wisatawan asing dan wisatawan domestik yang mengunjungi situs pariwisata di Sumatera Barat untuk pertama kalinya. Sedangkan pengujian hipotesis menggunakan SEM. Dalam penelitian ini semua hubungan antar variabel ditemukan positif dan signifikan. Implikasi dari penelitian ini adalah bahwa manajer pariwisata tidak hanya memperhatikan layanan dan kebersihan tetapi juga menyediakan media sosial.</em></p>

scholarly journals Are pilot trials useful for predicting randomisation and attrition rates in definitive studies: A review of publicly funded trials

2018 ◽  
Vol 15 (2) ◽  
pp. 189-196 ◽  
Author(s):  
Cindy L Cooper ◽  
Amy Whitehead ◽  
Edward Pottrill ◽  
Steven A Julious ◽  
Stephen J Walters
Keyword(s):  
Randomised Controlled Trials ◽  
Pilot Trial ◽  
Attrition Rate ◽  
Controlled Trials ◽  
Publicly Funded ◽  
Limits Of Agreement ◽  
Main Trial ◽  
Attrition Rates ◽  
Pilot Trials ◽  
Randomised Controlled

Background/aims: External pilot trials are recommended for testing the feasibility of main or confirmatory trials. However, there is little evidence that progress in external pilot trials actually predicts randomisation and attrition rates in the main trial. To assess the use of external pilot trials in trial design, we compared randomisation and attrition rates in publicly funded randomised controlled trials with rates in their pilots. Methods: Randomised controlled trials for which there was an external pilot trial were identified from reports published between 2004 and 2013 in the Health Technology Assessment Journal. Data were extracted from published papers, protocols and reports. Bland–Altman plots and descriptive statistics were used to investigate the agreement of randomisation and attrition rates between the full and external pilot trials. Results: Of 561 reports, 41 were randomised controlled trials with pilot trials and 16 met criteria for a pilot trial with sufficient data. Mean attrition and randomisation rates were 21.1% and 50.4%, respectively, in the pilot trials and 16.8% and 65.2% in the main. There was minimal bias in the pilot trial when predicting the main trial attrition and randomisation rate. However, the variation was large: the mean difference in the attrition rate between the pilot and main trial was −4.4% with limits of agreement of −37.1% to 28.2%. Limits of agreement for randomisation rates were −47.8% to 77.5%. Conclusion: Results from external pilot trials to estimate randomisation and attrition rates should be used with caution as comparison of the difference in the rates between pilots and their associated full trial demonstrates high variability. We suggest using internal pilot trials wherever appropriate.

Sign in / Sign up

Export Citation Format

Share Document