The statistical interpretation of pilot trials: should significance thresholds be reconsidered?

BMC Medical Research Methodology

Steven A Julious¹  Richard M Jacques¹  Amy L Whitehead¹  Ellen C Lee¹ 

[1] Medical Statistics Group, School of Health and Related Research (ScHARR), University of Sheffield, 30 Regent Street, Sheffield S1 4DA, UK

关键词: Bayesian methods;    Significance;    Confidence interval;    Type I error;    Power;    Pilot trial;   

Others  :  866380
DOI  :  10.1186/1471-2288-14-41

 received in 2013-10-18, accepted in 2014-03-12,  发布年份 2014

【 摘 要 】

Background

In an evaluation of a new health technology, a pilot trial may be undertaken prior to a trial that makes a definitive assessment of benefit. The objective of pilot studies is to provide sufficient evidence that a larger definitive trial can be undertaken and, at times, to provide a preliminary assessment of benefit.

Methods

We describe significance thresholds, confidence intervals and surrogate markers in the context of pilot studies and how Bayesian methods can be used in pilot trials. We use a worked example to illustrate the issues raised.

Results

We show how significance levels other than the traditional 5% should be considered to provide preliminary evidence for efficacy and how estimation and confidence intervals should be the focus to provide an estimated range of possible treatment effects. We also illustrate how Bayesian methods could also assist in the early assessment of a health technology.

Conclusions

We recommend that in pilot trials the focus should be on descriptive statistics and estimation, using confidence intervals, rather than formal hypothesis testing and that confidence intervals other than 95% confidence intervals, such as 85% or 75%, be used for the estimation. The confidence interval should then be interpreted with regards to the minimum clinically important difference. We also recommend that Bayesian methods be used to assist in the interpretation of pilot trials. Surrogate endpoints can also be used in pilot trials but they must reliably predict the overall effect on the clinical outcome.

【 授权许可】

   
2014 Lee et al.; licensee BioMed Central Ltd.

【 图 表 】

【 参考文献 】

• [1]Wood J, Lambert M: Sample size calculations for trials in health services research. J Health Serv Res Policy 1999, 4(4):226-229.
• [2]Julious SA, Patterson SD: Sample sizes for estimation in clinical research. Pharm Stat 2004, 3(3):213-215.
• [3]Biomarkers Definitions Working Group: Biomarkers and surrogate endpoints: preferred definitions and conceptual framework. Clin Pharmacol Ther 2001, 69(3):89-95.
• [4]Lancaster GA, Dodd S, Williamson PR: Design and analysis of pilot studies: recommendations for good practice. J Eval Clin Pract 2004, 10(2):307-312.
• [5]Thabane L, Ma J, Chu R, Cheng J, Ismaila A, Rios LP, Robson R, Thabane M, Giangregorio L, Goldsmith CH: A tutorial on pilot studies: the what, why and how. BMC Med Res Methodol 2010, 10:1. BioMed Central Full Text
• [6]Arain M, Campbell MJ, Cooper CL, Lancaster GA: What is a pilot or feasibility study? A review of current practice and editorial policy. BMC Med Res Methodol 2010, 10:67. BioMed Central Full Text
• [7]Kianifard F, Islam MZ: A guide to the design and analysis of small clinical studies. Pharm Stat 2011, 10(4):363-368.
• [8]Stallard N: Optimal sample sizes for phase II clinical trials and pilot studies. Stat Med 2012, 31:1031-1042.
• [9]Schoenfeld D: Statistical considerations for pilot-studies. Int J Radiat Oncol Biol Phys 1980, 6(3):371-374.
• [10]Papadakis S, Aitken D, Gocan S, Riley D, Laplante MA, Bhatnagar-Bost A, Cousineau D, Simpson D, Edjoc R, Pipe AL, Sharma M, Reid RD: A randomised controlled pilot study of standardised counselling and cost-free pharmacotherapy for smoking cessation among stroke and TIA patients. BMJ Open 2011, 1(2):e000366.
• [11]Legault C, Jennings JM, Katula JA, Dagenbach D, Gaussoin SA, Sink KM, Rapp SR, Rejeski WJ, Shumaker SA, Espeland MA: Designing clinical trials for assessing the effects of cognitive training and physical activity interventions on cognitive outcomes: the Seniors Health and Activity Research Program Pilot (SHARP-P) study, a randomized controlled trial. BMC Geriatr 2011, 11:27. BioMed Central Full Text
• [12]Walters SJ: Consultants’ forum: should post hoc sample size calculations be done? Pharm Stat 2009, 8(2):163-169.
• [13]Walters SJ, Morrell CJ, Dixon S: Measuring health-related quality of life in patients with venous leg ulcers. Qual Life Res 1999, 8(4):327-336.
• [14]Morrell CJ, Walters SJ, Dixon S, Collins KA, Brereton LML, Peters J, Brooker CGD: Cost effectiveness of community leg ulcer clinics: randomised controlled trial. Br Med J 1998, 316(7143):1487-1491.
• [15]Collins K, Morrell J, Peters J, Walters S, Brooker C, Brereton L: Problems associated with patient satisfaction surveys. Bri J Commun Health Nurs 2007, 2(3):156-163.
• [16]Carpenter JR, Kenward MG: Multiple Imputation and its Application. Chichester: Wiley; 2013.
• [17]De Gruttola VG, Clax P, DeMets DL, Downing GJ, Ellenberg SS, Friedman L, Gail MH, Prentice R, Wittes J, Zeger SL: Considerations in the evaluation of surrogate endpoints in clinical trials: Summary of a National Institutes of Health Workshop. Control Clin Trials 2001, 22(5):485-502.
• [18]Prentice RL: Surrogate endpoints in clinical-trials - definition and operational criteria. Stat Med 1989, 8(4):431-440.
• [19]International Conference on Harmonisation: ICH E9 statistical principals for clinical trials. 1998. http://www.ich.org/fileadmin/Public_Web_Site/ICH_Products/Guidelines/Efficacy/E9/Step4/E9_Guideline.pdf webcite
• [20]Fleming TR, DeMets DL: Surrogate end points in clinical trials: are we being misled? Ann Intern Med 1996, 125(7):605-613.
• [21]Temple R: Are surrogate markers adequate to assess cardiovascular disease drugs? J Am Med Assoc 1999, 282(8):790-795.
• [22]Julious SA, Machin D, Tan SB: An Introduction to Statistics in Early Phase Trials. Oxford: Wiley-Blackwell; 2010.
• [23]Julious SA, Swank DJ: Moving statistics beyond the individual clinical trial: applying decision science to optimize a clinical development plan. Pharm Stat 2005, 4(1):37-46.
• [24]Chuang-Stein C, Kirby S, French J, Kowalski K, Marshall S, Smith MK, Bycott P, Beltangady M: A quantitative approach for making go/no-go decisions in drug development. Drug Inform J 2011, 45(2):187-202.
• [25]O’Hagan A, Stevens JW, Campbell MJ: Assurance in clinical trial design. Pharm Stat 2005, 4(3):187-201.
• [26]Chuang-Stein C: Sample size and the probability of a successful trial. Pharm Stat 2006, 5(4):305-309.
• [27]Parmar MKB, Ungerleider RS, Simon R: Assessing whether to perform a confirmatory randomized clinical trial. J Natl Canc Inst 1996, 88(22):1645-1651.
• [28]Spiegelhalter DJ, Abrams KR, Myles JP: Bayesian Approaches to Clinical Trials and Health-Care Evaluation. Chichester: John Wiley & Sons; 2004.
• [29]Lee PM: Bayesian Statistics: An Introduction. New York: Oxford University Press; Edward Arnold; 1989.