Accuracy of the Berger-Exner test for detecting third-order selection bias in randomised controlled trials: a simulation-based investigation

BMC Medical Research Methodology

Vance W Berger⁴  Sheila Gudehithlu²  Bo Fu³  Steffen Mickenautsch¹ 

[1] SYSTEM Initiative/Department of Community Dentistry, Faculty of Health Sciences, University of the Witwatersrand, 7 York Rd., Parktown, Johannesburg 2193, South Africa;Department of Statistics, University of Illinois at Urbana-Champaign, Champaign, IL, USA;Department of Biostatistics, GSPH, University of Pittsburgh, Pittsburgh, PA, USA;National Cancer Institute, National Institute of Health, Rockville, MD, USA

关键词: ROC curve;    Specificity;    Sensitivity;    Berger-Exner test;    Selection bias;    Randomised trials;   

Others  :  1090764
DOI  :  10.1186/1471-2288-14-114

 received in 2013-10-21, accepted in 2014-09-23,  发布年份 2014

【 摘 要 】

Background

Randomised controlled trials (RCT) are highly influential upon medical decisions. Thus RCTs must not distort the truth. One threat to internal trial validity is the correct prediction of future allocations (selection bias). The Berger-Exner test detects such bias but has not been widely utilized in practice. One reason for this non-utilisation may be a lack of information regarding its test accuracy. The objective of this study is to assess the accuracy of the Berger-Exner test on the basis of relevant simulations for RCTs with dichotomous outcomes.

Methods

Simulated RCTs with various parameter settings were generated, using R software, and subjected to bias-free and selection bias scenarios. The effect size inflation due to bias was quantified. The test was applied in both scenarios and the pooled sensitivity and specificity, with 95% confidence intervals for alpha levels of 1%, 5%, and 20%, were computed. Summary ROC curves were generated and the relationships of parameters with test accuracy were explored.

Results

An effect size inflation of 71% - 99% was established. Test sensitivity was 1.00 (95% CI: 0.99 – 1.00) for alpha level 1%, 5%, and 20%; test specificity was 0.94 (95% CI: 0.93 – 0.96); 0.82 (95% CI: 0.80 – 0.84), and 0.56 (95% CI: 0.54 – 0.58) for alpha 1%, 5%, and 20%, respectively. Test accuracy was best with the maximal procedure used with a maximum tolerated imbalance (MTI) = 2 as the randomisation method at alpha 1%.

Conclusions

The results of this simulation study suggest that the Berger-Exner test is generally accurate for identifying third-order selection bias.

【 授权许可】

   
2014 Mickenautsch et al.; licensee BioMed Central Ltd.

【 图 表 】

Figure 1.

Figure 2.

Figure 3.

【 参考文献 】

• [1]Green S, Higgins J: Glossary. Cochrane handbook for systematic reviews of interventions 4.2. 5. 2009. [Updated May 2005]
• [2]Berger VW: Selection bias and covariate imbalances in randomised clinical trials. Chichester, UK: John Wiley & Sons, Ltd.; 2005.
• [3]Gluud LL: Bias in clinical intervention research. Am J Epidemiol 2006, 163:493-501.
• [4]Chalmers TC, Matta RJ, Smith H Jr, Kunzler AM: Evidence favoring the use if anticoagulants in the hospital phase of acute myocardial infarction. N Engl J Med 1977, 297:1091-1096.
• [5]Schulz KF, Chalmers I, Hayes RJ, Altman DG: Empirical evidence of bias. Dimensions of methodological quality associated with estimates of treatment effects in controlled trials. J Am Med Assoc 1995, 273:408-412.
• [6]Kjaergard LL, Villumsen J, Gluud C: Reported methodologic quality and discrepancies between large and small randomized trials in meta-analyses. Ann Intern Med 2001, 135:982-989.
• [7]Crossley NA, Sena E, Goehler J, Horn J, van der Worp B, Bath PM, Macleod M, Dirnagl U: Empirical evidence of bias in the design of experimental stroke studies: a meta-epidemiologic approach. Stroke 2008, 39:929-934.
• [8]Egger M, Jüni P, Bartlett C, Holenstein F, Sterne J: How important are comprehensive literature searches and the assessment of trial quality in systematic reviews? Empirical study. Health Technol Assess 2003, 7:1.
• [9]Berger VW, Ivanova A, Knoll MD: Minimizing predictability while retaining balance through the use of less restrictive randomization procedures. Stat Med 2003, 22:3017-3028.
• [10]Berger VW, Exner DV: Detecting selection bias in randomized clinical trials. Contr Clin Trials 1999, 20:319-327.
• [11]Berger VW: The reverse propensity score to detect selection bias and correct for baseline imbalances. Stat Med 2005, 24:2777-2787.
• [12]Asarnow JR, Jaycox LH, Duan N, LaBorde AP, Rea MM, Murray P, Anderson M, Landon C, Tang L, Wells KB: Effectiveness of a quality improvement intervention for adolescent depression in primary care clinics: a randomized controlled trial. JAMA 2005, 293:311-319.
• [13]Collins ED, Kleber HD, Whittington RA, Heitler NE: Anesthesia-assisted vs buprenorphine- or clonidine-assisted heroin detoxification and naltrexone induction: a randomized trial. JAMA 2005, 294:903-913.
• [14]Luabeya KK, Mpontshane N, Mackay M, Ward H, Elson I, Chhagan M, Tomkins A, Van den Broeck J, Bennish ML: Zinc or multiple micronutrient supplementation to reduce diarrhea and respiratory disease in South African children: a randomized controlled trial. PLoS One 2007, 2:e541.
• [15]Nava S, Grassi M, Fanfulla F, Domenighetti G, Carlucci A, Perren A, Dell'Orso D, Vitacca M, Ceriana P, Karakurt Z, Clini E: Non-invasive ventilation in elderly patients with acute hypercapnic respiratory failure: a randomised controlled trial. Age Ageing 2011, 40:444-450.
• [16]Swaminathan S, Menon PA, Gopalan N, Perumal V, Santhanakrishnan RK, Ramachandran R, Chinnaiyan P, Iliayas S, Chandrasekaran P, Navaneethapandian PD, Elangovan T, Pho MT, Wares F, Paranji Ramaiyengar N: Efficacy of a six-month versus a 36-month regimen for prevention of tuberculosis in HIV-infected persons in India: a randomized clinical trial. PLoS One 2012, 7:e47400.
• [17]Gupta SK, Mi D, Dubé MP, Saha CK, Johnson RM, Stein JH, Clauss MA, Mather KJ, Desta Z, Liu Z: Pentoxifylline, inflammation, and endothelial function in HIV-infected persons: a randomized, placebo-controlled trial. PLoS One 2013, 8:e60852.
• [18]Ganderton L, Jenkins S, Gain K, Fowler R, Winship P, Lunt D, Gabbay E: Short term effects of exercise training on exercise capacity and quality of life in patients with pulmonary arterial hypertension: protocol for a randomised controlled trial. BMC Pulm Med 2011, 11:25. BioMed Central Full Text
• [19]Deeks JJ: Systematic reviews of evaluations of diagnostic and screening tests. In Systematic reviews in health care. Edited by Egger M, Smith GD, Altman DG. London, UK: BMJ Publishing group; 2001:248.
• [20]Glas AS, Lijmer JG, Prins MH, Bonsel GJ, Bossuyt PM: The diagnostic odds ratio: a single indicator of test performance. J Clin Epidemiol 2003, 56:1129-1135.
• [21]Moher D, Cook DJ, Jadad AR, Tugwell P, Moher M, Jones A, Pham B, Klassen TP: Assessing the quality of reports of randomised trials: implications for the conduct of meta-analyses. Health Technol Assess 1999, 3:1-98.
• [22]Zamora J, Abraira V, Muriel A, Khan K, Coomarasamy A: Meta-DiSc: a software for meta-analysis of test accuracy data. BMC Med Res Methodol 2006, 6:31. BioMed Central Full Text
• [23]Higgins JPT, Green S: Cochrane Handbook for Systematic Reviews of Interventions Version 5.1.0 [updated March 2011]. Table 8.5.d. The Cochrane Collaboration; 2011. Available from http://www.cochrane-handbook.org webcite
• [24]Odgaard-Jensen J, Vist GE, Timmer A, Kunz R, Akl EA, Schünemann H, Briel M, Nordmann AJ, Pregno S, Oxman AD: Randomisation to protect against selection bias in healthcare trials. Cochrane Database Syst Rev 2011, 4:MR000012.
• [25]Ivanova A, Barrier RC Jr, Berger VW: Adjusting for observable selection bias in block randomized trials. Stat Med 2005, 24:1537-1546.
• [26]Guyatt GH, Oxman AD, Vist G, Kunz R, Brozek J, Alonso-Coello P, Montori V, Akl EA, Djulbegovic B, Falck-Ytter Y, Norris SL, Williams JW Jr, Atkins D, Meerpohl J, Schünemann HJ: GRADE guidelines: 4. Rating the quality of evidence–study limitations (risk of bias). J Clin Epidemiol 2011, 64:407-715.
• [27]Turner L, Boutron I, Hróbjartsson A, Altman DG, Moher D: The evolution of assessing bias in Cochrane systematic reviews of interventions: celebrating methodological contributions of the Cochrane Collaboration. Syst Rev 2013, 2:79. BioMed Central Full Text