Simulation evaluation of statistical properties of methods for indirect and mixed treatment comparisons

BMC Medical Research Methodology

Jim Maas¹  Max O Bachmann¹  Allan Clark¹  Fujian Song¹ 

[1] Norwich Medical School, Faculty of Medicine and Health Science, University of East Anglia, Norwich, Norfolk, NR4 7TJ, UK

关键词: Simulation evaluation;    Statistical power;    Type I error;    Bias;    Inconsistency;    Network meta-analysis;    Mixed treatment comparison;    Indirect comparison;   

Others  :  1126798
DOI  :  10.1186/1471-2288-12-138

 received in 2012-06-15, accepted in 2012-09-04,  发布年份 2012

【 摘 要 】

Background

Indirect treatment comparison (ITC) and mixed treatment comparisons (MTC) have been increasingly used in network meta-analyses. This simulation study comprehensively investigated statistical properties and performances of commonly used ITC and MTC methods, including simple ITC (the Bucher method), frequentist and Bayesian MTC methods.

Methods

A simple network of three sets of two-arm trials with a closed loop was simulated. Different simulation scenarios were based on different number of trials, assumed treatment effects, extent of heterogeneity, bias and inconsistency. The performance of the ITC and MTC methods was measured by the type I error, statistical power, observed bias and mean squared error (MSE).

Results

When there are no biases in primary studies, all ITC and MTC methods investigated are on average unbiased. Depending on the extent and direction of biases in different sets of studies, ITC and MTC methods may be more or less biased than direct treatment comparisons (DTC). Of the methods investigated, the simple ITC method has the largest mean squared error (MSE). The DTC is superior to the ITC in terms of statistical power and MSE. Under the simulated circumstances in which there are no systematic biases and inconsistencies, the performances of MTC methods are generally better than the performance of the corresponding DTC methods. For inconsistency detection in network meta-analysis, the methods evaluated are on average unbiased. The statistical power of commonly used methods for detecting inconsistency is very low.

Conclusions

The available methods for indirect and mixed treatment comparisons have different advantages and limitations, depending on whether data analysed satisfies underlying assumptions. To choose the most valid statistical methods for research synthesis, an appropriate assessment of primary studies included in evidence network is required.

【 授权许可】

   
2012 Song et al.; licensee BioMed Central Ltd.

【 图 表 】

Figure 1.

Figure 2.

Figure 3.

Figure 4.

Figure 5.

Figure 6.

Figure 7.

Figure 8.

【 参考文献 】

• [1]Glenny AM, Altman DG, Song F, Sakarovitch C, Deeks JJ, D'Amico R, Bradburn M, Eastwood AJ: Indirect comparisons of competing interventions. Health Technol Assess 2005, 9(26):1-134. iii-iv
• [2]Song F, Loke YK, Walsh T, Glenny AM, Eastwood AJ, Altman DG: Methodological problems in the use of indirect comparisons for evaluating healthcare interventions: a survey of published systematic reviews. BMJ 2009, 338:b1147.
• [3]Donegan S, Williamson P, Gamble C, Tudur-Smith C: Indirect comparisons: a review of reporting and methodological quality. PLoS One 2010, 5(11):e11054.
• [4]Edwards SJ, Clarke MJ, Wordsworth S, Borrill J: Indirect comparisons of treatments based on systematic reviews of randomised controlled trials. Int J Clin Pract 2009, 63(6):841-854.
• [5]Bucher HC, Guyatt GH, Griffith LE, Walter SD: The results of direct and indirect treatment comparisons in meta-analysis of randomized controlled trials. J Clin Epidemiol 1997, 50(6):683-691.
• [6]Higgins JP, Whitehead A: Borrowing strength from external trials in a meta-analysis. Stat Med 1996, 15(24):2733-2749.
• [7]Lumley T: Network meta-analysis for indirect treatment comparisons. Stat Med 2002, 21(16):2313-2324.
• [8]Lu G, Ades AE: Combination of direct and indirect evidence in mixed treatment comparisons. Stat Med 2004, 23(20):3105-3124.
• [9]Lu G, Ades AE: Assessing evidence inconsistency in mixed treatment comparisons. J Am Stat Assoc 2006, 101(474):447-459.
• [10]Jansen JP, Fleurence R, Devine B, Itzler R, Barrett A, Hawkins N, Lee K, Boersma C, Annemans L, Cappelleri JC: Interpreting indirect treatment comparisons and network meta-analysis for health-care decision making: report of the ISPOR Task Force on Indirect Treatment Comparisons Good Research Practices: part 1. Value Health 2011, 14(4):417-428.
• [11]Jansen JP, Schmid CH, Salanti G: Directed acyclic graphs can help understand bias in indirect and mixed treatment comparisons. J Clin Epidemiol 2012, 65(7):798-807.
• [12]Song F, Altman DG, Glenny AM, Deeks JJ: Validity of indirect comparison for estimating efficacy of competing interventions: empirical evidence from published meta-analyses. BMJ 2003, 326(7387):472-475.
• [13]Song F, Xiong T, Parekh-Bhurke S, Loke YK, Sutton AJ, Eastwood AJ, Holland R, Chen YF, Glenny AM, Deeks JJ, et al.: Inconsistency between direct and indirect comparisons of competing interventions: meta-epidemiological study. BMJ 2011, 343:d4909.
• [14]Chou R, Fu R, Huffman LH, Korthuis PT: Initial highly-active antiretroviral therapy with a protease inhibitor versus a non-nucleoside reverse transcriptase inhibitor: discrepancies between direct and indirect meta-analyses. Lancet 2006, 368(9546):1503-1515.
• [15]Madan J, Stevenson MD, Cooper KL, Ades AE, Whyte S, Akehurst R: Consistency between direct and indirect trial evidence: is direct evidence always more reliable? Value Health 2011, 14(6):953-960.
• [16]Gartlehner G, Moore CG: Direct versus indirect comparisons: a summary of the evidence. Int J Technol Assess Health Care 2008, 24(2):170-177.
• [17]Dias S, Welton NJ, Caldwell DM, Ades AE: Checking consistency in mixed treatment comparison meta-analysis. Stat Med 2010, 29(7–8):932-944.
• [18]Cooper NJ, Sutton AJ, Morris D, Ades AE, Welton NJ: Addressing between-study heterogeneity and inconsistency in mixed treatment comparisons: application to stroke prevention treatments in individuals with non-rheumatic atrial fibrillation. Stat Med 2009, 28(14):1861-1881.
• [19]Salanti G, Marinho V, Higgins JP: A case study of multiple-treatments meta-analysis demonstrates that covariates should be considered. J Clin Epidemiol 2009, 62(8):857-864.
• [20]Mills EJ, Ghement I, O'Regan C, Thorlund K: Estimating the power of indirect comparisons: a simulation study. PLoS One 2011, 6(1):e16237.
• [21]Wells GA, Sultan SA, Chen L, Khan M, Coyle D: Indirect evidence: indirect treatment comparisons in meta-analysis. Ottawa, Canada: Canadian Agency for Drugs and Technologies in Health; 2009.
• [22]DerSimonian R, Laird N: Meta-analysis in clinical trials. Controlled Clin Trials 1986, 7:177-188.
• [23]Salanti G, Higgins JP, Ades AE, Ioannidis JP: Evaluation of networks of randomized trials. Stat Methods Med Res 2008, 17(3):279-301.
• [24]Dias S, Welton NJ, Sutton AJ, Caldwell DM, Lu G, Ades AE: NICE DSU Technical Support Document 4: Inconsistency in Network of Evidence Based on Randomised Controlled Trials. 2011. Available from http://www.nicedsu.org.uk webcite
• [25]Eckermann S, Coory M, Willan AR: Indirect comparison: relative risk fallacies and odds solution. J Clin Epidemiol 2009, 62(10):1031-1036.
• [26]Pullenayegum EM: An informed reference prior for between-study heterogeneity in meta-analyses of binary outcomes. Stat Med 2011, 30(26):3082-3094.
• [27]Higgins JP, Altman DG: Chapter 8: Assessing risk of bias in included studies. In Cochrane Handbook for Systematic Reviews of Interventions. Edited by Higgins J, Green S. Chichester: Wiley; 2008.
• [28]R_Development_Core_Team: R: A language and environment for statistical computing. In. Vienna, Austria. Vienna, Austria: R Foundation for Statistical Computing; 2008.
• [29]Song F, Harvey I, Lilford R: Adjusted indirect comparison may be less biased than direct comparison for evaluating new pharmaceutical interventions. J Clin Epidemiol 2008, 61(5):455-463.
• [30]Hardy RJ, Thompson SG: Detecting and describing heterogeneity in meta-analysis. Stat Med 1998, 17(8):841-856.
• [31]Nixon RM, Bansback N, Brennan A: Using mixed treatment comparisons and meta-regression to perform indirect comparisons to estimate the efficacy of biologic treatments in rheumatoid arthritis. Stat Med 2007, 26(6):1237-1254.
• [32]Dias S, Welton NJ, Sutton AJ, Ades AE: NICE DSU Technical Support Document 2: A Generalised Linear Modelling Framework for Pairwise and Network Meta-Analysis of Randomised Controlled Trials. 2011. Available from: http://www.nicedsu.org.uk webcite
• [33]Hartling L, Ospina M, Liang Y, Dryden DM, Hooton N, Krebs Seida J, Klassen TP: Risk of bias versus quality assessment of randomised controlled trials: cross sectional study. BMJ 2009, 339:b4012.
• [34]Juni P, Altman DG, Egger M: Systematic reviews in health care: assessing the quality of controlled clinical trials. BMJ 2001, 323(7303):42-46.
• [35]Schulz KF, Chalmers I, Hayes RJ, Altman DG: Empirical evidence of bias. Dimensions of methodological quality associated with estimates of treatment effects in controlled trials. JAMA 1995, 273(5):408-412.
• [36]Wood L, Egger M, Gluud LL, Schulz KF, Juni P, Altman DG, Gluud C, Martin RM, Wood AJ, Sterne JA: Empirical evidence of bias in treatment effect estimates in controlled trials with different interventions and outcomes: meta-epidemiological study. BMJ 2008, 336(7644):601-605.
• [37]Hrobjartsson A, Thomsen AS, Emanuelsson F, Tendal B, Hilden J, Boutron I, Ravaud P, Brorson S: Observer bias in randomised clinical trials with binary outcomes: systematic review of trials with both blinded and non-blinded outcome assessors. BMJ 2012, 344:e1119.
• [38]Chalmers I, Matthews R: What are the implications of optimism bias in clinical research? Lancet 2006, 367(9509):449-450.
• [39]Thompson S, Ekelund U, Jebb S, Lindroos AK, Mander A, Sharp S, Turner R, Wilks D: A proposed method of bias adjustment for meta-analyses of published observational studies. Int J Epidemiol 2011, 40(3):765-777.
• [40]Turner RM, Spiegelhalter DJ, Smith GC, Thompson SG: Bias modelling in evidence synthesis. J R Stat Soc A 2009, 172(1):21-47.
• [41]Welton NJ, Ades AE, Carlin JB, Altman DG, Sterne JA: Models for potentially biased evidence in meta-analysis using empirically based priors. J R Stat Soc A 2009, 172(Part 1):119-136.
• [42]Xiong T, Parekh-Burke S, Loke YK, Abdelhamid A, Sutton AJ, Eastwood AJ, Holland R, Chen YF, Walsh T, Glenny AM, et al.: Assessment of trial similarity and evidence consistency for indirect treatment comparison: an empirical investigation. J Clin Epidemiol 2012. In press