The Hartung-Knapp-Sidik-Jonkman method for random effects meta-analysis is straightforward and considerably outperforms the standard DerSimonian-Laird method

BMC Medical Research Methodology

George F Borm²  John PA Ioannidis¹  Joanna IntHout² 

[1] Department of Statistics, Stanford University School of Humanities and Sciences, Stanford, CA 94305, USA;Department for Health Evidence (HEV), Radboud University Medical Center, Huispost 133, P.O. box 9101, Nijmegen, HB 6500, The Netherlands

关键词: Cochrane Database of Systematic Reviews;    Random effects;    Type I error;    Heterogeneity;    Trial size;    Clinical trial;    Meta-analysis;   

Others  :  866434
DOI  :  10.1186/1471-2288-14-25

 received in 2013-11-04, accepted in 2014-01-06,  发布年份 2014

【 摘 要 】

Background

The DerSimonian and Laird approach (DL) is widely used for random effects meta-analysis, but this often results in inappropriate type I error rates. The method described by Hartung, Knapp, Sidik and Jonkman (HKSJ) is known to perform better when trials of similar size are combined. However evidence in realistic situations, where one trial might be much larger than the other trials, is lacking. We aimed to evaluate the relative performance of the DL and HKSJ methods when studies of different sizes are combined and to develop a simple method to convert DL results to HKSJ results.

Methods

We evaluated the performance of the HKSJ versus DL approach in simulated meta-analyses of 2–20 trials with varying sample sizes and between-study heterogeneity, and allowing trials to have various sizes, e.g. 25% of the trials being 10-times larger than the smaller trials. We also compared the number of “positive” (statistically significant at p < 0.05) findings using empirical data of recent meta-analyses with > = 3 studies of interventions from the Cochrane Database of Systematic Reviews.

Results

The simulations showed that the HKSJ method consistently resulted in more adequate error rates than the DL method. When the significance level was 5%, the HKSJ error rates at most doubled, whereas for DL they could be over 30%. DL, and, far less so, HKSJ had more inflated error rates when the combined studies had unequal sizes and between-study heterogeneity. The empirical data from 689 meta-analyses showed that 25.1% of the significant findings for the DL method were non-significant with the HKSJ method. DL results can be easily converted into HKSJ results.

Conclusions

Our simulations showed that the HKSJ method consistently results in more adequate error rates than the DL method, especially when the number of studies is small, and can easily be applied routinely in meta-analyses. Even with the HKSJ method, extra caution is needed when there are = <5 studies of very unequal sizes.

【 授权许可】

   
2014 IntHout et al.; licensee BioMed Central Ltd.

【 图 表 】

【 参考文献 】

• [1]DerSimonian R, Laird N: Meta-analysis in clinical trials. Control Clin Trials 1986, 7(3):177-188.
• [2]The Cochrane Collaboration: Review Manager (RevMan) 5.1.4. Copenhagen: The Nordic Cochrane Centre; 2011.
• [3]Borenstein M, Hedges L, Higgins J, Rothstein H: Comprehensive Meta-analysis Version 2. Englewood NJ: Biostat; 2005.
• [4]Hartung J: An alternative method for meta-analysis. Biom J 1999, 901-916.
• [5]Hartung J, Knapp G: A refined method for the meta analysis of controlled clinical trials with binary outcome. Stat Med 2001, 20(24):3875-3889.
• [6]Hartung J, Knapp G: On tests of the overall treatment effect in meta analysis with normally distributed responses. Stat Med 2001, 20(12):1771-1782.
• [7]Follmann DA, Proschan MA: Valid inference in random effects meta-analysis. Biometrics 1999, 55(3):732-737.
• [8]Hartung J, Makambi KH: Reducing the number of unjustified significant results in meta-analysis. Commun Stat Simul Comput 2003, 32(4):1179-1190.
• [9]Makambi KH: The effect of the heterogeneity variance estimator on some tests of treatment efficacy. J Biopharm Stat 2004, 14(2):439-449.
• [10]Sidik K, Jonkman JN: Robust variance estimation for random effects meta-analysis. Comput Stat Data Anal 2006, 50(12):3681-3701.
• [11]Sidik K, Jonkman JN: A simple confidence interval for meta-analysis. Stat Med 2002, 21(21):3153-3159.
• [12]Sidik K, Jonkman JN: On constructing confidence intervals for a standardized mean difference in meta-analysis. Commun Stat Simul Comput 2003, 32(4):1191-1203.
• [13]Sánchez-Meca J, Marín-Martínez F: Confidence intervals for the overall effect size in random-effects meta-analysis. Psychol Meth 2008, 13(1):31.
• [14]Sidik K, Jonkman JN: Simple heterogeneity variance estimation for meta analysis. J Roy Stat Soc 2005, 54(2):367-384.
• [15]Davey J, Turner R, Clarke M, Higgins J: Characteristics of meta-analyses and their component studies in the Cochrane Database of Systematic Reviews: a cross-sectional, descriptive analysis. BMC Med Res Methodol 2011, 11(1):160. BioMed Central Full Text
• [16]Higgins J, Thompson SG, Deeks JJ, Altman DG: Measuring inconsistency in meta-analyses. Bmj 2003, 327(7414):557.
• [17]Ioannidis JP, Patsopoulos NA, Evangelou E: Uncertainty in heterogeneity estimates in meta-analyses. Bmj 2007, 335(7626):914-916.
• [18]IntHout J, Ioannidis JP, Borm GF: Obtaining evidence by a single well-powered trial or several modestly powered trials. Stat Methods Med Res 2012. [Epub ahead of print]
• [19]Borm GF, Lemmers O, Fransen J, Donders R: The evidence provided by a single trial is less reliable than its statistical analysis suggests. J Clin Epidemiol 2009, 62(7):711-715. e711
• [20]Viechtbauer W: Conducting meta-analyses in R with the metafor package. J Stat Softw 2010, 36(3):1-48.
• [21]Harbord RM, Higgins JP: Meta-regression in Stata. Meta 2008, 8(4):493-519.
• [22]Higgins JPT, Thompson SG: Quantifying heterogeneity in a meta-analysis. Stat Med 2002, 21(11):1539-1558.
• [23]The Cochrane Collaboration. http://www.cochrane.org/cochrane-reviews webcite
• [24]Singh M, Das RR: Zinc for the common cold. Cochrane Database Syst Rev 2011, 2:CD001364.
• [25]Pidala J, Djulbegovic B, Anasetti C, Kharfan‒Dabaja M, Kumar A: Allogeneic hematopoietic cell transplantation for acute lymphoblastic leukemia (ALL) in first complete remission. Cochrane Library 2011, 10:CD008818.
• [26]Thorlund K, Wetterslev J, Awad T, Thabane L, Gluud C: Comparison of statistical inferences from the DerSimonian–Laird and alternative random-effects model meta-analyses – an empirical assessment of 920 Cochrane primary outcome meta-analyses. Res Synth Meth 2011, 2(4):238-253.
• [27]Kontopantelis E, Reeves D: Performance of statistical methods for meta-analysis when true study effects are non-normally distributed: a simulation study. Stat Methods Med Res 2012, 21(4):409-426.
• [28]Brockwell SE, Gordon IR: A simple method for inference on an overall effect in meta-analysis. Stat Med 2007, 26(25):4531-4543.
• [29]Hardy RJ, Thompson SG: A likelihood approach to meta-analysis with random effects. Stat Med 1996, 15(6):619-629.
• [30]Shuster JJ: Empirical vs natural weighting in random effects meta-analysis. Stat Med 2010, 29(12):1259-1265.
• [31]Henmi M, Copas JB: Confidence intervals for random effects meta analysis and robustness to publication bias. Stat Med 2010, 29(29):2969-2983.
• [32]Guolo A: Higher-order likelihood inference in meta-analysis and meta-regression. Stat Med 2012, 31(4):313-327.
• [33]Borenstein M, Hedges LV, Higgins JPT, Rothstein HR: Introduction to Meta-Analysis. Chichester, UK: Wiley; 2009.
• [34]Knapp G, Hartung J: Improved tests for a random effects meta-regression with a single covariate. Stat Med 2003, 22:2693-2710.