Prediction models for clustered data: comparison of a random intercept and standard regression model

BMC Medical Research Methodology

Yvonne Vergouwe³  Karel GM Moons³  Wilton A van Klei¹  Teus H Kappen¹  Jos WR Twisk²  Walter Bouwmeester³ 

[1]Department of Perioperative Care and Emergency Medicine, University Medical Center Utrecht, Utrecht, The Netherlands

[2]Department of Methodology and Applied Biostatistics, Institute of Health Sciences, Vrije Universiteit, Amsterdam, The Netherlands

[3]Julius Center for Health Sciences and Primary Care, University Medical Center Utrecht, Str. 6.131, PO Box 85500, Utrecht, GA 3508, The Netherlands

关键词: Validation;    Prediction model with random intercept;    Logistic regression analysis;   

Others  :  1126197
DOI  :  10.1186/1471-2288-13-19

 received in 2011-10-29, accepted in 2012-12-16,  发布年份 2013

【 摘 要 】

Background

When study data are clustered, standard regression analysis is considered inappropriate and analytical techniques for clustered data need to be used. For prediction research in which the interest of predictor effects is on the patient level, random effect regression models are probably preferred over standard regression analysis. It is well known that the random effect parameter estimates and the standard logistic regression parameter estimates are different. Here, we compared random effect and standard logistic regression models for their ability to provide accurate predictions.

Methods

Using an empirical study on 1642 surgical patients at risk of postoperative nausea and vomiting, who were treated by one of 19 anesthesiologists (clusters), we developed prognostic models either with standard or random intercept logistic regression. External validity of these models was assessed in new patients from other anesthesiologists. We supported our results with simulation studies using intra-class correlation coefficients (ICC) of 5%, 15%, or 30%. Standard performance measures and measures adapted for the clustered data structure were estimated.

Results

The model developed with random effect analysis showed better discrimination than the standard approach, if the cluster effects were used for risk prediction (standard c-index of 0.69 versus 0.66). In the external validation set, both models showed similar discrimination (standard c-index 0.68 versus 0.67). The simulation study confirmed these results. For datasets with a high ICC (≥15%), model calibration was only adequate in external subjects, if the used performance measure assumed the same data structure as the model development method: standard calibration measures showed good calibration for the standard developed model, calibration measures adapting the clustered data structure showed good calibration for the prediction model with random intercept.

Conclusion

The models with random intercept discriminate better than the standard model only if the cluster effect is used for predictions. The prediction model with random intercept had good calibration within clusters.

【 授权许可】

   
2013 Bouwmeester et al.; licensee BioMed Central Ltd.

【 图 表 】

Figure 1.

【 参考文献 】

• [1]Steyerberg EW: Clinical prediction models; a practical approach to development, validation, and updating. New York: Springer; 2009.
• [2]Bouwmeester W: Reporting and methods in clinical prediction research: a systematic review. In Prediction models: systematic reviews and clustered study data. Utrecht: Igitur archive; 2012.
• [3]Sullivan LM, Dukes KA, Losina E: Tutorial in biostatistics. An introduction to hierarchical linear modelling. Stat Med 1999, 18:855-888.
• [4]Goldstein H: Multilevel statistical models. London: Edward Arnold; 1995.
• [5]Moerbeek M, van Breukelen GJ, Berger MP: A comparison between traditional methods and multilevel regression for the analysis of multicenter intervention studies. J Clin Epidemiol 2003, 56:341-350.
• [6]Twisk JWR: Applied multilevel analysis. New York: Cambridge University Press; 2006.
• [7]Guo G, Zhao H: Multilevel modeling for binary data. Annu Rev Sociol 2000, 26:441-462.
• [8]Bardenheier BH, Shefer A, Barker L, Winston CA, Sionean CK: Public health application comparing multilevel analysis with logistic regression: immunization coverage among long-term care facility residents. Ann Epidemiol 2005, 15:749-755.
• [9]Liu I, Agresti A: The analysis of ordered categorical data: an overview and a survey of recent developments. Sociedad de Estadistica e Investigacion Operative Test 2005, 14:1-73.
• [10]Carriere I, Bouyer J: Choosing marginal or random-effects models for longitudinal binary responses: application to self-reported disability among older persons. BMC Med Res Methodol 2002, 2:15. BioMed Central Full Text
• [11]Apfel CC, Laara E, Koivuranta M, Greim CA, Roewer N: A simplified risk score for predicting postoperative nausea and vomiting: conclusions from cross-validations between two centers. Anesthesiology 1999, 91:693-700.
• [12]Kappen TH, Vergouwe Y, van Klei WA, van Wolfswinkel L, Kalkman CJ, Moons KG: Adaptation of clinical prediction models for application in local settings. Med Decis Making 2012, 32(3):E1-10.
• [13]Van den Bosch JE, Moons KG, Bonsel GJ, Kalkman CJ: Does measurement of preoperative anxiety have added value for predicting postoperative nausea and vomiting? Anesth Analg 2005, 100:1525-1532. Table
• [14]Goldstein H, Browne W, Rasbash J: Multilevel modelling of medical data. Stat Med 2002, 21:3291-3315.
• [15]Skrondal A, Rabe-Hesketh S: Prediction in multilevel generalized linear models. J R Stat Soc A Stat Soc 2009, 172:659-687.
• [16]Harrell FE Jr, Lee KL, Mark DB: Multivariable prognostic models: issues in developing models, evaluating assumptions and adequacy, and measuring and reducing errors. Stat Med 1996, 15:361-387.
• [17]Miller ME, Hui SL, Tierney WM: Validation techniques for logistic regression models. Stat Med 1991, 10:1213-1226.
• [18]Van Houwelingen HC, Thorogood J: Construction, validation and updating of a prognostic model for kidney graft survival. Stat Med 1995, 14:1999-2008.
• [19]Van OR, Lesaffre E: An application of Harrell's C-index to PH frailty models. Stat Med 2010, 29:3160-3171.
• [20]Senn S: Some controversies in planning and analysing multi-centre trials. Stat Med 1998, 17:1753-1765.
• [21]Scott AJ, Holt D: The effect of Two-stage sampling on ordinary least squares methods. J Am Stat Assoc 1982, 77:848-854.
• [22]R Development Core Team: R: a language and environment for statistical computing. Vienna, Austria: R Foundation for Statistical Computing; 2008. Ref Type: Computer
• [23]Bates D, Maechler M: lme4: Linear mixed-effects models using S4 classes. 2009. 14-12-2009. R CRAN Project. Accessible via http://lme4.r-forge.r-project.org/ webcite. Ref Type: Computer Program
• [24]Harrell F: Design. 2009. 18-9-2009. R CRAN Project. Accessible via http://biostat.mc.vanderbilt.edu/s/Design webcite. Ref Type: Computer Program
• [25]Vergouwe Y, Moons KG, Steyerberg EW: External validity of risk models: Use of benchmark values to disentangle a case-mix effect from incorrect coefficients. Am J Epidemiol 2010, 172:971-980.
• [26]Hedeker D, Gibbons R, Davis J: Random regression models for multicenter clinical trial data. Psychopharmacol Bull 1991, 27:73-77.
• [27]Turrell G, Sanders AE, Slade GD, Spencer AJ, Marcenes W: The independent contribution of neighborhood disadvantage and individual-level socioeconomic position to self-reported oral health: a multilevel analysis. Community Dent Oral Epidemiol 2007, 35:195-206.
• [28]Diez-Roux AV, Nieto FJ, Muntaner C, Tyroler HA, Comstock GW, Shahar E: Neighborhood environments and coronary heart disease: a multilevel analysis. Am J Epidemiol 1997, 146:48-63.