Assessing discriminative ability of risk models in clustered data

BMC Medical Research Methodology

Yvonne Vergouwe²  Pablo Perel¹  Ewout W Steyerberg²  David van Klaveren² 

[1] Department of Population Health, London School of Hygiene and Tropical Medicine, Keppel Street, London WC1E 7HT, UK;Department of Public Health, Erasmus MC, Dr. Molewaterplein 50, Rotterdam 3015 GE, The Netherlands

关键词: Risk model;    Prediction;    Meta-analysis;    Discrimination;    Concordance;    Clustered data;   

Others  :  866503
DOI  :  10.1186/1471-2288-14-5

 received in 2013-10-07, accepted in 2014-01-08,  发布年份 2014

【 摘 要 】

Background

The discriminative ability of a risk model is often measured by Harrell’s concordance-index (c-index). The c-index estimates for two randomly chosen subjects the probability that the model predicts a higher risk for the subject with poorer outcome (concordance probability). When data are clustered, as in multicenter data, two types of concordance are distinguished: concordance in subjects from the same cluster (within-cluster concordance probability) and concordance in subjects from different clusters (between-cluster concordance probability). We argue that the within-cluster concordance probability is most relevant when a risk model supports decisions within clusters (e.g. who should be treated in a particular center). We aimed to explore different approaches to estimate the within-cluster concordance probability in clustered data.

Methods

We used data of the CRASH trial (2,081 patients clustered in 35 centers) to develop a risk model for mortality after traumatic brain injury. To assess the discriminative ability of the risk model within centers we first calculated cluster-specific c-indexes. We then pooled the cluster-specific c-indexes into a summary estimate with different meta-analytical techniques. We considered fixed effect meta-analysis with different weights (equal; inverse variance; number of subjects, events or pairs) and random effects meta-analysis. We reflected on pooling the estimates on the log-odds scale rather than the probability scale.

Results

The cluster-specific c-index varied substantially across centers (IQR = 0.70-0.81; I² = 0.76 with 95% confidence interval 0.66 to 0.82). Summary estimates resulting from fixed effect meta-analysis ranged from 0.75 (equal weights) to 0.84 (inverse variance weights). With random effects meta-analysis – accounting for the observed heterogeneity in c-indexes across clusters – we estimated a mean of 0.77, a between-cluster variance of 0.0072 and a 95% prediction interval of 0.60 to 0.95. The normality assumptions for derivation of a prediction interval were better met on the probability than on the log-odds scale.

Conclusion

When assessing the discriminative ability of risk models used to support decisions at cluster level we recommend meta-analysis of cluster-specific c-indexes. Particularly, random effects meta-analysis should be considered.

【 授权许可】

   
2014 van Klaveren et al.; licensee BioMed Central Ltd.

【 图 表 】

【 参考文献 】

• [1]Steyerberg EW, Vickers AJ, Cook NR, Gerds T, Gonen M, Obuchowski N, Pencina MJ, Kattan MW: Assessing the performance of prediction models: a framework for traditional and novel measures. Epidemiology 2010, 21(1):128-138.
• [2]Harrell FE Jr, Califf RM, Pryor DB, Lee KL, Rosati RA: Evaluating the yield of medical tests. JAMA 1982, 247(18):2543-2546.
• [3]Pencina MJ, D'Agostino RB: Overall C as a measure of discrimination in survival analysis: model specific population value and confidence interval estimation. Stat Med 2004, 23(13):2109-2123.
• [4]Bouwmeester W, Twisk JW, Kappen TH, van Klei WA, Moons KG, Vergouwe Y: Prediction models for clustered data: comparison of a random intercept and standard regression model. BMC Med Res Methodol 2013, 13:19. BioMed Central Full Text
• [5]Gelman A, Hill J: Data analysis using regression and multilevel/hierarchical models. Cambridge: Cambridge University Press; 2007.
• [6]Duchateau L, Janssen P: The Frailty Model. New York: Springer; 2008.
• [7]Van Oirbeek R, Lesaffre E: An application of Harrell's C-index to PH frailty models. Stat Med 2010, 29(30):3160-3171.
• [8]Van Oirbeek R, Lesaffre E: Assessing the predictive ability of a multilevel binary regression model. Comput Stat Data Anal 2012, 56(6):1966-1980.
• [9]Collaborators MCT, Perel P, Arango M, Clayton T, Edwards P, Komolafe E, Poccock S, Roberts I, Shakur H, Steyerberg E, et al.: Predicting outcome after traumatic brain injury: practical prognostic models based on large cohort of international patients. BMJ 2008, 336(7641):425-429.
• [10]Steyerberg EW, Mushkudiani N, Perel P, Butcher I, Lu J, McHugh GS, Murray GD, Marmarou A, Roberts I, Habbema JD, et al.: Predicting outcome after traumatic brain injury: development and international validation of prognostic scores based on admission characteristics. PLoS Med 2008, 5(8):e165.
• [11]Edwards P, Arango M, Balica L, Cottingham R, El-Sayed H, Farrell B, Fernandes J, Gogichaisvili T, Golden N, Hartzenberg B, et al.: Final results of MRC CRASH, a randomised placebo-controlled trial of intravenous corticosteroid in adults with head injury-outcomes at 6 months. Lancet 2005, 365(9475):1957-1959.
• [12]Teasdale G, Jennett B: Assessment of coma and impaired consciousness. A practical scale. Lancet 1974, 2(7872):81-84.
• [13]Jennett B, Bond M: Assessment of outcome after severe brain damage. Lancet 1975, 1(7905):480-484.
• [14]R Development Core Team: R: A Language and Environment for Statistical Computing. Vienna, Austria: R Foundation for Statistical Computing; 2011. ISBN 3-900051-07-0, URL http://www.R-project.org/
• [15]Therneau T, original Splus->R port by Lumley T: survival: Survival analysis, including penalised likelihood. R package version 2.36-9. 2011. http://CRAN.R-project.org/package=survival webcite
• [16]Quade D: Nonparametric partial correlation. Volume No. 526. North Carolina: Institute of Statistics Mimeo; 1967. Volume 526
• [17]Higgins JP, Thompson SG: Quantifying heterogeneity in a meta-analysis. Stat Med 2002, 21(11):1539-1558.
• [18]DerSimonian R, Laird N: Meta-analysis in clinical trials. Control Clin Trials 1986, 7(3):177-188.
• [19]DerSimonian R, Kacker R: Random-effects model for meta-analysis of clinical trials: an update. Contemp Clin Trials 2007, 28(2):105-114.
• [20]Higgins JPT, Thompson SG, Spiegelhalter DJ: A re-evaluation of random-effects meta-analysis. J R Soc Health Series A 2009, 172(1):137-159.
• [21]Riley RD, Higgins JPT, Deeks JJ: Interpretation of random effects meta-analyses. BMJ 2011, 342:d549.
• [22]Hardy RJ, Thompson SG: Detecting and describing heterogeneity in meta-analysis. Stat Med 1998, 17(8):841-856.
• [23]Lumley T: rmeta: Meta-analysis. R package version 2.16. 2009. http://CRAN.R-project.org/package=rmeta webcite
• [24]Vergouwe Y, Moons KG, Steyerberg EW: External validity of risk models: use of benchmark values to disentangle a case-mix effect from incorrect coefficients. Am J Epidemiol 2010, 172(8):971-980.
• [25]Steyerberg EW: Clinical Prediction Models: A Practical Approach to Development, Validation, and Updating. New York: Springer; 2009.
• [26]Gönen M, Heller G: Concordance probability and discriminatory power in proportional hazards regression. Biometrika 2005, 92(4):965-970.
• [27]Uno H, Cai T, Pencina MJ, D'Agostino RB, Wei LJ: On the C-statistics for evaluating overall adequacy of risk prediction procedures with censored survival data. Stat Med 2011, 30(10):1105-1117.
• [28]Hanley JA, McNeil BJ: The meaning and use of the area under a receiver operating characteristic (ROC) curve. Radiology 1982, 143(1):29-36.
• [29]DeLong ER, DeLong DM, Clarke-Pearson DL: Comparing the areas under two or more correlated receiver operating characteristic curves: a nonparametric approach. Biometrics 1988, 44(3):837-845.
• [30]Nam BH, D'Agostino RB: Discrimination Index, the Area under the ROC Curve. In Goodness-of-Fit Tests and Model Validity. Boston: Birkhauser; 2002:267-279.
• [31]Efron B, Tibshirani R: An Introduction to the Bootstrap. Boca Raton, FL: CRC press; 1993.