External validation of a Cox prognostic model: principles and methods

BMC Medical Research Methodology

Douglas G Altman¹  Patrick Royston² 

[1] Centre for Statistics in Medicine, University of Oxford Wolfson College, Linton Road, Oxford OX2 6UD, UK;Hub for Trials Methodology Research, MRC Clinical Trials Unit and University College London, Aviation House 125, Kingsway country London, WC2B 6NH UK

关键词: Calibration;    Discrimination;    External validation;    Cox proportional hazards model;    Prognostic models;    Time to event data;   

Others  :  1126103
DOI  :  10.1186/1471-2288-13-33

 received in 2012-07-10, accepted in 2013-02-15,  发布年份 2013

【 摘 要 】

Background

A prognostic model should not enter clinical practice unless it has been demonstrated that it performs a useful role. External validation denotes evaluation of model performance in a sample independent of that used to develop the model. Unlike for logistic regression models, external validation of Cox models is sparsely treated in the literature. Successful validation of a model means achieving satisfactory discrimination and calibration (prediction accuracy) in the validation sample. Validating Cox models is not straightforward because event probabilities are estimated relative to an unspecified baseline function.

Methods

We describe statistical approaches to external validation of a published Cox model according to the level of published information, specifically (1) the prognostic index only, (2) the prognostic index together with Kaplan-Meier curves for risk groups, and (3) the first two plus the baseline survival curve (the estimated survival function at the mean prognostic index across the sample). The most challenging task, requiring level 3 information, is assessing calibration, for which we suggest a method of approximating the baseline survival function.

Results

We apply the methods to two comparable datasets in primary breast cancer, treating one as derivation and the other as validation sample. Results are presented for discrimination and calibration. We demonstrate plots of survival probabilities that can assist model evaluation.

Conclusions

Our validation methods are applicable to a wide range of prognostic studies and provide researchers with a toolkit for external validation of a published Cox model.

【 授权许可】

   
2013 Royston and Altman; licensee BioMed Central Ltd.

【 图 表 】

Figure 1.

Figure 2.

Figure 3.

Figure 4.

Figure 5.

Figure 6.

【 参考文献 】

• [1]Altman DG, Royston P: What do we mean by validating a prognostic model? Stat Med 2000, 19:453-473.
• [2]Moons KGM, Royston P, Vergouwe Y, Altman DG: Prognosis and prognostic research: what, why, and how? Br Med J 2009, 338:b375.
• [3]Moons KGM, Altman DG, Vergouwe Y, Royston P: Prognosis and prognostic research: Application and impact of prognostic models in clinical practice. Br Med J 2009, 338:b606.
• [4]Miller ME, Hui SL: Validation techniques for logistic regression models. Stat Med 1991, 10:1213-1226.
• [5]Hosmer DW, Lemeshow S: Applied Logistic Regression. New York: Wiley; 2000.
• [6]Harrell FE: Regression Modeling Strategies, with Applications to Linear Models, Logistic Regression, and Survival Analysis. New York: Springer; 2001.
• [7]Steyerberg EW: Clinical Prediction Models. Heidelberg: Springer; 2009.
• [8]Vergouwe Y, Steyerberg EW, Eijkemans MJC, Habbema JDF: Substantial effective sample sizes were required for external validation studies of predictive logistic regression models. J Clin Epidemiol 2005, 58:475-483.
• [9]Altman DG, Vergouwe Y, Royston P, Moons KGM: Prognosis and prognostic research: validating a prognostic model. Brit Med J 2009, 338:b605.
• [10]Feinstein AR: Multivariable Analysis. New Haven: Yale University Press; 1996.
• [11]Justice AC, Covinsky KE, Berlin JA: Assessing the generalizability of prognostic information. Ann Intern Med 1999, 130:515-524.
• [12]van Houwelingen H C: Validation, calibration, revision and combination of prognostic survival models. Stat Med 2000, 19:3401-3415.
• [13]Burton A, Altman DG: Missing covariate data within cancer prognostic studies: a review of current reporting and proposed guidelines. Brit J Cancer 2004, 91:4-8.
• [14]Mallett S, Royston P, Dutton S, Waters R, Altman DG: Reporting methods in studies developing prognostic models in cancer: a review. BMC Med 2010, 8:20. BioMed Central Full Text
• [15]Royston P, Lambert PC: Flexible Parametric Survival Analysis Using Stata: Beyond the Cox model. StataPress: College Station; 2011.
• [16]Foekens J, Peters H, Look M, Portengen H, Schmitt M, Kramer M, Brunner N, Jänicke F, Meijer-van Gelder M, Henzen-Logmans S, van Putten W, Klijn J: The urokinase system of plasminogen activation and prognosis in 2780 breast cancer patients. Cancer Res 2000, 60:636-643.
• [17]Valsecchi MG, Miller ME, Hui SL: Evaluation of long-term survival: use of diagnostic and robust estimators with Cox’s proportional hazards model. Stat Med 1996, 15:2763-2780.
• [18]Schumacher M, Bastert G, Bojar H, Hübner K, Olschweski M, Sauerbrei W, Schmoor C, Beyerle C, Neumann RLA, Rauschecker HF: Randomized 2×2 trial evaluating hormonal treatment and the duration of chemotherapy in node-positive breast cancer patients. J Clin Oncol 1994, 12:2086-2093.
• [19]Royston P, Sauerbrei W: Multivariable Model-Building: A Pragmatic Approach to Regression Analysis Based on Fractional Polynomials for Modelling Continuous Variables. Chichester: Wiley; 2008.
• [20]Durrleman S, Simon R: Flexible regression-models with cubic-splines. Stat Med 1989, 8:551-561.
• [21]Royston P, Altman DG, Sauerbrei W: Dichotomizing continuous predictors in multiple regression: a bad idea. Stat Med 2006, 25:127-141.
• [22]Mallett S, Royston P, Waters R, Dutton S, Altman DG: Reporting performance of prognostic models in cancer: a review. BMC Med 2010, 8:21. BioMed Central Full Text
• [23]Teasdale G, Jennett B: Assessment of coma and impaired consciousness. A practical scale. Lancet 1974, 304:81-84.
• [24]Anyanwu AC, Rogers CA, Murday AJ: A simple approach to risk stratification in adult heart disease. Eur J Cardiothorac Surg 1999, 16:424-428.
• [25]Kent JT, O’Quigley J: Measures of dependence for censored survival data. Biometrika 1988, 75:525-534.
• [26]Royston P, Sauerbrei W: A new measure of prognostic separation in survival data. Stat Med 2004, 23:723-748.
• [27]Kalbfleisch JD, Prentice RL: The Statistical Analysis of Failure Time Data. New York: Wiley; 2002.
• [28]StataCorp Stata Release 12. Stata Press; 2011.
• [29]Altman DG: Prognostic models: a methodological framework and review of models for breast cancer. Cancer Invest 2009, 27:235-243.
• [30]Cox DR: Note on grouping. J Am Stat Assoc 1957, 52:543-547.
• [31]Vergouwe Y, Moons KGM, Steyerberg EW: External validity of risk models: use of benchmark values to disentangle a case-mix effect from incorrect coefficients. Am J Epidemiol 2010, 172:971-980.
• [32]Choodari-Oskooei B, Royston P, Parmar MKB, A simulation study of predictive ability measures in a survival model I: explained variation measures. Stat Med 2012, 31:2627-2643.
• [33]Hielscher T, Zucknick M, Werft W, Benner A: On the prognostic value of survival models with application to gene expression signatures. Stat Med 2010, 29:818-829.
• [34]Harrell FE, Califf RM, Prior DB, Lee KL, Rosati RA: Evaluating the yield of medical tests. J Am Med Assoc 1982, 247:2543-2546.
• [35]Gönen M, Heller G: Concordance probability and discriminatory power in proportional hazards regression. Biometrika 2005, 92:965-970.
• [36]Graf E, Schmoor C, Sauerbrei W, Schumacher M: Assessment and comparison of prognostic classification schemes for survival data. Stat Med 1999, 18:2529-2545.
• [37]Zheng Y, Cai T, Pepe MS, Levy WC: Time-dependent predictive values of prognostic biomarkers with failure time outcome. J Am Stat Assoc 2008, 103:362-368.
• [38]Bland MJ, Altman DG: Statistical methods for assessing agreement between two methods of clinical measurement. Lancet 1986, 8:307-310.
• [39]Harrell FE: rms:S functions for biostatistical/epidemiologic modeling, testing, estimation, validation, graphics, prediction, and typesetting by storing enhanced model design attributes in the fit. Implements methods in Regression Modeling Strategies. New York: Springer; 2001. Available from [http://biostat.mc.vanderbilt.edu/rms webcite] 2013
• [40]Henderson R, Keiding N: Individual survival time prediction using statistical models. J Med Ethics 2005, 31:703-706.
• [41]Janssen KJ, Moons KG, Kalkman CJ, Grobbee DE, Vergouwe Y: Updating methods improved the performance of a clinical prediction model in new patients. J Clin Epidemiol 2008, 61:76-86.
• [42]Ivanov J, Tu JV, Naylor C: Ready-made, recalibrated, or remodeled? Issues in the use of risk indexes for assessing mortality after coronary artery bypass graft surgery. Circulation 1999, 99:2098-2104.
• [43]Jinks RC: Sample size for multivariable prognostic models. PhD thesis. London: University College; 2012.
• [44]Dunkler D, Michiels S, Schemper M: Gene expression profiling: Does it add predictive accuracy to clinical characteristics in cancer prognosis? Eur J Cancer 2007, 43:745-751.
• [45]Royston P, Parmar MKB: Flexible proportional-hazards and proportional-odds models for censored survival data, with application to prognostic modelling and estimation of treatment effects. Stat Med 2002, 21:2175-2197.