An assessment of the relationship between clinical utility and predictive ability measures and the impact of mean risk in the population

BMC Medical Research Methodology

Patrick MM Bossuyt¹  Les Irwig³  Petra Macaskill³  Kevin McGeechan² 

[1] Department of Clinical Epidemiology, Biostatistics and Bioinformatics, Academic Medical Centre (AMC), University of Amsterdam, Amsterdam, The Netherlands;Sydney School of Public Health, The University of Sydney, Sydney, Australia;The Screening and Test Evaluation Program, The University of Sydney, Sydney, Australia

关键词: Prediction;    Risk assessment;    Event free life years (EFLY);    Net benefit;    Area under curve (AUC);    Net reclassification improvement (NRI);    Biomarkers;   

Others  :  865376
DOI  :  10.1186/1471-2288-14-86

 received in 2013-11-11, accepted in 2014-06-26,  发布年份 2014

【 摘 要 】

Background

Measures of clinical utility (net benefit and event free life years) have been recommended in the assessment of a new predictor in a risk prediction model. However, it is not clear how they relate to the measures of predictive ability and reclassification, such as the c-statistic and Net Reclassification Improvement (NRI), or how these measures are affected by differences in mean risk between populations when a fixed cutpoint to define high risk is assumed.

Methods

We examined the relationship between measures of clinical utility (net benefit, event free life years) and predictive ability (c-statistic, binary c-statistic, continuous NRI(0), NRI with two cutpoints, binary NRI) using simulated data and the Framingham dataset.

Results

In the analysis of simulated data, the addition of a new predictor tended to result in more people being treated when the mean risk was less than the cutpoint, and fewer people being treated for mean risks beyond the cutpoint. The reclassification and clinical utility measures showed similar relationships with mean risk when the mean risk was less than the cutpoint and the baseline model was not strong. However, when the mean risk was greater than the cutpoint, or the baseline model was strong, the reclassification and clinical utility measures diverged in their relationship with mean risk.

Although the risk of CVD was lower for women compared to men in the Framingham dataset, the measures of predictive ability, reclassification and clinical utility were both larger for women. The difference in these results was, in part, due to the larger hazard ratio associated with the additional risk predictor (systolic blood pressure) for women.

Conclusion

Measures such as the c-statistic and the measures of reclassification do not capture the consequences of implementing different prediction models. We do not recommend their use in evaluating which new predictors may be clinically useful in a particular population. We recommend that a measure such as net benefit or EFLY is calculated and, where appropriate, the measure is weighted to account for differences in the distribution of risks between the study population and the population in which the new predictors will be implemented.

【 授权许可】

   
2014 McGeechan et al.; licensee BioMed Central Ltd.

【 图 表 】

【 参考文献 】

• [1]Moons KG, Royston P, Vergouwe Y, Grobbee DE, Altman DG: Prognosis and prognostic research: what, why, and how? BMJ 2009, 338:b375.
• [2]Kannel WB, D'Agostino RB, Sullivan L, Wilson PW: Concept and usefulness of cardiovascular risk profiles. Am Heart J 2004, 148(1):16-26.
• [3]Gail MH, Brinton LA, Byar DP, Corle DK, Green SB, Schairer C, Mulvihill JJ: Projecting individualized probabilities of developing breast cancer for white females who are being examined annually. J Natl Cancer Inst 1989, 81(24):1879-1886.
• [4]Helfand M, Buckley DI, Freeman M, Fu R, Rogers K, Fleming C, Humphrey LL: Emerging risk factors for coronary heart disease: a summary of systematic reviews conducted for the U.S. Preventive Services Task Force. Ann Intern Med 2009, 151(7):496-507.
• [5]Pencina MJ, D'Agostino RB, Pencina KM, Janssens AC, Greenland P: Interpreting incremental value of markers added to risk prediction models. Am J Epidemiol 2012, 176(6):473-481.
• [6]Rapsomaniki E, White IR, Wood AM, Thompson SG: A framework for quantifying net benefits of alternative prognostic models. Stat Med 2012, 31(2):114-130.
• [7]Cook NR, Paynter NP: Performance of reclassification statistics in comparing risk prediction models. Biom J 2011, 53(2):237-258.
• [8]Pepe MS: Problems with risk reclassification methods for evaluating prediction models. Am J Epidemiol 2011, 173(11):1327-1335.
• [9]Vickers AJ, Elkin EB: Decision curve analysis: a novel method for evaluating prediction models. Med Decis Making 2006, 26(6):565-574.
• [10]Hlatky MA, Hlatky MA, Greenland P, Arnett DK, Ballantyne CM, Criqui MH, Elkind MS, Go AS, Harrell FE Jr, Hong Y, Howard BV, Howard VJ, Hsue PY, Kramer CM, McConnell JP, Normand SL, O'Donnell CJ, Smith SC Jr, Wilson PW: Criteria for evaluation of novel markers of cardiovascular risk: a scientific statement from the American Heart Association. Circulation 2009, 119(17):2408-2416.
• [11]Steyerberg EW, Pencina MJ, Lingsma HF, Kattan MW, Vickers AJ, Van Calster B: Assessing the incremental value of diagnostic and prognostic markers: a review and illustration. Eur J Clin Invest 2012, 42(2):216-228.
• [12]Leening MJ, Vedder MM, Witteman JC, Pencina MJ, Steyerberg EW: Net reclassification improvement: computation, interpretation, and controversies: a literature review and clinician's guide. Ann Intern Med 2014, 160(2):122-131.
• [13]Cook NR: Use and misuse of the receiver operating characteristic curve in risk prediction. Circulation 2007, 115(7):928-935.
• [14]Mihaescu R, van Zitteren M, van Hoek M, Sijbrands EJ, Uitterlinden AG, Witteman JC, Hofman A, Hunink MG, van Duijn CM, Janssens AC: Improvement of risk prediction by genomic profiling: reclassification measures versus the area under the receiver operating characteristic curve. Am J Epidemiol 2010, 172(3):353-361.
• [15]Greenland S: The need for reorientation toward cost-effective prediction: comments on 'evaluating the added predictive ability of a new marker: from area under the ROC curve to reclassification and beyond' by M. J. Pencina et al., Statistics in Medicine (DOI: 10.1002/sim.2929). Stat Med 2008, 27(2):199-206.
• [16]Van Calster B, Steyerberg EW, D'Agostino RB Sr, Pencina MJ: Sensitivity and specificity can change in opposite directions when new predictive markers are added to risk models. Med Decis Making 2014, 34(4):513-522.
• [17]JBS 2: Joint British Societies' guidelines on prevention of cardiovascular disease in clinical practice. Heart 2005, 91(Suppl 5):v1-v52.
• [18]National Heart Foundation of New Zealand, Stroke Foundation of New Zealand, New Zealand Ministry of Health, New Zealand Guidelines Group: The assessment and management of cardiovascular risk. Wellington, NZ: New Zealand Guidelines Group; 2003:190.
• [19]National Cholesterol Education Program (U.S.). Expert Panel on Detection Evaluation and Treatment of High Blood Cholesterol in Adults: Third report of the National Cholesterol Education Program (NCEP) Expert Panel on Detection, Evaluation, and Treatment of High Blood Cholesterol in Adults (adult treatment panel III): final report. NIH publication; no. 02–52152002. Bethesda, Md: National Cholesterol Education Program, National Heart, Lung, and Blood Institute, National Institutes of Health. 1 v. (various pagings); 2002.
• [20]Graham I, Atar D, Borch-Johnsen K, Boysen G, Burell G, Cifkova R, Dallongeville J, De Backer G, Ebrahim S, Gjelsvik B, Herrmann-Lingen C, Hoes A, Humphries S, Knapton M, Perk J, Priori SG, Pyorala K, Reiner Z, Ruilope L, Sans-Menendez S, Op Reimer WS, Weissberg P, Wood D, Yarnell J, Zamorano JL: European guidelines on cardiovascular disease prevention in clinical practice: executive summary. Atherosclerosis 2007, 194(1):1-45.
• [21]Danesh J, Erqou S, Walker M, Thompson SG, Tipping R, Ford C, Pressel S, Walldius G, Jungner I, Folsom AR, Chambless LE, Knuiman M, Whincup PH, Wannamethee SG, Morris RW, Willeit J, Kiechl S, Santer P, Mayr A, Wald N, Ebrahim S, Lawlor DA, Yarnell JW, Gallacher J, Casiglia E, Tikhonoff V, Nietert PJ, Sutherland SE, Bachman DL, Keil JE: The emerging risk factors collaboration: analysis of individual data on lipid, inflammatory and other markers in over 1.1 million participants in 104 prospective studies of cardiovascular diseases. Eur J Epidemiol 2007, 22(12):839-869.
• [22]D'Agostino RB Sr, Vasan RS, Pencina MJ, Wolf PA, Cobain M, Massaro JM, Kannel WB: General cardiovascular risk profile for use in primary care: the Framingham Heart Study. Circulation 2008, 117(6):743-753.
• [23]Harrell FE Jr, Lee KL, Mark DB: Multivariable prognostic models: issues in developing models, evaluating assumptions and adequacy, and measuring and reducing errors. Stat Med 1996, 15(4):361-387.
• [24]Pencina MJ, D'Agostino RB Sr, Steyerberg EW: Extensions of net reclassification improvement calculations to measure usefulness of new biomarkers. Stat Med 2011, 30(1):11-21.
• [25]Steyerberg EW, Pencina MJ: Reclassification calculations for persons with incomplete follow-up. Ann Intern Med 2010, 152(3):195-196. author reply 196–7
• [26]Vickers AJ, Cronin AM, Elkin EB, Gonen M: Extensions to decision curve analysis, a novel method for evaluating diagnostic tests, prediction models and molecular markers. BMC Med Inform Decis Mak 2008, 8:53.
• [27]Baigent C, Keech A, Kearney PM, Blackwell L, Buck G, Pollicino C, Kirby A, Sourjina T, Peto R, Collins R, Simes R: Efficacy and safety of cholesterol-lowering treatment: prospective meta-analysis of data from 90,056 participants in 14 randomised trials of statins. Lancet 2005, 366(9493):1267-1278.
• [28]National Institute for Health and Clinical Excellence: Social Value Judgements: Principles for the Development of NICE Guidance, 2nd edition. London: National Institute for Health and Clinical Excellence. 2008. Available from: http://www.nice.org.uk/Media/Default/About/what-we-do/Research-and-development/Social-Value-Judgements-principles-for-the-development-of-NICE-guidance.pdf webcite. Accessed 8 July 2014
• [29]Bender R, Augustin T, Blettner M: Generating survival times to simulate Cox proportional hazards models. Stat Med 2005, 24(11):1713-1723.
• [30]Rothman KJ, Greenland S, Lash TL: Modern epidemiology. 3rd edition. Philadelphia: Wolters Kluwer Health/Lippincott Williams & Wilkins; 2008:758.
• [31]Paynter NP, Everett BM, Cook NR: Cardiovascular disease risk prediction in women: is there a role for novel biomarkers? Clin Chem 2014, 60(1):88-97.
• [32]Baker SG, Cook NR, Vickers A, Kramer BS: Using relative utility curves to evaluate risk prediction. J R Stat Soc Ser A Stat Soc 2009, 172(4):729-748.
• [33]Van Calster B, Vickers AJ, Pencina MJ, Baker SG, Timmerman D, Steyerberg EW: Evaluation of markers and risk prediction models: overview of relationships between NRI and decision-analytic measures. Med Decis Making 2013, 33(4):490-501.
• [34]Pepe MS, Fan J, Seymour CW: Estimating the receiver operating characteristic curve in studies that match controls to cases on covariates. Acad Radiol 2013, 20(7):863-873.
• [35]Di Angelantonio E, Gao P, Pennells L, Kaptoge S, Caslake M, Thompson A, Butterworth AS, Sarwar N, Wormser D, Saleheen D, Ballantyne CM, Psaty BM, Sundstrom J, Ridker PM, Nagel D, Gillum RF, Ford I, Ducimetiere P, Kiechl S, Koenig W, Dullaart RP, Assmann G, D'Agostino RB Sr, Dagenais GR, Cooper JA, Kromhout D, Onat A, Tipping RW, Gomez-de-la-Camara A, Rosengren A: Lipid-related markers and cardiovascular disease prediction. JAMA 2012, 307(23):2499-2506.
• [36]Kaptoge S, Di Angelantonio E, Pennells L, Wood AM, White IR, Gao P, Walker M, Thompson A, Sarwar N, Caslake M, Butterworth AS, Amouyel P, Assmann G, Bakker SJ, Barr EL, Barrett-Connor E, Benjamin EJ, Bjorkelund C, Brenner H, Brunner E, Clarke R, Cooper JA, Cremer P, Cushman M, Dagenais GR, D'Agostino RB Sr, Dankner R, Davey-Smith G, Deeg D, Dekker JM: C-reactive protein, fibrinogen, and cardiovascular disease prediction. N Engl J Med 2012, 367(14):1310-1320.
• [37]Rousson V, Zumbrunn T: Decision curve analysis revisited: overall net benefit, relationships to ROC curve analysis, and application to case–control studies. BMC Med Inform Decis Mak 2011, 11:45.