Developing and validating risk prediction models in an individual participant data meta-analysis

BMC Medical Research Methodology

Richard D Riley²  Karel GM Moons¹  Thomas PA Debray¹  Ikhlaaq Ahmed³ 

[1] Julius Center for Health Sciences and Primary Care, University Medical Center Utrecht, Utrecht, The Netherlands;School of Health and Population Sciences, Public Health Building, University of Birmingham, Edgbaston, Birmingham B15 2TT, UK;Midlands Hub for Trials Methodology Research, School of Health and Population Sciences, University of Birmingham, Edgbaston, Birmingham B15 2TT, UK

关键词: Reporting;    Review;    Individual participant (patient) data;    Prognosis;    Prognostic factor;    Meta-analysis;   

Others  :  866516
DOI  :  10.1186/1471-2288-14-3

 received in 2013-07-17, accepted in 2013-12-20,  发布年份 2014

【 摘 要 】

Background

Risk prediction models estimate the risk of developing future outcomes for individuals based on one or more underlying characteristics (predictors). We review how researchers develop and validate risk prediction models within an individual participant data (IPD) meta-analysis, in order to assess the feasibility and conduct of the approach.

Methods

A qualitative review of the aims, methodology, and reporting in 15 articles that developed a risk prediction model using IPD from multiple studies.

Results

The IPD approach offers many opportunities but methodological challenges exist, including: unavailability of requested IPD, missing patient data and predictors, and between-study heterogeneity in methods of measurement, outcome definitions and predictor effects. Most articles develop their model using IPD from all available studies and perform only an internal validation (on the same set of data). Ten of the 15 articles did not allow for any study differences in baseline risk (intercepts), potentially limiting their model’s applicability and performance in some populations. Only two articles used external validation (on different data), including a novel method which develops the model on all but one of the IPD studies, tests performance in the excluded study, and repeats by rotating the omitted study.

Conclusions

An IPD meta-analysis offers unique opportunities for risk prediction research. Researchers can make more of this by allowing separate model intercept terms for each study (population) to improve generalisability, and by using ‘internal-external cross-validation’ to simultaneously develop and validate their model. Methodological challenges can be reduced by prospectively planned collaborations that share IPD for risk prediction.

【 授权许可】

   
2014 Ahmed et al.; licensee BioMed Central Ltd.

【 图 表 】

【 参考文献 】

• [1]Steyerberg EW: Clinical prediction models: a practical approach to development, validation, and updating. New York: Springer; 2009.
• [2]Royston P, Moons KGM, Altman DG, Vergouwe Y: Prognosis and prognostic research: developing a prognostic model. BMJ 2009, 338(b604):1373-1377.
• [3]Steyerberg EW, Moons KG, van der Windt DA, Hayden JA, Perel P, Schroter S, Riley RD, Hemingway H, Altman DG: Prognosis research strategy (PROGRESS) 3: prognostic model research. PLoS Med 2013, 10(2):e1001381.
• [4]Riley RD, Hayden JA, Steyerberg EW, Moons KGM, Abrams KR, Kyzas PA, Malats N, Briggs A, Schroter S, Altman DG, et al.: Prognosis research strategy (PROGRESS) 2: prognostic factor research. PLoS Med 2013, 10(2):e1001380.
• [5]Noordzij JP, Lee SL, Bernet VJ, Payne RJ, Cohen SM, McLeod IK, Hier MP, Black MJ, Kerr PD, Richards ML, et al.: Early prediction of hypocalcemia after thyroidectomy using parathyroid hormone: an analysis of pooled individual patient data from nine observational studies. J Am Coll Surg 2007, 205(6):748-754.
• [6]Gail MH, Brinton LA, Byar DP, Corle DK, Green SB, Schairer C, Mulvihill JJ: Projecting individualized probabilities of developing breast cancer for white females who are being examined annually. J Natl Cancer Inst 1989, 81(24):1879-1886.
• [7]Gail MH, Benichou J: Validation studies on a model for breast cancer risk. J Natl Cancer Inst 1994, 86(8):573-575.
• [8]Haybittle JL, Blamey RW, Elston CW, Johnson J, Doyle PJ, Campbell FC, Nicholson RI, Griffiths K: A prognostic index in primary breast cancer. Br J Cancer 1982, 45(3):361-366.
• [9]Moons KG, Altman DG, Vergouwe Y, Royston P: Prognosis and prognostic research: application and impact of prognostic models in clinical practice. BMJ 2009, 338:b606.
• [10]Moons KG, Kengne AP, Grobbee DE, Royston P, Vergouwe Y, Altman DG, Woodward M: Risk prediction models: II. External validation, model updating, and impact assessment. Heart 2012, 98(9):691-698.
• [11]Altman DG, Vergouwe Y, Royston P, Moons KGM: Prognosis and prognostic research: validating a prognostic model. BMJ 2009, 338:b605.
• [12]Moons KG, Kengne AP, Woodward M, Royston P, Vergouwe Y, Altman DG, Grobbee DE: Risk prediction models: I. Development, internal validation, and assessing the incremental value of a new (bio)marker. Heart 2012, 98(9):683-690.
• [13]Harrell FE: Regression modeling strategies, with applications to linear models, logistic regression, and survival analysis. New York: Springer; 2001.
• [14]Bleeker SE, Moll HA, Steyerberg EW, Donders ART, Derksen-Lubsen G, Grobbee DE, Moons KGM: External validation is necessary in, prediction research: a clinical example. J Clin Epidemiol 2003, 56(9):826-832.
• [15]Justice AC, Covinsky KE, Berlin JA: Assessing the generalizability of prognostic information. Ann Intern Med 1999, 130(6):515-524.
• [16]Debray TP, Moons KG, Ahmed I, Koffijberg H, Riley RD: A framework for developing, implementing, and evaluating clinical prediction models in an individual participant data meta-analysis. Stat Med 2013, 32(18):3158-3180.
• [17]Royston P, Parmar MKB, Sylvester R: Construction and validation of a prognostic model across several studies, with an application in superficial bladder cancer. Stat Med 2004, 23:907-926.
• [18]Steyerberg EW, Mushkudiani N, Perel P, Butcher I, Lu J, McHugh GS, Murray GD, Marmarou A, Roberts I, Habbema JDF, et al.: Predicting outcome after traumatic brain injury: development and international validation of prognostic scores based on admission characteristics. PLoS Med 2008, 5(8):1251-1261.
• [19]Blak BT, Thompson M, Dattani H, Bourke A: Generalisability of The Health Improvement Network (THIN) database: demographics, chronic disease prevalence and mortality rates. Inform Prim Care 2011, 19(4):251-255.
• [20]Hippisley-Cox J, Coupland C, Vinogradova Y, Robson J, May M, Brindle P: Derivation and validation of QRISK, a new cardiovascular disease risk score for the United Kingdom: prospective open cohort study. BMJ 2007, 335(7611):136.
• [21]Riley RD, Lambert PC, Abo-Zaid G: Meta-analysis of individual participant data: rationale, conduct, and reporting. BMJ 2010, 340:c221.
• [22]Riley RD, Simmonds MC, Look MP: Evidence synthesis combining individual patient data and aggregate data: a systematic review identified current practice and possible methods. J Clin Epidemiol 2007, 60(5):431-439.
• [23]Asia Pacific Cohort Studies Collaboration: An evaluation of metabolic risks for coronary death in the Asia Pacific region. Diabetes Res Clin Pract 2006, 74(3):274-281.
• [24]Chau I, Norman AR, Cunningham D, Waters JS, Oates J, Ross PJ: Multivariate prognostic factor analysis in locally advanced and metastatic esophago-gastric cancer–pooled analysis from three multicenter, randomized, controlled trials using individual patient data. J Clin Oncol 2004, 22(12):2395-2403.
• [25]Fowkes G, Fowkes FGR, Murray GD, Butcher I, Heald CL, Lee RJ, Chambless LE, Folsom AR, Hirsch AT, Dramaix M, et al.: Ankle brachial index combined with Framingham risk score to predict cardiovascular events and mortality: a meta-analysis. JAMA 2008, 300(2):197-208.
• [26]Heffner JE, Nietert PJ, Barbieri C: Pleural fluid pH as a predictor of pleurodesis failure: analysis of primary data. Chest 2000, 117(1):87-95.
• [27]Horn J, Naylor AR, Laman DM, Chambers BR, Stork JL, Schroeder TV, Nielsen M-Y, Dunne VG, Ackerstaff RGA: Identification of patients at risk for ischaemic cerebral complications after carotid endarterectomy with TCD monitoring. Eur J Vasc Endovasc Surg 2005, 30(3):270-274.
• [28]Nieder C, Grosu AL, Andratschke NH, Molls M: Proposal of human spinal cord reirradiation dose based on collection of data from 40 patients. Int J Radiat Oncol Biol Phys 2005, 61(3):851-855.
• [29]Pagliaro L, Craxi A, Cammaa C, Tine F, Di Marco V, Lo Iacono O, Almasio P: Interferon-alpha for chronic hepatitis C: an analysis of pretreatment clinical predictors of response. Hepatology 1994, 19(4):820-828.
• [30]Raboud JM, Rae S, Montaner JS: Predicting HIV RNA virologic outcome at 52-weeks follow-up in antiretroviral clinical trials. The INCAS and AVANTI study groups. JAIDS 2000, 24(5):433-439.
• [31]Rovers MM, Glasziou P, Appelman CL, Burke P, McCormick DP, Damoiseaux RA, Little P, Le SN, Hoes AW: Predictors of pain and/or fever at 3 to 7 days for children with acute otitis media not treated initially with antibiotics: a meta-analysis of individual patient data. Pediatrics 2007, 119(3):579-585.
• [32]Schaich M, Schlenk RF, Al-Ali HK, Dohner H, Ganser A, Heil G, Illmer T, Krahl R, Krauter J, Sauerland C, et al.: Prognosis of acute myeloid leukemia patients up to 60 years of age exhibiting trisomy 8 within a non-complex karyotype: individual patient data-based meta-analysis of the German acute myeloid leukemia intergroup. Haematologica 2007, 92(6):763-770.
• [33]Sylvester RJ, Van Der Meijden APM, Oosterlinck W, Witjes JA, Bouffioux C, Denis L, Newling DWW, Kurth K: Predicting recurrence and progression in individual patients with stage Ta T1 bladder cancer using EORTC risk tables: a combined analysis of 2596 patients from seven EORTC trials. Eur Urol 2006, 49(3):466-475.
• [34]Terwee CB, Nieveen Van Dijkum EJ, Gouma DJ, Bakkevold KE, Klinkenbijl JH, Wade TP, van Wagensveld BA, Wong A, van der Meulen JH: Pooling of prognostic studies in cancer of the pancreatic head and periampullary region: the Triple-P study. Triple-P study group. Eur J Surg 2000, 166(9):706-712.
• [35]Yap YG, Duong T, Bland M, Malik M, Torp-Pedersen C, Kober L, Connolly SJ, Gallagher MM, Camm AJ: Potential demographic and baselines variables for risk stratification of high-risk post-myocardial infarction patients in the era of implantable cardioverter-defibrillator - a prognostic indicator. Int J Cardiol 2008, 126(1):101-107.
• [36]Abo-Zaid G, Guo B, Deeks JJ, Debray TPA, Steyerberg EW, Moons KGM, Riley RD: Individual participant data meta-analyses should not ignore clustering. J Clin Epidemiol 2013, 66(8):865-873.
• [37]Abo-Zaid G, Sauerbrei W, Riley RD: Individual participant data meta-analysis of prognostic factor studies: state of the art? BMC Med Res Methodol 2012, 12:56. BioMed Central Full Text
• [38]Mallett S, Royston P, Waters R, Dutton S, Altman DG: Reporting performance of prognostic models in cancer: a review. BMC Med 2010, 8:21. BioMed Central Full Text
• [39]Bouwmeester W, Zuithoff NP, Mallett S, Geerlings MI, Vergouwe Y, Steyerberg EW, Altman DG, Moons KG: Reporting and methods in clinical prediction research: a systematic review. PLoS Med 2012, 9(5):1-12.
• [40]Collins G: BMJ Group Blogs. 2011. http://blogs.bmj.com/bmj/2011/08/03/gary-collins-opening-up-multivariable-prediction-models/ webcite
• [41]Royston P, Parmar MKB: Flexible parametric proportional-hazards and proportional-odds models for censored survival data, with application to prognostic modelling and estimation of treatment effects. Stat Med 2002, 21:2175-2197.
• [42]Royston P, Lambert PC: Flexible Parametric Survival Analysis Using Stata: Beyond the Cox Model. College Station, TX: CRC Press; 2011.
• [43]Ahmed I, Sutton AJ, Riley RD: Assessment of publication bias, selection bias and unavailable data in meta-analyses using individual participant data: a database survey. BMJ 2012, 344:d7762.