Formal and informal prediction of recurrent stroke and myocardial infarction after stroke: a systematic review and evaluation of clinical prediction models in a new cohort

BMC Medicine

William N Whiteley²  Cathie LM Sudlow²  Martin Dennis²  Gordon D Murray¹  Douglas D Thompson¹ 

[1] Edinburgh MRC Hub for Trials Methodology Research, Centre for Population Health Sciences, University of Edinburgh Medical School, Teviot Place, Edinburgh EH8 9AG, UK;Division of Clinical Neurosciences, University of Edinburgh, Bramwell Dott Building, Western General Hospital, Edinburgh EH4 2XU, UK

关键词: Development;    Evaluation;    Statistical modelling;    Prediction;    Stroke;    Meta-analysis;    Systematic review;   

Others  :  854923
DOI  :  10.1186/1741-7015-12-58

 received in 2013-11-21, accepted in 2014-02-28,  发布年份 2014

【 摘 要 】

Background

The objective of this study was to: (1) systematically review the reporting and methods used in the development of clinical prediction models for recurrent stroke or myocardial infarction (MI) after ischemic stroke; (2) to meta-analyze their external performance; and (3) to compare clinical prediction models to informal clinicians’ prediction in the Edinburgh Stroke Study (ESS).

Methods

We searched Medline, EMBASE, reference lists and forward citations of relevant articles from 1980 to 19 April 2013. We included articles which developed multivariable clinical prediction models for the prediction of recurrent stroke and/or MI following ischemic stroke. We extracted information to assess aspects of model development as well as metrics of performance to determine predictive ability. Model quality was assessed against a pre-defined set of criteria. We used random-effects meta-analysis to pool performance metrics.

Results

We identified twelve model development studies and eleven evaluation studies. Investigators often did not report effective sample size, regression coefficients, handling of missing data; typically categorized continuous predictors; and used data dependent methods to build models. A meta-analysis of the area under the receiver operating characteristic curve (AUROCC) was possible for the Essen Stroke Risk Score (ESRS) and for the Stroke Prognosis Instrument II (SPI-II); the pooled AUROCCs were 0.60 (95% CI 0.59 to 0.62) and 0.62 (95% CI 0.60 to 0.64), respectively. An evaluation among minor stroke patients in the ESS demonstrated that clinicians discriminated poorly between those with and those without recurrent events and that this was similar to clinical prediction models.

Conclusions

The available models for recurrent stroke discriminate poorly between patients with and without a recurrent stroke or MI after stroke. Models had a similar discrimination to informal clinicians' predictions. Formal prediction may be improved by addressing commonly encountered methodological problems.

【 授权许可】

   
2014 Thompson et al.; licensee BioMed Central Ltd.

【 图 表 】

【 参考文献 】

• [1]Mohan KM, Wolfe CD, Rudd AG, Heuschmann PU, Kolominsky-Rabas PL, Grieve AP: Risk and cumulative risk of stroke recurrence. Stroke 2011, 42:1489-1494.
• [2]Woodward M, Brindle P, Tunstall-Pedoe H: Adding social deprivation and family history to cardiovascular risk assessment: the ASSIGN score from the Scottish Heart Health Extended Cohort (SHHEC). Heart 2007, 93:172-176.
• [3]Anderson KM, Odell PM, Wilson PW, Kannel WB: Cardiovascular disease risk profiles. Am Heart J 1991, 121:293-298.
• [4]Gage BF, Waterman AD, Shannon W, Boechler M, Rich MW, Radford MJ: Validation of clinical classification schemes for predicting stroke: Results from the national registry of atrial fibrillation. JAMA 2001, 285:2864-2870.
• [5]Whiteley W, Murray G, Thompson D: The performance of multivariate prognostic models to predict occlusive and haemorrhagic vascular events after stroke; a systematic review. Available at: http://www.dcn.ed.ac.uk/dcn/documents/profile_protocols/whiteley_p2.pdf webcite
• [6]Collins GS, Mallett S, Omar O, Yu LM: Developing risk prediction models for type 2 diabetes: a systematic review of methodology and reporting. BMC Med 2011, 9:103. BioMed Central Full Text
• [7]Counsell C, Dennis M: Systematic review of prognostic models in patients with acute stroke. Cerebrovasc Dis 2001, 12:159-170.
• [8]Google Scholar http://scholar.google.co.uk/ webcite
• [9]Hayden JA, Côté P, Bombardier C: Evaluation of the quality of prognosis studies in systematic reviews. Ann Intern Med 2006, 144:427-437.
• [10]Mallett S, Royston P, Dutton S, Waters R, Altman D: Reporting methods in studies developing prognostic models in cancer: a review. BMC Med 2010, 8:20. BioMed Central Full Text
• [11]Mushkudiani NA, Hukkelhoven CW, Hernández AV, Murray GD, Choi SC, Maas AI, Steyerberg EW: A systematic review finds methodological improvements necessary for prognostic models in determining traumatic brain injury outcomes. J Clin Epidemiol 2008, 61:331-343.
• [12]Kwakkel G, Wagenaar RC, Kollen BJ, Lankhorst GJ: Predicting disability in stroke—a critical review of the literature. Age Ageing 1996, 1996:479-489.
• [13]Laupacis A, Sekar N, Stiell LG: Clinical prediction rules. JAMA 1997, 277:488-494.
• [14]Moons KG, Royston P, Vergouwe Y, Grobbee DE, Altman DG: Prognosis and prognostic research: what, why, and how? BMJ 2009, 338:b375.
• [15]Clark TG, Altman DG, Stavola BL: Quantification of the completeness of follow-up. Lancet 2002, 359:1309-1310.
• [16]Rubin DB: Multiple imputation after 18+ years. J Am Stat Assoc 1996, 91:473-489.
• [17]Vergouwe Y, Royston P, Moons KG, Altman DG: Development and validation of a prediction model with missing predictor data: a practical approach. J Clin Epidemiol 2010, 63:205-214.
• [18]Harrell FE: Regression Modeling Strategies: With Applications to Linear Models, Logistic Regression, and Survival Analysis. Springer; 2001.
• [19]Sun GW, Shook TL, Kay GL: Inappropriate use of bivariable analysis to screen risk factors for use in multivariable analysis. J Clin Epidemiol 1996, 49:907-916.
• [20]Steyerberg EW, Eijkemans MJ, Habbema JD: Stepwise selection in small data sets: a simulation study of bias in logistic regression analysis. J Clin Epidemiol 1999, 52:935-942.
• [21]Altman DG: Problems in dichotomizing continuous variables. Am J Epidemiol 1994, 139:442-445.
• [22]Peduzzi P, Concato J, Kemper E, Holford TR, Feinstein AR: A simulation study of the number of events per variable in logistic regression analysis. J Clin Epidemiol 1996, 49:1373-1379.
• [23]Steyerberg EW, Vickers AJ, Cook NR, Gerds T, Gonen M, Obuchowski N, Pencina MJ, Kattan MW: Assessing the performance of prediction models: a framework for traditional and novel measures. Epidemiology 2010, 21:128-138.
• [24]DerSimonian R, Laird N: Meta-analysis in clinical trials. Control Clin Trials 1986, 7:177-188.
• [25]Viechtbauer W: Conducting meta-analyses in R with the metafor package. J Stat Softw 2010, 36:1-48.
• [26]Riley RD, Higgins JPT, Deeks JJ: Interpretation of random effects meta-analyses. BMJ 2011, 342:d549.
• [27]Peters JL, Sutton AJ, Jones DR, Abrams KR, Rushton L: Contour-enhanced meta-analysis funnel plots help distinguish publication bias from other causes of asymmetry. J Clin Epidemiol 2008, 61:991-996.
• [28]Whiteley W, Jackson C, Lewis S, Lowe G, Rumley A, Sandercock P, Wardlaw J, Dennis M, Sudlow C: Inflammatory markers and poor outcome after stroke: a prospective cohort study and systematic review of interleukin-6. PLoS Med 2009, 6:e1000145.
• [29]Alvarez-Sabin J, Quintana M, Rodriguez M, Arboix A, Ramirez JM, Fuentes B: Validation of the Essen risk scale and its adaptation to the Spanish population, Modified Essen risk scale. Neurologia 2008, 23:209-214.
• [30]Ay H, Gungor L, Arsava EM, Rosand J, Vangel M, Benner T, Schwamm LH, Furie KL, Koroshetz WJ, Sorensen AG: A score to predict early risk of recurrence after ischemic stroke. Neurology 2010, 74:128-135.
• [31]Kamouchi M, Kumagai N, Okada Y, Origasa H, Yamaguchi T, Kitazono T: Risk score for predicting recurrence in patients with ischemic stroke: the Fukuoka Stroke Risk Score for Japanese. Cerebrovasc Dis 2012, 34:351-357.
• [32]Kernan WN, Horwitz RI, Brass LM, Viscoli CM, Taylor KJ: A prognostic system for transient ischemia or minor stroke. Ann Intern Med 1991, 114:552-557.
• [33]van Wijk I, Kappelle LJ, van Gijn J, Koudstaal PJ, Franke CL, Vermeulen M, Gorter JW, Algra A: Long-term survival and vascular event risk after transient ischaemic attack or minor ischaemic stroke: a cohort study. Lancet 2005, 365:2098-2104.
• [34]Pezzini A, Grassi M, Del Zotto E, Lodigiani C, Ferrazzi P, Spalloni A, Patella R, Giossi A, Volonghi I, Iacoviello L, Magoni M, Rota LL, Rasura M, Padovani A: Common genetic markers and prediction of recurrent events after ischemic stroke in young adults. Neurology 2009, 73:717-723.
• [35]Putaala J, Haapaniemi E, Metso AJ, Metso TM, Artto V, Kaste M, Tatlisumak T: Recurrent ischemic events in young adults after first-ever ischemic stroke. Ann Neurol 2010, 68:661-671.
• [36]Dhamoon MS, Tai W, Boden-Albala B, Rundek T, Paik MC, Sacco RL, Elkind MS: Risk of myocardial infarction or vascular death after first ischemic stroke: the Northern Manhattan Study. Stroke 2007, 38:1752-1758.
• [37]Sumi S, Origasa H, Houkin K, Terayama Y, Uchiyama S, Daida H, Shigematsu H, Goto S, Tanaka K, Miyamoto S, Minematsu K, Matsumoto M, Okada Y, Sato M, Suzuki N: A modified Essen stroke risk score for predicting recurrent cardiovascular events: development and validation. Int J Stroke 2013, 8:251-257.
• [38]Suzuki N, Sato M, Houkin K, Terayama Y, Uchiyama S, Daida H, Shigematsu H, Goto S, Tanaka K, Origasa H, Miyamoto S, Minematsu K, Matsumoto M, Okada Y: One-year atherothrombotic vascular events rates in outpatients with recent non-cardioembolic ischemic stroke: the EVEREST (Effective Vascular Event REduction after STroke) Registry. J Stroke Cerebrovasc Dis 2012, 21:245-253.
• [39]Stahrenberg R, Niehaus CF, Edelmann F, Mende M, Wohlfahrt J, Wasser K, Seegers J, Hasenfuß G, Gröschel K, Wachter R: High-sensitivity troponin assay improves prediction of cardiovascular risk in patients with cerebral ischaemia. J Neurol Neurosurg Psychiatry 2013, 84:479-487.
• [40]Diener HC, Ringleb PA, Savi P: Clopidogrel for the secondary prevention of stroke. Expert Opin Pharmacother 2005, 6:755-764.
• [41]Kernan WN, Viscoli CM, Brass LM, Makuch RW, Sarrel PM, Roberts RS, Gent M, Rothwell P, Sacco RL, Liu RC, Boden-Albala B, Horwitz RI: The stroke prognosis instrument II (SPI-II): a clinical prediction instrument for patients with transient ischemia and nondisabling ischemic stroke. Stroke 2000, 31:456-462.
• [42]Steyerberg EW: Clinical Prediction Models: A Practical Approach to Development, Validation, and Updating. New York: Springer; 2009.
• [43]Fitzek S, Leistritz L, Witte OW, Heuschmann PU, Fitzek C: The Essen Stroke Risk Score in one-year follow-up acute ischemic stroke patients. Cerebrovasc Dis 2011, 31:400-407.
• [44]Weimar C, Diener HC, Alberts MJ, Steg PG, Bhatt DL, Wilson PW, Mas JL, Röther J, REduction of Atherothrombosis for Continued Health Registry Investigators: The Essen Stroke Risk Score predicts recurrent cardiovascular events. Stroke 2009, 40:350-354.
• [45]Weimar C, Goertler M, Rother J, Ringelstein EB, Darius H, Nabavi DG, Kim IH, Benemann J, Diener HC, SCALA Study Group: Predictive value of the Essen Stroke Risk Score and Ankle Brachial Index in acute ischaemic stroke patients from 85 German stroke units. J Neurol Neurosurg Psychiatry 2008, 79:1339-1343.
• [46]Navi BB, Kamel H, Sidney S, Klingman JG, Nguyen-Huynh MN, Johnston SC: Validation of the Stroke Prognostic Instrument-II in a large, modern, community-based cohort of ischemic stroke survivors. Stroke 2011, 42:3392-3396.
• [47]Wijnhoud AD, Maasland L, Lingsma HF, Steyerberg EW, Koudstaal PJ, Dippel DW: Prediction of major vascular events in patients with transient ischemic attack or ischemic stroke. Stroke 2010, 41:2178-2185.
• [48]Chandratheva A, Geraghty OC, Rothwell PM: Poor performance of current prognostic scores for early risk of recurrence after minor stroke. Stroke 2011, 42:632-637.
• [49]Meng X, Wang Y, Zhao X, Wang C, Li H, Liu L, Zhou Y, Xu J, Wang Y: Validation of the Essen Stroke Risk Score and the Stroke Prognosis Instrument II in Chinese patients. Stroke 2011, 42:3619-3620.
• [50]Weimar C, Benemann J, Michalski D, Muller M, Luckner K, Katsarava Z, Weber R, Diener HC, German Stroke Study Collaboration: Prediction of recurrent stroke and vascular death in patients with transient ischemic attack or nondisabling stroke: a prospective comparison of validated prognostic scores. Stroke 2010, 41:487-493.
• [51]Weimar C, Siebler M, Brandt T, Römer D, Rosin L, Bramlage P, Sander D: Vascular risk prediction in ischemic stroke patients undergoing in-patient rehabilitation – insights from the investigation of patients with ischemic stroke in neurologic rehabilitation (INSIGHT) registry. Int J Stroke 2013, 8:503-509.
• [52]Maier IL, Bauerle M, Kermer P, Helms HJ, Buettner T: Risk prediction of very early recurrence, death and progression after acute ischaemic stroke. Eur J Neurol 2013, 20:599-604.
• [53]Johnston SC, Rothwell PM, Nguyen-Huynh MN, Giles MF, Elkins JS, Bernstein AL, Sidney S: Validation and refinement of scores to predict very early stroke risk after transient ischaemic attack. Lancet 2007, 369:283-292.
• [54]Kurth T, Stapf C: Here comes the sun? Neurology 2010, 74:102-103.
• [55]Veerbeek JM, Kwakkel G, van Wegen EE, Ket JC, Heymans MW: Early prediction of outcome of activities of daily living after stroke: a systematic review. Stroke 2011, 42:1482-1488.
• [56]Counsell C, Dennis M, McDowall M: Predicting functional outcome in acute stroke: comparison of a simple six variable model with other predictive systems and informal clinical prediction. J Neurol Neurosurg Psychiatry 2004, 75:401-405.
• [57]Pencina MJ, D'Agostino RB, Vasan RS: Evaluating the added predictive ability of a new marker: from area under the ROC curve to reclassification and beyond. Stat Med 2008, 27:157-172.
• [58]Bouwmeester W, Zuithoff NP, Mallett S, Geerlings MI, Vergouwe Y, Steyerberg EW, Altman DG, Moons KG: Reporting and methods in clinical prediction research: a systematic review. PLoS Med 2012, 9:e1001221.
• [59]Collins G: Opening up multivariable prediction models: consensus-based guidelines for transparent reporting. BMJ 2011.