Variable performance of models for predicting methicillin-resistant Staphylococcus aureus carriage in European surgical wards

BMC Infectious Diseases

Ben S Cooper⁷  José Antonio Martínez⁵  Silvia Garilli²  George L Daikos⁶  Annie Chalfine⁸  Andrea Patroni³  Stephan Harbarth¹  Angelo Pan²  Andie S Lee⁴ 

[1] Infection Control Program, University of Geneva Hospitals and Faculty of Medicine, Geneva, Switzerland;Infectious and Tropical Diseases Unit, Istituti Ospitalieri di Cremona, Cremona, Italy;General Medicine Unit, Ospedale di Esine, Esine, Italy;Departments of Infectious Diseases and Microbiology, Royal Prince Alfred Hospital, Sydney, Australia;Service of Infectious Diseases, Hospital Clinic de Barcelona, Barcelona, Spain;First Department of Propaedeutic Medicine, Laiko General Hospital, Athens, Greece;Centre for Clinical Vaccinology and Tropical Medicine, Nuffield Department of Clinical Medicine, University of Oxford, Oxford, UK;Infection Control Unit, Groupe Hospitalier Paris Saint-Joseph, Paris, France

关键词: Predictive models;    Screening;    Methicillin-resistant Staphylococcus aureus;   

Others  :  1135656
DOI  :  10.1186/s12879-015-0834-y

 received in 2014-09-20, accepted in 2015-02-12,  发布年份 2015

【 摘 要 】

Background

Predictive models to identify unknown methicillin-resistant Staphylococcus aureus (MRSA) carriage on admission may optimise targeted MRSA screening and efficient use of resources. However, common approaches to model selection can result in overconfident estimates and poor predictive performance. We aimed to compare the performance of various models to predict previously unknown MRSA carriage on admission to surgical wards.

Methods

The study analysed data collected during a prospective cohort study which enrolled consecutive adult patients admitted to 13 surgical wards in 4 European hospitals. The participating hospitals were located in Athens (Greece), Barcelona (Spain), Cremona (Italy) and Paris (France). Universal admission MRSA screening was performed in the surgical wards. Data regarding demographic characteristics and potential risk factors for MRSA carriage were prospectively collected during the study period. Four logistic regression models were used to predict probabilities of unknown MRSA carriage using risk factor data: “Stepwise” (variables selected by backward elimination); “Best BMA” (model with highest posterior probability using Bayesian model averaging which accounts for uncertainty in model choice); “BMA” (average of all models selected with BMA); and “Simple” (model including variables selected >50% of the time by both Stepwise and BMA approaches applied to repeated random sub-samples of 50% of the data). To assess model performance, cross-validation against data not used for model fitting was conducted and net reclassification improvement (NRI) was calculated.

Results

Of 2,901 patients enrolled, 111 (3.8%) were newly identified MRSA carriers. Recent hospitalisation and presence of a wound/ulcer were significantly associated with MRSA carriage in all models. While all models demonstrated limited predictive ability (mean c-statistics <0.7) the Simple model consistently detected more MRSA-positive individuals despite screening fewer patients than the Stepwise model. Moreover, the Simple model improved reclassification of patients into appropriate risk strata compared with the Stepwise model (NRI 6.6%, P = .07).

Conclusions

Though commonly used, models developed using stepwise variable selection can have relatively poor predictive value. When developing MRSA risk indices, simpler models, which account for uncertainty in model selection, may better stratify patients’ risk of unknown MRSA carriage.

【 授权许可】

   
2015 Lee et al.; licensee BioMed Central.

【 图 表 】

Figure 1.

Figure 2.

Figure 3.

【 参考文献 】

• [1]Weber SG, Huang SS, Oriola S, Huskins WC, Noskin GA, Harriman K, et al.: Legislative mandates for use of active surveillance cultures to screen for methicillin-resistant Staphylococcus aureus and vancomycin-resistant enterococci: Position statement from the Joint SHEA and APIC Task Force. Am J Infect Control 2007, 35:73-85.
• [2]UK Department of Health. MRSA Screening - Operational Guidance 2. [http://webarchive.nationalarchives.gov.uk/20130107105354/http://www.dh.gov.uk/en/Publicationsandstatistics/Lettersandcirculars/Dearcolleagueletters/DH_092844]
• [3]Murthy A, De Angelis G, Pittet D, Schrenzel J, Uckay I, Harbarth S: Cost-effectiveness of universal MRSA screening on admission to surgery. Clin Microbiol Infect 2010, 16:1747-1753.
• [4]Robotham JV, Graves N, Cookson BD, Barnett AG, Wilson JA, Edgeworth JD, et al.: Screening, isolation, and decolonisation strategies in the control of meticillin resistant Staphylococcus aureus in intensive care units: cost effectiveness evaluation. BMJ 2011, 343:d5694.
• [5]Collins J, Raza M, Ford M, Hall L, Brydon S, Gould FK: Review of a three-year meticillin-resistant Staphylococcus aureus screening programme. J Hosp Infect 2011, 78:81-85.
• [6]Kang J, Mandsager P, Biddle AK, Weber DJ: Cost-effectiveness analysis of active surveillance screening for methicillin-resistant Staphylococcus aureus in an academic hospital setting. Infect Control Hosp Epidemiol 2012, 33:477-486.
• [7]Safdar N, Bradley EA: The risk of infection after nasal colonization with Staphylococcus aureus. Am J Med 2008, 121:310-315.
• [8]Lee AS, Cooper BS, Malhotra-Kumar S, Chalfine A, Daikos GL, Fankhauser C, et al.: Comparison of strategies to reduce meticillin-resistant Staphylococcus aureus rates in surgical patients: a controlled multicentre intervention trial. BMJ Open 2013, 3:e003126.
• [9]Schweizer M, Perencevich E, McDanel J, Carson J, Formanek M, Hafner J, et al.: Effectiveness of a bundled intervention of decolonization and prophylaxis to decrease Gram positive surgical site infections after cardiac or orthopedic surgery: systematic review and meta-analysis. BMJ 2013, 346:f2743.
• [10]Furuno JP, Harris AD, Wright MO, McGregor JC, Venezia RA, Zhu J, et al.: Prediction rules to identify patients with methicillin-resistant Staphylococcus aureus and vancomycin-resistant enterococci upon hospital admission. Am J Infect Control 2004, 32:436-440.
• [11]Harbarth S, Sax H, Fankhauser-Rodriguez C, Schrenzel J, Agostinho A, Pittet D: Evaluating the probability of previously unknown carriage of MRSA at hospital admission. Am J Med 2006, 119:275. e215-223
• [12]Furuno JP, McGregor JC, Harris AD, Johnson JA, Johnson JK, Langenberg P, et al.: Identifying groups at high risk for carriage of antibiotic-resistant bacteria. Arch Int Med 2006, 166:580-585.
• [13]Haley CC, Mittal D, Laviolette A, Jannapureddy S, Parvez N, Haley RW: Methicillin-resistant Staphylococcus aureus infection or colonization present at hospital admission: multivariable risk factor screening to increase efficiency of surveillance culturing. J Clin Microbiol 2007, 45:3031-3038.
• [14]Harbarth S, Sax H, Uckay I, Fankhauser C, Agostinho A, Christenson JT, et al.: A predictive model for identifying surgical patients at risk of methicillin-resistant Staphylococcus aureus carriage on admission. J Am Coll Surg 2008, 207:683-689.
• [15]Riedel S, Von Stein D, Richardson K, Page J, Miller S, Winokur P, et al.: Development of a prediction rule for methicillin-resistant Staphylococcus aureus and vancomycin-resistant enterococcus carriage in a Veterans Affairs Medical Center population. Infect Control Hosp Epidemiol 2008, 29:969-971.
• [16]Morgan DJ, Day HR, Furuno JP, Young A, Johnson JK, Bradham DD, et al.: Improving efficiency in active surveillance for methicillin-resistant Staphylococcus aureus or vancomycin-resistant Enterococcus at hospital admission. Infect Control Hosp Epidemiol 2010, 31:1230-1235.
• [17]Robicsek A, Beaumont JL, Wright MO, Thomson RB Jr, Kaul KL, Peterson LR: Electronic prediction rules for methicillin-resistant Staphylococcus aureus colonization. Infect Control Hosp Epidemiol 2011, 32:9-19.
• [18]Yeung KY, Bumgarner RE, Raftery AE: Bayesian model averaging: development of an improved multi-class, gene selection and classification tool for microarray data. Bioinformatics 2005, 21:2394-2402.
• [19]Wang D, Zhang W, Bakhai A: Comparison of Bayesian model averaging and stepwise methods for model selection in logistic regression. Stat Med 2004, 23:3451-3467.
• [20]Pan A, Lee A, Cooper B, Chalfine A, Daikos GL, Garilli S, et al.: Risk factors for previously unknown meticillin-resistant Staphylococcus aureus carriage on admission to 13 surgical wards in Europe. J Hosp Infect 2013, 83:107-113.
• [21]Van Heirstraeten L, Cortinas Abrahantes J, Lammens C, Lee A, Harbarth S, Molenberghs G, et al.: Impact of a short period of pre-enrichment on detection and bacterial loads of methicillin-resistant Staphylococcus aureus from screening specimens. J Clin Microbiol 2009, 47:3326-3328.
• [22]Gazin M, Lee A, Derde L, Kazma M, Lammens C, Ieven M, et al.: Culture-based detection of methicillin-resistant Staphylococcus aureus by a network of European laboratories: an external quality assessment study. Eur J Clin Microbiol Infect Dis 2012, 31:1765-1770.
• [23]Cook NR: Statistical evaluation of prognostic versus diagnostic models: beyond the ROC curve. Clin Chem 2008, 54:17-23.
• [24]Pencina MJ, D'Agostino RB Sr, D'Agostino RB Jr, Vasan RS: Evaluating the added predictive ability of a new marker: from area under the ROC curve to reclassification and beyond. Stat Med 2008, 27:157-172. discussion 207–112
• [25]Raftery A, Hoeting J, Volinsky C, Painter I, Yeung KY: BMA: Bayesian Model Averaging. R package version 3.14.1. 2011.
• [26]R Development Core Team. R. A language and environment for statistical computing. R Foundation for Statistical Computing, Vienna, Austria. 2010. ISBN 3-900051-07-0 [http://www.R-project.org/]
• [27]Royston P, Moons KG, Altman DG, Vergouwe Y: Prognosis and prognostic research: Developing a prognostic model. BMJ 2009, 338:b604.
• [28]Hesterberg T, Choi NH, Meier L, Fraley C: Least angle and l1 penalized regression: A review. Statist Surv 2008, 2:61-93.
• [29]Babyak MA: What you see may not be what you get: a brief, nontechnical introduction to overfitting in regression-type models. Psychosom Med 2004, 66:411-421.
• [30]Steyerberg EW, Harrell FE Jr, Borsboom GJ, Eijkemans MJ, Vergouwe Y, Habbema JD: Internal validation of predictive models: efficiency of some procedures for logistic regression analysis. J Clin Epidemiol 2001, 54:774-781.
• [31]Genell A, Nemes S, Steineck G, Dickman PW: Model selection in medical research: a simulation study comparing Bayesian model averaging and stepwise regression. BMC Med Res Methodol 2010, 10:108. BioMed Central Full Text
• [32]Concato J, Feinstein AR, Holford TR: The risk of determining risk with multivariable models. Ann Intern Med 1993, 118:201-210.
• [33]Steyerberg EW, Vickers AJ, Cook NR, Gerds T, Gonen M, Obuchowski N, et al.: Assessing the performance of prediction models: a framework for traditional and novel measures. Epidemiology 2010, 21:128-138.
• [34]McKinnell JA, Huang SS, Eells SJ, Cui E, Miller LG: Quantifying the impact of extranasal testing of body sites for methicillin-resistant Staphylococcus aureus colonization at the time of hospital or intensive care unit admission. Infect Control Hosp Epidemiol 2013, 34:161-170.
• [35]Daum RS: Clinical practice. Skin and soft-tissue infections caused by methicillin-resistant Staphylococcus aureus. N Eng J Med 2007, 357:380-390.
• [36]Voss A, Loeffen F, Bakker J, Klaassen C, Wulf M: Methicillin-resistant Staphylococcus aureus in pig farming. Emerg Infect Dis 2005, 11:1965-1966.
• [37]Tsiodras S, Daikos GL, Lee A, Plachouras D, Antoniadou A, Ploiarchopoulou F, et al.: Risk factors for Community-Associated MRSA in a large metropolitan area in Greece: An Epidemiological Study Using Two Case Definitions. J Global Antimicrobial Resistance 2014, 2:27-33.
• [38]Altman DG, Vergouwe Y, Royston P, Moons KG: Prognosis and prognostic research: validating a prognostic model. BMJ 2009, 338:b605.
• [39]WHO. Report on the burden of endemic health care-associated infection worldwide. [http://www.who.int/gpsc/country_work/burden_hcai/en/]
• [40]Lee AS, Huttner B, Harbarth S: Control of methicillin-resistant Staphylococcus aureus. Infect Dis Clin North Am 2011, 25:155-179.
• [41]Bode LG, Kluytmans JA, Wertheim HF, Bogaers D, Vandenbroucke-Grauls CM, Roosendaal R, et al.: Preventing surgical-site infections in nasal carriers of Staphylococcus aureus. N Eng J Med 2010, 362:9-17.