Identifying unusual performance in Australian and New Zealand intensive care units from 2000 to 2010

BMC Medical Research Methodology

John L Moran¹  Jessica Kasza²  Patricia J Solomon³ 

[1] Department of Intensive Care Medicine, The Queen Elizabeth Hospital, 28 Woodville Road, Woodville, 5011 Adelaide, Australia;Department of Epidemiology and Preventive Medicine, Monash University, 3004 Melbourne, Australia;School of Mathematical Sciences, University of Adelaide, North Terrace, 5005 Adelaide, Australia

关键词: Variance components;    Seasonal effects;    Risk-adjusted mortality;    Multiple comparisons;    Intensive care performance;    Hospital comparisons;    Hierarchical models;   

Others  :  866357
DOI  :  10.1186/1471-2288-14-53

 received in 2014-03-06, accepted in 2014-04-14,  发布年份 2014

【 摘 要 】

Background

The Australian and New Zealand Intensive Care Society (ANZICS) Adult Patient Database (APD) collects voluntary data on patient admissions to Australian and New Zealand intensive care units (ICUs). This paper presents an in-depth statistical analysis of risk-adjusted mortality of ICU admissions from 2000 to 2010 for the purpose of identifying ICUs with unusual performance.

Methods

A cohort of 523,462 patients from 144 ICUs was analysed. For each ICU, the natural logarithm of the standardised mortality ratio (log-SMR) was estimated from a risk-adjusted, three-level hierarchical model. This is the first time a three-level model has been fitted to such a large ICU database anywhere. The analysis was conducted in three stages which included the estimation of a null distribution to describe usual ICU performance. Log-SMRs with appropriate estimates of standard errors are presented in a funnel plot using 5% false discovery rate thresholds. False coverage-statement rate confidence intervals are also presented. The observed numbers of deaths for ICUs identified as unusual are compared to the predicted true worst numbers of deaths under the model for usual ICU performance.

Results

Seven ICUs were identified as performing unusually over the period 2000 to 2010, in particular, demonstrating high risk-adjusted mortality compared to the majority of ICUs. Four of the seven were ICUs in private hospitals. Our three-stage approach to the analysis detected outlying ICUs which were not identified in a conventional (single) risk-adjusted model for mortality using SMRs to compare ICUs. We also observed a significant linear decline in mortality over the decade. Distinct yearly and weekly respiratory seasonal effects were observed across regions of Australia and New Zealand for the first time.

Conclusions

The statistical approach proposed in this paper is intended to be used for the review of observed ICU and hospital mortality. Two important messages from our study are firstly, that comprehensive risk-adjustment is essential in modelling patient mortality for comparing performance, and secondly, that the appropriate statistical analysis is complicated.

【 授权许可】

   
2014 Solomon et al.; licensee BioMed Central Ltd.

【 图 表 】

【 参考文献 】

• [1]Schoenfeld D: Survival methods, including those using competing risk analysis, are not appropriate for intensive care unit outcome studies. Crit Care 2006, 10:103.
• [2]Freemantle N, Richardson M, Wood J, Ray D, Khosla S, Shahian D, Roche W, Stephens I, Keogh B, Pagano D: Weekend hospitalization and additional risk of death: an analysis of inpatient data. J R Soc Med 2012, 105:74-84.
• [3]Stow PJ, Hart GK, Higlett T, George C, Herkes R, McWilliam D, Bellomo R: Development and implementation of a high-quality clinical database: the Australian and New Zealand Intensive Care Society Adult Patient Database. J Crit Care 2006, 21:133-41.
• [4]Knaus WA, Draper EA, Wagner DP, Zimmerman JE: APACHE II: a severity of disease classification system. Crit Care Med 1985, 13:818-29.
• [5]Knaus WA, Wagner DP, Draper EA, Zimmerman JE, Bergner M, Bastos PG, Sirio CA, Murphy DJ, Lotring T, Damiano A: The APACHE III prognostic system. Risk prediction of hospital mortality for critically ill hospitalized adults. Chest 1991, 100:1619-36.
• [6]Le Gall JR, Lemeshow S, Saulnier F: A new simplified acute physiology score (SAPS II) based on a European/North American multicenter study. JAMA 1993, 270:2957-63.
• [7]Ash AS, Feinberg SE, Louis TA, Norman SL, Stukel TA, Utts J: Statistical Issues in Assessing Hospital Performance. Commissioned by the Committee of Presidents of Statistical Statistical Societies for the Centers for Medicare and Medicaid Services (CMS), January 27, 2012. [ http://www.cms.gov/Medicare/Quality-Initiatives-Patient-Assessment-Instruments/HospitalQualityInits/Downloads/Statistical-Issues-in-Assessing-Hospital-Performance.pdf webcite]
• [8]Gallagher MP, Krumholz HM: Public reporting of hospital outcomes: a challenging road ahead. MJA 2011, 194:658-60.
• [9]Goldstein H, Spiegelhalter DJ: League tables and their limitations: statistical issues in comparisons of institutional performance. JRSS A 1996, 159:285-443.
• [10]DeLong ER, Peterson ED, DeLong DM, Muhlbaier LH, Hackett S, Mark DB: Comparing risk-adjustment methods for provider profiling. Statist Med 1997, 16:2645-64.
• [11]Jones HE, Spiegelhalter DJ: The identification of ‘unusual’ health-care providers from a hierarchical model. Am Stat 2011, 65:154-63.
• [12]Normand S-LT, Shahian DM: Statistical and clinical aspects of hospital outcomes profiling. Stat Sci 2007, 22:206-26.
• [13]Ohlssen DI, Sharples L, Spiegelhalter DJ: A hierarchical modelling framework for identifying unusual performance in health care providers. JRSS A 2007, 170:265-90.
• [14]Kasza J, Moran JL, Solomon PJ: Evaluating the performance of Australian and New Zealand intensive care units in 2009 and 2010. Statist Med 2013, 13:3720-36.
• [15]Kalbfleisch JD, Wolfe R: On monitoring outcomes of medical providers. Stat Bio 2013, 5:286-302.
• [16]Pouw ME, Peelen LM, Lingsma HF, Pieter D, Steyerberg E, Kalkman CJ, Moons KGM: Hospital standardized mortality ratio: consequences of adjusting hospital mortality with indirect standardization. PLOS One 2013, 8(4):e59160. doi:10.1371/journal.pone.0059160
• [17]Seaton SE, Barker L, Lingsm HF, Steyerberg EW, Manktelow BN: What is the probability of detecting poorly performing hospitals using funnel plots? BMJ Qual Saf 2013. doi:10.1136/bmjqs-2012-001689
• [18]Bryan ML, Jenkins SP: Regression analysis of country effects using multilevel data: a cautionary tale. ISER Working Paper Series 2013-14. Colchester, University of Essex. [ https://www.iser.essex.ac.uk/publications/workingpapers/iser/2013-14 webcite]
• [19]Moran JL, Bristow P, Solomon PJ, George C, Hart G: Mortality and length-of-stay outcomes, 1993-2003 in the binational Australian and New Zealand intensive care Adult Patient Database. Crit Care Med 2008, 36:46-61.
• [20]Hosmer DW, Lemeshow S: Confidence interval estimates of an index of quality performance based on logistic regression models. Statist Med 1995, 14:2161-72.
• [21]StataCorp: Stata™: Release 12. College Station, TX, USA: StataCorp LP, Statistical Software; 2011.
• [22]Iezzoni L: The risks of risk-adjustment. JAMA 1997, 278:1600-7.
• [23]Sauerbrei W, Royston P, Binder H: Selection of important variables and determination of functional form for continuous predictors in multivariable model building. Stat Med 2007, 26:5512-8.
• [24]Moran JL, Solomon PJ: Conventional and advanced time series estimation: application to the Australian and New Zealand Intensive Care Society Adult Patient Database 1993-2006. JECP 2011, 17:45-60.
• [25]Bhonagiri D, Pilcher DV, Bailey MJ: Increased mortality associated with after-hours and weekend admission to the intensive care unit: a retrospective analysis. MJA 2011, 194:287-92.
• [26]Diggle PJ: Time Series: a Biostatistical Introduction. Oxford: Oxford University Press; 1990.
• [27]Stolwijk AM, Straatman H, Zielhuis GA: Studying seasonality by using sine and cosine functions in regression analysis. J Epid Comm H 1999, 53:235-38.
• [28]Paul P, Pennell ML, Lemeshow S: Standardizing the power of the Hosmer-Lemeshow goodness of fit test in large data sets. Stat Med 2013, 32:67-80.
• [29]Gelman A, Hill J: Data Analysis Using Regression and Multilevel/hierarchical Models. Cambridge: Cambridge University Press; 2007.
• [30]Verbeke G, Molenberghs G: The gradient function as an exploratory goodness-of-fit assessment of the random-effects distribution in mixed models. Biostat 2013, 14:477-90.
• [31]Benjamini Y, Yekutieli D: False discovery rate-adjusted multiple confidence intervals for selected parameters. JASA 2005, 100:71-93.
• [32]Benjamini Y, Hochberg Y: Controlling the false discovery rate: a practical and powerful approach to multiple testing. JRSS B 1995, 57:289-300.
• [33]Harrison DA, Lertsithichai P, Brady AR, Carpenter JR, Rowan K: Winter excess mortality in intensive care in the UK: an analysis of outcome adjusted for patient casemix and unit workload. Int Care Med 2004, 30:1900-7.
• [34]Twisk J, de Boer M, de Vente W, Heymans M: Multiple imputation of missing values was not necessary before performing a longitudinal mixed-model analysis. J Clin Epi 2013, 66:1022-28.
• [35]ANZICS CORE: ANZICS Centre for Outcome and Resource Evaluation. 2011. Outlier management policy at: [ http://www.anzics.com.au/core/core-management-governance webcite]