The probability of being identified as an outlier with commonly used funnel plot control limits for the standardised mortality ratio

BMC Medical Research Methodology

Bradley N Manktelow¹  Sarah E Seaton¹ 

[1] Department of Health Sciences, University of Leicester, 22-28 Princess Road West, Leicester, LE1 6TP, UK

关键词: Outlier;    Probability;    Poisson;    SMR;    Funnel plot;   

Others  :  1136559
DOI  :  10.1186/1471-2288-12-98

 received in 2011-08-31, accepted in 2012-06-27,  发布年份 2012

【 摘 要 】

Background

Emphasis is increasingly being placed on the monitoring of clinical outcomes for health care providers. Funnel plots have become an increasingly popular graphical methodology used to identify potential outliers. It is assumed that a provider only displaying expected random variation (i.e. ‘in-control’) will fall outside a control limit with a known probability. In reality, the discrete count nature of these data, and the differing methods, can lead to true probabilities quite different from the nominal value. This paper investigates the true probability of an ‘in control’ provider falling outside control limits for the Standardised Mortality Ratio (SMR).

Methods

The true probabilities of an ‘in control’ provider falling outside control limits for the SMR were calculated and compared for three commonly used limits: Wald confidence interval; ‘exact’ confidence interval; probability-based prediction interval.

Results

The probability of falling above the upper limit, or below the lower limit, often varied greatly from the nominal value. This was particularly apparent when there were a small number of expected events: for expected events ≤50 the median probability of an ‘in-control’ provider falling above the upper 95% limit was 0.0301 (Wald), 0.0121 (‘exact’), 0.0201 (prediction).

Conclusions

It is important to understand the properties and probability of being identified as an outlier by each of these different methods to aid the correct identification of poorly performing health care providers. The limits obtained using probability-based prediction limits have the most intuitive interpretation and their properties can be defined a priori. Funnel plot control limits for the SMR should not be based on confidence intervals.

【 授权许可】

   
2012 Seaton and Manktelow; licensee BioMed Central Ltd.

【 图 表 】

Figure 1.

Figure 2.

Figure 3.

【 参考文献 】

• [1]Spiegelhalter DJ: Surgical audit: statistical lessons from Nightingale and Codman. Journal Of The Royal Statistical Society Series A-Statistics In Society 1999, 162(Pt1):45-58.
• [2]Department of Health: Equity and excellence: Liberating the NHS. Department of Health, London; 2010.
• [3]Liddell FDK: Simple Exact Analysis of the Standardized Mortality Ratio. J Epidemiol Community Health 1984, 38(1):85-88.
• [4]Spiegelhalter D: Funnel plots for institutional comparison. Qual Saf Health Care 2002, 11(4):390-391.
• [5]Tekkis PP, Mcculloch P, Steger AC, Benjamin IS, Poloniecki JD: Mortality Control Charts for Comparing Performance of Surgical Units: Validation Study Using Hospital Mortality Data. Br Med J 2003, 326(7393):786-788A.
• [6]Spiegelhalter D: Funnel plots for comparing institutional performance. Stat Med 2005, 24(8):1185-1202.
• [7]Mohammed MA, Deeks JJ: In the context of performance monitoring, the caterpillar plot should be mothballed in favor of the funnel plot. Ann Thorac Surg 2008, 86(1):348.
• [8]Mayer EK, Bottle A, Rao C, Darzi AW, Athanasiou T: Funnel plots and their emerging application in surgery. Ann Surg 2009, 249(3):376-383.
• [9]van Dishoeck AM, Looman CWN, van der Wilden-van Lier ECM, Mackenbach JP, Steyerberg EW: Displaying random variation in comparing hospital performance. BMJ Quality & Safety 2011, 20(8)):651-657.
• [10]Department of Health/Healthcare Quality Improvement Partnership: Detection and Management of Outliers. Department of Health, London; 2011.
• [11]Flowers J: Technical Briefing 2: Statistical process control methods in public health intelligence. Association of Public Health Observatories, York; 2007.
• [12]Gini R, Forni S: Funnel plots for institutional comparisons. In United Kingdom Stata Users' Group Meetings 2009. Stata Users Group,  ; 2009. Available at: http://www.stata.com/meeting/uk09/uk09_gini_forni.pdf webcite (accessed 12 July 2012)
• [13]Centre for Maternal and Child Enquiries (CEMACE): Perinatal Mortality 2008. CEMACE, United Kingdom. London; 2010.
• [14]Kirkham JJ, Bouamra O: The use of statistical process control for monitoring institutional performance in trauma care. Journal of Trauma® Injury, Infection, and Critical Care 2008, 65(6):1494-1501.
• [15]Association of Public Health Observatories. http://www.apho.org.uk/default.aspx?RID=39403 webcite (accessed 26 July 2011)
• [16]Bottle A, Aylin P: Application of AHRQ patient safety indicators to English hospital data. Qual Saf Health Care 2009, 18(4):303-308.
• [17]Eastern Region Public Health Observatory. http://www.erpho.org.uk/topics/tools/funnel.aspx#16009 webcite (accessed 26 July 2011)
• [18]Barker L: A comparison of nine confidence intervals for a Poisson parameter when the expected number of events ≤ 5. The American Statistician 2002, 56(2):85-89.
• [19]Garwood F: Fiducial limits for the Poisson distribution. Biometrika 1936, 28(3/4):437-442.
• [20]Neyman J: On the problem of confidence intervals. Annals of Mathematical Statistics 1935, 6(3):111-116.
• [21]Hart MK, Hart RF, Schmaltz S: Control limits for p control charts with small subgroup sizes. Qual Manag Health Care 2007, 16(2):123-129.
• [22]Jones HE, Ohlssen DI, Spiegelhalter DJ: Use of the false discovery rate when comparing multiple health care providers. J Clin Epidemiol 2008, 61(3):232-240.
• [23]Schulman J, Spiegelhalter DJ, Parry G: How to interpret your dot: decoding the message of clinical performance indicators. J Perinatol 2008, 28(9):588-596.