Outlier identification and monitoring of institutional or clinician performance: an overview of statistical methods and application to national audit data

BMC Health Services Research

Research

Uday Trivedi¹  Andrew T. Goodwin²  Gareth Ambler³  Rumana Z. Omar³  Menelaos Pavlou³  Mark de Belder⁴  Peter Ludman⁵ 

[1] Department of Cardiac Surgery, University Hospital Sussex NHS Foundation Trust, Brighton, UK;Department of Cardiothoracic Surgery, South Tees Hospitals NHS Foundation Trust, Middlesbrough, UK;National Institute for Cardiovascular Outcomes Research (NICOR), Barts Health NHS Trust, London, UK;Department of Statistical Science, UCL, London, UK;National Institute for Cardiovascular Outcomes Research (NICOR), Barts Health NHS Trust, London, UK;National Institute for Cardiovascular Outcomes Research (NICOR), Barts Health NHS Trust, London, UK;Institute of Cardiovascular Sciences, University of Birmingham, Birmingham, UK;

关键词: Outlier detection;    Funnel plot;    Random effects model;    Overdispersion;   

DOI  :  10.1186/s12913-022-08995-z

 received in 2022-07-15, accepted in 2022-12-20,  发布年份 2022

来源: Springer

【 摘 要 】

BackgroundInstitutions or clinicians (units) are often compared according to a performance indicator such as in-hospital mortality. Several approaches have been proposed for the detection of outlying units, whose performance deviates from the overall performance.MethodsWe provide an overview of three approaches commonly used to monitor institutional performances for outlier detection. These are the common-mean model, the ‘Normal-Poisson’ random effects model and the ‘Logistic’ random effects model. For the latter we also propose a visualisation technique. The common-mean model assumes that the underlying true performance of all units is equal and that any observed variation between units is due to chance. Even after applying case-mix adjustment, this assumption is often violated due to overdispersion and a post-hoc correction may need to be applied. The random effects models relax this assumption and explicitly allow the true performance to differ between units, thus offering a more flexible approach. We discuss the strengths and weaknesses of each approach and illustrate their application using audit data from England and Wales on Adult Cardiac Surgery (ACS) and Percutaneous Coronary Intervention (PCI).ResultsIn general, the overdispersion-corrected common-mean model and the random effects approaches produced similar p-values for the detection of outliers. For the ACS dataset (41 hospitals) three outliers were identified in total but only one was identified by all methods above. For the PCI dataset (88 hospitals), seven outliers were identified in total but only two were identified by all methods. The common-mean model uncorrected for overdispersion produced several more outliers. The reason for observing similar p-values for all three approaches could be attributed to the fact that the between-hospital variance was relatively small in both datasets, resulting only in a mild violation of the common-mean assumption; in this situation, the overdispersion correction worked well.ConclusionIf the common-mean assumption is likely to hold, all three methods are appropriate to use for outlier detection and their results should be similar. Random effect methods may be the preferred approach when the common-mean assumption is likely to be violated.

【 授权许可】

CC BY   
© The Author(s) 2023

【 图 表 】

Fig. 1

41116_2022_35_Article_IEq231.gif

41116_2022_35_Article_IEq230.gif

41116_2022_35_Article_IEq229.gif

41116_2022_35_Article_IEq227.gif

41116_2022_35_Article_IEq226.gif

41116_2022_35_Article_IEq225.gif

41116_2022_35_Article_IEq224.gif

41116_2022_35_Article_IEq223.gif

41116_2022_35_Article_IEq222.gif

41116_2022_35_Article_IEq221.gif

41116_2022_35_Article_IEq220.gif

41116_2022_35_Article_IEq219.gif

41116_2022_35_Article_IEq218.gif

41116_2022_35_Article_IEq217.gif

41116_2022_35_Article_IEq216.gif

41116_2022_35_Article_IEq215.gif

Fig. 1

41116_2022_35_Article_IEq213.gif

41116_2022_35_Article_IEq212.gif

41116_2022_35_Article_IEq211.gif

41116_2022_35_Article_IEq210.gif

41116_2022_35_Article_IEq209.gif

40249_2023_1056_Article_IEq5.gif

41116_2022_35_Article_IEq208.gif

41116_2022_35_Article_IEq207.gif

41116_2022_35_Article_IEq206.gif

41116_2022_35_Article_IEq205.gif

41116_2022_35_Article_IEq204.gif

41116_2022_35_Article_IEq203.gif

41116_2022_35_Article_IEq201.gif

Fig. 2

41116_2022_35_Article_IEq199.gif

Fig. 1

41116_2022_35_Article_IEq197.gif

41116_2022_35_Article_IEq196.gif

Fig. 2

41116_2022_35_Article_IEq194.gif

41116_2022_35_Article_IEq193.gif

41116_2022_35_Article_IEq192.gif

41116_2022_35_Article_IEq191.gif

Fig. 1

41116_2022_35_Article_IEq189.gif

41116_2022_35_Article_IEq188.gif

41116_2022_35_Article_IEq186.gif

Fig. 44

41116_2022_35_Article_IEq185.gif

Fig. 3

41116_2022_35_Article_IEq180.gif

41116_2022_35_Article_IEq178.gif

41116_2022_35_Article_IEq175.gif

41116_2022_35_Article_IEq173.gif

Fig. 2

Fig. 4