Improved assessment of multiple sclerosis lesion segmentation agreement via detection and outline error estimates

BMC Medical Imaging

Robert Zivadinov³  Guy Poloni²  Deepa Ramasamy²  Sara Hussein²  Laura Ranza⁴  Carol Di Perri⁴  Niels Bergsland²  Michael G Dwyer²  David S Wack¹ 

[1]Buffalo Neuroimaging Analysis Center, Jacobs Neurological Institute, State University of NY at Buffalo, 100 High St., Buffalo, NY, 14203, USA

[2]Buffalo Neuroimaging Analysis Center, Dept. of Neurology, University at Buffalo, State University of New York at Buffalo, Buffalo, NY, USA

[3]The Jacobs Neurological Institute, Dept. of Neurology, University at Buffalo, State University of New York at Buffalo, Buffalo, NY, USA

[4]Department of Neuroradiology, IRCCS, C. Mondino, University of Pavia, Pavia, Italy

关键词: ROI;    MRI;    Lesion;    Kappa;    Index;    Measure;    Similarity index;    Jaccard Index;    Metric;    Operator agreement;    Rater agreement;    Detection and outline error estimates;    Multiple sclerosis;   

Others  :  1092078
DOI  :  10.1186/1471-2342-12-17

 received in 2011-11-08, accepted in 2012-07-19,  发布年份 2012

【 摘 要 】

Background

Presented is the method “Detection and Outline Error Estimates” (DOEE) for assessing rater agreement in the delineation of multiple sclerosis (MS) lesions. The DOEE method divides operator or rater assessment into two parts: 1) Detection Error (DE) -- rater agreement in detecting the same regions to mark, and 2) Outline Error (OE) -- agreement of the raters in outlining of the same lesion.

Methods

DE, OE and Similarity Index (SI) values were calculated for two raters tested on a set of 17 fluid-attenuated inversion-recovery (FLAIR) images of patients with MS. DE, OE, and SI values were tested for dependence with mean total area (MTA) of the raters' Region of Interests (ROIs).

Results

When correlated with MTA, neither DE (ρ = .056, p=.83) nor the ratio of OE to MTA (ρ = .23, p=.37), referred to as Outline Error Rate (OER), exhibited significant correlation. In contrast, SI is found to be strongly correlated with MTA (ρ = .75, p < .001). Furthermore, DE and OER values can be used to model the variation in SI with MTA.

Conclusions

The DE and OER indices are proposed as a better method than SI for comparing rater agreement of ROIs, which also provide specific information for raters to improve their agreement.

【 授权许可】

   
2012 Wack et al.; licensee BioMed Central Ltd.

【 图 表 】

Figure 1.

Figure 2.

Figure 3.

Figure 4.

Figure 5.

Figure 6.

【 参考文献 】

• [1]Hunttenlocher D, Klanderman G, Ruckelidge W: Comparing images using the Hausdorff distance. IEEE Trans Pattern Anal Mach Intell 1993, 15(9):850-863.
• [2]Liu C, Frazier P, Kumar L: Comparative assessment of the measures of thematic classification accuracy. Remote Sens Environ 2007, 107(4):606-616.
• [3]Udupa J, Leblanc V, Schmidt H, Imielinska C, Saha K, Grevera G, Zhuge Y, Molholt P, Currie L, Jin Y: A methodology for evaluating image segmentation algorithms. In SPIE Medical Imaging 2002, 4684:266-277.
• [4]Udupa J, LeBlanc V, Zhuge Y, Imielinska C, Schmidt H, Currie L, Hirsch B, Woodburn J: A framework for evaluating image segmentation algorithms. Comput Med Imaging Graph 2006, 30(2):75-87.
• [5]Chang HH, Zhuang AH, Valentino DJ, Chu WC: Performance measure characterization for evaluating neuroimage segmentation algorithms. Neuroimage 2009, 47(1):122-135.
• [6]Zijdenbos A, Dawant B, Margolin R, Palmer A: Morphometric analysis of white matter lesions in mr images: methodand validation. IEEE Trans Med Imaging 1994, 13(4):716-724.
• [7]Filippi M, Horsfield M, Bressi S, Martinelli V, Baratti C, Reganati P, Campi A, Miller D, Comi G: Intra-and inter-observer agreement of brain MRI lesion volume measurements in multiple sclerosis: a comparison of techniques. Brain 1995, 118(6):1593-1600.
• [8]Gerig G, Jomier M, Chakos M: Valmet: A new validation tool for assessing and improving 3D object segmentation. In Proceedings of the 4th International Conference on Medical Image Computing and Computer-Assisted Intervention: 2001. Springer-Verlag London, UK; 2001:516-523.
• [9]Warfield S, Zou K, Wells W: Simultaneous truth and performance level estimation (STAPLE): an algorithm for the validation of image segmentation. IEEE Trans Med Imaging 2004, 23(7):903-921.
• [10]Adams H, Wagner S, Sobel D, Slivka L, Sipe J, Romine J, Beutler E, Koziol J: Hypointense and hyperintense lesions on magnetic resonance imaging in secondary-progressive MS patients. Eur Neurol 1999, 42:52-63.
• [11]Johnston B, Atkins M, Mackiewich B, Anderson M: Segmentation of multiple sclerosis lesions in intensity correctedmultispectral MRI. IEEE Trans Med Imaging 1996, 15(2):154-169.
• [12]Udupa J, Wei L, Samarasekera S, Miki Y, Van Buchem M, Grossman R: Multiple sclerosis lesion quantification using fuzzy-connectednessprinciples. IEEE Trans Med Imaging 1997, 16(5):598-609.
• [13]Admiraal-Behloul F, Van Den Heuvel D, Olofsen H, van Osch M, van der Grond J, Van Buchem M, Reiber J: Fully automatic segmentation of white matter hyperintensities in MR images of the elderly. Neuroimage 2005, 28(3):607-617.
• [14]Dyrby T, Rostrup E, Baaré W, van Straaten E, Barkhof F, Vrenken H, Ropele S, Schmidt R, Erkinjuntti T, Wahlund L: Segmentation of age-related white matter changes in a clinical multi-center study. Neuroimage 2008, 41(2):335-345.
• [15]Kurtzke J: Rating neurologic impairment in multiple sclerosis: an expanded disability status scale (EDSS). Neurology 1983, 33(11):1444-1452.
• [16]Polman C, Reingold S, Edan G, Filippi M, Hartung H, Kappos L, Lublin F, Metz L, McFarland H, O'Connor P: Diagnostic criteria for multiple sclerosis: 2005 revisions to the" McDonald Criteria". Ann Neurol 2005, 58(6):840-846.
• [17]Di Perri C, Dwyer M, Wack D, Cox J, Hashmi K, Saluste E, Hussein S, Schirda C, Stosic M, Durfee J: Signal abnormalities on 1.5 and 3 Tesla brain MRI in multiple sclerosis patients and healthy controls. A morphological and spatial quantitative comparison study. Neuroimage 2009, 47(4):1352-1362.
• [18]Akaike H: A new look at the statistical model identification. IEEE Trans Automatic Control 1974, 19(6):716-723.
• [19]Anbeek P, Vincken K, van Osch M, Bisschops R, van der Grond J: Probabilistic segmentation of white matter lesions in MR imaging. Neuroimage 2004, 21(3):1037-1044.