Evaluating current automatic de-identification methods with Veteran’s health administration clinical documents

BMC Medical Research Methodology

Stéphane M Meystre²  Matthew H Samore²  F Jeffrey. Friedlin¹  Shuying Shen²  Brett R South²  Oscar Ferrández² 

[1] Medical Informatics, Regenstrief Institute, Inc, Indianapolis, IN, USA;IDEAS Center SLCVA Healthcare System, Salt Lake City, UT, USA

关键词: United States department of veterans affairs [I01.409.137.500.700];    Electronic health records [E05.318.308.940.968.625.500];    Anonymization;    De-identification;    Health insurance portability and accountability act [N03.219.521.576.343.349];    Natural language processing [L01.224.065.580];    Confidentiality, patient data privacy [MeSH F04.096.544.335.240];   

Others  :  1136465
DOI  :  10.1186/1471-2288-12-109

 received in 2012-02-08, accepted in 2012-07-10,  发布年份 2012

【 摘 要 】

Background

The increased use and adoption of Electronic Health Records (EHR) causes a tremendous growth in digital information useful for clinicians, researchers and many other operational purposes. However, this information is rich in Protected Health Information (PHI), which severely restricts its access and possible uses. A number of investigators have developed methods for automatically de-identifying EHR documents by removing PHI, as specified in the Health Insurance Portability and Accountability Act “Safe Harbor” method.

This study focuses on the evaluation of existing automated text de-identification methods and tools, as applied to Veterans Health Administration (VHA) clinical documents, to assess which methods perform better with each category of PHI found in our clinical notes; and when new methods are needed to improve performance.

Methods

We installed and evaluated five text de-identification systems “out-of-the-box” using a corpus of VHA clinical documents. The systems based on machine learning methods were trained with the 2006 i2b2 de-identification corpora and evaluated with our VHA corpus, and also evaluated with a ten-fold cross-validation experiment using our VHA corpus. We counted exact, partial, and fully contained matches with reference annotations, considering each PHI type separately, or only one unique ‘PHI’ category. Performance of the systems was assessed using recall (equivalent to sensitivity) and precision (equivalent to positive predictive value) metrics, as well as the F₂-measure.

Results

Overall, systems based on rules and pattern matching achieved better recall, and precision was always better with systems based on machine learning approaches. The highest “out-of-the-box” F₂-measure was 67% for partial matches; the best precision and recall were 95% and 78%, respectively. Finally, the ten-fold cross validation experiment allowed for an increase of the F₂-measure to 79% with partial matches.

Conclusions

The “out-of-the-box” evaluation of text de-identification systems provided us with compelling insight about the best methods for de-identification of VHA clinical documents. The errors analysis demonstrated an important need for customization to PHI formats specific to VHA documents. This study informed the planning and development of a “best-of-breed” automatic de-identification application for VHA clinical text.

【 授权许可】

   
2012 Ferrández et al.; licensee BioMed Central Ltd.

【 图 表 】

Figure 1 .

Figure 2 .

Figure 3 .

Figure 4 .

【 参考文献 】

• [1]: GPO US: 45 C.F.R. § 46 Protection of Human Subjects. , ; . http://www.access.gpo.gov/nara/cfr/waisidx_08/45cfr46_08.html
• [2]: GPO US: 45 C.F.R. § 164 Security and Privacy. , ; . http://www.access.gpo.gov/nara/cfr/waisidx_08/45cfr164_08.html
• [3]Meystre SM, Friedlin FJ, South BR, Shen S, Samore MH: Automatic de-identification of textual documents in the electronic health record: a review of recent research. BMC Med Res Methodol 2010, 10:70. BioMed Central Full Text
• [4]Meystre SM, Savova GK, Kipper-Schuler KC, Hurdle JF: Extracting information from textual documents in the electronic health record: a review of recent research. Yearb Med Inform 2008, 47:128-144.
• [5]Grishman R, Sundheim B: Message understanding conference-6: A brief history. In New York. Assoc for Comput Linguist 1996, 1:466-471.
• [6]Beckwith BA, Mahaadevan R, Balis UJ, Kuo F: Development and evaluation of an open source software tool for deidentification of pathology reports. BMC Med Inform Decis Mak 2006, 6:12. BioMed Central Full Text
• [7]Friedlin FJ, McDonald CJ: A software tool for removing patient identifying information from clinical documents. J Am Med Inform Assoc 2008, 15:601-610.
• [8]Gupta D, Saul M, Gilbertson J: Evaluation of a Deidentification (De-Id) Software Engine to Share Pathology Reports and Clinical Documents for Research. Am J Clin Pathol 2004, 121:176-186.
• [9]Neamatullah I, Douglass MM, Lehman L-wei H, Reisner A, Villarroel M, Long WJ, Szolovits P, Moody GB, Mark RG, Clifford GD: Automated de-identification of free-text medical records. BMC Med Inform Decis Mak 2008, 8:32. BioMed Central Full Text
• [10]Aberdeen J, Bayer S, Yeniterzi R, Wellner B, Clark C, Hanauer D, Malin B, Hirschman L: The MITRE Identification Scrubber Toolkit: design, training, and assessment. Int J Med Inform 2010, 79:849-859.
• [11]Aramaki E, Imai T, Miyo K, Ohe K: Automatic deidentification by using sentence features and label consistency. In i2b2 Workshop on Challenges in Natural Language Processing for Clinical Data. , Washington, DC; 2006.
• [12]Gardner J, Xiong L: An integrated framework for de-identifying unstructured medical data. Data Knowl Eng 2009, 68:1441-1451.
• [13]Guo Y, Gaizauskas R, Roberts I, Demetriou G, Hepple M: Identifying Personal Health Information Using Support Vector Machines Training sub-system Testing sub-system The Preprocessing sub-system. In i2b2 Workshop on Challenges in Natutal Language Processing for Clinical Data. , Washington, DC; 2006.
• [14]Hara K: Applying a SVM Based Chunker and a Text Classifier to the Deid Challenge. In 2b2 Workshop on Challenges in Natural Language Processing for Clinical Data. , Washington, DC; 2006.
• [15]Szarvas G, Farkas R, Busa-Fekete R: State-of-the-art anonymization of medical records using an iterative machine learning framework. J Am Med Inform Assoc 2007, 14:574-580.
• [16]Uzuner O, Sibanda TC, Luo Y, Szolovits P: A de-identifier for medical discharge summaries. Artif Intell Med 2008, 42:13-35.
• [17]: Frequently Occurring First Names and Surnames From the 1990 Census. , ; . http://www.census.gov/genealogy/names/
• [18]: Health Level 7. HL7 Web site. , ; . http://www.hl7.org/
• [19]Uzuner O, Luo Y, Szolovits P: Evaluating the state-of-the-art in automatic de-identification. J Am Med Inform Assoc 2007, 14:550-563.
• [20]Dalianis H, Velupillai S: De-identifying Swedish clinical text - refinement of a gold standard and experiments with Conditional random fields. J Biomed Semantics 2010, 1:6. BioMed Central Full Text
• [21]Ohm P: Broken Promises of Privacy: Responding to the Surprising Failure of Anonymisation. Ucla Law Review 2009, 57:1701-1777.
• [22]Berman JJ: Concept-match medical data scrubbing. How pathology text can be used in research. Arch Pathol Lab Med 2003, 127:680-686.