Improving a gold standard: treating human relevance judgments of MEDLINE document pairs

BMC Bioinformatics

Research

Won Kim¹  W John Wilbur¹ 

[1] National Center for Biotechnology Information, National Library of Medicine, National Institute of Health, 20894, Bethesda, MD, USA;

关键词: Regularization Parameter;    Maximum Entropy Method;    Human Judgment;    True Label;    Document Pair;   

DOI  :  10.1186/1471-2105-12-S3-S5

来源: Springer

【 摘 要 】

Given prior human judgments of the condition of an object it is possible to use these judgments to make a maximal likelihood estimate of what future human judgments of the condition of that object will be. However, if one has a reasonably large collection of similar objects and the prior human judgments of a number of judges regarding the condition of each object in the collection, then it is possible to make predictions of future human judgments for the whole collection that are superior to the simple maximal likelihood estimate for each object in isolation. This is possible because the multiple judgments over the collection allow an analysis to determine the relative value of a judge as compared with the other judges in the group and this value can be used to augment or diminish a particular judge’s influence in predicting future judgments. Here we study and compare five different methods for making such improved predictions and show that each is superior to simple maximal likelihood estimates.

【 授权许可】

CC BY   
© The Author(s) 2011. This article is published under license to BioMed Central Ltd. This article is in the public domain.