Increasing the Efficiency of High-Recall Information Retrieval

Information Retrieval;High Recall;Machine Learning;Active Learning

Zhang, Haotianadvisor:Cormack, Gordon ; advisor:Smucker, Mark ; affiliation1:Faculty of Mathematics ; Smucker, Mark ; Cormack, Gordon ;

University of Waterloo

关键词: High Recall;    Information Retrieval;    Machine Learning;    Doctoral Thesis;    Active Learning;   

Others  :  https://uwspace.uwaterloo.ca/bitstream/10012/14594/1/Zhang_Haotian.pdf

瑞士|英语

来源: UWSPACE Waterloo Institutional Repository

【 摘 要 】

The goal of high-recall information retrieval (HRIR) is to find all, or nearly all, relevant documents while maintaining reasonable assessment effort. Achieving high recall is a key problem in the use of applications such aselectronic discovery, systematic review, and construction of test collections forinformation retrieval tasks.State-of-the-art HRIR systems commonly rely on iterative relevance feedback in whichhuman assessors continually assess machine learning-selected documents.The relevance of the assessed documents is then fed back to the machine learning model to improve its ability to select the next set ofpotentially relevant documents for assessment. In many instances, thousands of human assessments might be required to achieve high recall. These assessments represent the main cost of such HRIRapplications. Therefore, their effectiveness in achieving high recallis limited by their reliance on human input when assessing the relevance ofdocuments. In this thesis, we test different methods in order to improve the effectiveness andefficiency of finding relevant documents using state-of-the-art HRIRsystem. With regard to the effectiveness, we try to build a machine-learnedmodel that retrieves relevant documents more accurately.For efficiency, we try to help human assessors makerelevance assessments more easily and quickly via our HRIR system.Furthermore, we try to establish a stopping criteria for the assessment process so as to avoid excessive assessment.In particular, we hypothesize that total assessment effort to achieve highrecall can be reduced by using shorter document excerpts(e.g., extractive summaries) in place of full documents for the assessment ofrelevance and using a high-recall retrieval system based on continuous activelearning (CAL). In order to test this hypothesis, we implemented ahigh-recall retrieval system based on state-of-the-art implementation of CAL. This high-recall retrieval system could display either full documents or short document excerpts for relevance assessment.A search engine was also integrated into our system to provideassessors the option of conducting interactive search and judging. We conducted a simulation study, and separately, a 50-person controlled user study to test our hypothesis.The results of the simulation study show that judging even a singleextracted sentence for relevance feedback may be adequate for CALto achieve high recall. The results of the controlled user studyconfirmed that human assessors were able to finda significantly larger number of relevant documents within limited time when they used thesystem with paragraph-length document excerpts as opposed to full documents.In addition, we found that allowing participants to compose and execute theirown search queries did not improve their ability to find relevantdocuments and, by some measures, impaired performance.Moreover, integrating sampling methods with activelearning can yield accurate estimates of the number of relevant documents, and thus avoid excessive assessments.