The Effect of Topic Sampling on Sensitivity Comparisons of Information Retrieval Metrics

the Fifth NTCIR Workshop Meeting on Evaluation of Information Access Technologies: Information Retrieval, Question Answering and Cross-Lingual Information Access

计算机科学;图书情报档案学

Tetsuya Sakai

Others  :  http://research.nii.ac.jp/ntcir/workshop/OnlineProceedings5/data/OPEN/NTCIR5-OPEN-SakaiT.pdf
PID  :  52341

来源: CEUR

【 摘 要 】

The Voorhees/Buckley swap method is useful for comparing the discrimination power of Information Retrieval (IR) and Question Answering (QA) metrics. Given a test collection, a set of runs and an evaluation metric, it derives the swap rate, the chance of observing inconsistencies when two completely different topic sets are used for comparing a pair of runs. Recently, however, Sanderson and Zobel claimed that the method overestimates swap rates as it samples topics without replacement. The main question we address in this paper is whether sampling with and without replacement produce any different results for the purpose of comparing the sensitivity of different metrics. Our IR and QA experiments show that the two methods do generally yield similar results, which suggests that the original Voorhees/Buckley method is valid.