Beware the Null Hypothesis: Critical Value Tables for Evaluating Classifiers

Forman, George ; Cohen, Ira

HP Development Company

关键词: statistical significance;    critical value tables;    benchmark comparison;    F-measure;    precision in the top 10;    small test sets;    skewed/unbalanced class distribution;   

RP-ID  :  HPL-2005-70

学科分类：计算机科学（综合）

美国|英语

来源: HP Labs

【 摘 要 】

Scientists regularly decide the statistical significance of their findings by determining whether they can, with sufficient confidence, rule out the possibility that their findings could be attributed to random variation—the ‘null hypothesis.’ For this, they rely on tables with critical values pre- computed for the normal distribution, the t- distribution, etc. This paper provides such tables (and methods for generating them) for the performance metrics of binary classification: accuracy, F-measure, area under the ROC curve (AUC), and true positives in the top ten. Given a test set of a certain size, the tables provide the critical value for accepting or rejecting the null hypothesis that the score of the best classifier would be consistent with taking the best of a set of random classifiers. The tables are appropriate to consult when a researcher, practitioner or contest manager selects the best of many classifiers measured against a common test set. The risk of the null hypothesis is especially high when there is a shortage of positives or negatives in the testing set (irrespective of the training set size), as is the case for many medical and industrial classification tasks with highly skewed class distributions. Notes: Copyright 2005 Springer-Verlag. Published in and presented at the 16th European Conference on Machine Learning, 3-7 October 2005, Porto, Portugal. http://ecmlpkdd05.liacc.up.pt/ 28 Pages