Exact p-values for pairwise comparison of Friedman rank sums, with application to comparing classifiers

BMC Bioinformatics

Methodology Article

Ben Pelzer¹  Manfred Te Grotenhuis¹  Rob Eisinga¹  Tom Heskes² 

[1] Department of Social Science Research Methods, Radboud University Nijmegen, PO Box 9104,, 6500 HE, Nijmegen, The Netherlands;Institute for Computing and Information Sciences, Radboud University Nijmegen, Nijmegen, The Netherlands;

关键词: Friedman test;    p;    Rank sum difference;    Multiple comparison;    Nonparametric statistics;    Classifier comparison;    Machine learning;   

DOI  :  10.1186/s12859-017-1486-2

 received in 2016-07-17, accepted in 2017-01-11,  发布年份 2017

来源: Springer

【 摘 要 】

BackgroundThe Friedman rank sum test is a widely-used nonparametric method in computational biology. In addition to examining the overall null hypothesis of no significant difference among any of the rank sums, it is typically of interest to conduct pairwise comparison tests. Current approaches to such tests rely on large-sample approximations, due to the numerical complexity of computing the exact distribution. These approximate methods lead to inaccurate estimates in the tail of the distribution, which is most relevant for p-value calculation.ResultsWe propose an efficient, combinatorial exact approach for calculating the probability mass distribution of the rank sum difference statistic for pairwise comparison of Friedman rank sums, and compare exact results with recommended asymptotic approximations. Whereas the chi-squared approximation performs inferiorly to exact computation overall, others, particularly the normal, perform well, except for the extreme tail. Hence exact calculation offers an improvement when small p-values occur following multiple testing correction. Exact inference also enhances the identification of significant differences whenever the observed values are close to the approximate critical value. We illustrate the proposed method in the context of biological machine learning, were Friedman rank sum difference tests are commonly used for the comparison of classifiers over multiple datasets.ConclusionsWe provide a computationally fast method to determine the exact p-value of the absolute rank sum difference of a pair of Friedman rank sums, making asymptotic tests obsolete. Calculation of exact p-values is easy to implement in statistical software and the implementation in R is provided in one of the Additional files and is also available at http://www.ru.nl/publish/pages/726696/friedmanrsd.zip.

【 授权许可】

CC BY   
© The Author(s). 2017

【 图 表 】

12864_2016_2682_Article_IEq39.gif

12864_2015_2118_Article_IEq5.gif

12894_2015_Article_81_TeX2GIF_IEq1.gif

12864_2016_3263_Article_IEq17.gif

12864_2017_3487_Article_IEq3.gif

12711_2017_365_Article_IEq133.gif

12880_2015_Article_68_TeX2GIF_IEq4.gif

12864_2016_2682_Article_IEq33.gif

12864_2017_4358_Article_IEq1.gif

12864_2016_2682_Article_IEq32.gif

12864_2016_2821_Article_IEq12.gif

12864_2017_3783_Article_IEq1.gif

12864_2017_4309_Article_IEq7.gif