Domain-Based Benchmark Experiments: Exploratory and Inferential Analysis

Austrian Journal of Statistics

article

Manuel J. A. Eugster¹  Torsten Hothorn¹  Friedrich Leisch² 

[1] Institut für Statistik, LMU München;Institut für Angewandte Statistik und EDV, BOKU Wien

关键词: Benchmark Experiment;    Learning Algorithm;    Visualisation;    Inference;    Mixed-Effects Model;    Ranking.;   

DOI  :  10.17713/ajs.v41i1.185

学科分类：医学（综合）

来源: Austrian Statistical Society

【 摘 要 】

Benchmark experiments are the method of choice to compare learning algorithms empirically. For collections of data sets, the empirical performance distributions of a set of learning algorithms are estimated, compared, and ordered. Usually this is done for each data set separately. The present manuscript extends this single data set-based approach to a joint analysis for the complete collection, the so called problem domain. This enablesto decide which algorithms to deploy in a specific application or to compare newly developed algorithms with well-known algorithms on established problem domains.Specialized visualization methods allow for easy exploration of huge amounts of benchmark data. Furthermore, we take the benchmark experiment design into account and use mixed-effects models to provide a formal statistical analysis. Two domain-based benchmark experiments demonstrate our methods: the UCI domain as a well-known domain when one is developing a new algorithm; and the Grasshopper domain as a domain where we want to find the  best learning algorithm for a prediction component in an enterprise application software system.

【 授权许可】

CC BY