Austrian Journal of Statistics | |
Domain-Based Benchmark Experiments: Exploratory and Inferential Analysis | |
article | |
Manuel J. A. Eugster1  Torsten Hothorn1  Friedrich Leisch2  | |
[1] Institut für Statistik, LMU München;Institut für Angewandte Statistik und EDV, BOKU Wien | |
关键词: Benchmark Experiment; Learning Algorithm; Visualisation; Inference; Mixed-Effects Model; Ranking.; | |
DOI : 10.17713/ajs.v41i1.185 | |
学科分类:医学(综合) | |
来源: Austrian Statistical Society | |
【 摘 要 】
Benchmark experiments are the method of choice to compare learning algorithms empirically. For collections of data sets, the empirical performance distributions of a set of learning algorithms are estimated, compared, and ordered. Usually this is done for each data set separately. The present manuscript extends this single data set-based approach to a joint analysis for the complete collection, the so called problem domain. This enablesto decide which algorithms to deploy in a specific application or to compare newly developed algorithms with well-known algorithms on established problem domains.Specialized visualization methods allow for easy exploration of huge amounts of benchmark data. Furthermore, we take the benchmark experiment design into account and use mixed-effects models to provide a formal statistical analysis. Two domain-based benchmark experiments demonstrate our methods: the UCI domain as a well-known domain when one is developing a new algorithm; and the Grasshopper domain as a domain where we want to find the best learning algorithm for a prediction component in an enterprise application software system.
【 授权许可】
CC BY
【 预 览 】
Files | Size | Format | View |
---|---|---|---|
RO202105240000206ZK.pdf | 510KB | download |