Stochastic m-estimators:controlling accuracy-cost tradeoffs in machine learning

Approximate inference;Semi-supervised learning;Parameter learning;Structured prediction;Graphical models

Dillon, Joshua V. ; Computing

University:Georgia Institute of Technology

Department:Computing

关键词: Approximate inference;    Semi-supervised learning;    Parameter learning;    Structured prediction;    Graphical models;   

Others  :  https://smartech.gatech.edu/bitstream/1853/42913/1/dillon_joshua_v_201112_phd.pdf

美国|英语

来源: SMARTech Repository

【 摘 要 】

m-Estimation represents a broad class of estimators, including least-squares and maximum likelihood, and is a widely used tool for statistical inference. Its successful application however, often requires negotiating physical resources for desired levels of accuracy. These limiting factors, which we abstractly refer as costs, may be computational, such as time-limited cluster access for parameter learning, or they may be financial, such as purchasing human-labeled training data under a fixed budget. This thesis explores these accuracy- cost tradeoffs by proposing a family of estimators that maximizes a stochastic variation of the traditional m-estimator.Such "stochastic m-estimators" (SMEs) are constructed by stitching together different m-estimators, at random. Each such instantiation resolves the accuracy-cost tradeoff differently, and taken together they span a continuous spectrum of accuracy-cost tradeoff resolutions. We prove the consistency of the estimators and provide formulas for their asymptotic variance and statistical robustness. We also assess their cost for two concerns typical to machine learning: computational complexity and labeling expense.For the sake of concreteness, we discuss experimental results in the context of a variety of discriminative and generative Markov random fields, including Boltzmann machines, conditional random fields, model mixtures, etc. The theoretical and experimental studies demonstrate the effectiveness of the estimators when computational resources are insufficient or when obtaining additional labeled samples is necessary.We also demonstrate that in some cases the stochastic m-estimator is associated with robustness thereby increasing its statistical accuracy and representing a win-win.