Dual Free Adaptive Minibatch SDCA for Empirical Risk Minimization

Frontiers in Applied Mathematics and Statistics

He, Xi¹  Taká¹  Tappenden, Rachael² 

[1]Industrial and Systems Engineering, Lehigh University, United States

[2]School of Mathematics and Statistics, University of Canterbury, New Zealand

关键词: SDCA;    Importance sampling;    Non-uniform sampling;    mini-batch;    Adaptive sampling;   

DOI  :  10.3389/fams.2018.00033

学科分类：数学（综合）

来源: Frontiers

【 摘 要 】

In this paper, we develop an adaptive dual free Stochastic Dual Coordinate Ascent (adfSDCA) algorithm for regularized empirical risk minimization problems. This is motivated by the recent work on dual free SDCA of Shalev-Shwartz (2016). The novelty of our approach is that the coordinates to update at each iteration are selected non-uniformly from an adaptive probability distribution, and this extends the previously mentioned work which only allowed for a uniform selection of ``dual" coordinates from a fixed probability distribution. We describe an efficient iterative procedure for generating the non-uniform samples, where the scheme selects the coordinate with the greatest potential to decrease the sub-optimality of the current iterate. We also propose a heuristic variant of adfSDCA that is more aggressive than the standard approach. Furthermore, in order to utilize multi-core machines, we consider a mini-batch adfSDCA algorithm and develop complexity results that guarantee the algorithm's convergence. The work is concluded with several numerical experiments to demonstrate the practical benefits of the proposed approach.

【 授权许可】

CC BY