On the iteration complexity analysis of Stochastic Primal-Dual Hybrid Gradient approach with high probability

NEUROCOMPUTING

卷:307

Article

Qiao, Linbo¹  Lin, Tianyi²  Qin, Qi³  Lu, Xicheng¹ 

[1] Natl Univ Def Technol, Natl Lab Parallel & Distributed Proc, Changsha 410073, Hunan, Peoples R China

[2] Univ Calif Berkeley, Dept Ind Engn & Operat Res, Berkeley, CA 94720 USA

[3] Stanford Univ, Dept Stat, Stanford, CA 94305 USA

关键词: Stochastic Primal-Dual Hybrid Gradient;    Iteration complexity;    High probability;    Graph-guided regularized logistic regression;   

DOI  :  10.1016/j.neucom.2018.03.066

来源: Elsevier

【 摘 要 】

In this paper, we propose a stochastic Primal-Dual Hybrid Gradient (PDHG) approach for solving a wide spectrum of regularized stochastic minimization problems, where the regularization term is composite with a linear function. It has been recognized that solving this kind of problem is challenging since the closed-form solution of the proximal mapping associated with the regularization term is not available due to the imposed linear composition, and the per-iteration cost of computing the full gradient of the expected objective function is extremely high when the number of input data samples is considerably large. Our new approach overcomes these issues by exploring the special structure of the regularization term and sampling a few data points at each iteration. Rather than analyzing the convergence in expectation, we provide the detailed iteration complexity analysis for the cases of both uniformly and nonuniformly averaged iterates with high probability. This strongly supports the good practical performance of the proposed approach. Numerical experiments demonstrate that the efficiency of stochastic PDHG, which outperforms other competing algorithms, as expected by the high-probability convergence analysis. Keywords: Stochastic Primal-Dual Hybrid Gradient Iteration complexity High probability Graph-guided regularized logistic regression (C) 2018 Elsevier B.V. All rights reserved.

【 授权许可】

Free