Review of second-order optimization techniques in artificial neural networks backpropagation

11th Curtin University Technology, Science and Engineering (CUTSE) International Conference

工业技术（总论）

Tan, Hong Hui^1 ; Lim, King Hann^1

Department of Electrical and Computer Engineering, Curtin University Malaysia, Sarawak, Miri

98009, Malaysia^1

关键词: Curvature information;    Fast convergence;    Hyper-parameter;    Levenberg-Marquardt;    Neural network training;    Objective functions;    Optimization techniques;    Second order optimization;   

Others  :  https://iopscience.iop.org/article/10.1088/1757-899X/495/1/012003/pdf
DOI  :  10.1088/1757-899X/495/1/012003

学科分类：工业工程学

来源: IOP

【 摘 要 】

Second-order optimization technique is the advances of first-order optimization in neural networks. It provides an addition curvature information of an objective function that adaptively estimate the step-length of optimization trajectory in training phase of neural network. With the additional information, it reduces training iteration and achieves fast convergence with less tuning of hyper-parameter. The current improved memory allocation and computing power further motivates machine learning practitioners to revisit the benefits of second-order optimization techniques. This paper covers the review on second-order optimization techniques that involve Hessian calculation for neural network training. It reviews the basic theory of Newton method, quasi-Newton, Gauss-Newton, Levenberg-Marquardt, Approximate Greatest Descent and Hessian-Free optimization. This paper summarizes the feasibility and performance of optimization techniques in deep neural network training. Comments and suggestions are highlighted for second-order optimization techniques in artificial neural network training in term of advantages and limitations.