Distributed optimization for deep learning with gossip exchange

NEUROCOMPUTING

卷:330

Article

Blot, Michael¹  Picard, David^1,2  Thome, Nicolas^1,3  Cord, Matthieu¹ 

[1] Sorbonne Univ, CNRS, UPMC Univ Paris 06, LIP6 UMR 7606, 4 Pl Jussieu, F-75005 Paris, France

[2] Univ Cergy Pontoise, CNRS, Univ Paris Seine, ENSEA,ETIS UMR 8051, Paris, France

[3] Conservatoire Natl Arts & Metiers, CEDRIC, 292 Rue St Martin, F-75003 Paris, France

关键词: Optimization;    Distributed gradient descent;    Gossip;    Deep;    Learning;    Neural networks;   

DOI  :  10.1016/j.neucom.2018.11.002

来源: Elsevier

【 摘 要 】

We address the issue of speeding up the training of convolutional neural networks by studying a distributed method adapted to stochastic gradient descent. Our parallel optimization setup uses several threads, each applying individual gradient descents on a local variable. We propose a new way of sharing information between different threads based on gossip algorithms that show good consensus convergence properties. Our method called GoSGD has the advantage to be fully asynchronous and decentralized. (C) 2018 Published by Elsevier B.V.

【 授权许可】

Free