Training binary neural networks with knowledge transfer

NEUROCOMPUTING

卷:396

Article

Leroux, Sam¹  Vankeirsbilck, Bert¹  Verbelen, Tim¹  Simoens, Pieter¹  Dhoedt, Bart¹ 

[1] Ghent Univ IMEC, Dept Informat Technol, IDLab, iGent Tower Technol Pk Zwijnaarde 15, B-9052 Ghent, Belgium

关键词: Binary neural networks;    Deep learning;    Knowledge transfer;   

DOI  :  10.1016/j.neucom.2018.09.103

来源: Elsevier

【 摘 要 】

Binary Neural Networks (BNNs) use binary values for both weights and activations instead of 32 bit floating point numbers typically used in deep neural networks. This reduces the memory footprint by a factor of 32 and allows a very efficient implementation in hardware. BNNs are trained using regular gradient descent but are harder to optimise, take longer to train and generally require a more careful tuning of hyperparameters such as the learning rate decay schedule than floating point versions. We propose to use Knowledge Transfer techniques to make it easier to train BNNs. Knowledge transfer is a general technique that tries to transfer the knowledge stored in a large network (the teacher) to a smaller (student) network. In our case the teacher is a network trained with floating point weights and activations while the student is a BNN. We apply different Knowledge Transfer techniques to the task of training a BNN. We introduce a novel similarity based Knowledge Transfer algorithm and show that this technique results in a higher test accuracy on different benchmark datasets compared to training the BNN from scratch. (C) 2019 Elsevier B.V. All rights reserved.

【 授权许可】

Free