Build a compact binary neural network through bit-level sensitivity and data pruning

NEUROCOMPUTING

卷:398

Article

Li, Yixing¹  Zhang, Shuai²  Zhou, Xichuan²  Ren, Fengbo¹ 

[1] Arizona State Univ, Sch Comp Informat & Decis Syst Engn, Tempe, AZ 85281 USA

[2] Chongqing Univ, Sch Microelect & Commun Engn, Chongqing, Peoples R China

关键词: Binary neural networks;    Deep neural networks;    Deep learning;    Neural network compression;   

DOI  :  10.1016/j.neucom.2020.02.012

来源: Elsevier

【 摘 要 】

Due to the high computational complexity and memory storage requirement, it is hard to directly deploy a full-precision convolutional neural network (CNN) on embedded devices. The hardware-friendly designs are needed for resource-limited and energy-constrained embedded devices. Emerging solutions are adopted for the neural network compression, e.g., binary/ternary weight network, pruned network and quantized network. Among them, binary neural network (BNN) is believed to be the most hardware-friendly framework due to its small network size and low computational complexity. No existing work has further shrunk the size of BNN. In this work, we explore the redundancy in BNN and build a compact BNN (CBNN) based on the bit-level sensitivity analysis and bit-level data pruning. The input data is converted to a high dimensional bit-sliced format. In the post-training stage, we analyze the impact of different bit slices to the accuracy. By pruning the redundant input bit slices and shrinking the network size, we are able to build a more compact BNN. Our result shows that we can further scale down the network size of the BNN up to 3.9x with no more than 1% accuracy drop. The actual runtime can be reduced up to 2x and 9.9x compared with the baseline BNN and its full-precision counterpart, respectively. (C) 2020 Elsevier B.V. All rights reserved.

【 授权许可】

Free