Hypercolumn Sparsification for Low-Power Convolutional Neural Networks

ACM JOURNAL ON EMERGING TECHNOLOGIES IN COMPUTING SYSTEMS

Article; Proceedings Paper

Pilly, Praveen K.¹  Stepp, Nigel D.¹  Liapis, Yannis^1,2  Payton, David W.¹  Srinivasa, Narayan^1,3 

[1] HRL Labs LLC, Ctr Auton Comp, Informat & Syst Sci Lab, Malibu, CA 90265 USA.;Enthought Inc, Austin, TX 78701 USA.;Eta Compute, Westlake Village, CA 91362 USA.

关键词: Convolutional neural networks;    machine learning;    low precision;    sparse;    quantization;    low energy;    embedded systems;    object recognition;   

DOI  :  10.1145/3304104

来源: SCIE

【 摘 要 】

We provide here a novel method, called hypercolumn sparsification, to achieve high recognition performance for convolutional neural networks (CNNs) despite low-precision weights and activities during both training and test phases. This method is applicable to any CNN architecture that operates on signal patterns (e.g., audio, image, video) to extract information such as class membership. It operates on the stack of feature maps in each of the cascading feature matching and pooling layers through the processing hierarchy of the CNN by an explicit competitive process (k-WTA, winner take all) that generates a sparse feature vector at each spatial location. This principle is inspired by local brain circuits, where neurons tuned to respond to different patterns in the incoming signals from an upstream region inhibit each other using interneurons, such that only the ones that are maximally activated survive the quenching threshold. We show this process of sparsification is critical for probabilistic learning of low-precision weights and bias terms, thereby making pattern recognition amenable for energy-efficient hardware implementations. Further, we show that hypercolumn sparsification could lead to more data-efficient learning as well as having an emergent property of significantly pruning down the number of connections in the network. A theoretical account and empirical analysis are provided to understand these effects better.

【 授权许可】

Free