Bayesian tensorized neural networks with automatic rank selection

NEUROCOMPUTING

卷:453

Article

Hawkins, Cole¹  Zhang, Zheng² 

[1] Univ Calif Santa Barbara, Dept Math, Santa Barbara, CA 93106 USA

[2] Univ Calif Santa Barbara, Dept Elect & Comp Engn, Santa Barbara, CA 93106 USA

关键词: Low-rank tensor;    Tensor rank determination;    Neural network compression;   

DOI  :  10.1016/j.neucom.2021.04.117

来源: Elsevier

【 摘 要 】

Tensor decomposition is an effective approach to compress over-parameterized neural networks and to enable their deployment on resource-constrained hardware platforms. However, directly applying tensor compression in the training process is a challenging task due to the difficulty of choosing a proper tensor rank. In order to address this challenge, this paper proposes a low-rank Bayesian tensorized neural network. Our Bayesian method performs automatic model compression via an adaptive tensor rank determination. We also present approaches for posterior density calculation and maximum a posteriori (MAP) estimation for the end-to-end training of our tensorized neural network. We provide experimental validation on a two-layer fully connected neural network, a 6-layer CNN and a 110-layer residual neural network where our work produces 7.4x to 137x more compact neural networks directly from the training while achieving high prediction accuracy. (C) 2021 Elsevier B.V. All rights reserved.

【 授权许可】

Free