Training Restricted Boltzmann Machines Using Modified Objective Function Based on Limiting the Free Energy Value

IEEE Access

Song Guo¹  Changjun Zhou¹  Bin Wang²  Xuedong Zheng² 

[1] College of Mathematics and Computer Science, Zhejiang Normal University, Jinhua, China;Key Laboratory of Advanced Design and Intelligent Computing, Ministry of Education, Dalian University, Dalian, China;

关键词: Restricted Boltzmann machines;    deep learning;    feature extractor;    loss function;   

DOI  :  10.1109/ACCESS.2018.2885071

来源: DOAJ

【 摘 要 】

A restricted Boltzmann machine (RBM) is a generative model that can capture the probability distribution of data relevant to the problem domain and is usually deployed as the fundamental building block to form more complex deep architectures, such as deep belief network, deep Boltzmann machine, and deep stacked auto-encoder. In addition, the RBM itself can be used as a feature extractor to learn features from raw data. In addition, an RBM is a special type of energy-based model. This paper proposes a modified loss function as an example of an energy-based model defined by restricting the free energy value of the training data. This restriction punishes very low free energy value to reduce the model complexity, which is helpful to the training procedure of the RBM. We validate our method using the MNIST and MNIST-ROTATION datasets. Experiments reveal that the modified loss function behaves better in learning discriminative features as well as in providing better parameters when used to initialize deep feed-forward neural networks (DNN). The convergence speed of the DNN can improve by 44% on both datasets.

【 授权许可】

Unknown