Abstract：To resolve the problems of feature homogeneity in unsupervised training of Restricted Boltzmann Machine (RBM) and non-adaptiveness of Sparse Restricted Boltzmann Machine (SRBM), a new sparse mechanism method of RBM based on competitive learning was designed. Firstly, a distance measurement was designed based on the cosine value between the neuron weight vector and the input vector to evaluate the similarity. Secondly, the optimal matching implicit unit based on distance measurement was selected for different samples during training. Thirdly, the sparse penalty for other hidden units was calculated according to the activation state of the optimal matching hidden unit. Finally, the parameters were updated and the competitive sparseness was applied to the construction of Deep Boltzmann Machine (DBM) based on the deep model training process. The handwritten number recognition results show that, compared with the mechanism using the sum of squared errors as the regularization factor, the classification accuracy of DBM based on new sparse mechanism is improved by 0.74%, and the average sparsity measurement is increased by 5.6%, without the need to set sparse parameters. Therefore, the proposed sparse mechanism can improve the training efficiency of unsupervised training model, such as RBM, and can be applied into the construction of deep models.
[1] 曾安,张艺楠,潘丹,等.基于稀疏降噪自编码器的深度置信网络[J].计算机应用,2017,37(9):2585-2589.(ZENG A, ZHANG Y N, PAN D, et al. Deep belief networks based on sparse denoising auto encoders[J]. Journal of Computer Applications, 2017, 37(9):2585-2589.) [2] 张娟,杨建功,汪西莉.条件深度玻尔兹曼机人脸图像分割模型[J].小型微型计算机系统,2017,38(5):1130-1133.(ZHANG J, YANG J G, WANG X L. Conditional deep Boltzmann machine face image segmentation model[J]. Journal of Chinese Computer Systems, 2017, 38(5):1130-1133.) [3] 张立民,刘凯.基于深度玻尔兹曼机的文本特征提取研究[J].微电子学与计算机,2015,32(2):142-147.(ZHANG L M, LIU K. Document features extraction based on DBM[J]. Microelectronics & Computer, 2015, 32(2):142-147.) [4] 李楠,卢钢,李新利,等.基于集成深度玻尔兹曼机和最小二乘支持向量回归的燃烧过程NOx预测算法[J].动力工程学报,2016,36(8):615-620.(LI N, LU G, LI X L, et al. Nox emission prediction based on deep Boltzmann machine integrated with least square support vector regression[J]. Journal of Chinese Society of Power Engineering, 2016, 36(8):615-620.) [5] 刘帅师,程曦,郭文燕,等.深度学习方法研究新进展[J].智能系统学报,2016,11(5):567-577.(LIU S S, CHENG X, GUO W Y, et al. Progress report on new research in deep learning[J]. CAAI Transactions on Intelligent Systems, 2016, 11(5):567-577.) [6] 尹宝才,王文通,王立春.深度学习研究综述[J].北京工业大学学报,2015,41(1):48-59.(YI B C, WANG W T, WANG L C. Review of deep learning[J]. Journal of Beijing University of Technology. 2015, 41(1):48-59.) [7] ZHAO Z J, GU J W. Recognition of digital modulation signals based on hybrid three-order restricted Boltzmann machine[C]//Proceedings of the 2016 International Conference on Communication Technology. Piscataway, NJ:IEEE, 2016:166-169. [8] NAKASHIKA T, MIMAMI Y. Generative acoustic-phonemic-speaker model based on three-way restricted Boltzmann machine[C]//Proceedings of the 2016 International Conference on INTERSPEECH. Berlin:Springer, 2016:1487-1491. [9] DENTON E, CHINTALA S, SZLAMA, et al. Deep generative image models using a laplacian pyramid of adversarial networks[C]//Proceedings of the 2015 International Conference on Neural Information Processing Systems. Montréal:NIPS, 2015:1486-1494. [10] SRIVASTAVA N, HINTON G, KRIZHEVSKY A, et al. Dropout:a simple way to prevent neural networks from overfitting[J]. Journal of Machine Learning Research, 2014, 15(1):1929-1958. [11] 刘凯,张立民,周立军.随机受限玻尔兹曼机组设计[J].上海交通大学学报(自然版),2017,51(10):1235-1240.(LIU K, ZHANG L M, ZHOU L J. Design of random restricted boltzmann machine group[J]. Journal of Shanghai Jiaotong University (Science), 2017, 51(10):1235-1240.) [12] SALAKHUTDINOV R, HINTON G. An efficient learning procedure for deep Boltzmann machines[J]. Neural Computation, 2014, 24(8):1967-2006. [13] 刘凯,张立民,孙永威.基于遗传算法的RBM优化设计[J].微电子学与计算机,2015,32(6):96-100.(LIU K, ZHANG L M, SUN Y W. RBM optimization based on genetic algorithms[J]. Microelectronics & Computer, 2015, 32(6):96-100.) [14] CARLSON D, CEVHER V, CARIN L. Stochastic spectral descent for restricted Boltzmann machines[C]//Proceedings of the 2015 International Conference on Artificial Intelligence and Statistics. Berlin:Springer, 2015:111-119. [15] LEE H, EKANADHAM C, NG A Y. Sparse deep belief net model for visual area V2[C]//Proceedings of the 2008 International Conference on Advances in Neural Information Processing Systems. New York:Curran Associates Inc, 2008:873-880. [16] LUO H, SHEN R, NIU C. Sparse group restricted Boltzmann machines[C]//Proceedings of the 25th American Association for Artificial Intelligence Conference on Artificial Intelligence and the 23rd Innovative Applications of Artificial Intelligence Conference. San Francisco:AI Access Foundation, 2011:429-434. [17] JIN N, ZHANG J S, ZHANG C X. A sparse-response deep belief network based on rate distortion theory[J]. Pattern Recognition, 2014, 47(9):3179-3191. [18] TIELEMAN T, HINTON G E. Using fast weights to improve persistent contrastive divergence[C]//Proceedings of the 26th International Conference on Machine Learning. Madison:Omni Press, 2009:1033-1040. [19] 刘凯,张立民,张超.受限玻尔兹曼机的新混合稀疏惩罚机制[J].浙江大学学报(工学版),2015,49(6):1070-1078.(LIU K, ZHANG L M, ZHANG C. New hybrid sparse penalty mechanism of restricted Boltzmann machine[J]. Journal of Zhejiang University (Engineering Science), 2015, 49(6):1070-1078.) [20] 许曈,凌有铸,陈孟元.一种融合DGSOM神经网络的仿生算法研究[J].智能系统学报,2017,12(3):405-412.(XU T, LING Y Z, CHEN M Y. A bio-inspired algorithm integrated with DGSOM neural network[J]. CAAI Transactions on Intelligent Systems, 2017, 12(3):405-412.) [21] 王蕾,王连明.一种改进的基于STDP规则的SOM脉冲神经网络[J].东北师大学报(自然科学版),2017,49(3):52-56.(WANG L, WANG L M. An improved self-organizing map spiking neural networks based on STDP rule[J]. Journal of Northeast Normal University (Natural Science Edition), 2017, 49(3):52-56.) [22] LW C, BOTTOU L, BENGIO Y, et al. Gradient-based learning applied to document recognition[J]. Proceedings of the IEEE, 1998, 86(11):2278-2324. [23] CHANG C C, LIN C J. LIBSVM:a library for support vector machines[J]. ACM Transactions on Intelligent Systems and Technology, 2011, 2(3):27-31. [24] SALAKHUTDINOV R, HINTON G. Deep Boltzmann machines[J]. Journal of Machine Learning Research, 2009, 5(2):1967-2006. [25] HOYER P O. Non-negative matrix factorization with sparseness constraints[J]. Journal of Machine Learning Research, 2004, 5(1):1457-1469.
2018, Vol. 38 
