基于多新息理论的深度信念网络算法

doi:10.11772/j.issn.1001-9081.2016.09.2521

计算机应用 ›› 2016, Vol. 36 ›› Issue (9): 2521-2525.DOI: 10.11772/j.issn.1001-9081.2016.09.2521

基于多新息理论的深度信念网络算法

李萌, 秦品乐, 李传朋

中北大学计算机与控制工程学院, 太原 030051

收稿日期:2016-02-24 修回日期:2016-03-30 出版日期:2016-09-10 发布日期:2016-09-08
通讯作者: 秦品乐
作者简介:李萌(1990-),女,山西运城人,硕士研究生,主要研究方向:计算机视觉、数字图像处理、深度学习;秦品乐(1978-),男,山西太原人,副教授,博士,CCF会员,主要研究方向:大数据、机器视觉、三维重建;李传朋(1991-),男,山东济南人,硕士研究生,主要研究方向:计算机视觉、数字图像处理、深度学习。
基金资助:
山西省自然科学基金资助项目（2015011045）。

Deep belief network algorithm based on multi-innovation theory

LI Meng, QIN Pingle, LI Chuanpeng

School of Computer Science and Control Engineering, North University of China, Taiyuan Shanxi 030051, China

Received:2016-02-24 Revised:2016-03-30 Online:2016-09-10 Published:2016-09-08
Supported by:
This work is partially supported by the Natural Science Foundation of Shanxi Province (2015011045).

摘要/Abstract

摘要： 针对深度信念网络（DBN）算法在采用反向传播修正网络的连接权值和偏置的过程中，容易产生梯度小、学习率低、误差收敛速度慢等问题，提出一种结合多新息理论对标准DBN算法进行改进的算法，即多新息DBN（MI-DBN）。MI-DBN算法是对标准DBN算法中反向传播的过程重新建模，使得算法在原先只利用单个新息的情况下，扩展为能够充分利用之前多个周期的新息，从而大幅提高误差收敛速度。通过实验对MI-DBN算法和其他分类算法进行了数据集分类的比较，实验结果表明，MI-DBN算法相较其他分类算法，其误差收敛速度较快，而且最终对MNIST数据集和Caltech101数据集的识别中误差结果相对更小。

关键词: 深度信念网络算法, 误差收敛速度, 多新息理论, 反向传播

Abstract: Aiming at the problem of small gradient, low learning rate, slow convergence of error during the process of using Deep Belief Network (DBN) algorithm to correct connection weight and bias of network by the method of back propagation, a new algorithm called Multi-Innovation DBN (MI-DBN) was proposed based on combination of standard DBN algorithm with multi-innovation theory. The back propagation process in standard DBN algorithm was remodeled to make full use of multiple innovations in previous cycles, while the original algorithm can only use single innovation. Thus, the convergence rate of error was significantly increased. MI-DBN algorithm and other representative classifiers were compared through experiments of datasets classification. Experimental results show that MI-DBN algorithm has a faster convergence rate than other sorting algorithms; especially when identifying MNIST and Caltech101 dataset, MI-DBN algorithm has the fewest inaccuracies among all the algorithms.

Key words: Deep Belief Network (DBN) algorithm, error convergence rate, multi-innovation theory, back-propagation

中图分类号:

TP389.1

李萌, 秦品乐, 李传朋. 基于多新息理论的深度信念网络算法[J]. 计算机应用, 2016, 36(9): 2521-2525.

LI Meng, QIN Pingle, LI Chuanpeng. Deep belief network algorithm based on multi-innovation theory[J]. Journal of Computer Applications, 2016, 36(9): 2521-2525.

参考文献

[1] LECUN Y, BENGIO Y, HINTON G E, et al. Deep learning [J]. Nature, 2015, 521(7553): 436-444.
[2] HINTON G E, OSINDERO S, TEH Y W. A fast learning algorithm for deep belief nets [J]. Neural Computation, 2006, 18(7): 1527-1554.
[3] LAROCHELLE H, ERHAN D, COURVILLE A, et al. An empirical evaluation of deep architectures on problems with many factors of variation [C]// ICML '07: Proceedings of the 2007 24th International Conference on Machine Learning. New York: ACM, 2007: 473-480.
[4] KEYVANRAD M A, HOMAYOUNPOUR M M. Deep belief network training improvement using elite samples minimizing free energy [EB/OL]. [2015-11-22]. http://xueshu.baidu.com/s?wd=paperuri%3A%282e0ed0ef0b45da606b7629105f1f17ed%29&filter=sc_long_sign&tn=SE_xueshusource_2kduw22v&sc_vurl=http%3A%2F%2Farxiv.org%2Fpdf%2F1411.4046v1&ie=utf-8&sc_us=758889154434608003.
[5] LIU Y, ZHOU S, CHEN Q. Discriminative deep belief networks for visual data classification [J]. Pattern Recognition, 2011, 44(10/11): 2287-2296.
[6] HINTON G E, SALAKHUTDINOV R. Reducing the dimensionality of data with neural networks [J]. Science, 2016, 313(5786): 504-507.
[7] 丁锋,萧德云,丁韬.多新息随机梯度辨识方法[J].控制理论与应用,2003,20(6):870-874.(DING F, XIAO D Y, DING T. Multi-innovation stochastic gradient identification methods [J]. Control Theory and Applications, 2003, 20(6): 870-874.)
[8] DING F. Several multi-innovation identification methods [J]. Digital Signal Processing, 2010, 20(4): 1027-1039.
[9] HINTON G E. A practical guide to training restricted Boltzmann machines [C]// Neural Networks: Tricks of the Trade, LNCS 7700. Berlin: Springer, 2012: 599-619.
[10] SWERSKY K, CHEN B, MARLIN B, et al. A tutorial on stochastic approximation algorithms for training restricted Boltzmann machines and deep belief nets [C]// Proceedings of the 2010 Information Theory and Applications Workshop. Piscataway, NJ: IEEE, 2010:1-10.
[11] 丁洁,谢莉,丁锋. 非均匀采样系统多新息随机梯度辨识性能分析[J].控制与决策,2011,26(9):1338-1342.(DING J, XIE L, DING F. Performance analysis of multi-innovation stochastic gradient identification for non-uniformly sampled systems [J]. Control and Decision, 2011, 26(9): 1338-1342.)
[12] 丁锋.系统辨识(6):多新息辨识理论与方法[J].南京信息工程大学学报,2012,4(1):1-28.(DING F. System identification. Part F: multi-innovation identification theory and methods [J]. Journal of Nanjing University of Information Science and Technology, 2012, 4(1):1-28.)
[13] 丁锋,杨家本.衰减激励条件下确定性系统多新息辨识的收敛性分析[J].清华大学学报(自然科学版),1998,38(9):111-115.(DING F, YANG J B. Convergence of multi-innovation identification under attenuating excitation conditions for deterministic systems [J]. Journal of Tsinghua University (Science and Technology), 1998,38(9):111-115.)
[14] LEE H, EKANADHAM C, NG A Y. Sparse deep belief net model for visual area V2 [EB/OL]. [2015-11-26]. http://web.eecs.umich.edu/~honglak/nips07-sparseDBN.pdf.
[15] KRIZHEVSKY A, Learning multiple layers of features from tiny images [D]. Toronto: University of Toronto, 2009:17.
[16] LECUN Y, CORTES C. The MNIST database of handwritten digits [DB/OL]. [2011-12-20]. http://yann.lecun.com/exdb/mnist/index.html.
[17] LI F F, FERGUS R, PERONA P. Learning generative visual models from few training examples: an incremental Bayesian approach tested on 101 object categories [C]// CVPRW '04: Proceedings of the 2004 Conference on Computer Vision and Pattern Recognition. Washington, DC: IEEE Computer Society, 2004,12: 178.
[18] KEYVANRAD M A, HOMAYOUNPOUR M M. A brief survey on deep belief networks and introducing a new object oriented Matlab toolbox (DeeBNetV2.2) [EB/OL]. [2015-11-06]. https://www.researchgate.net/publication/264790642_A_brief_survey_on_deep_belief_networks_and_introducing_a_new_object_oriented_MATLAB_toolbox_DeeBNet_V20.
[19] COLLOBERT R, SINZ F, WESTON J, et al. Large scale transductive SVMs [J]. Journal of Machine Learning Research, 2006, 7: 1687-1712.
[20] HAGAN M T, DEMUTH H B, BEALE M. Neural Network Design [M]. Beijing: China Machine Press, 2002: 357.
[21] SALAKHUTDINOV R, HINTON G E. Learning a nonlinear embedding by preserving class neighbourhood structure [J]. Journal of Machine Learning Research, 2007, 2: 412-419.
[22] TIELEMAN T, HINTON G. Using fast weights to improve persistent contrastive divergence [C]// ICML'09: Proceedings of the 26th Annual International Conference on Machine Learning. New York: ACM, 2009:1033-1040.

基于多新息理论的深度信念网络算法

Deep belief network algorithm based on multi-innovation theory

PDF

可视化

摘要/Abstract

引用本文

使用本文

参考文献

相关文章 14

编辑推荐

Metrics

[1]	褚苏荣, 牛之贤, 宋春花, 牛保宁. 面向移动端的渐进网格简化算法[J]. 计算机应用, 2020, 40(3): 806-811.
[2]	韩颖, 赵萌, 陈胜勇, 王照锡. 基于最大截面特征的病变宫颈细胞核的自动筛查[J]. 计算机应用, 2019, 39(4): 1189-1195.
[3]	陈浩广, 王银河. 基于扩展BP神经网络的一类非线性系统自适应控制设计[J]. 计算机应用, 2017, 37(6): 1670-1673.
[4]	刘巧玲, 李劲, 肖人彬. 基于参数反演的网络舆情传播趋势预测——以新浪微博为例[J]. 计算机应用, 2017, 37(5): 1419-1423.
[5]	征察, 吉立新, 李邵梅, 高超. 基于多模态信息融合的新闻图像人脸标注[J]. 计算机应用, 2017, 37(10): 3006-3011.
[6]	李世宝, 陈瑞祥, 刘建航, 陈海华, 丁淑妍, 龚琛. 基于反向传播神经网络改进的增益修改卡尔曼滤波算法[J]. 计算机应用, 2016, 36(5): 1196-1200.
[7]	赵虎, 杨宇. 基于迭代式MapReduce的误差反向传播算法[J]. 计算机应用, 2016, 36(4): 923-926.
[8]	邵良杉, 郭雅婵. 基于Codebook的视频火焰识别算法[J]. 计算机应用, 2015, 35(5): 1483-1487.
[9]	罗丹, 罗海勇. 基于随机森林的跌倒检测算法[J]. 计算机应用, 2015, 35(11): 3157-3160.
[10]	张玲王玲吴桐. 基于改进的粒子群算法优化反向传播神经网络的热舒适度预测模型[J]. 计算机应用, 2014, 34(3): 775-779.
[11]	徐红艳党晓婉冯勇李军平. 基于BP神经网络的Deep Web实体识别方法[J]. 计算机应用, 2013, 33(03): 776-779.
[12]	孟红飞牛建强杨瑞瑞段世忠. 基于香蕉雷达回波图像的特征提取与识别[J]. 计算机应用, 2011, 31(02): 413-415.
[13]	马东昱孙龙清. 基于图像特征的籽棉品级分级模型的研究[J]. 计算机应用, 2010, 30(8): 2235-2238.
[14]	张宏斌贾志新. 基于逆向思维的系统可靠性精确分配模型[J]. 计算机应用, 2009, 29(09): 2468-2470.