Deep belief networks based on sparse denoising auto encoders

doi:10.11772/j.issn.1001-9081.2017.09.2585

Journal of Computer Applications ›› 2017, Vol. 37 ›› Issue (9): 2585-2589.DOI: 10.11772/j.issn.1001-9081.2017.09.2585

Previous Articles Next Articles

Deep belief networks based on sparse denoising auto encoders

ZENG An¹, ZHANG Yinan¹, PAN Dan², Xiao-wei SONG³

1. Faculty of Computer Science, Guangdong University of Technology, Guangzhou Guangdong 510006, China;
2. Modern Education Technical Center, Guangdong Construction Polytechnic, Guangzhou Guangdong 510440, China;
3. ImageTech Lab, Simon Fraser University, Vancouver V6B 5K3, Canada

Received:2017-03-28 Revised:2017-06-07 Online:2017-09-10 Published:2017-09-13
Supported by:
This work is partially supported by the National Natural Science Foundation of China (61300107), the Natural Science Foundation of Guangdong, China (S2012010010212), the Science and Technology Program of Guangzhou (201504301341059,201505031501397).

基于稀疏降噪自编码器的深度置信网络

曾安¹, 张艺楠¹, 潘丹², Xiao-Wei Song³

1. 广东工业大学计算机学院, 广州 510006;
2. 广东建设职业技术学院现代教育技术中心, 广州 510440;
3. 西蒙弗雷泽大学影像技术实验室, 加拿大温哥华 V6B 5K3

通讯作者: 潘丹,2656351065@qq.com
作者简介:曾安(1978-),女,湖南新化人,教授,博士,CCF会员,主要研究方向:人工智能、数据挖掘;张艺楠(1993-),女,广东兴宁人,硕士研究生,主要研究方向:数据挖掘;潘丹(1975-),男,广东兴宁人,高级工程师,博士,主要研究方向:人工智能、数据挖掘、大数据;Xiao-Wei Song (1962-),女,北京人,研究员,博士,主要研究方向:脑科学、神经影像。
基金资助:
国家自然科学基金资助项目（61300107）；广东省自然科学基金资助项目（S2012010010212）；广州市科技计划资助项目（201504301341059， 201505031501397）。

Abstract

Abstract: The conventional Deep Belief Network (DBN) often utilizes the method of randomly initializing the weights and bias of Restricted Boltzmann Machine(RBM) to initialize the network. Although it could overcome the problems of local optimality and long training time to some extent, it is still difficult to further achieve higher accuracy and better learning efficiency owing to the huge difference between reconstruction and original input resulting from random initialization. In view of the above-mentioned problem, a kind of DBN model based on Sparse Denoising AutoEncoder (SDAE) was proposed. The advantage of the advocated model was the feature extraction by SDAE. Firstly, SDAE was trained, and then, the obtained weights and bias were utilized to initialize DBN. Finally, DBN was trained. Experiments were performed on card game data set of Poker hand and handwriting data sets of MNIST and USPS to verify the performance of the proposed model. In Poker hand data set, compared with the conventional DBN, the error rate of the proposed model is lowered by 46.4%, the accuracy rate and the recall rate are improved by 15.56% and 14.12% respectively. The results exhibit that the proposed method is superior to other existing methods in recognition performance.

Key words: Deep Belief Network (DBN), Restricted Boltzmann Machine (RBM), Sparse Denoising AutoEncoder (SDAE), deep learning

摘要： 传统的深度置信网络（DBN）采用随机初始化受限玻尔兹曼机（RBM）的权值和偏置的方法初始化网络。虽然这在一定程度上克服了由BP算法带来的易陷入局部最优和训练时间长的问题，但随机初始化仍然会导致网络重构和原始输入的较大差别，这使得网络无论在准确率还是学习效率上都无法得到进一步提升。针对以上问题，提出一种基于稀疏降噪自编码器（SDAE）的深度网络模型，其核心是稀疏降噪自编码器对数据的特征提取。首先，训练稀疏降噪自编码；然后，用训练后得到的权值和偏置来初始化深度置信网络；最后，训练深度置信网络。在Poker Hand 纸牌游戏数据集和MNIST、USPS手写数据集上测试模型性能，在Poker Hand数据集下，方法的误差率比传统的深度置信网络降低46.4%，准确率和召回率依次提升15.56%和14.12%。实验结果表明，所提方法能有效地改善模型性能。

关键词: 深度置信网络, 受限玻尔兹曼机, 稀疏降噪自编码器, 深度学习

CLC Number:

TP183

ZENG An, ZHANG Yinan, PAN Dan, Xiao-wei SONG. Deep belief networks based on sparse denoising auto encoders[J]. Journal of Computer Applications, 2017, 37(9): 2585-2589.

曾安, 张艺楠, 潘丹, Xiao-Wei Song. 基于稀疏降噪自编码器的深度置信网络[J]. 计算机应用, 2017, 37(9): 2585-2589.

References

[1] LECUN Y, BENGIO Y, HINTON G. Deep learning[J]. Nature, 2015, 521(7553):436-444.
[2] ERHAN D, BENGIO Y, COURVILLE A, et al. Why does unsupervised pre-training help deep learning?[J]. Journal of Machine Learning Research, 2010, 11(3):625-660.
[3] MOHAMED A R, DAHL G E, HINTON G. Acoustic modeling using deep belief networks[J]. IEEE Transactions on Audio Speech & Language Processing, 2012, 20(1):14-22.
[4] WALID R, LASFAR A. Handwritten digit recognition using sparse deep architectures[C]//Proceedings of the 20149th International Conference on Intelligent Systems:Theories and Applications. Piscataway, NJ:IEEE, 2014:1-6.
[5] BU S, LIU Z, HAN J, et al. Learning high-level feature by deep belief networks for 3-D model retrieval and recognition[J]. IEEE Transactions on Multimedia, 2014, 16(8):2154-2167.
[6] SARIKAYA R, HINTON G E, DEORAS A. Application of deep belief networks for natural language understanding[J]. IEEE/ACM Transactions on Audio Speech & Language Processing, 2014, 22(4):778-784.
[7] HINTON G. A practical guide to training restricted Boltzmann machines[EB/OL].[2016-12-12]. http://www.csri.utoronto.ca/~hinton/absps/guideTR.pdf.
[8] HINTON G E. Training products of experts by minimizing contrastive divergence[J]. Neural Computation, 2002, 14(8):1771-1800.
[9] 潘广源,柴伟,乔俊飞.DBN网络的深度确定方法[J].控制与决策,2015,30(2):256-260.(PAN G Y, CHAI W, QIAO J F. Calculation for depth of deep belief network[J]. Control and Decision, 2015, 30(2):256-260.)
[10] SRIVASTAVA N, HINTON G, KRIZHEVSKY A, et al. Dropout:a simple way to prevent neural networks from overfitting[J]. Journal of Machine Learning Research, 2014, 15(1):1929-1958.
[11] RANZATO M, BOUREAU Y L, LECUN Y. Sparse feature learning for deep belief networks[J]. Advances in Neural Information Processing Systems, 2007, 20:1185-1192.
[12] 胡振,傅昆,张长水.基于深度学习的作曲家分类问题[J].计算机研究与发展,2014,51(9):1945-1954.(HU Z, FU K, ZHANG C S. Audio classical composer identification by deep neural network[J]. Journal of Computer Research and Development, 2014, 51(9):1945-1954.)
[13] BENGIO Y, LAMBLIN P, POPOVICI D, et al. Greedy layer-wise training of deep networks[C]//Proceedings of the 19th International Conference on Neural Information Processing Systems. Cambridge, MA:MIT Press, 2006:153-160.
[14] VINCENT P, LAROCHELLE H, BENGIO Y, et al. Extracting and composing robust features with denoising autoencoders[C]//Proceedings of the 25th International Conference on Machine Learning. New York:ACM, 2008:1096-1103.
[15] SALAKHUTDINOV R, HINTON G. Deep Boltzmann machines[J]. Journal of Machine Learning Research, 2009, 5(2):1967-2006.
[16] BENGIO Y, COURVILLE A C, VINCENT P. Unsupervised feature learning and deep learning:a review and new perspectives[EB/OL].[2016-12-22]. http://docs.huihoo.com/deep-learning/Representation-Learning-A-Review-and-New-Perspectives-v1.pdf.
[17] VINCENT P, LAROCHELLE H, LAJOIE I, et al. Stacked denoising autoencoders:learning useful representations in a deep network with a local denoising criterion[J]. Journal of Machine Learning Research, 2010, 11:3371-3408.
[18] LIU Y, ZHOU S, CHEN Q. Discriminative deep belief networks for visual data classification[J]. Pattern Recognition, 2011, 44(10/11):2287-2296.
[19] XIE J, XU L, CHEN E. Image denoising and inpainting with deep neural networks[EB/OL].[2016-11-27]. http://staff.ustc.edu.cn/~linlixu/papers/nips12.pdf.
[20] DENG L. A tutorial survey of architectures, algorithms, and applications for deep learning[EB/OL].[2017-01-10]. https://www.cambridge.org/core/services/aop-cambridge-core/content/view/S2048770314000043.
[21] HINTON G E, SALAKHUTDINOV R R. Reducing the dimensionality of data with neural networks[J]. Science, 2006, 313(5786):504-507.

Deep belief networks based on sparse denoising auto encoders

基于稀疏降噪自编码器的深度置信网络

PDF

Knowledge

Abstract

Cite this article

share this article

References

Related Articles 15

Recommended Articles

Metrics

[1]	XIE Defeng, JI Jianmin. Syntax-enhanced semantic parsing with syntax-aware representation [J]. Journal of Computer Applications, 2021, 41(9): 2489-2495.
[2]	DAI Yurou, YANG Qing, ZHANG Fengli, ZHOU Fan. Trajectory prediction model of social network users based on self-supervised learning [J]. Journal of Computer Applications, 2021, 41(9): 2545-2551.
[3]	CHEN Chengrui, SUN Ning, HE Shibiao, LIAO Yong. Deep learning-based joint channel estimation and equalization algorithm for C-V2X communications [J]. Journal of Computer Applications, 2021, 41(9): 2687-2693.
[4]	ZHAO Hong, KONG Dongyi. Chinese description of image content based on fusion of image feature attention and adaptive attention [J]. Journal of Computer Applications, 2021, 41(9): 2496-2503.
[5]	XU Jianglang, LI Linyan, WAN Xinjun, HU Fuyuan. Indoor scene recognition method combined with object detection [J]. Journal of Computer Applications, 2021, 41(9): 2720-2725.
[6]	ZHENG Zhiqiang, HU Xin, WENG Zhi, WANG Yuhe, CHENG Xi. Cattle eye image feature extraction method based on improved DenseNet [J]. Journal of Computer Applications, 2021, 41(9): 2780-2784.
[7]	HE Zhenghai, XIAN Yantuan, WANG Meng, YU Zhengtao. Case reading comprehension method combining syntactic guidance and character attention mechanism [J]. Journal of Computer Applications, 2021, 41(8): 2427-2431.
[8]	CAO Yuhong, XU Hai, LIU Sun'ao, WANG Zixiao, LI Hongliang. Review of deep learning-based medical image segmentation [J]. Journal of Computer Applications, 2021, 41(8): 2273-2287.
[9]	QIN Binbin, PENG Liangkang, LU Xiangming, QIAN Jiangbo. Research progress on driver distracted driving detection [J]. Journal of Computer Applications, 2021, 41(8): 2330-2337.
[10]	GAO Qinquan, HUANG Bingcheng, LIU Wenzhe, TONG Tong. Bamboo strip surface defect detection method based on improved CenterNet [J]. Journal of Computer Applications, 2021, 41(7): 1933-1938.
[11]	LI Yafang, LIANG Ye, FENG Weiwei, ZU Baokai, KANG Yujian. Deep network embedding method based on community optimization [J]. Journal of Computer Applications, 2021, 41(7): 1956-1963.
[12]	WANG Yue, JIANG Yiming, LAN Julong. Intrusion detection based on improved triplet network and K-nearest neighbor algorithm [J]. Journal of Computer Applications, 2021, 41(7): 1996-2002.
[13]	DU Yan, LYU Liangfu, JIAO Yichen. Fuzzy prototype network based on fuzzy reasoning [J]. Journal of Computer Applications, 2021, 41(7): 1885-1890.
[14]	HOU Xiaohan, JIN Guodong, TAN Lining, XUE Yuanliang. Synthetic aperture radar ship detection method based on self-adaptive and optimal features [J]. Journal of Computer Applications, 2021, 41(7): 2150-2155.
[15]	SHEN Xuewen, WANG Xiaodong, YAO Yu. Spatial frequency divided attention network for ultrasound image segmentation [J]. Journal of Computer Applications, 2021, 41(6): 1828-1835.