Deep learning algorithm optimization based on combination of auto-encoders

doi:10.11772/j.issn.1001-9081.2016.03.697

Journal of Computer Applications ›› 2016, Vol. 36 ›› Issue (3): 697-702.DOI: 10.11772/j.issn.1001-9081.2016.03.697

Previous Articles Next Articles

Deep learning algorithm optimization based on combination of auto-encoders

DENG Junfeng^1,2, ZHANG Xiaolong^1,2

1. School of Computer Science and Technology, Wuhan University of Science and Technology, Wuhan Hubei 430065, China;
2. Hubei Province Key Laboratory of Intelligent Information Processing and Real-time Industrial System, Wuhan Hubei 430065, China

Received:2015-08-13 Revised:2015-10-14 Online:2016-03-10 Published:2016-03-17
Supported by:
This work is partially supported by the National Natural Science Foundation of China (61273225) and the National Key Technology R&D Program (2012BAC22B01).

基于自动编码器组合的深度学习优化方法

邓俊锋^1,2, 张晓龙^1,2

1. 武汉科技大学计算机科学与技术学院, 武汉 430065;
2. 智能信息处理与实时工业系统湖北省重点实验室, 武汉 430065

通讯作者: 张晓龙
作者简介:邓俊锋(1989-),男,湖北黄冈人,硕士研究生,主要研究方向:机器学习、数据挖掘;张晓龙(1963-),男,江西永新人,教授,博士生导师,主要研究方向:数据挖掘、机器学习、生物信息处理。
基金资助:
国家自然科学基金资助项目(61273225);国家科技支撑计划项目(2012BAC22B01)。

Abstract

Abstract: In order to improve the learning accuracy of Auto-Encoder (AE) algorithm and further reduce the classification error rate, Sparse marginalized Denoising Auto-Encoder (SmDAE) was proposed combined with Sparse Auto-Encoder (SAE) and marginalized Denoising Auto-Encoder (mDAE). SmDAE is an auto-encoder which was added the constraint conditions of SAE and mDAE and has the characteristics of SAE and mDAE, so as to enhance the ability of deep learning. Experimental results show that SmDAE outperforms both SAE and mDAE in the given classification tasks; comparative experiments with Convolutional Neural Network (CNN) show that SmDAE with marginalized denoising and a more robust model outperforms convolutional neural network.

Key words: deep learning, Auto-Encoder (AE), Sparse Auto-Encoder (SAE), Denoising Auto-Encoder (DAE), Convolutional Neural Network (CNN)

摘要： 为了提高自动编码器算法的学习精度,更进一步降低分类任务的分类错误率,提出一种组合稀疏自动编码器(SAE)和边缘降噪自动编码器(mDAE)从而形成稀疏边缘降噪自动编码器(SmDAE)的方法,将稀疏自动编码器和边缘降噪自动编码器的限制条件加载到一个自动编码器(AE)之上,使得这个自动编码器同时具有稀疏自动编码器的稀疏性约束条件和边缘降噪自动编码器的边缘降噪约束条件,提高自动编码器算法的学习能力。实验表明,稀疏边缘降噪自动编码器在多个分类任务上的学习精度都高于稀疏自动编码器和边缘降噪自动编码器的分类效果;与卷积神经网络(CNN)的对比实验也表明融入了边缘降噪限制条件,而且更加鲁棒的SmDAE模型的分类精度比CNN还要好。

关键词: 深度学习, 自动编码器, 稀疏自动编码器, 降噪自动编码器, 卷积神经网络

CLC Number:

TP392

DENG Junfeng, ZHANG Xiaolong. Deep learning algorithm optimization based on combination of auto-encoders[J]. Journal of Computer Applications, 2016, 36(3): 697-702.

邓俊锋, 张晓龙. 基于自动编码器组合的深度学习优化方法[J]. 计算机应用, 2016, 36(3): 697-702.

References

[1] RUMELHART D E, HINTON G E, WILLIAMS R J. Learning representations by back-propagating errors [J]. Nature, 1986,323(9):533-536.
[2] BALDI P, HORNIK K. Neural networks and principal component analysis: learning from examples without local minima [J]. Neural Networks, 1989,2(1):53-58.
[3] BENGIO Y, LAMBLIN P, POPOVICI D, et al. Personal communications with Will Zou. learning optimization Greedy layer-wise training of deep networks [C]//Proceedings of the 20th Annual Conference on Neural Information Processing System. Cambridge, MA: MIT Press, 2006:153-160.
[4] BENGIO Y. Learning deep architectures for AI [J]. Foundations & Trends® in Machine Learning, 2009, 2(1): 1-127.
[5] VINCENT P, LAROCHELLE H, BENGIO Y, et al. Extracting and composing robust features with denoising autoencoders [C]//Proceedings of the 2008 25th International Conference on Machine Learning. New York: ACM, 2008: 1096-1103.
[6] CHEN M, WEINBERGER K, SHA F, et al. Marginalized denoising auto-encoders for nonlinear representations [C]//Proceedings of the 2014 31th International Conference on Machine Learning. New York: ACM, 2014: 1476-1484.
[7] VINCENT P, LAROCHELLE H, LAJOIE I, et al. Stacked denoising autoencoders: learning useful representations in a deep network with a local denoising criterion [J]. Journal of Machine Learning Research, 2010,11(6): 3371-3408.
[8] LeCUN Y, BOTTOU L, BENGIO Y, et al. Gradient-based learning applied to document recognition [J]. Proceedings of the IEEE, 1998, 86(11): 2278-2324.
[9] FARABET C, COUPRIE C, NAJMAN L, et al. Learning hierarchical features for scene labeling [J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2013, 35(8): 1915-1929.
[10] MOHAMED A, DAHL G E, HINTON G. Acoustic modeling using deep belief networks [J]. IEEE Transactions on Audio, Speech, and Language Processing, 2012, 20(1): 14-22.
[11] LeCUN Y, BOTTOU L, ORR G B, et al. Efficient BackProp [M]//ORR G B, MVLLER K-R. Neural Networks: Tricks of the Trade, LNCS 1524. Berlin: Springer, 1998:9-50.
[12] HINTON G E, OSINDERO S, TEH Y W. A fast learning algorithm for deep belief nets [J]. Neural Computation, 2006,18(7): 1527-1554.
[13] JAITLY N, HINTON G E. Using an autoencoder with deformable templates to discover features for automated speech recognition [EB/OL]. [2015-04-07]. http://www.cs.toronto.edu/~ndjaitly/jaitly-interspeech13.pdf.
[14] TSURUOKA Y, TSUJII J, ANANIADOU S. Stochastic gradient descent training for L1-regularized log-linear models with cumulative penalty [EB/OL]. [2015-04-07]. http://aye.comp.nus.edu.sg/~antho/P/P09/P09-1054.pdf.
[15] LE Q V, NGIAM J, COATES A, et al. On optimization methods for deep learning [EB/OL]. [2015-04-07]. http://ai.stanford.edu/~ang/papers/icml11-OptimizationForDeepLearning.pdf.

Deep learning algorithm optimization based on combination of auto-encoders

基于自动编码器组合的深度学习优化方法

PDF

Knowledge

Abstract

Cite this article

share this article

References

Related Articles 15

Recommended Articles

Metrics

[1]	WANG Hebing, ZHANG Chunmei. Facial landmark detection based on ResNeXt with asymmetric convolution and squeeze excitation [J]. Journal of Computer Applications, 2021, 41(9): 2741-2747.
[2]	ZHENG Zhiqiang, HU Xin, WENG Zhi, WANG Yuhe, CHENG Xi. Cattle eye image feature extraction method based on improved DenseNet [J]. Journal of Computer Applications, 2021, 41(9): 2780-2784.
[3]	CHEN Chengrui, SUN Ning, HE Shibiao, LIAO Yong. Deep learning-based joint channel estimation and equalization algorithm for C-V2X communications [J]. Journal of Computer Applications, 2021, 41(9): 2687-2693.
[4]	SONG Zhongshan, LIANG Jiarui, ZHENG Lu, LIU Zhenyu, TIE Jun. Remote sensing scene classification based on bidirectional gated scale feature fusion [J]. Journal of Computer Applications, 2021, 41(9): 2726-2735.
[5]	LI Kangkang, ZHANG Jing. Multi-layer encoding and decoding model for image captioning based on attention mechanism [J]. Journal of Computer Applications, 2021, 41(9): 2504-2509.
[6]	ZHANG Yongbin, CHANG Wenxin, SUN Lianshan, ZHANG Hang. Detection method of domains generated by dictionary-based domain generation algorithm [J]. Journal of Computer Applications, 2021, 41(9): 2609-2614.
[7]	ZHAO Hong, KONG Dongyi. Chinese description of image content based on fusion of image feature attention and adaptive attention [J]. Journal of Computer Applications, 2021, 41(9): 2496-2503.
[8]	XU Jianglang, LI Linyan, WAN Xinjun, HU Fuyuan. Indoor scene recognition method combined with object detection [J]. Journal of Computer Applications, 2021, 41(9): 2720-2725.
[9]	XIE Defeng, JI Jianmin. Syntax-enhanced semantic parsing with syntax-aware representation [J]. Journal of Computer Applications, 2021, 41(9): 2489-2495.
[10]	DAI Yurou, YANG Qing, ZHANG Fengli, ZHOU Fan. Trajectory prediction model of social network users based on self-supervised learning [J]. Journal of Computer Applications, 2021, 41(9): 2545-2551.
[11]	ZENG Xiangyin, ZHENG Bochuan, LIU Dan. Detection of left and right railway tracks based on deep convolutional neural network and clustering [J]. Journal of Computer Applications, 2021, 41(8): 2324-2329.
[12]	HE Zhenghai, XIAN Yantuan, WANG Meng, YU Zhengtao. Case reading comprehension method combining syntactic guidance and character attention mechanism [J]. Journal of Computer Applications, 2021, 41(8): 2427-2431.
[13]	CAO Yuhong, XU Hai, LIU Sun'ao, WANG Zixiao, LI Hongliang. Review of deep learning-based medical image segmentation [J]. Journal of Computer Applications, 2021, 41(8): 2273-2287.
[14]	QIN Binbin, PENG Liangkang, LU Xiangming, QIAN Jiangbo. Research progress on driver distracted driving detection [J]. Journal of Computer Applications, 2021, 41(8): 2330-2337.
[15]	HUANG Chengcheng, DONG Xiaoxiao, LI Zhao. Deep pipeline 5×5 convolution method based on two-dimensional Winograd algorithm [J]. Journal of Computer Applications, 2021, 41(8): 2258-2264.