计算机应用 ›› 2019, Vol. 39 ›› Issue (10): 2809-2814.DOI: 10.11772/j.issn.1001-9081.2019040624

• 人工智能 • 上一篇    下一篇

改进的弹性网模型在深度神经网络中的应用

冯明皓1, 张天伦1, 王林辉1, 陈荣1, 连少静2   

  1. 1. 大连海事大学 信息科学技术学院, 辽宁 大连 116026;
    2. 河北大学 数学与信息科学学院, 河北 保定 071002
  • 收稿日期:2019-04-15 修回日期:2019-07-03 出版日期:2019-10-10 发布日期:2019-08-21
  • 通讯作者: 陈荣
  • 作者简介:冯明皓(1995-),男,天津人,硕士研究生,主要研究方向:深度学习、最优化算法;张天伦(1991-),男,河北保定人,博士研究生,主要研究方向:机器学习、计算视觉;王林辉(1995-),男,山东烟台人,硕士研究生,主要研究方向:深度学习、文本分类;陈荣(1969-),男,辽宁大连人,教授,博士,CCF会员,主要研究方向:人工智能、软件工程;连少静(1990-),女,河北邯郸人,硕士,主要研究方向:机器学习。
  • 基金资助:
    国家自然科学基金资助项目(61672122,61402070,61602077);辽宁省自然科学基金资助项目(20170540097,2015020023);辽宁省科学事业公益研究基金资助项目(GY-2017-0005);中央高校基本科研业务费资助项目(3132019207,3132019355)。

Improved elastic network model for deep neural network

FENG Minghao1, ZHANG Tianlun1, WANG Linhui1, CHEN Rong1, LIAN Shaojing2   

  1. 1. College of Information Science and Technology, Dalian Maritime University, Dalian Liaoning 116026, China;
    2. College of Mathematics and Information Science, Hebei University, Baoding Hebei 071002, China
  • Received:2019-04-15 Revised:2019-07-03 Online:2019-10-10 Published:2019-08-21
  • Supported by:
    This work is partially supported by the National Natural Science Foundation of China (61672122, 61402070, 61602077), the Natural Science Foundation of Liaoning Province (20170540097, 2015020023), the Public Research Funds for Scientific Ventures of Liaoning Province (GY-2017-0005), the Fundamental Research Funds for the Central Universities (3132019207, 3132019355).

摘要: 由于具有较高的模型复杂度,深层神经网络容易产生过拟合问题,为了减少该问题对网络性能的不利影响,提出一种基于改进的弹性网模型的深度学习优化方法。首先,考虑到变量之间的相关性,对弹性网模型中的L1范数的不同变量进行自适应加权,从而得到L2范数与自适应加权的L1范数的线性组合。其次,将改进的弹性网络模型与深度学习的优化模型相结合,给出在这种新正则项约束下求解神经网络参数的过程。然后,推导出改进的弹性网模型在神经网络优化中具有群组选择能力和Oracle性质,进而从理论上保证该模型是一种更加鲁棒的正则化方法。最后,在多个回归问题和分类问题的实验中,相对于L1、L2和弹性网正则项,该方法的回归测试误差可分别平均降低87.09、88.54和47.02,分类测试准确度可分别平均提高3.98、2.92和3.58个百分点。由此,在理论和实验两方面验证了改进的弹性网模型可以有效地增强深层神经网络的泛化能力,提升优化算法的性能,解决深度学习的过拟合问题。

关键词: 神经网络模型, 深度学习, 正则化方法, 弹性网模型, 过拟合

Abstract: Deep neural networks tend to suffer from overfitting problem because of the high complexity of the model. To reduce the adverse eeffects of the problem on the network performance, an improved elastic network model based deep learning optimization method was proposed. Firstly, considering the strong correlation between the variables, the adaptive weights were assigned to different variables of L1-norm in elastic network model, so that the linear combination of the L2-norm and the adaptively weighted L1-norm was obtained. Then, the solving process of neural network parameters under this new regularization term was given by combining improved elastic network model with the deep learning optimization model. Moreover, the robustness of this proposed model was theoretically demonstrated by showing the grouping selection ability and Oracle property of the improved elastic network model in the optimization of neural network. At last, in regression and classification experiments, the proposed model was compared with L1-norm, L2-norm and elastic network regularization term, and had the regression error decreased by 87.09, 88.54 and 47.02 and the classification accuracy improved by 3.98, 2.92 and 3.58 percentage points respectively. Thus, theory and experimental results prove that the improved elastic network model can effectively improve the generalization ability of deep neural network model and the performance of optimization algorithm, and solve the overfitting problem of deep learning.

Key words: neural network model, deep learning, regularization method, elastic network model, overfitting

中图分类号: