Four-layer multiple kernel learning method based on random feature mapping

doi:10.11772/j.issn.1001-9081.2021010171

Abstract

Abstract:

Since there is no perfect theoretical basis for the selection of kernel function in single kernel network models， and the network node size of Four-layer Neural Network based on Randomly Feature Mapping （FRFMNN） is excessively large， a Four-layer Multiple Kernel Neural Network based on Randomly Feature Mapping （MK-FRFMNN） algorithm was proposed. Firstly， the original input features were transformed into randomly mapped features by a specific random mapping algorithm. Then， multiple basic kernel matrices were generated through different random kernel mappings. Finally， the synthetic kernel matrix formed by basic kernel matrices was linked to the output layer through the output weights. Since the weights of random mapping of original features were randomly generated according to the random continuous sampling probability distribution randomly， without the need of updates of the weights， and the weights of the output layer were quickly solved by the ridge regression pseudo inverse algorithm， thus avoiding the time-consuming training process of the repeated iterations. Different random weight matrices were introduced into the basic kernel mapping of MK-FRFMNN. the generated synthetic kernel matrix was able to not only synthesize the advantages of various kernel functions， but also integrate the characteristics of various random distribution functions， to obtain better feature selection and expression effect in the new feature space. Theoretical and experimental analyses show that， compared with the single kernel models such as Broad Learning System （BLS） and FRMFNN， MK-FRMFNN model has the node size reduced by about 2/3 with stable classification performance； compared with mainstream multiple kernel models， MK-FRMFNN model can learn large sample datasets， and has better performance in classification.

Key words: random feature mapping, sparse autoencoder, multiple kernel learning, ridge regression, regularization

摘要：

针对单核网络模型的核函数选择无理论依据以及基于随机特征映射的四层神经网络（FRMFNN）节点规模过大的问题，提出了一种基于随机特征映射的四层多核学习神经网络（MK-FRMFNN）算法。首先，把原始输入特征通过特定的随机映射算法转化为随机映射特征；然后，经过不同的随机核映射生成多个基本核矩阵；最后，将基本核矩阵组成合成核矩阵，并通过输出权重连接到输出层。对原始特征进行随机映射的权重是根据任意连续采样概率分布随机生成的，不需要训练更新，且对输出层的权重使用岭回归伪逆算法进行快速求解，从而避免了反复迭代耗时的训练过程。MK-FRMFNN在基本核映射时引入了不同的随机权重矩阵，生成的合成核矩阵不仅可以综合各种核函数的优势，而且可以集合各种随机分布函数的特性，使数据在新的特征空间中获得更好的特征选择和表达效果。理论和实验分析表明，与宽度学习系统（BLS）及FRMFNN等单核模型相比，MK-FRMFNN模型的节点规模减小了2/3左右，且分类性能稳定；与主流的多种多核模型相比，MK-FRMFNN模型能够对大样本数据集进行学习，并且分类性能明显更优。

关键词: 随机特征映射, 稀疏自动编码器, 多核学习, 岭回归, 正则化

CLC Number:

TP183

Yue YANG, Shitong WANG. Four-layer multiple kernel learning method based on random feature mapping[J]. Journal of Computer Applications, 2022, 42(1): 16-25.

杨悦, 王士同. 基于随机特征映射的四层多核学习方法[J]. 《计算机应用》唯一官方网站, 2022, 42(1): 16-25.

Figures/Tables 8

Fig.1 Four-layer neural network framework based on randomly feature mapping

Fig.2 four-layer kernel method framework based on randomly feature mapping

Fig.3 Multiple kernel learning framework based on randomly feature mapping

Tab. 1 Details of datasets

数据集	样本数	特征数	类别数
Letter	20 000	16	26
Robot	5 456	24	4
Ecoli	336	7	8
ACT	46 728	8	2
Adult	48 841	14	2
Australian	690	14	2
Eye state	14 980	15	2
Magic	19 020	11	2
Car	1 594	6	2

Table 2 Experimental results with different regularization parameters

C值	Ecoli		Australian		Car
C值	Train_acc	Test_acc	Train_acc	Test_acc	Train_acc	Test_acc
2^-24	87.36±1.37	82.18±2.38	88.34±0.71	85.17±0.81	92.11±0.24	90.79±0.30
2^-22	86.72±1.70	80.30±2.27	87.87±0.48	84.78±0.82	91.78±0.18	90.79±0.40
2^-20	87.40±1.66	83.37±1.46	87.99±0.39	84.69±0.92	91.63±0.21	90.42±0.48
2^-18	86.77±1.78	81.49±2.47	87.50±0.50	84.44±0.89	91.39±0.33	90.77±0.58
2^-16	85.06±2.32	79.21±3.07	87.68±0.65	84.93±0.60	90.98±0.33	90.29±0.34
2^-14	85.32±1.21	80.89±1.66	87.41±0.50	84.59±0.76	90.75±0.36	90.27±0.46
2^-12	84.55±2.64	79.21±3.04	87.62±0.50	84.69±0.75	90.54±0.31	89.96±0.36
2^-10	82.47±1.54	78.81±2.43	87.37±0.46	84.35±0.44	90.25±0.36	89.60±0.43

Tab.3 Parameter settings of BLS， FRMFNN and MK-FRMFNN models

数据集	BLS	FRMFNN	MK-FRMFNN
数据集	［N_f×N_m，N_e，C］	［N_f×N_m，N_e，C］	［N_f×N_m，N_e， $σ 1$ ， $σ 2$ ，C］
Letter	［11×9， 2 500， 2^-24］	［11×11， 2 940， 2^-24］	［11×10， 960， 10^-2， 1， 2^-24］
Robot	［5×6， 2 930， 2^-24］	［3×9， 2 690， 2^-24］	［3×7， 940， 10^-2， 10， 2^-24］
Ecoli	［2×2， 245， 2^-24］	［2×2， 235， 2^-24］	［2×2， 41， 10^-2， 10^-1， 2^-24］
ACT	［3×8， 1 040， 2^-24］	［4×5， 990， 2^-24］	［4×5， 300， 10^-1， 10， 2^-24］
Adult	［4×7， 1 890， 2^-24］	［8×5， 1 020， 2^-24］	［7×4， 330， 10²， 1， 2^-24］
Australian	［7×6， 84， 2^-24］	［7×4， 80， 2^-24］	［6×3， 28， 10， 10²， 2^-24］
Eye state	［11×10， 2 880， 2^-24］	［11×11， 2 000， 2^-24］	［10×10， 675， 10^-1， 10^-1， 2^-24］
Magic	［10×9， 1 900， 2^-24］	［11×9， 810， 2^-24］	［10×8， 310， 1， 1， 2^-24］
Car	［4×2， 1 520， 2^-24］	［5×3， 1 130， 2^-24］	［5×2， 390， 1， 10²， 2^-24］

Tab.3 Parameter settings of BLS， FRMFNN and MK-FRMFNN models

数据集	BLS	FRMFNN	MK-FRMFNN
数据集	［N_f×N_m，N_e，C］	［N_f×N_m，N_e，C］	［N_f×N_m，N_e， $σ 1$ ， $σ 2$ ，C］
Letter	［11×9， 2 500， 2^-24］	［11×11， 2 940， 2^-24］	［11×10， 960， 10^-2， 1， 2^-24］
Robot	［5×6， 2 930， 2^-24］	［3×9， 2 690， 2^-24］	［3×7， 940， 10^-2， 10， 2^-24］
Ecoli	［2×2， 245， 2^-24］	［2×2， 235， 2^-24］	［2×2， 41， 10^-2， 10^-1， 2^-24］
ACT	［3×8， 1 040， 2^-24］	［4×5， 990， 2^-24］	［4×5， 300， 10^-1， 10， 2^-24］
Adult	［4×7， 1 890， 2^-24］	［8×5， 1 020， 2^-24］	［7×4， 330， 10²， 1， 2^-24］
Australian	［7×6， 84， 2^-24］	［7×4， 80， 2^-24］	［6×3， 28， 10， 10²， 2^-24］
Eye state	［11×10， 2 880， 2^-24］	［11×11， 2 000， 2^-24］	［10×10， 675， 10^-1， 10^-1， 2^-24］
Magic	［10×9， 1 900， 2^-24］	［11×9， 810， 2^-24］	［10×8， 310， 1， 1， 2^-24］
Car	［4×2， 1 520， 2^-24］	［5×3， 1 130， 2^-24］	［5×2， 390， 1， 10²， 2^-24］

Table 4 Accuracy comparison of BLS， FRMFNN and MK-FRMFNN models on datasets for classification

数据集	BLS		FRMFNN		MK-FRMFNN
数据集	Train_acc	Test_acc	Train_acc	Test_acc	Train_acc	Test_acc
Letter	96.58±0.57	93.42±0.53	96.59±0.006 0	93.41±0.005 3	96.58±0.004 7	93.43±0.004 3
Robot	96.78±0.27	90.19±0.32	96.35±0.002 3	90.00±0.004 8	94.36±0.005 8	89.54±0.006 3
Ecoli	89.11±1.05	82.08±1.50	88.00±1.42	82.57±1.73	87.36±1.37	82.18±2.38
ACT	99.21±0.01	99.32±0.01	99.21±0.01	99.31±0.01	99.19±0.03	99.29±0.04
Adult	79.54±0.08	79.72±0.16	79.68±0.05	79.78±0.07	79.60±0.08	79.71±0.11
Australian	90.48±0.61	83.53±1.25	89.83±0.41	84.06±0.86	88.34±0.71	85.17±0.81
Eye state	86.19±0.36	85.12±0.28	86.07±0.60	85.07±0.57	86.10±0.31	85.07±0.25
Magic	83.40±0.21	82.85±0.26	83.43±0.28	82.82±0.24	82.58±0.17	82.11±0.17
Car	92.05±0.27	90.94±0.42	92.07±0.35	90.67±0.38	92.11±0.24	90.79±0.30

Tab. 5 Experimental parameters and results on three datasets

数据集	算法	参数	精度/%
Ecoli	simpleMKL	C=2³	73.30
	easyMKL	λ=0.3	75.26
	GLMKL	C=2⁴	73.29
	NLMKL	C=2⁴	75.26
	MK-FRMFNN	C=2^-24	82.18
Car	simpleMKL	C=2³	81.36
	easyMKL	λ=0.1	83.33
	GLMKL	C=2^-1	80.48
	NLMKL	C=2⁵	83.15
	MK-FRMFNN	C=2^-24	90.79
Australian	simpleMKL	C=2⁴	78.18
	easyMKL	λ=0.1	80.28
	GLMKL	C=2⁰	79.43
	NLMKL	C=2⁵	80.12
	MK-FRMFNN	C=2^-24	85.17

References 26

1	SCHÖLKOPF B， SMOLA A J. Learning with Kernels： Support Vector Machines， Regularization， Optimization， and Beyond［M］. 1st ed. Cambridge： MIT Press， 2001：10-41.
2	WANG Z， CHEN S C， SUN T K. MultiK-MHKS： a novel multiple kernel learning algorithm［J］. IEEE Transactions on Pattern Analysis and Machine Intelligence， 2008， 30（2）：348-353. 10.1109/tpami.2007.70786
3	ZHENG H， YE Q L， JIN Z. A novel multiple kernel sparse representation based classification for face recognition［J］. KSII Transactions on Internet and Information Systems， 2014， 8（4）： 1463-1480. 10.3837/tiis.2014.04.017
4	SAUNDERS C， STITSON M O， WESTON J， et al. Support vector machine［J］. Computer Science， 2002， 1（4）：1-28.
5	SUYKENS J A K， VANDEWALLE J. Least squares support vector machine classifiers［J］. Neural Processing Letters， 1999， 9（3）：293-300. 10.1023/a:1018628609742
6	陶剑文，王士同. 领域适应核支持向量机［J］. 自动化学报， 2012， 38（5）：797-811. 10.3724/SP.J.1004.2012.00797
	TAO J W， WANG S T. Kernel support vector machine for domain adaptation［J］. Acta Automatica Sinica， 2012， 38（5）：797-811. 10.3724/SP.J.1004.2012.00797
7	HUANG G B， ZHOU H M， DING X J， et al. Extreme learning machine for regression and multiclass classification［J］. IEEE Transactions on Systems， Man， and Cybernetics， Part B （Cybernetics）， 2012， 42（2）：513-529. 10.1109/tsmcb.2011.2168604
8	CHEN C L P， LIU Z L. Broad learning system： an effective and efficient incremental learning system without the need for deep architecture［J］. IEEE Transactions on Neural Networks and Learning Systems， 2018， 29（1）：10-24. 10.1109/tnnls.2017.2716952
9	杨悦，王士同. 随机特征映射的四层神经网络及其增量学习［J］. 计算机科学与探索， 2021， 15（7）：1265-1278. 10.3389/fpsyg.2021.760060
	YANG Y， WANG S T. A novel four-layer neural network and its incremental learning based on randomly mapped features［J］. Journal of Frontiers of Computer Science and Technology， 2021， 15（7）：1265-1278. 10.3389/fpsyg.2021.760060
10	HOERL A E， KENNARD R W. Ridge regression： biased estimation for nonorthogonal problems［J］. Technometrics， 1970， 12（1）：55-67. 10.1080/00401706.1970.10488634
11	BEN-HUR A， NOBLE W S. Kernel methods for predicting protein-protein interactions［J］. Bioinformatics， 2005， 21（S1）：i38-i46. 10.1093/bioinformatics/bti1016
12	钟志鹏，张立保. 基于多核学习特征融合的人脸表情识别［J］. 计算机应用， 2015， 35（S2）：245-249. 10.1007/s10489-021-02575-0
	ZHONG Z P， ZHANG L B. Facial expression recognition based on multi-feature fusion by multi-kernel SVM［J］. Journal of Computer Applications， 2015， 35（S2）：245-249. 10.1007/s10489-021-02575-0
13	KINGSBURY N， TAY D B H， PALANISWAMI M. Multi-scale kernel methods for classification［C］// Proceedings of the 2005 IEEE Workshop on Machine Learning for Signal Processing. Piscataway： IEEE， 2005：43-48. 10.1109/mlsp.2005.1532872
14	DAMOULAS T， GIROLAMI M A， NOTES A. Probabilistic multi-class multi-kernel learning： on protein fold recognition and remote homology detection［J］. Bioinformatics， 2008， 24（10）：1264-1270. 10.1093/bioinformatics/btn112
15	汪洪桥，孙富春，蔡艳宁，等. 多核学习方法［J］. 自动化学报， 2010， 36（8）：1037-1050. 10.3724/SP.J.1004.2010.01037
	WANG H Q， SUN F C， CAI Y N， et al. On multiple kernel learning methods［J］. Acta Automatica Sinica， 2010， 36（8）：1037-1050. 10.3724/SP.J.1004.2010.01037
16	周志华. 机器学习［M］. 北京：清华大学出版社， 2016：63-66， 137-140. 10.21436/inbom.12382432
	ZHOU Z H. Machine Learning［M］. Beijing： Tsinghua University Press， 2016：63-66， 137-140. 10.21436/inbom.12382432
17	AIOLLI F， DONINI M. EasyMKL： a scalable multiple kernel learning algorithm［J］. Neurocomputing， 2015， 169：215-224. 10.1016/j.neucom.2014.11.078
18	RAKOTOMAMONJY A， BACH F R， CANU S， et al. Simple MKL［J］. Journal of Machine Learning Research， 2008， 9： 2491-2521.
19	VARMA M， BABU B R. More generality in efficient multiple kernel learning［C］// Proceedings of the 26th International Conference on Machine Learning. New York： ACM， 2009： 1065-1072. 10.1145/1553374.1553510
20	CORTES C， MOHRI M， ROSTAMIZADEH A. Learning non-linear combinations of kernels［C］// Proceedings of the 22nd International Conference on Neural Information Processing Systems. Red Hook， NY： Curran Associates Inc.， 2009：396-404. 10.1109/icassp.2003.1198859
21	XU Z L， JIN R， YANG H Q， et al. Simple and efficient multiple kernel learning by group lasso［C］// Proceedings of the 27th International Conference on Machine Learning. Madison， WI： Omnipress， 2010： 1175-1182.
22	TIBSHIRANI R. Regression shrinkage and selection via the lasso［J］. Journal of the Royal Statistical Society： Series B （Methodological）， 1996， 58（1）：267-288. 10.1111/j.2517-6161.1996.tb02080.x
23	BOYD S， PARIKH N， CHU E， et al. Distributed optimization and statistical learning via the alternating direction method of multipliers［J］. Foundations and Trends in Machine Learning， 2010， 3（1）：1-122. 10.1561/2200000016
24	GOLDSTEIN T， O’DONOGHUE B， SETZER S， et al. Fast alternating direction optimization methods［J］. SIAM Journal on Imaging Sciences， 2014， 7（3） 1588-1623. 10.1137/120896219
25	CRISTIANINTNELLO. An introduction to support vector machines and other kernel-based learning methods［J］. Kybernetes， 2000， 30（1）：103-115. 10.1108/k.2001.30.1.103.6
26	DUA D， GRAFF C. UCI repository of machine learning repository［DS/OL］. ［2021-01-12］..

[1]	Shuaihua ZHANG, Shufen ZHANG, Mingchuan ZHOU, Chao XU, Xuebin CHEN. Malicious traffic detection model based on semi-supervised federated learning [J]. Journal of Computer Applications, 2024, 44(11): 3487-3494.
[2]	Jie WU, Xuezhong QIAN, Wei SONG. Personalized federated learning based on similarity clustering and regularization [J]. Journal of Computer Applications, 2024, 44(11): 3345-3353.
[3]	Mengjie LAN, Jianping CAI, Lan SUN. Self-regularization optimization methods for Non-IID data in federated learning [J]. Journal of Computer Applications, 2023, 43(7): 2073-2081.
[4]	Wenbo LI, Bo LIU, Lingling TAO, Fen LUO, Hang ZHANG. Deep spectral clustering algorithm with L1 regularization [J]. Journal of Computer Applications, 2023, 43(12): 3662-3667.
[5]	Xiangzhou QI, Hongjie XING. Centered kernel alignment based multiple kernel one-class support vector machine [J]. Journal of Computer Applications, 2022, 42(2): 349-356.
[6]	Lili FAN, Guifu LU, Ganyi TANG, Dan YANG. Low-rank representation subspace clustering method based on Hessian regularization and non-negative constraint [J]. Journal of Computer Applications, 2022, 42(1): 115-122.
[7]	LIN Junchao, WAN Yuan. Self-adaptive multi-measure unsupervised feature selection method with structured graph optimization [J]. Journal of Computer Applications, 2021, 41(5): 1282-1289.
[8]	Qian LIU, Hongyuan WANG, Liang CAO, Boyan SUN, Yu XIAO, Ji ZHANG. Cloth-changing person re-identification based on joint loss capsule network [J]. Journal of Computer Applications, 2021, 41(12): 3596-3601.
[9]	Xian CHEN, Liying HU, Xiaowei LIN, Lifei CHEN. Directed graph clustering algorithm based on kernel nonnegative matrix factorization [J]. Journal of Computer Applications, 2021, 41(12): 3447-3454.
[10]	Hua LI, Guifu LU, Qinru YU. Manifold regularized nonnegative matrix factorization based on clean data [J]. Journal of Computer Applications, 2021, 41(12): 3492-3498.
[11]	SHEN Marui, LI Jincheng, ZHANG Ya, ZOU Jian. Magnetic resonance image reconstruction algorithm via non-convex total variation regularization [J]. Journal of Computer Applications, 2020, 40(8): 2358-2364.
[12]	WANG Zhiyuan, JIANG Ailian, MUHAMMAD Osman. Unsupervised feature selection method based on regularized mutual representation [J]. Journal of Computer Applications, 2020, 40(7): 1896-1900.
[13]	XIE Qi, XU Xu, CHENG Gengguo, CHEN Heping. Feature selection algorithm based on new forest optimization algorithm [J]. Journal of Computer Applications, 2020, 40(5): 1266-1271.
[14]	GENG Yuanqian, WU Chuansheng, LIU Wen. Mixed non-convex and non-smooth regularization constraint based blind image restoration [J]. Journal of Computer Applications, 2020, 40(4): 1171-1176.
[15]	Meng ZENG, Bin NING, Zhihua CAI, Qiong GU. Hyperspectral band selection based on deep adversarial subspace clustering [J]. Journal of Computer Applications, 2020, 40(2): 381-385.