Robust weight matrix combination selection method of broad learning system

doi:10.11772/j.issn.1001-9081.2023101422

Journal of Computer Applications ›› 2024, Vol. 44 ›› Issue (10): 3032-3038.DOI: 10.11772/j.issn.1001-9081.2023101422

• Artificial intelligence • Previous Articles Next Articles

Robust weight matrix combination selection method of broad learning system

Han WANG¹, Yuan WAN¹, Dong WANG¹, Yiming DING²()

^1.School of Science，Wuhan University of Technology，Wuhan Hubei 430070，China
^2.School of Sciences，Wuhan University of Science and Technology，Wuhan Hubei 430081，China

Received:2023-10-24 Revised:2024-03-04 Accepted:2024-04-03 Online:2024-10-15 Published:2024-10-10
Contact: Yiming DING
About author:WANG Han， born in 2000， M. S. candidate. Her research interests include broad learning system， machine learning.
WAN Yuan， born in 1976， Ph. D.， professor. Her research interests include artificial intelligence， machine learning， data mining.
WANG Dong， born in 1998， M. S. candidate. His research interests include neural network， data analysis.
Supported by:
National Key Research and Development Program of China(2020YFA0714200)

宽度学习系统中鲁棒性权值矩阵组合的筛选方法

汪韩¹, 万源¹, 王东¹, 丁义明²()

^1.武汉理工大学理学院，武汉 430070
^2.武汉科技大学理学院，武汉 430081

通讯作者: 丁义明
作者简介:汪韩（2000—），女，湖北孝感人，硕士研究生，主要研究方向：宽度学习系统、机器学习
万源（1976—），女，湖北武汉人，教授，博士，CCF会员，主要研究方向：人工智能、机器学习、数据挖掘
王东（1998—），男，河南南阳人，硕士研究生，主要研究方向：神经网络、数据分析
丁义明（1972—），男，江西丰城人，教授，博士，主要研究方向：动力系统、统计学习 dingym@wust.edu.cn。
基金资助:
国家重点研发计划项目(2020YFA0714200)

Abstract

Abstract:

Broad Learning System （BLS） has excellent computational efficiency and prediction accuracy. However， in the traditional BLS framework， the weight matrix is randomly generated with the risk of unstable learning results. Therefore， a Robust Weight matrix Selection of BLS（RWS-BLS） method was proposed. Firstly， the significant difference of the randomized weight matrix in the overall training error of the samples was revealed by the validation of four sets of function data. Secondly， the combination forms of the weight matrices were studied， the strict optimal restriction of the screening conditions was relaxed， the optimal restriction was converted into the better restriction， the minimum value of the error was limited in a specified range， and the conditions such as the elite combinations were defined. Finally， the combinations of the reliable weight matrices were obtained， so that the influence of randomness was effectively reduced and a robust model could be established. The experimental results show that on 16 sets of simulated data， NORB dataset and 5 sets of UCI regression dataset， with data replacement or subjected to noise perturbation， the proposed method decreases the Mean Square Error （MSE） by 7.32%， 8.73% and 1.63% compared with the BLS method. RWS-BLS provides a direction for model smoothness study for BLS， which can improve the efficiency and stability of models with stochastic parameters， and is useful for other machine learning methods with stochastic parameters.

Key words: Broad Learning System (BLS), weight matrix combination, feature node, enhancement node, robustness analysis

摘要：

宽度学习系统（BLS）具有出色的计算效率和预测准确性；然而，在传统BLS框架中，权值矩阵采用随机生成的方式，存在学习结果不稳定的风险。因此，设计一种BLS中鲁棒性权值矩阵组合的筛选方法（RWS-BLS）。首先，通过4组函数数据的验证，揭示随机权值矩阵在样本整体训练误差上的显著差异性；其次，研究权值矩阵组合的形式，放宽筛选条件的严格最优限制，将最优转换为较优，并将误差最小值限定在指定范围内，定义精英组合等条件；最后，得到可靠的权值矩阵的组合，有效降低随机性影响，并建立稳健的模型。实验结果表明，在16组模拟数据、NORB数据集和5组UCI回归数据集上，在数据更换或受噪声扰动的情况下，与BLS方法相比，所提方法的均方误差（MSE）下降了7.32%、8.73%和1.63%。RWS-BLS为BLS提供了一种模型平稳性研究的方向，提高了含有随机参数模型的效率和稳定性，并对涉及随机参数的其他机器学习方法具有借鉴作用。

关键词: 宽度学习系统, 权值矩阵组合, 特征节点, 增强节点, 鲁棒性分析

CLC Number:

TP389.1

Han WANG, Yuan WAN, Dong WANG, Yiming DING. Robust weight matrix combination selection method of broad learning system[J]. Journal of Computer Applications, 2024, 44(10): 3032-3038.

汪韩, 万源, 王东, 丁义明. 宽度学习系统中鲁棒性权值矩阵组合的筛选方法[J]. 《计算机应用》唯一官方网站, 2024, 44(10): 3032-3038.

Figures/Tables 15

Fig. 1 Structure of broad learning system

Tab. 1 Four datasets used in significance test

数据集	数据和噪声分布	多维输入输出函数模型
Dataset1	$x ∼ U (- 2,2)$ ， $ε ∼ N 0,0.1$	$y 1 = 1 + 5 (x 1 + x 2 + x 3 + x 4 + x 5)$
		$y 2 = 1 + s i n (x 1 + x 2) + 4 (x 3 + x 4 + x 5)$
		$y 3 = 1 + c o s (x 1 + x 2) + 3 (x 3 + x 4 + x 5)$
Dataset2	$x ∼ U (- 2,2)$ ， $ε ∼ N 0,0.1$	$y 1 = 1 + 2 x 1 + 3 (x 2 + x 3) 2 + 4 (x 4 + x 5) 3$
		$y 2 = 1 + x 1 3 + s i n (x 2 + x 3 + x 4 + x 5)$
		$y 3 = 1 + (x 1 + x 2) 3 + c o s (x 3 + x 4 + x 5)$
Dataset3	$x ∼ U (- 2,2)$ ， $ε ∼ N 0,0.1$	$y 1 = e 2 (x 1 + x 2 + x 3 + x 4 + x 5) + 1$
		$y 2 = 0.5 e 2 (x 1 + x 2 + x 3 + x 4 + x 5) + 1$
		$y 3 = 2 e 2 (x 1 + x 2 + x 3 + x 4 + x 5) + 1$
Dataset4	$x ∼ U (- 2,2)$ ， $ε ∼ N (0,0.1)$	$y 1 = 100 s i n (x 3 + x 4 + x 5) / (x 1 + x 2)$
		$y 2 = 100 (x 3 + x 4 + x 5) / (x 1 + x 2)$
		$y 3 = 100 / (x 1 + x 2 + x 3 + x 4 + x 5)$

Tab. 1 Four datasets used in significance test

数据集	数据和噪声分布	多维输入输出函数模型
Dataset1	$x ∼ U (- 2,2)$ ， $ε ∼ N 0,0.1$	$y 1 = 1 + 5 (x 1 + x 2 + x 3 + x 4 + x 5)$
		$y 2 = 1 + s i n (x 1 + x 2) + 4 (x 3 + x 4 + x 5)$
		$y 3 = 1 + c o s (x 1 + x 2) + 3 (x 3 + x 4 + x 5)$
Dataset2	$x ∼ U (- 2,2)$ ， $ε ∼ N 0,0.1$	$y 1 = 1 + 2 x 1 + 3 (x 2 + x 3) 2 + 4 (x 4 + x 5) 3$
		$y 2 = 1 + x 1 3 + s i n (x 2 + x 3 + x 4 + x 5)$
		$y 3 = 1 + (x 1 + x 2) 3 + c o s (x 3 + x 4 + x 5)$
Dataset3	$x ∼ U (- 2,2)$ ， $ε ∼ N 0,0.1$	$y 1 = e 2 (x 1 + x 2 + x 3 + x 4 + x 5) + 1$
		$y 2 = 0.5 e 2 (x 1 + x 2 + x 3 + x 4 + x 5) + 1$
		$y 3 = 2 e 2 (x 1 + x 2 + x 3 + x 4 + x 5) + 1$
Dataset4	$x ∼ U (- 2,2)$ ， $ε ∼ N (0,0.1)$	$y 1 = 100 s i n (x 3 + x 4 + x 5) / (x 1 + x 2)$
		$y 2 = 100 (x 3 + x 4 + x 5) / (x 1 + x 2)$
		$y 3 = 100 / (x 1 + x 2 + x 3 + x 4 + x 5)$

Fig. 2 Distribution of mean square error for datasets

Fig. 3 P value results of t test for datasets

Fig. 4 Changes of training error of data model

Fig. 5 Selecting process of RWS-BLS method

Fig. 6 Rankings of Dataset1 under four groups of noise

Tab. 2 Proportion of rankings for 10 pairs of weight matrix combinations selected against 2 500 pairs of random combinations under four noise disturbances

SNR/dB	排名占比/%
SNR/dB	组合 1	组合 2	组合 3	组合 4	组合 5	组合 6	组合 7	组合 8	组合 9	组合 10
50	0.12	0.16	0.20	0.04	0.08	0.28	0.36	0.24	0.32	0.40
40	0.44	0.56	0.32	0.28	3.56	4.24	4.84	4.64	1.84	4.40
30	1.60	0.20	0.48	2.32	1.84	4.08	1.44	12.60	5.60	2.16
20	0.44	3.80	0.40	4.84	3.52	3.56	2.60	3.44	2.52	10.40

Fig. 7 Image categories of NORB dataset

Fig. 8 Influence of combination change of weight matrix on training error

Tab. 3 Noise added to NORB dataset

数据类别	加入形式	噪声分布
data1	$y n o i s e = x ⋅ ε$	$ε ∼ N 1,0.25$
data2	$y n o i s e = x ⋅ ε$	$ε ∼ U 0.8,1.2$
data3	$y n o i s e = x ⋅ ε$	$ε ∼ U 0.9,1.1$
data4	$y n o i s e = x ⋅ ε$	$ε ∼ t r i a n g u l a r 1.2,0.8 . 1$

Tab. 3 Noise added to NORB dataset

数据类别	加入形式	噪声分布
data1	$y n o i s e = x ⋅ ε$	$ε ∼ N 1,0.25$
data2	$y n o i s e = x ⋅ ε$	$ε ∼ U 0.8,1.2$
data3	$y n o i s e = x ⋅ ε$	$ε ∼ U 0.9,1.1$
data4	$y n o i s e = x ⋅ ε$	$ε ∼ t r i a n g u l a r 1.2,0.8 . 1$

Tab. 4 Proportions of training error ranking for selecting combinations under four groups of noise

$C b e s t$	data1	data2	data3	data4
组合1	0.48	0.32	0.56	0.68
组合2	0.36	0.28	0.20	0.24
组合3	1.48	0.56	1.92	1.00
组合4	0.56	0.36	0.48	0.44
组合5	0.88	0.48	0.84	0.52
组合6	2.40	1.56	2.76	2.24
组合7	1.80	1.20	1.52	1.60
组合8	1.68	2.80	1.08	1.36

Tab. 4 Proportions of training error ranking for selecting combinations under four groups of noise

$C b e s t$	data1	data2	data3	data4
组合1	0.48	0.32	0.56	0.68
组合2	0.36	0.28	0.20	0.24
组合3	1.48	0.56	1.92	1.00
组合4	0.56	0.36	0.48	0.44
组合5	0.88	0.48	0.84	0.52
组合6	2.40	1.56	2.76	2.24
组合7	1.80	1.20	1.52	1.60
组合8	1.68	2.80	1.08	1.36

Fig. 9 Training error values of combination 1 under four kinds of noise

Tab. 5 Regression datasets

数据集	属性数	训练集样本数	测试集样本数
Abalone	8	997	427
Bodyfat	14	177	75
Bike sharing	14	511	221
Cancer	30	399	171
Slump	7	73	31

Tab. 6 Performance of various algorithms on regression datasets

数据集	算法	训练时间/s	RMSE
Abalone	ELM	0.029 8±0.003 2	0.075 7±0.000 4
	BLS	0.277 8±0.024 5	0.075 4±0.001 6
	WBLS	0.097 1±0.002 4	0.075 3±0.000 7
	RWS-BLS	1.694 9±0.104 2	0.054 5±0.000 2
Bodyfat	ELM	0.055 7±0.001 2	0.123 8±0.001 4
	BLS	0.033 8±0.002 4	0.122 9±0.004 5
	WBLS	0.065 8±0.000 3	0.122 0±0.000 8
	RWS-BLS	0.065 8±0.001 1	0.105 1±0.001 0
Bike sharing	ELM	0.041 8±0.000 7	0.087 8±0.000 4
	BLS	0.981 8±0.002 8	0.089 8±0.001 2
	WBLS	0.058 1±0.003 0	0.076 1±0.000 8
	RWS-BLS	1.014 4±0.004 4	0.073 4±0.001 4
Cancer	ELM	0.005 0±0.000 1	0.135 7±0.002 1
	BLS	0.147 9±0.003 4	0.141 1±0.001 8
	WBLS	0.676 4±0.000 9	0.132 8±0.000 3
	RWS-BLS	0.103 9±0.001 6	0.131 5±0.000 7
Slump	ELM	0.050 7±0.000 2	0.166 5±0.001 1
	BLS	0.144 5±0.001 4	0.178 4±0.002 4
	WBLS	0.070 7±0.000 3	0.161 9±0.001 3
	RWS-BLS	0.101 9±0.002 7	0.161 2±0.002 0

References 22

1	CHEN C L P， LIU Z. Broad learning system： an effective and efficient incremental learning system without the need for deep architecture［J］. IEEE Transactions on Neural Networks and Learning Systems， 2018， 29（1）：10-24.
2	CHEN C L P， LIU Z. Broad learning system： a new learning paradigm and system without going deep ［C］// Proceedings of the 2017 32nd Youth Academic Annual Conference of Chinese Association of Automation. Piscataway： IEEE， 2017：1271-1276.
3	ZHANG L， SUGANTHAN P N. A comprehensive evaluation of random vector functional link networks ［J］. Information Sciences， 2016， 367/368： 1094-1105.
4	CHEN C L P， LIU Z， FENG S. Universal approximation capability of broad learning system and its structural variations［J］. IEEE Transactions on Neural Networks and Learning Systems， 2019， 30（4）： 1191-1204.
5	ZHAO H， ZHENG J， DENG W， et al. Semi-supervised broad learning system based on manifold regularization and broad network［J］. IEEE Transactions on Circuits and Systems Ⅰ： Regular Papers， 2020， 67（3）： 983-994.
6	贾贺姿.基于宽度学习和深度集成的图像分类［D］.西安：西安电子科技大学， 2019： 38-61.
	JIA H Z. Image classification based on broad learning and deep ensemble［D］. Xi’an： Xidian University， 2019： 38-61.
7	JIN J-W， CHEN C L P. Regularized robust broad learning system for uncertain data modeling ［J］. Neurocomputing， 2018， 322：58-69.
8	郑云飞，陈霸东.基于最小p-范数的宽度学习系统［J］.模式识别与人工智能，2019，32（1）：51-57.
	ZHENG Y F， CHEN B D. Least p-norm broad learning system［J］. Pattern Recognition and Artificial Intelligence， 2019， 32（1）： 51-57.
9	ZHOU Q， HE X. Broad learning model based on enhanced features learning ［J］. IEEE Access， 2019， 7： 42536-42550.
10	杨永娇，邱宇，占力超.基于宽度学习的智能电网数据服务器流量异常检测算法［J］.计算机与现代化，2019（9）：77-82，89.
	YANG Y J， QIU Y， ZHAN L C. An anomaly detection approach on servers traffic in smart grid based on breadth learning algorithm［J］. Computer and Modernization， 2019（9）： 77-82， 89.
11	LIN Z D， CHEN H P， YANG Q， et al. A flexible approach for human activity recognition based on broad learning system ［C］// Proceedings of the 2019 11th International Conference on Machine Learning and Computing. New York： ACM， 2019：368-373.
12	徐鹏飞，王敏，刘金平，等.基于数据分布特性的代价敏感宽度学习系统［J］. 控制与决策，2021， 36（7）： 1686-1692.
	XU P F， WANG M， LIU J P，et al. Data distribution-based cost-sensitive broad learning system［J］. Control and Decision， 2021， 36（7）： 1686-1692.
13	ZONG W， HUANG G-B， CHEN Y. Weighted extreme learning machine for imbalance learning［J］. Neurocomputing， 2013， 101： 229-242.
14	CHU F， LIANG T， CHEN C L P， et al. Weighted broad learning system and its application in nonlinear industrial process modeling［J］. IEEE Transactions Neural Networks and Learning Systems， 2020， 31（8）： 3017-3031.
15	王萌铎，续欣莹，阎高伟，等.基于AdaBoost集成加权宽度学习系统的不平衡数据分类［J］.计算机工程， 2020，48（4）：99-105，112.
	WANG M D， XU X Y， YAN G W， et al. Imbalanced data classification based on ensemble weighted broad learning system with AdaBoost［J］. Computer Engineering， 2020， 48（4）： 99-105， 112.
16	郭威，徐涛.基于M-estimator的鲁棒宽度学习系统［J］.控制与决策，2023， 38 （4）： 1039-1046.
	GUO W， XU T. M-estimator-based robust broad learning system［J］. Control and Decision， 2023， 38（4）： 1039-1046.
17	任长娥，袁超，孙彦丽，等. 宽度学习系统研究进展［J］. 计算机应用研究， 2021， 38（8）： 2258-2267.
	REN C E， YUAN C， SUN Y L， et al. Research of broad learning system ［J］. Application Research of Computers， 2021， 38（8）： 2258-2267.
18	HUANG G-B， ZHU Q-Y， C-K SIEW. Extreme learning machine： theory and applications ［J］. Neurocomputing， 2006， 70（1/2/3）： 489-501.
19	BEN-ISRAEL A， GREVILLE T N E. Generalized Inverses： Theory And Applications［M］. New York： Springer-Verlag， 2003：56-98.
20	LV Y， LI B， YU J， et al. Reduce training error of extreme learning machine by selecting appropriate hidden layer output matrix［J］. Journal of Systems Science and Systems Engineering， 2021， 30： 552-571.
21	LeCUN Y， HUANG F J， BOTTOU L. Learning methods for generic object recognition with invariance to pose and lighting ［C］// Proceedings of the 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. Washington， DC： IEEE Computer Society， 2004： 97-104.
22	BACHE K， LICHMAN M. UCI machine learning repository［DB/OL］. （2013-10-10）［2023-10-01］. .

Robust weight matrix combination selection method of broad learning system

宽度学习系统中鲁棒性权值矩阵组合的筛选方法

RichHTML

PDF

Knowledge

Abstract

Cite this article

share this article

Figures/Tables 15

References 22

Related Articles 1

Recommended Articles

Metrics