Structure-fuzzy multi-class support vector machine algorithm based on pinball loss

doi:10.11772/j.issn.1001-9081.2021010062

Abstract

Abstract:

The Multi-Class Support Vector Machine （MSVM） has the defects such as strong sensitivity to noise， instability to resampling data and lower generalization performance. In order to solve the problems， the pinball loss function， sample fuzzy membership degree and sample structural information were introduced into the Simplified Multi-Class Support Vector Machine （SimMSVM） algorithm， and a structure-fuzzy multi-class support vector machine algorithm based on pinball loss， namely Pin-SFSimMSVM， was proposed. Experimental results on synthetic datasets， UCI datasets and UCI datasets adding different proportions of noise show that， the accuracy of the proposed Pin-SFSimMSVM algorithm is increased by 0~5.25 percentage points compared with that of SimMSVM algorithm. The results also show that the proposed algorithm not only has the advantages of avoiding indivisible areas of multi-class data and fast calculation speed， but also has good insensitivity to noise and stability to resampling data. At the same time， the proposed algorithm considers the fact that different data samples play different roles in classification and the important prior knowledge contained in the data， so that the classifier training is more accurate.

Key words: multi-class, Support Vector Machine (SVM), pinball loss, structural information, fuzzy membership degree

摘要：

针对多分类支持向量机（MSVM）对噪声较强的敏感性、对重采样数据的不稳定性以及泛化性能低等缺陷，将pinball损失函数、样本模糊隶属度以及样本结构信息引入到简化的多分类支持向量机（SimMSVM）算法中，构建了基于pinball损失的结构模糊多分类支持向量机算法Pin-SFSimMSVM。在人工数据集、UCI数据集以及添加不同比例噪声的UCI数据集上的实验结果显示：所提出的Pin-SFSimMSVM算法与SimMSVM算法相比，准确率均提升了0~5.25个百分点；所提出的算法不仅具有避免多类数据存在不可分区域和计算速度快的优点，而且具有对噪声较好的不敏感性以及对重采样数据的稳定性，同时考虑了不同数据样本在分类时扮演不同角色的事实以及数据中包含的重要先验知识，从而使分类器训练更准确。

关键词: 多分类, 支持向量机, pinball损失, 结构信息, 模糊隶属度

CLC Number:

TP391

Kai LI, Jie LI. Structure-fuzzy multi-class support vector machine algorithm based on pinball loss[J]. Journal of Computer Applications, 2021, 41(11): 3104-3112.

李凯, 李洁. 基于pinball损失的结构模糊多分类支持向量机算法[J]. 《计算机应用》唯一官方网站, 2021, 41(11): 3104-3112.

Figures/Tables 14

References 30

1	VAPNIK V N. The Nature of Statistical Learning Theory ［M］. New York： Springer， 1995： 96. 10.1007/978-1-4757-2440-0_7
2	JAYADEVA， KHEMCHANDANI R， CHANDRA S. Twin support vector machines for pattern classification ［J］. IEEE Transactions on Pattern Analysis and Machine Intelligence， 2007， 29（5）： 905-910. 10.1109/tpami.2007.1068
3	HUANG H J， WEI X X， ZHOU Y Q. Twin support vector machines： a survey ［J］. Neurocomputing， 2018， 300： 34-43. 10.1016/j.neucom.2018.01.093
4	DING S F， ZHANG N， ZHANG X K， et al. Twin support vector machine： theory， algorithm and applications ［J］. Neural Computing and Applications， 2017， 28（11）： 3119-3130. 10.1007/s00521-016-2245-4
5	TANVEER M， SHARMA A， SUGANTHAN P N. General twin support vector machine with pinball loss function ［J］. Information Sciences， 2019， 494： 311-327. 10.1016/j.ins.2019.04.032
6	XU Y T， YANG Z J， PAN X L. A novel twin support-vector machine with pinball loss ［J］. IEEE Transactions on Neural Networks and Learning Systems， 2017， 28（2）： 359-370. 10.1109/tnnls.2015.2513006
7	SHARMA S， RASTOGI R， CHANDRA S. Large-scale twin parametric support vector machine using pinball loss function ［J］. IEEE Transactions on Systems， Man， and Cybernetics： Systems， 2021， 51（2）： 987-1003. 10.1109/tsmc.2019.2896642
8	方佳艳，刘峤，吴德，等.基于模糊C-均值的相似性特征转换光滑支持向量机［J］.电子学报，2018，46（11）：2714-2724. 10.3969/j.issn.0372-2112.2018.11.019
	FANG J Y， LIU Q， WU D， et al. Smooth support vector machine with similarity-based feature transformation technique and fuzzy C-means clustering ［J］. Acta Electronica Sinica， 2018， 46（11）： 2714-2724. 10.3969/j.issn.0372-2112.2018.11.019
9	XU Y T， PAN X L， ZHOU Z J， et al. Structural least square twin support vector machine for classification ［J］. Applied Intelligence， 2015， 42（3）： 527-536. 10.1007/s10489-014-0611-4
10	CHEN D D， TIAN Y J， LIU X H. Structural nonparallel support vector machine for pattern recognition ［J］. Pattern Recognition， 2016， 60： 296-305. 10.1016/j.patcog.2016.04.017
11	YU L， ZHOU R T， TANG L， et al. A DBN-based resampling SVM ensemble learning paradigm for credit classification with imbalanced data ［J］. Applied Soft Computing， 2018， 69： 192-202. 10.1016/j.asoc.2018.04.049
12	丁世飞，张健，张谢锴，等.多分类孪生支持向量机研究进展［J］.软件学报，2018，29（1）：89-108. 10.13328/j.cnki.jos.005319
	DING S F， ZHANG J， ZHANG X K， et al. Survey on multi class twin support vector machines ［J］. Journal of Software， 2018， 29（1）： 89-108. 10.13328/j.cnki.jos.005319
13	SHAO Y H， CHEN W J， WANG Z， et al. Weighted linear loss twin support vector machine for large-scale classification ［J］. Knowledge-Based Systems， 2015， 73：276-288. 10.1016/j.knosys.2014.10.011
14	XIE J Y， HONE K， LIU W X， et al. Extending twin support vector machine classifier for multi-category classification problems ［J］. Intelligent Data Analysis， 2013， 17（4）：649-664. 10.3233/ida-130598
15	TOMAR D， AGARWAL S. Direct acyclic graph based multi-class twin support vector machine for pattern classification ［C］// Proceedings of the 2015 2nd ACM IKDD Conference on Data Sciences. New York： ACM， 2015： 80-85. 10.1145/2732587.2732598
16	DON D R， IACOB I E. DCSVM： fast multi-class classification using support vector machines ［J］. International Journal of Machine Learning and Cybernetics， 2020， 11（2）： 433-447. 10.1007/s13042-019-00984-9
17	LIU Y， BI J W， FAN Z P. A method for multi-class sentiment classification based on an improved One-Vs-One （OVO） strategy and the Support Vector Machine （SVM） algorithm ［J］. Information Sciences， 2017， 394/395： 38-52. 10.1016/j.ins.2017.02.016
18	YANG Z M， WU H J， LI C N， et al. Least squares recursive projection twin support vector machine for multi-class classification ［J］. International Journal of Machine Learning and Cybernetics， 2016， 7（3）：411-426. 10.1007/s13042-015-0394-x
19	XU Y T， GUO R， WANG L S. A twin multi-class classification support vector machine ［J］. Cognitive Computation， 2013， 5（4）： 580-588. 10.1007/s12559-012-9179-7
20	YANG Z X， SHAO Y H， ZHANG X S. Multiple birth support vector machine for multi-class classification ［J］. Neural Computing and Applications， 2013， 22（S1）： 153-161. 10.1007/s00521-012-1108-x
21	CHU M X， LIU X P， GONG R F， et al. Multi-class classification method using twin support vector machines with multi-information for steel surface defects ［J］. Chemometrics and Intelligent Laboratory Systems， 2018， 176： 108-118. 10.1016/j.chemolab.2018.03.014
22	JU H M， JING L. An improved fuzzy multi-class twin support vector machine ［C］// Proceedings of the 6th International Conference on Systems and Informatics. Piscataway： IEEE， 2019： 393-397. 10.1109/icsai48974.2019.9010402
23	TOMAR D， AGARWAL S. A comparison on multi-class classification methods based on least squares twin support vector machine ［J］. Knowledge-Based Systems， 2015， 81：131-147. 10.1016/j.knosys.2015.02.009
24	TANG L， TIAN Y J， PARDALOS P M. A novel perspective on multiclass classification： regular simplex support vector machine ［J］. Information Sciences， 2019， 480： 324-338. 10.1016/j.ins.2018.12.026
25	GUO H S， WANG W J. An active learning-based SVM multi-class classification model ［J］. Pattern Recognition， 2015， 48（5）： 1577-1597. 10.1016/j.patcog.2014.12.009
26	WESTON J， WATKINS C. Multi-class support vector machine： CSD-TR-98-04 ［R］. Egham， Surrey： Royal Holloway， University of London， 1998.
27	CRAMMER K， SINGER Y. On the algorithmic implementation of multiclass kernel-based vector machines ［J］. Journal of Machine Learning Research， 2001， 2： 265-292. 10.1007/3-540-44581-1_7
28	WANG Z， XUE X Y. Multi-class support vector machine ［M］// MA Y Q， GUO G D. Support Vector Machines Applications. Cham： Springer， 2014： 23-48. 10.1007/978-3-319-02300-7_2
29	DUA D， GRAFF C. UCI machine learning repository ［DS/OL］. ［2020-12-08］. .
30	RIPLEY B D. Pattern Recognition and Neural Networks ［DB/OL］. ［2020-12-08］. . 10.1017/cbo9780511812651

数据集	样本数	特征数	类别数	每类包含的簇数
Data1	400	2	4	1，1，1，1
Data2	300	2	3	2，3，3
Data3	500	3	5	1，1，1，1，1

数据集	样本数	特征数	类别数	每类包含的簇数
Data1	400	2	4	1，1，1，1
Data2	300	2	3	2，3，3
Data3	500	3	5	1，1，1，1，1

数据集	样本数	特征数	类别数
Iris	150	4	3
Zoo	101	16	7
Glass	214	9	6
Seeds	210	7	3
Ecoli	327	7	5
Balance	625	4	3
Soybean	47	35	4
CMC	1 473	9	3

数据集	样本数	特征数	类别数
Iris	150	4	3
Zoo	101	16	7
Glass	214	9	6
Seeds	210	7	3
Ecoli	327	7	5
Balance	625	4	3
Soybean	47	35	4
CMC	1 473	9	3

数据集	SimMSVM	OVO-TWSVM	OVA-TWSVM	Twin-KSVC	MBSVM	Pin-SFSimMSVM （层次聚类）	Pin-SFSimMSVM （C均值聚类）
Ripley	86.200 0±4.467 2	88.000 0±3.887 3	85.800 0±4.756 3	85.000 0±3.681 8	85.600 0±4.263 5	88.000 0±4.422 2	88.100 0±4.756 3
Data1	84.150 0±2.000 7	84.250 0±3.197 8	82.500 0±3.015 6	80.000 0±2.500 0	79.625 0±3.387 6	84.450 0±2.420 4	84.450 0±2.420 4
Data2	74.166 7±3.773 0	74.166 7±3.418 3	75.333 3±3.279 3	72.166 7±3.187 5	71.666 7±3.015 9	76.833 3±4.142 5	74.500 0±4.037 4
Data3	78.375 0±2.208 7	87.125 0±1.608 4	84.400 0±4.880 8	83.500 0±3.634 8	84.125 0±3.227 5	79.027 8±2.365 1	78.625 0±3.653 9