Semi-supervised classification algorithm based on weight diversity

doi:10.11772/j.issn.1001-9081.2020111872

Abstract

Abstract: In real life, many data samples of systems can be easily obtained, but only a small part of accurate laabels can be obtained. In order to obtain a better classification learning model, a semi-supervised classification algorithm based on weight diversity was proposed by introducing semi-supervised learning and improving Unlabeled Data to Enhance Ensemble Diversity (UDEED), namely UDEED⁺. In UDEED⁺, based on the prediction disagreement of unlabeled data by base learners, the loss of weight diversity was proposed. The disagreement between base learners was represented by the cosine similarity of the weights of base learners. The diversity of model was fully expanded from different perspectives of loss function, and the unlabeled data were used to encourage the diversity representation of ensemble learners in the process of model training, so as to improve the performance and generalization of the classification learning model. Comparative experiments were conducted on 8 UCI public datasets with semi-supervised algorithms of UDEED algorithm, Safe Semi-Supervised Support Vector Machine (S4VM) and Semi-Supervised Weak-Label (SSWL). Compared with UDEED, UDEED⁺ has the accuracy and F1 score improved by 1.4 percentage points and 1.1 percentage points respectively; compared with S4VM, UDEED⁺ has the accuracy and F1 score improved by 1.3 percentage points and 3.1 percentage points respectively; compared with UDEED, UDEED⁺ has the accuracy and F1 score improved by 0.7 percentage points and 1.5 percentage points respectively. Experimental results illustrate that the increase of weight diversity can improve the classification performance of the model, verifying its positive effect on the improvement of the classification performance of UDEED⁺.

Key words: classification machine learning, unlabeled data, semi-supervised learning, ensemble learning, diversity

摘要： 在实际生活中，可以很容易地获得大量系统数据样本，却只能获得很小一部分的准确标签。为了获得更好的分类学习模型，引入半监督学习的处理方式，对基于未标注数据强化集成多样性（UDEED）算法进行改进，提出了UDEED⁺——一种基于权值多样性的半监督分类算法。UDEED⁺主要的思路是在基学习器对未标注数据的预测分歧的基础上提出权值多样性损失，通过引入基学习器权值的余弦相似度来表示基学习器之间的分歧，并且从损失函数的不同角度充分扩展模型的多样性，使用未标注数据在模型训练过程中鼓励集成学习器的多样性的表示，以此达到提升分类学习模型性能和泛化性的目的。在8个UCI公开数据集上，与UDEED算法、S4VM（Safe Semi-Supervised Support Vector Machine）和SSWL（Semi-Supervised Weak-Label）半监督算法进行了对比，相较于UDEED算法，UDEED⁺在正确率和F1分数上分别提升了1.4个百分点和1.1个百分点；相较于S4VM，UDEED⁺在正确率和F1分数上分别提升了1.3个百分点和3.1个百分点；相较于SSWL，UDEED⁺在正确率和F1分数上分别提升了0.7个百分点和1.5个百分点。实验结果表明，权值多样性的提升可以改善UDEED⁺算法的分类性能，验证了其对所提算法UDEED⁺的分类性能提升的正向效果。

关键词: 分类机器学习, 未标注数据, 半监督学习, 集成学习, 多样性

CLC Number:

TP391.4

MAO Mingze, CAO Ruihao, YAN Chungang. Semi-supervised classification algorithm based on weight diversity[J]. Journal of Computer Applications, 2021, 41(9): 2473-2480.

毛铭泽, 曹芮浩, 闫春钢. 基于权值多样性的半监督分类算法[J]. 计算机应用, 2021, 41(9): 2473-2480.

References

[1] ZHOU Z H. A brief introduction to weakly supervised learning[J]. National Science Review,2018,5(1):44-53.
[2] PRINCE M. Does active learning work? A review of the research[J]. Journal of Engineering Education,2004,93(3):223-231.
[3] ZHOU Z H,LI M. Semi-supervised learning by disagreement[J]. Knowledge and Information Systems,2010,24(3):415-439.
[4] PAPANDREOU G,CHEN L C,MURPHY K P,et al. Weaklyand semi-supervised learning of a deep convolutional network for semantic image segmentation[C]//Proceedings of the 2015 IEEE International Conference on Computer Vision. Piscataway:IEEE, 2015:1742-1750.
[5] CHAPELLE O, SCHÖLKOPF B, ZIEN A. Semi-Supervised Learning[M]. Cambridge:MIT Press,2006:1-8.
[6] DIETTERICH T G,LATHROP R H,LOZANO-PÉREZ T. Solving the multiple instance problem with axis-parallel rectangles[J]. Artificial Intelligence,1997,89(1/2):31-71.
[7] ZHANG Z Y,ZHAO P,JIANG Y,et al. Learning from incomplete and inaccurate supervision[C]//Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. New York:ACM,2019:1017-1025.
[8] WANG D, TAN X Y. Label-denoising auto-encoder for classification with inaccurate supervision information[C]//Proceedings of the 22nd International Conference on Pattern Recognition. Piscataway:IEEE,2014:3648-3653.
[9] DEMPSTER A P,LAIRD N M,RUBIN D B. Maximum likelihood from incomplete data via the EM algorithm[J]. Journal of the Royal Statistical Society:Series B (Methodological),1977,39(1):1-38.
[10] MILLER D J,UYAR H S. A mixture of experts classifier with learning based on both labelled and unlabelled data[C]//Proceedings of the 9th International Conference on Neural Information Processing Systems. Cambridge:MIT Press,1996:571-577.
[11] NIGAM K, MCCALLUM A K, THRUN S, et al. Text classification from labeled and unlabeled documents using EM[J]. Machine Learning,2000,39(2/3):103-134.
[12] BLUM A,CHAWLA S. Learning from labeled and unlabeled data using graph mincuts[C]//Proceedings of the 18th International Conference on Machine Learning. San Francisco:Morgan Kaufmann Publishers Inc.,2001:19-26.
[13] ZHOU D Y,BOUSQUET O,LAL T N,et al. Learning with local and global consistency[C]//Proceedings of the 16th International Conference on Neural Information Processing Systems. Cambridge:MIT Press,2003:321-328.
[14] ZHU X J,GHAHRAMANI Z,LAFFERTY J. Semi-supervised learning using Gaussian fields and harmonic functions[C]//Proceedings of the 20th International Conference on Machine Learning. Palo Alto,CA:AAAI Press,2003:912-919.
[15] CHAPELLE O,ZIEN A. Semi-supervised classification by low density separation[C]//Proceedings of the 10th International Workshop on Artificial Intelligence and Statistics. New York:JMLR. org,2005:57-64.
[16] JOACHIMS T. Transductive inference for text classification using support vector machines[C]//Proceedings of the 16th International Conference on Machine Learning. San Francisco:Morgan Kaufmann Publishers Inc.,1999:200-209.
[17] LI Y F,TSANG I W,KWOK J T,et al. Convex and scalable weakly labeled SVMs[J]. Journal of Machine Learning Research, 2013,14:2151-2188.
[18] BLUM A,MITCHELL T. Combining labeled and unlabeled data with co-training[C]//Proceedings of the 11th Annual Conference on Computational Learning Theory. New York:ACM,1998:92-100.
[19] CHEN D D, WANG W, GAO W, et al. Tri-net for semisupervised deep learning[C]//Proceedings of the 27th International Joint Conference on Artificial Intelligence. Palo Alto,CA:AAAI Press,2018:2014-2020.
[20] ZHOU Z H. When semi-supervised learning meets ensemble learning[J]. Frontiers of Electrical and Electronic Engineering in China,2011,6(1):6-16.
[21] 周志华. 基于分歧的半监督学习[J]. 自动化学报,2013,39(11):1871-1878. (ZHOU Z H. Disagreement-based semisupervised learning[J]. Acta Automatica Sinica,2013,39(11):1871-1878.)
[22] ZHANG M L,ZHOU Z H. Exploiting unlabeled data to enhance ensemble diversity[J]. Data Mining and Knowledge Discovery, 2013,26(1):98-129.
[23] JOHNSON R W. An introduction to the bootstrap[J]. Teaching Statistics,2001,23(2):49-54.
[24] DUA D,GRAFF C. UCI machine learning repository[DS/OL].[2020-09-08]. http://archive.ics.uci.edu/ml.
[25] LI Y F,ZHOU Z H. Towards making unlabeled data never hurt[C]//Proceedings of the 28th International Conference on Machine Learning. Madison,WI:Omnipress,2011:1081-1088.
[26] DONG H C,LI Y F,ZHOU Z H. Learning from semi-supervised weak-label data[C]//Proceedings of the 32nd AAAI Conference on Artificial Intelligence. Palo Alto,CA:AAAI,2018:2926-2933.