Multi-level feature selection algorithm based on mutual information

doi:10.11772/j.issn.1001-9081.2020060871

Journal of Computer Applications ›› 2020, Vol. 40 ›› Issue (12): 3478-3484.DOI: 10.11772/j.issn.1001-9081.2020060871

• 2020 China Conference on Granular Computing and Knowledge Discovery（CGCKD 2020） • Previous Articles Next Articles

Multi-level feature selection algorithm based on mutual information

YONG Juya^1,2, ZHOU Zhongmei^1,2

1. School of Computer Science, Minnan Normal University, Zhangzhou Fujian 363000, China;
2. Key Laboratory of Data Science and Intelligence Application, Fujian Province University, Zhangzhou Fujian 363000, China

Received:2020-06-12 Revised:2020-08-20 Online:2020-12-10 Published:2020-10-20
Supported by:
This work is partially supported by the Natural Science Foundation of Fujian Province （2018J01545）.

基于互信息的多级特征选择算法

雍菊亚^1,2, 周忠眉^1,2

1. 闽南师范大学计算机学院, 福建漳州 363000;
2. 数据科学与智能应用福建省高等学校重点实验室, 福建漳州 363000

通讯作者: 周忠眉(1965-),女,福建漳州人,教授,博士,CCF会员,主要研究方向:数据挖掘、机器学习。64523040@qq.com
作者简介:雍菊亚(1994-),女,江苏扬州人,硕士研究生,CCF会员,主要研究方向:数据挖掘
基金资助:
福建省自然科学基金资助项目（2018J01545）。

Abstract

Abstract: Focusing on the problem that the process of removing redundancy will be very complicated due to the large number of the selected features, and the problem that some features only can have strong correlation with label after being combined with other features in the feature selection, a Multi-Level Feature Selection algorithm based on Mutual Information (MI_MLFS) was proposed. Firstly, the features were divided into strongly correlated, sub-strongly correlated and other features according to the degrees of correlations between features and label. Secondly, after selecting strongly correlated features, features with low redundancy in the sub-strongly correlated features were selected. Finally, the features which were able to enhance the correlation between the selected feature subset and label were selected. Among 15 datasets, MI_MLFS was compared with the algorithms of ReliefF, minimal-Redundancy-Maximal-Relevance criterion (mRMR), Joint Mutual Information (JMI), Conditional Mutual Information Maximization criterion (CMIM) and Double Input Symmetrical Relevance (DISR). The results show that MI_MLFS achieves the highest classification accuracy in 13 datasets and 11 datasets with Support Vector Machine (SVM) classifier and Classification And Regression Tree (CART) classifier respectively. MI_MLFS has better classification performance than many classical feature selection algorithms.

Key words: feature selection, mutual information, multi-level, correlation, redundancy, classification accuracy

摘要： 针对在特征选择中选取特征较多时造成的去冗余过程很复杂的问题，以及一些特征需与其他特征组合后才会与标签有较强相关度的问题，提出了一种基于互信息的多级特征选择算法（MI_MLFS）。首先，根据特征与标签的相关度，将特征分为强相关、次强相关和其他特征；其次，选取强相关特征后，在次强相关特征中，选取冗余度较低的特征；最后，选取能增强已选特征集合与标签相关度的特征。在15组数据集上，将MI_MLFS与ReliefF、最大相关最小冗余（mRMR）算法、基于联合互信息（JMI）算法、条件互信息最大化准则（CMIM）算法和双输入对称关联（DISR）算法进行对比实验，结果表明MI_MLFS在支持向量机（SVM）和分类回归树（CART）分类器上分别有13组和11组数据集获得了最高的分类准确率。相较多种经典特征选择方法，MI_MLFS算法有更好的分类性能。

关键词: 特征选择, 互信息, 多级, 相关度, 冗余度, 分类准确率

CLC Number:

TP181

YONG Juya, ZHOU Zhongmei. Multi-level feature selection algorithm based on mutual information[J]. Journal of Computer Applications, 2020, 40(12): 3478-3484.

雍菊亚, 周忠眉. 基于互信息的多级特征选择算法[J]. 计算机应用, 2020, 40(12): 3478-3484.

References

[1] GEORGE G, HAAS M R, PENTLAND A. Big data and management[J]. Academy of Management Journal,2014,57(2):321-326.
[2] 谢娟英, 谢维信. 基于特征子集区分度与支持向量机的特征选择算法[J]. 计算机学报, 2014, 37(8):1704-1718.(XIE J Y,XIE W X. Several feature selection algorithms based on the discernibility of a feature subset and support vector machines[J]. Chinese Journal of Computers,2014,37(8):1704-1718.)
[3] KWAK N,CHOI C H. Input feature selection for classification problems[J]. IEEE Transactions on Neural Networks,2002,13(1):143-159.
[4] ESTEVEZ P A,TESMER M,PEREZ C A,et al. Normalized mutual information feature selection[J]. IEEE Transactions on Neural Networks,2009,20(2):189-201.
[5] HOQUE N,BHATTACHARYYA D K,KALITA J K. MIFS-ND:a mutual information-based feature selection method[J]. Expert Systems with Applications,2014,41(14):6371-6385.
[6] YANG H H,MOODY J. Feature selection based on joint mutual information[EB/OL].[2020-01-03]. http://pdfs.semanticscholar.org/dd69/1540c3f28decb477a7738f16aa92709b0f59.pdf.
[7] VINH L T,LEE S,PARK Y T,et al. A novel feature selection method based on normalized mutual information[J]. Applied Intelligence,2012,37(1):100-120.
[8] LEE J,KIM D W. Mutual Information-based multi-label feature selection using interaction information[J]. Expert Systems with Applications,2015,42(4):2013-2025.
[9] KONONENKO I. Estimation attributes:analysis and extension of RELIEF[C]//Proceedings of the 1994 European Conference on Machine Learning,LNCS 784. Berlin:Springer,1994:171-182.
[10] FLEURET F. Fast binary feature selection with conditional mutual information[J]. Journal of Machine Learning Research,2004,5:1531-1555.
[11] YANG H H,MOODY J. Data visualization and feature selection:new algorithms for nongaussian data[C]//Proceedings of the 200012th International Conference on Neural Information Processing Systems. Cambridge:MIT Press,2000:687-693.
[12] PENG H,LONG F,DING C. Feature selection based on mutual information:criteria of max-dependency,max-relevance,and minredundancy[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence,2005,27(8):1226-1238.
[13] VINH L T,THANG N D,LEE Y K. An improved maximum relevance and minimum redundancy feature selection algorithm based on normalized mutual information[C]//Proceedings of the 201010th IEEE/IPSJ International Symposium on Applications and the Internet. Piscataway:IEEE,2010:395-398.
[14] MEYER P E, SCHRETTER C, BONTEMPI G. Informationtheoretic feature selection in microarray data using variablecomplementarity[J]. IEEE Journal of Selected Topics in Signal Processing,2008,2(3):261-274.
[15] 段宏湘, 张秋余, 张墨逸. 基于归一化互信息的FCBF特征选择算[J]. 华中科技大学学报(自然科学版), 2017, 45(1):52-56. (DUAN H X,ZHANG Q Y,ZHANG M Y. FCBF algorithm based on normalized mutual information for feature selection[J]. Journal of Huazhong University of Science and Technology (Natural Science Edition),2017,45(1):52-56.)
[16] JOHN G H,KOHAVI R,PFLEGER K. Irrelevant features and the subset selection problem[C]//Proceedings of the 199411th International Conference on Machine Learning. San Francisco:Morgan Kaufmann Publisher,1994:121-129.
[17] 张俐, 王枞. 基于最大相关最小冗余联合互信息的多标签特征选择算法[J]. 通信学报, 2018, 39(5):111-122.(ZHANG L, WANG C. Multi-label feature selection algorithm based on mutual information of max-relevance and min-redundancy[J]. Journal on Communications,2018,39(5):111-122.)
[18] YANG Y. Book Review of Elements of Information Theory by COVER T M,THOMAS J A[J]. Publications of the American Statal Association,2008,103:429-429.
[19] 张振海, 李士宁, 李志刚, 等. 一类基于信息熵的多标签特征选择算法[J]. 计算机研究与发展, 2013, 50(6):1177-1184. (ZHANG Z H,LI S N,LI Z G,et al. Multi-label feature selection algorithm based on information entropy[J]. Journal of Computer Research and Development,2013,50(6):1177-1184.)
[20] LIN Y,HU Q,LIU J,et al. Multi-label feature selection based on max-dependency and min-redundancy[J]. Neurocomputing, 2015,168:92-103.
[21] RAHMANINIA M,MORADI P. OSFSMI:online stream feature selection method based on mutual information[J]. Applied Soft Computing,2018,68:733-746.
[22] 牛晓太. 基于KNN算法和10折交叉验证法的支持向量选取算法[J]. 华中师范大学学报(自然科学版), 2014, 48(3):335-338.(NIU X T. Support vector extracted algorithm based on KNN and 10 fold cross-validation method[J]. Journal of Huazhong Normal University(Natural Sciences),2014,48(3):335-338.)

Multi-level feature selection algorithm based on mutual information

基于互信息的多级特征选择算法

PDF

Knowledge

Abstract

Cite this article

share this article

References

Related Articles 15

Recommended Articles

Metrics

[1]	ZHANG Shipeng, LI Yongzhong, DU Xiangtong. Intrusion detection model based on semi-supervised learning and three-way decision [J]. Journal of Computer Applications, 2021, 41(9): 2602-2608.
[2]	ZHAN Hang, HE Lang, HUANG Zhangcan, LI Huafeng, ZHANG Qiang, TAN Qing. Improved feature selection and classification algorithm for gene expression programming based on layer distance [J]. Journal of Computer Applications, 2021, 41(9): 2658-2667.
[3]	SUN Haoyi, WANG Chuanmei, DING Yiming. Extreme learning machine optimization based on hidden layer output matrix [J]. Journal of Computer Applications, 2021, 41(9): 2481-2488.
[4]	JI Zhangjian, REN Xingwang. Object tracking algorithm of fully-convolutional Siamese networks with rotation and scale estimation [J]. Journal of Computer Applications, 2021, 41(9): 2705-2711.
[5]	ZHU Cheng, ZHAO Xiaoqi, ZHAO Liping, JIAO Yuhong, ZHU Yafei, CHENG Jianying, ZHOU Wei, TAN Ying. Classification of functional magnetic resonance imaging data based on semi-supervised feature selection by spectral clustering [J]. Journal of Computer Applications, 2021, 41(8): 2288-2293.
[6]	LI Mengmeng, QIN Wei, LIU Yi, DIAO Xingchun. Hybrid ant colony optimization algorithm with brain storm optimization [J]. Journal of Computer Applications, 2021, 41(8): 2412-2417.
[7]	ZHOU Xianbing, FAN Xiaochao, REN Ge, YANG Yong. Automated English essay scoring method based on multi-level semantic features [J]. Journal of Computer Applications, 2021, 41(8): 2205-2211.
[8]	YAN Haisheng, MA Xinqiang. Feature construction algorithm for multi-target regression via radial basis function [J]. Journal of Computer Applications, 2021, 41(8): 2219-2224.
[9]	LIU Fangming, ZHANG Hong. Cross-modal retrieval algorithm based on multi-level semantic discriminative guided hashing [J]. Journal of Computer Applications, 2021, 41(8): 2187-2192.
[10]	YANG Su, OUYANG Zhi, DU Nisuo. Unsupervised parallel hash image retrieval based on correlation distance [J]. Journal of Computer Applications, 2021, 41(7): 1902-1907.
[11]	LI Guorong, YE Jimin, ZHEN Yuanting. Time series clustering based on new robust similarity measure [J]. Journal of Computer Applications, 2021, 41(5): 1343-1347.
[12]	JIA Heming, JIANG Zichao, LI Yao, SUN Kangjian. Simultaneous feature selection optimization based on improved spotted hyena optimizer algorithm [J]. Journal of Computer Applications, 2021, 41(5): 1290-1298.
[13]	LIN Junchao, WAN Yuan. Self-adaptive multi-measure unsupervised feature selection method with structured graph optimization [J]. Journal of Computer Applications, 2021, 41(5): 1282-1289.
[14]	WANG Ning, SONG Huihui, ZHANG Kaihua. Accurate object tracking algorithm based on distance weighting overlap prediction and ellipse fitting optimization [J]. Journal of Computer Applications, 2021, 41(4): 1100-1105.
[15]	ZHANG Zhihao, LIN Yaojin, LU Shun, GUO Chen, WANG Chenxi. Multi-label feature selection based on label-specific feature with missing labels [J]. Journal of Computer Applications, 2021, 41(10): 2849-2857.