Expert counter-evaluation model with three-way decision and entropy weight TOPSIS

doi:10.11772/j.issn.1001-9081.2024060819

Journal of Computer Applications ›› 2025, Vol. 45 ›› Issue (5): 1424-1431.DOI: 10.11772/j.issn.1001-9081.2024060819

• 2024 China Granular Computing and Knowledge Discovery Conference (CGCKD2024) • Previous Articles

Expert counter-evaluation model with three-way decision and entropy weight TOPSIS

Ying YU¹(), Feng ZHU¹, Hongjian FU², Yiwen LUO², Jin QIAN¹, Yuchao ZHENG¹

^1.School of Software，East China Jiaotong University，Nanchang Jiangxi 330013，China
^2.Scientific and Technological Information Research Institute，China Academy of Railway Sciences Corporation Limited，Beijing 100081，China

Received:2024-06-18 Revised:2024-07-08 Accepted:2024-07-11 Online:2024-07-25 Published:2025-05-10
Contact: Ying YU
About author:YU Ying， born in 1979， Ph. D.， professor. Her research interests include machine learning， computer vision， granular computing.
ZHU Feng， born in 1999， M. S. candidate. His research interests include computer vision， crowd counting.
FU Hongjian， born in 1986， senior engineer. His research interests include science and technology management.
LUO Yiwen， born in 1995， M. S.， engineer. Her research interests include science and technology management.
QIAN Jin， born in 1975， Ph. D.， professor. His research interests include granular computing， big data mining， machine learning.
ZHENG Yuchao， born in 1986， M. S.， lecturer. His research interests include machine learning.
Supported by:
Science and Technology Research and Development Program of National Railway Corporation(2023F005);National Natural Science Foundation of China(62163016);Natural Science Foundation of Jiangxi Province(20212ACB202001);High-Level Foreign Experts Recruitment Project of the Ministry of Science and Technology(G2023022005L);Science and Technology Research Program of Jiangxi Provincial Department of Education(GJJ2200644)

联合三支决策与熵权TOPSIS的专家反评估模型

余鹰¹(), 朱锋¹, 付红剑², 罗逸文², 钱进¹, 郑宇超¹

^1.华东交通大学软件学院，南昌 330013
^2.中国铁道科学研究院集团有限公司科学技术信息研究所，北京 100081

通讯作者: 余鹰
作者简介:余鹰（1979—），女，江西上饶人，教授，博士，CCF会员，主要研究方向：机器学习、计算机视觉、粒计算
朱锋（1999—），男，河南信阳人，硕士研究生，主要研究方向：计算机视觉、人群计数
付红剑（1986—），男，江西上饶人，高级工程师，主要研究方向：科技管理
罗逸文（1995—），女，陕西宝鸡人，工程师，硕士，主要研究方向：科技管理
钱进（1975—），男，江苏常州人，教授，博士，CCF会员，主要研究方向：粒计算、大数据挖掘、机器学习
郑宇超（1986—），男，江西南昌人，讲师，硕士，主要研究方向：机器学习。
基金资助:
国铁集团科技研究开发计划项目(2023F005);国家自然科学基金资助项目(62163016);江西省自然科学基金资助项目(20212ACB202001);科技部高端外国专家引进计划项目(G2023022005L);江西省教育厅科技计划项目(GJJ2200644)

Abstract

Abstract:

During the evaluation and review process of scientific and technological projects， the quality of evaluation and review conducted by experts significantly influences the accuracy and credibility of the final evaluation results. To ensure the fairness and objectivity of the evaluation results， it is necessary to conduct a counter-evaluation of the evaluation experts. By analyzing three aspects including the basic personal situation， professional level and evaluation performance of evaluation experts， an evaluation index system for counter-evaluation of experts was constructed. Based on this， an expert counter-evaluation model that combined three-way decisions and entropy weight based Technique for Order Preference by Similarity to Ideal Solution （TOPSIS） was proposed， which used the three-way decision theory to solve the problem of weight distortion due to the over-reliance of entropy weight method on data. When the index weights were abnormal， the group of experts to be evaluated was divided into positive， negative and boundary domains according to the threshold， and then the expert-in， expert-out and delayed evaluation strategies were adopted respectively. Once the index weights were normal or corrected， TOPSIS was used to rank the evaluation experts. Through empirical analysis of the historical evaluation and review data of scientific and technological projects from an enterprise， it can be seen that the proposed model can integrate the empirical judgment of decision makers with inherent information of the experts to be evaluated to realize the unity of subjectivity and objectivity， ensuring a scientific and fair evaluation of the evaluation experts， and providing decision-making references for construction of a high-quality expert database.

Key words: three-way decision, entropy weight method, Technique for Order Preference by Similarity to Ideal Solution (TOPSIS), scientific and technological project evaluation and review, expert counter-evaluation

摘要：

在科技项目评审活动中，评审专家的评审质量对于最终评价结果的准确性和公信力具有十分重要的影响。为确保评价结果的公正客观，对评审专家进行反评估显得尤为必要。通过对评审专家的个人基本情况、专业水平和评审业绩3个方面进行分析，构建了一套专家反评估评价指标体系。在此基础上，提出一种联合三支决策与基于熵权的逼近理想解排序法（TOPSIS）的专家反评估模型。该模型利用三支决策理论解决由于熵权法对数据的过度依赖而造成的权重失真问题。当指标权重出现异常时，模型将根据设定的阈值将待评估专家集合划分为正域、负域和边界域，并分别实施专家入库、专家出库和延迟评价策略；当指标权重正常或者得到修正后，则采用TOPSIS对评审专家进行评分排序。通过对某企业科技项目历史评审数据的实证分析可知，所提模型可以融合决策者的经验判断与待评专家的固有信息，实现主客观的统一，从而确保对评审专家进行科学、公正的评价，为构建高质量的专家库提供决策参考。

关键词: 三支决策, 熵权法, 逼近理想解排序法, 科技项目评审, 专家反评估

CLC Number:

TP301.6

Ying YU, Feng ZHU, Hongjian FU, Yiwen LUO, Jin QIAN, Yuchao ZHENG. Expert counter-evaluation model with three-way decision and entropy weight TOPSIS[J]. Journal of Computer Applications, 2025, 45(5): 1424-1431.

余鹰, 朱锋, 付红剑, 罗逸文, 钱进, 郑宇超. 联合三支决策与熵权TOPSIS的专家反评估模型[J]. 《计算机应用》唯一官方网站, 2025, 45(5): 1424-1431.

Figures/Tables 12

Fig. 1 Three-way decision model

Fig. 2 Entropy weight based TOPSIS

Fig. 3 Evaluation index system

Tab. 1 Evaluation index attributes

指标名称	指标属性	指标名称	指标属性
学历	极大型	科研项目	极大型
职称	极大型	科研成果	极大型
工龄	区间型	评审量	极大型
学术称号	极大型	偏离度	极小型

Tab. 2 Calculation methods of basic situation indexes

指标	计算描述
学历	博士、硕士、学士及其他，分别按照（100，80，60，40）评分
职称	正高级、副高级、中级、初级及无职称，分别按照（100，80，60，40，20）评分
工龄	工龄=当前年-参加工作年，区间型指标，25~35为理想值

Fig. 4 Evaluation model combining three-way decision with entropy weight based TOPSIS

Tab. 3 Initial values of evaluation indexes for sample experts

专家序号	基本情况			专业水平			评审业绩
专家序号	学历	职称	工龄	学术称号	科研项目	科研成果	评审量	偏离度
1	80	100	42	60	340	410	52	1.42
2	100	80	23	60	370	290	124	1.44
3	60	100	38	20	140	60	60	3.65
4	100	100	35	60	380	340	140	1.10
5	80	100	29	40	230	190	46	2.08
6	80	100	37	60	220	280	76	1.23
7	60	80	17	20	90	70	40	3.45
8	60	80	20	20	150	90	72	3.46
9	100	100	29	80	540	670	94	0.82
10	100	80	35	40	270	240	90	2.48
11	100	80	18	40	180	260	96	2.04
12	80	100	33	40	290	170	52	2.50
13	100	100	36	80	520	550	168	0.95
14	100	80	25	60	230	240	122	1.84
15	100	100	29	60	220	280	42	1.95
16	100	100	34	80	580	720	108	0.62
17	60	80	22	20	170	80	94	3.10
18	80	100	34	60	310	250	92	1.77
19	100	100	27	60	310	230	112	1.70
20	80	100	40	40	280	230	46	2.15
21	100	100	24	80	490	510	134	1.00
22	100	100	27	40	220	260	176	2.07
23	80	100	19	40	290	210	108	2.46
24	100	100	25	80	630	690	94	0.70
25	80	100	38	40	210	240	44	2.38

Tab. 4 Weight of each index

指标名称	信息熵	信息效用值	权重	是否异常
学历	0.995 3	0.004 7	0.020 0	否
职称	0.998 5	0.001 5	0.006 2	否
工龄	0.957 6	0.042 4	0.179 4	否
学术称号	0.975 6	0.024 4	0.103 2	否
科研项目	0.968 1	0.031 9	0.134 8	否
科研成果	0.945 1	0.054 9	0.232 3	是
评审量	0.972 9	0.027 1	0.114 7	否
偏离度	0.950 5	0.049 5	0.209 5	否

Fig. 5 Distribution of scientific achievement measures

Tab. 5 Decision rule table

专家序号	科研成果	规则条件	决策
16	0.406 4	$n e r r ≥ 0.3$	专家入库（正域）
24	0.389 5
9	0.378 2
13	0.310 4
21	0.287 9	$0.06 < n e r r < 0.3$	延迟评价（边界域）
1	0.231 4
4	0.191 9
2	0.163 7
6	0.158 0
15	0.158 0
11	0.146 7
22	0.146 7
18	0.141 1
10	0.135 5
14	0.135 5
25	0.135 5
19	0.129 8
20	0.129 8
23	0.118 5
5	0.107 2
12	0.096 0
8	0.050 8	$n e r r ≤ 0.06$	专家出库（负域）
17	0.045 2
7	0.039 5
3	0.033 9

Tab. 5 Decision rule table

专家序号	科研成果	规则条件	决策
16	0.406 4	$n e r r ≥ 0.3$	专家入库（正域）
24	0.389 5
9	0.378 2
13	0.310 4
21	0.287 9	$0.06 < n e r r < 0.3$	延迟评价（边界域）
1	0.231 4
4	0.191 9
2	0.163 7
6	0.158 0
15	0.158 0
11	0.146 7
22	0.146 7
18	0.141 1
10	0.135 5
14	0.135 5
25	0.135 5
19	0.129 8
20	0.129 8
23	0.118 5
5	0.107 2
12	0.096 0
8	0.050 8	$n e r r ≤ 0.06$	专家出库（负域）
17	0.045 2
7	0.039 5
3	0.033 9

Tab. 6 Corrected index weights

指标名称	信息熵	信息效用值	修正后权重
学历	0.997 8	0.002 2	0.008 5
职称	0.998 5	0.001 5	0.005 8
工龄	0.930 4	0.069 6	0.275 0
学术称号	0.990 7	0.009 3	0.036 7
科研项目	0.988 1	0.011 9	0.046 9
科研成果	0.986 3	0.013 7	0.054 1
评审量	0.967 0	0.033 0	0.130 5
偏离度	0.888 1	0.111 9	0.442 5

Tab. 7 Evaluation results of TOPSIS

专家序号	正理想解距离	负理想解距离	相对贴近度
21	0.043 3	0.355 3	0.891 4
4	0.057 0	0.340 0	0.856 5
6	0.129 2	0.284 2	0.687 5
2	0.121 6	0.257 2	0.679 0
19	0.166 7	0.236 2	0.586 2
18	0.184 3	0.222 7	0.547 3
14	0.191 6	0.219 3	0.533 8
1	0.212 7	0.227 1	0.516 3
22	0.230 7	0.215 8	0.483 3
15	0.236 3	0.194 8	0.451 8
5	0.262 0	0.178 8	0.405 7
10	0.318 9	0.163 9	0.339 5
12	0.334 3	0.158 3	0.321 4
11	0.284 4	0.106 5	0.272 4
20	0.293 1	0.086 8	0.228 5
25	0.321 2	0.094 2	0.226 7
23	0.340 0	0.066 5	0.163 5

References 30

1	张尧，林春.专家评审特征分析及专家反评估方法研究［J］.工业工程，2020，23（6）：124-130，154.
	ZHANG Y， LIN C. The review characteristics of expert and anti-assessment research［J］. Industrial Engineering Journal， 2020， 23（6）： 124-130， 154.
2	汪建，王裴裴，丁俊.科技项目专家评审的元评价综合模型研究［J］.科研管理，2020，41（2）：183-192.
	WANG J， WANG P P， DING J. A comprehensive model on the meta-evaluation of science and technology projects by review experts［J］. Science Research Management， 2020， 41（2）： 183-192.
3	杨健安，刘传斌，余乐安.基于相对逆序数的评审专家可信度评价方法［J］. 统计与决策，2022，38（3）： 184-188.
	YANG J A， LIU C B， YU L A. Relative inverse ordinal number based credibility evaluation method for evaluation experts［J］. Statistics and Decision Making， 2022， 38（3）： 184-188.
4	徐锦英，葛慧丽.同行评议专家信誉评价体系构建及实证研究［J］.科技管理研究，2017，37（21）：35-39.
	XU J Y， GE H L. Construction and empirical research on credibility evaluation system of peer reviewers［J］. Science and Technology Management Research， 2017， 37（21）： 35-39.
5	LI T， LI G， ZHANG M， et al. Research on performance and dynamic competency evaluation of bid evaluation experts based on weight interval number［J］. PLoS ONE， 2022， 17（7）： No.e0269467.
6	WANG J， MA Q， LIU H C. A meta-evaluation model on science and technology project review experts using IVIF-BWM and MULTIMOORA［J］. Expert Systems with Applications， 2021， 168： No.114236.
7	游庆根，张金隆，马辉民.基于熵权法的评审专家遴选模型研究［C］// 第十五届全国计算机模拟与信息技术学术会议论文集.长沙：管理学报，2015：275-282.
	YOU Q G， ZHANG J L， MA H M. Research on selection model of evaluation experts based on entropy weight method［C］// Proceedings of the 15th National Academic Conference on Computer Simulation and Information Technology. Changsha： Journal of Management， 2015： 275-282.
8	YAO Y. Three-way decisions with probabilistic rough sets［J］. Information Sciences， 2010， 180（3）： 341-353.
9	PAWLAK Z， SKOWRON A. Rudiments of rough sets［J］. Information Sciences， 2007， 177（1）： 3-27.
10	YAO Y. Decision-theoretic rough set models［C］// Proceedings of the 2007 International Conference on Rough Sets and Knowledge Technology， LNCS 4481. Berlin： Springer， 2007： 1-12.
11	XU Y， NIU G. Research on multi-view clustering algorithm based on sequential three-way decision［J］. Applied Soft Computing， 2024， 158： No.111590.
12	唐静昕.序贯多类三支决策模型研究［D］.合肥：安徽大学，2020：31-69.
	TANG J X. Research on sequential multi-class three-way decision model［D］. Hefei： Anhui University， 2020： 31-69.
13	袁国鑫，于洪.一种基于邮件头信息的三支决策邮件过滤方法［J］.计算机科学，2017，44（9）：74-77，114.
	YUAN G X， YU H. Method of three-way decision spam filtering based on head information of E-mail［J］. Computer Science， 2017， 44（9）： 74-77，114.
14	马新宇，黄春梅，姜春茂.基于三支决策的KNN渐进式文本分类方法［J］.计算机应用研究，2023，40（4）：1065-1069.
	MA X Y， HUANG C M， JIANG C M. KNN asymptotic text classification method based on three-way decision［J］. Application Research of Computers， 2023， 40（4）： 1065-1069.
15	蒋浩英，钱进，王滔滔，等.基于三支决策的新型分类匿名模型［J］.南京大学学报（自然科学），2023，59（6）：970-980.
	JIANG H Y， QIAN J， WANG T T， et al. A novel classified anonymity model based on the three-way decisions［J］. Journal of Nanjing University （Natural Science）， 2023， 59（6）： 970-980.
16	张师鹏，李永忠，杜祥通.基于半监督学习和三支决策的入侵检测模型［J］.计算机应用，2021，41（9）：2602-2608.
	ZHANG S P， LI Y Z， DU X T. Intrusion detection model based on semi-supervised learning and three-way decision［J］. Journal of Computer Applications， 2021， 41（9）： 2602-2608.
17	越凯强，李波，范盘龙.基于三支决策的飞机空战机动决策方法［J］.计算机应用，2022，42（2）：616-621.
	YUE K Q， LI B， FAN P L. Aircraft air combat maneuver decision method based on three-way decision［J］. Journal of Computer Applications， 2022， 42（2）： 616-621.
18	QIAN J， WANG T， JIANG H， et al. A TOPSIS method based on sequential three-way decision［J］. Applied Intelligence， 2023， 53（24）： 30661-30676.
19	HWANG C L， YOON K. Methods for multiple attribute decision making［M］// Multiple attribute decision making： methods and applications a state-of-the-art survey， LNE 186. Berlin： Springer， 1981： 58-191.
20	CHEN S J， HWANG C L. Fuzy multiple attribute decision making methods ［M］// Fuzzy multiple attribute decision making： methods and applications， LNE 375. Berlin： Springer， 1992： 289-486.
21	SALEHI V， ZAREI H， SHIRALI G A， et al. An entropy-based TOPSIS approach for analyzing and assessing crisis management systems in petrochemical industries［J］. Journal of Loss Prevention in the Process Industries， 2020， 67： No.104241.
22	NEOGI D. Performance appraisal of select nations in mitigation of COVID-19 pandemic using entropy based TOPSIS method［J］. Ciência and Saude Coletiva， 2021， 26（4）： 1419-1428.
23	丁文浩，胡明华，江斌.基于熵权法的平行跑道多模式进场效能分析［J］.华东交通大学学报，2023，40（1）：44-51.
	DING W H， HU M H， JIANG B. Multi-mode approach effectiveness analysis of parallel runways based on entropy weight method［J］. Journal of East China Jiaotong University， 2023， 40（1）： 44-51.
24	HE M， MA X， JIN Y. Station importance evaluation in dynamic bike-sharing rebalancing optimization using an entropy-based TOPSIS approach［J］. IEEE Access， 2021， 9： 38119-38131.
25	DEHGHAN RAHIMABADI P， BEHNIA M， NASABPOUR MOLAEI S， et al. Assessment of groundwater resources potential using Improved Water Quality Index （ImpWQI） and entropy-weighted TOPSIS model［J］. Sustainable Water Resources Management， 2024， 10： No.7.
26	KHAN H U， ABBAS M， ALRUWAILI O， et al. Selection of a smart and secure education school system based on the internet of things using entropy and TOPSIS approaches［J］. Computers in Human Behavior， 2024， 159： 108346.
27	DING S， LI R， GUO J. An entropy-based TOPSIS and optimized grey prediction model for spatiotemporal analysis in strategic emerging industry［J］. Expert Systems with Applications， 2023， 213（Pt C）： No.119169.
28	HUANG W， SHUAI B， XU Y， et al. Railway express freight train service sites planning： a two-stage entropy-TOPSIS approach［J］. Transportmetrica A： Transport Science， 2019， 15（2）： 807-823.
29	贺晓宇.基于TOPSIS的科技项目评审专家信用评价模型研究［J］.科技管理研究，2020，40（3）：32-38.
	HE X Y. Research on a TOPSIS credit evaluation model of peer reviewers in science and technology projects［J］. Science and Technology Management Research， 2020， 40（3）： 32-38.
30	张莹，王志浩.基于层次分析法的科技项目同行评议专家综合评价体系构建研究［J］.昆明理工大学学报（社会科学版）， 2021，21（5）：88-96.
	ZHANG Y， WANG Z H. Research on the establishment of comprehensive evaluation system for peer review experts of science and technology projects based on the analytic hierarchy process［J］. Journal of Kunming University of Science and Technology （Social Sciences）， 2021， 21（5）： 88-96.

[1]	Chunmao JIANG, Peng WU, Zhicong LI. Semi-supervised three-way clustering ensemble based on Seeds set and pairwise constraints [J]. Journal of Computer Applications, 2023, 43(5): 1481-1488.
[2]	Yanhao SUN, Wei XU, Tao ZHANG, Ningxin LIU. Software quality evaluation method considering decision maker’s psychological behaviors [J]. Journal of Computer Applications, 2022, 42(8): 2528-2533.
[3]	Kaiqiang YUE, Bo LI, Panlong FAN. Air combat maneuver decision method based on three-way decision [J]. Journal of Computer Applications, 2022, 42(2): 616-621.
[4]	Guopeng ZHANG, Xuebin CHEN, Haoshi WANG, Ran ZHAI, Zheng MA. K-Prototypes clustering method for local differential privacy [J]. Journal of Computer Applications, 2022, 42(12): 3813-3821.
[5]	ZHANG Shipeng, LI Yongzhong, DU Xiangtong. Intrusion detection model based on semi-supervised learning and three-way decision [J]. Journal of Computer Applications, 2021, 41(9): 2602-2608.
[6]	YANG Ling, JIANG Chunmao. Strategy of energy-aware virtual machine migration based on three-way decision [J]. Journal of Computer Applications, 2021, 41(4): 990-998.
[7]	REN Jie, MIN Fan, WANG Min. Cost-sensitive active learning through farthest distance sum sampling [J]. Journal of Computer Applications, 2019, 39(9): 2499-2504.
[8]	XU Jianfeng, ZOU Weikang, LIANG Wei, CHENG Gaojie, ZHANG Yuanjian. Three-way screening method of basic clustering for ensemble clustering [J]. Journal of Computer Applications, 2019, 39(11): 3120-3126.
[9]	WAN Zhichao, HU Feng, DENG Weibin. Feature selection method for imbalanced text sentiment classification based on three-way decisions [J]. Journal of Computer Applications, 2019, 39(11): 3127-3133.
[10]	LI Shuai, WANG Guoyin, YANG Jie. Three-way group decisions model based on cloud aggregation [J]. Journal of Computer Applications, 2019, 39(11): 3163-3171.
[11]	ZHAO Qingqing, HUANG Tianmin. Multi-objective decision making based on entropy weighted-Vague sets [J]. Journal of Computer Applications, 2018, 38(5): 1250-1253.
[12]	WANG Zewu, SUN Lei, GUO Songhui, SUN Ruichen. Scheduling method of virtual cipher machine based on entropy weight evaluation in cryptography cloud [J]. Journal of Computer Applications, 2018, 38(5): 1353-1359.
[13]	JIE Dan, HU Qiangqiang, XU Chengwu, GAO Baolu, LI Haifang. Edge quality evaluation based on fuzzy comprehensive evaluation [J]. Journal of Computer Applications, 2016, 36(9): 2580-2583.
[14]	YU Hong, MAO Chuankai. Automatic three-way decision clustering algorithm based on k-means [J]. Journal of Computer Applications, 2016, 36(8): 2061-2065.
[15]	XIONG Fang, ZHANG Xianyong. Regional attribute reduction and their structural heuristic algorithms for variable precision rough sets [J]. Journal of Computer Applications, 2016, 36(11): 2954-2957.

Expert counter-evaluation model with three-way decision and entropy weight TOPSIS

联合三支决策与熵权TOPSIS的专家反评估模型

RichHTML

PDF

Knowledge

Abstract

Cite this article

share this article

Figures/Tables 12

References 30

Related Articles 15

Recommended Articles

Metrics