Power work order classification in substation area based on MiniRBT-LSTM-GAT and label smoothing

doi:10.11772/j.issn.1001-9081.2024040533

Abstract

Abstract:

Record of power work orders in substation area serves as a reflection of substation operational conditions and user requirements， and is an important basis for establishing substation’s electricity safety management system and meeting the electricity demands of users. To address the issues of power work order classification in substation areas brought by high complexity and strong professionalism of the orders， a power work order classification in substation area， Mini RoBERTa-Long Short-Term Memory-Graph Attention neTwork （MiniRBT-LSTM-GAT） was proposed. Label Smoothing （LS） and a pre-trained language model were integrated by the proposed model. Firstly， a pre-trained model was utilized to calculate the character-level feature vector representation in the power work order text. Secondly， Bidirectional Long Short-Term Memory （BiLSTM） network was employed to capture the dependency within the power text sequence. Thirdly， Graph Attention neTwork （GAT） was applied to emphasize the feature information that contributes to text classification significantly. Finally， LS was used to modify the loss function， so as to improve the classification accuracy of the model. The proposed model was compared with mainstream text classification algorithms on Power Work Order dataset in Rural power Station area （RSPWO）， 95598 Power Work Order dataset in ZheJiang province （ZJPWO）， and THUCNews （TsingHua University Chinese News） dataset. Experimental results show that compared with Bidirectional Encoder Representations from Transformers （BERT） model for Electric Power Audit Text classification （EPAT-BERT）， the proposed model has an increase of 2.76 percentage points in precision and 2.02 percentage points in F1 value on RSPWO， and has an increase of 1.77 percentage points in precision and 1.40 percentage points in F1 value on ZJPWO. In comparison with capsule network based on BERT and dependency syntax （BRsyn-caps）， the proposed model has an increase of 0.76 percentage points in precision and 0.71 percentage points in accuracy on THUCNews dataset. The above confirms the effectiveness of the proposed model in enhancing the classification performance of power work orders in substation area， and the good performance of the proposed model on THUCNews dataset， verifying the generality of the model.

Key words: power work order in substation area, text classification, pre-trained model, Graph Attention neTwork (GAT), Label Smoothing (LS)

摘要：

台区电力工单记录反映了台区运行工况和用户需求，是制定台区用电安全管理制度和满足台区用户用电需求的重要依据。针对台区电力工单高复杂性和强专业性给台区工单分类带来的难题，提出一种融合标签平滑（LS）与预训练语言模型的台区电力工单分类模型（MiniRBT-LSTM-GAT）。首先，利用预训练模型计算电力工单文本中的字符级特征向量表示；其次，采用双向长短期记忆网络（BiLSTM）捕捉电力文本序列中的依赖关系；再次，通过图注意力网络（GAT）聚焦对文本分类贡献大的特征信息；最后，利用LS改进损失函数以提高模型的分类精度。所提模型与当前主流的文本分类算法在农网台区电力工单数据集（RSPWO）、浙江省95598电力工单数据集（ZJPWO）和THUCNews（TsingHua University Chinese News）数据集上的实验结果表明，与电力审计文本多粒度预训练语言模型（EPAT-BERT）相比，所提模型在RSPWO、ZJPWO上的查准率和F1值分别提升了2.76、2.02个百分点和1.77、1.40个百分点；与胶囊神经网络模型BRsyn-caps（capsule network based on BERT and dependency syntax）相比，所提模型在THUCNews数据集上的查准率和准确率分别提升了0.76和0.71个百分点。可见，所提模型有效提升了台区电力工单分类的性能，并在THUCNews数据集上表现良好，验证了模型的通用性。

关键词: 台区电力工单, 文本分类, 预训练模型, 图注意力网络, 标签平滑

CLC Number:

TP391.1

Jiaxin LI, Site MO. Power work order classification in substation area based on MiniRBT-LSTM-GAT and label smoothing[J]. Journal of Computer Applications, 2025, 45(4): 1356-1362.

李嘉欣, 莫思特. 基于MiniRBT-LSTM-GAT与标签平滑的台区电力工单分类[J]. 《计算机应用》唯一官方网站, 2025, 45(4): 1356-1362.

Figures/Tables 7

References 27

1	ZONG Y， WANG D， WANG X， et al. Characteristic portrait and analysis method of power grid work orders based on statistics［C］// Proceedings of the 2nd International Conference on Control and Intelligent Robotics. New York： ACM， 2022： 717-723.
2	YU Y， HE W， KANG Y M， et al. Leveraging continuous prompt for few-shot named entity recognition in electric power domain with meta-learning［J］. Data Intelligence， 2023， 5（2）： 494-509.
3	MOSCATO V， NAPOLANO G， POSTIGLIONE M， et al. Multi-task learning for few-shot biomedical relation extraction［J］. Artificial Intelligence Review， 2023， 56（11）： 13743-13763.
4	BANG J， PARK J， PARK J. GACaps-HTC： graph attention capsule network for hierarchical text classification［J］. Applied Intelligence， 2023， 53（17）： 20577-20594.
5	YIN Y， ZENG J， SU J， et al. Multi-modal graph contrastive encoding for neural machine translation［J］. Artificial Intelligence， 2023， 323： No.103986.
6	GEMAN S， BIENENSTOCK E， DOURSAT R. Neural networks and the bias/variance dilemma［J］. Neural Computation， 1992， 4（1）： 1-58.
7	WEI J， ZOU K. EDA： easy data augmentation techniques for boosting performance on text classification tasks［C］// Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing. Stroudsburg： ACL， 2019： 6382-6388.
8	YAO X， YANG Z， CUI Y， et al. MiniRBT： a two-stage distilled small Chinese pre-trained model［EB/OL］. ［2024-08-11］..
9	刘鹏. 基于深度学习的电力工单文本挖掘［D］. 南昌：南昌大学， 2022：6-23.
	LIU P. Text mining of electricity work order based on deep learning［D］. Nanchang： Nanchang University， 2022：6-23.
10	孙茂松，李景阳，郭志芃，等. THUCTC：一个高效的中文文本分类工具包［EB/OL］. ［2024-08-11］.，
	LI J Y， GUO Z P， et al. THUCTC： an efficient Chinese text classifier［EB/OL］. ［2024-08-11］.
11	曹湘，李誉坤，钱叶，等. 基于混合神经网络的电力短文本分类方法研究［J］. 计算机与数字工程， 2019， 47（5）：1145-1150.
	CAO X， LI Y K， QIAN Y， et al. Short text classification of electric power based on hybrid neural network［J］. Computer and Digital Engineering， 2019， 47（5）：1145-1150.
12	冯斌，张又文，唐昕，等. 基于BiLSTM-Attention神经网络的电力设备缺陷文本挖掘［J］. 中国电机工程学报， 2020， 40（S1）：1-10.
	FENG B， ZHANG Y W， TANG X， et al. Power equipment defect record text mining based on BiLSTM-Attention neural network［J］. Proceedings of the CSEE， 2020， 40（S1）：1-10.
13	MENG J， LI Y， LIU C， et al. Classification of customer service tickets in power system based on character and word level semantic understanding［C］// Proceedings of the 2021 China International Conference on Electricity Distribution. Piscataway： IEEE， 2021： 1062-1066.
14	MENG Q， SONG Y， MU J， et al. Electric power audit text classification with multi-grained pre-trained language model［J］. IEEE Access， 2023， 11： 13510-13518.
15	田园，原野，刘海斌，等. 基于BERT预训练语言模型的电网设备缺陷文本分类［J］. 南京理工大学学报， 2020， 44（4）：446-453.
	TIAN Y， YUAN Y， LIU H B， et al. BERT pre-trained language model for defective text classification of power grid equipment［J］. Journal of Nanjing University of Science and Technology， 2020， 44（4）：446-453.
16	付文杰，杨迪，马红明，等. 融合BTM和BERT的短文本分类方法［J］. 计算机工程与设计， 2022， 43（12）：3421-3427.
	FU W J， YANG D， MA H M， et al. Short text classification method based on BTM and BERT［J］. Computer Engineering and Design， 2022， 43（12）：3421-3427.
17	ZHANG S， TONG H， XU J， et al. Graph convolutional networks： a comprehensive review［J］. Computational Social Networks， 2019， 6： No.11.
18	VELIČKOVIĆ P， CUCURULL G， CASANOVA A， et al. Graph attention networks［EB/OL］. ［2024-08-11］..
19	CUI Y， CHE W， LIU T， et al. Pre-training with whole word masking for Chinese BERT［J］. IEEE/ACM Transactions on Audio， Speech， and Language Processing， 2021， 29： 3504-3514.
20	MÜLLER R， KORNBLITH S， HINTON G E. When does label smoothing help？［C］// Proceedings of the 31st International Conference on Neural Information Processing Systems. Red Hook： Curran Associates Inc.， 2019： 4694-4703.
21	周志华. 机器学习［M］. 北京：清华大学出版社， 2016：23-29.
	ZHOU Z H. Machine learning［M］. Beijing： Tsinghua University Press， 2016：23-29.
22	SRIVASTAVA N， HINTON G， KRIZHEVSKY A， et al. Dropout： a simple way to prevent neural networks from overfitting［J］. Journal of Machine Learning Research， 2014， 15： 1929-1958.
23	KIM Y. Convolutional neural networks for sentence classification［C］// Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing. Stroudsburg： ACL， 2014： 1746-1751.
24	LIU P， QIU X， HUANG X. Recurrent neural network for text classification with multi-task learning［C］// Proceedings of the 25th International Joint Conference on Artificial Intelligence. California： ijcai.org， 2016： 2873-2879.
25	DEVLIN J， CHANG M W， LEE K， et al. BERT： pre-training of deep bidirectional transformers for language understanding［C］// Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics： Human Language Technologies， Volume 1 （Long and Short Papers）. Stroudsburg： ACL， 2019： 4171-4186.
26	WAN C X， LI B. Financial causal sentence recognition based on BERT-CNN text classification［J］. The Journal of Supercomputing， 2022， 78（5）： 6503-6527.
27	LUO J， HE C， LUO H. BRsyn-Caps： Chinese text classification using capsule network based on BERT and dependency syntax［J］. IEICE Transactions on Information and Systems， 2024， E107-D（2）： 212-219.

数据集	类别数	合计	样本数
数据集	类别数	合计	训练集	验证集	测试集
RSPWO	15	9 779	6 260	1 563	1 956
ZJPWO	7	92 116	63 003	14 556	14 557
THUCNews	10	65 000	50 000	5 000	10 000

数据集	类别数	合计	样本数
数据集	类别数	合计	训练集	验证集	测试集
RSPWO	15	9 779	6 260	1 563	1 956
ZJPWO	7	92 116	63 003	14 556	14 557
THUCNews	10	65 000	50 000	5 000	10 000

模型	数据集	P	R	F1	Accuracy
TextCNN	RSPWO	93.75	90.03	91.59	92.93
	ZJPWO	89.94	93.48	90.94	96.41
	THUCNews	92.14	92.04	91.93	92.04
TextRNN	RSPWO	92.01	91.00	91.47	93.43
	ZJPWO	91.78	89.55	90.57	96.64
	THUCNews	89.72	88.44	87.91	88.44
GAT	RSPWO	95.05	93.35	94.03	94.68
	ZJPWO	89.36	89.22	88.61	95.31
	THUCNews	90.41	90.04	90.12	90.04
BERT	RSPWO	95.52	92.92	94.04	94.75
	ZJPWO	91.30	94.06	92.11	96.96
	THUCNews	93.81	93.76	93.69	93.76
BERT‑CNN	RSPWO	95.34	95.52	95.40	95.51
	ZJPWO	92.24	94.78	94.22	97.32
	THUCNews	93.88	93.73	93.65	93.73
MiniRBT	RSPWO	92.60	88.45	89.79	92.38
	ZJPWO	91.32	94.52	92.39	97.25
	THUCNews	95.28	95.25	95.22	95.25
MiniRBT‑CNN	RSPWO	89.15	90.67	89.66	91.54
	ZJPWO	92.55	92.59	91.91	96.98
	THUCNews	94.97	94.98	94.92	94.98
BRsyn-caps	RSPWO	96.31	96.79	96.52	96.37
	ZJPWO	90.91	96.67	93.35	97.34
	THUCNews	94.76	94.76	94.76	94.76
本文模型	RSPWO	97.39	97.24	97.31	96.98
	ZJPWO	92.57	97.16	94.61	97.36
	THUCNews	95.52	95.47	95.43	95.47

模型	数据集	P	R	F1	Accuracy
TextCNN	RSPWO	93.75	90.03	91.59	92.93
	ZJPWO	89.94	93.48	90.94	96.41
	THUCNews	92.14	92.04	91.93	92.04
TextRNN	RSPWO	92.01	91.00	91.47	93.43
	ZJPWO	91.78	89.55	90.57	96.64
	THUCNews	89.72	88.44	87.91	88.44
GAT	RSPWO	95.05	93.35	94.03	94.68
	ZJPWO	89.36	89.22	88.61	95.31
	THUCNews	90.41	90.04	90.12	90.04
BERT	RSPWO	95.52	92.92	94.04	94.75
	ZJPWO	91.30	94.06	92.11	96.96
	THUCNews	93.81	93.76	93.69	93.76
BERT‑CNN	RSPWO	95.34	95.52	95.40	95.51
	ZJPWO	92.24	94.78	94.22	97.32
	THUCNews	93.88	93.73	93.65	93.73
MiniRBT	RSPWO	92.60	88.45	89.79	92.38
	ZJPWO	91.32	94.52	92.39	97.25
	THUCNews	95.28	95.25	95.22	95.25
MiniRBT‑CNN	RSPWO	89.15	90.67	89.66	91.54
	ZJPWO	92.55	92.59	91.91	96.98
	THUCNews	94.97	94.98	94.92	94.98
BRsyn-caps	RSPWO	96.31	96.79	96.52	96.37
	ZJPWO	90.91	96.67	93.35	97.34
	THUCNews	94.76	94.76	94.76	94.76
本文模型	RSPWO	97.39	97.24	97.31	96.98
	ZJPWO	92.57	97.16	94.61	97.36
	THUCNews	95.52	95.47	95.43	95.47

模型	数据集	P	R	F1	Accuracy
文献［11］模型	RSPWO	94.05	91.11	92.45	93.47
文献［11］模型	ZJPWO	92.28	93.20	92.71	96.63
文献［12］模型	RSPWO	93.30	90.67	91.89	93.65
文献［12］模型	ZJPWO	91.97	92.42	92.07	97.03
文献［13］模型	RSPWO	92.88	92.21	92.47	93.54
文献［13］模型	ZJPWO	91.78	89.55	90.57	96.64
文献［15］模型	RSPWO	95.94	96.75	96.28	96.24
文献［15］模型	ZJPWO	92.22	94.25	93.05	97.04
EPAT-BERT^［14］	RSPWO	94.63	96.54	95.29	95.50
EPAT-BERT^［14］	ZJPWO	90.80	96.51	93.21	96.91
本文模型	RSPWO	97.39	97.24	97.31	96.98
本文模型	ZJPWO	92.57	97.16	94.61	97.36