Joint approach of intent detection and slot filling based on multi-task learning

doi:10.11772/j.issn.1001-9081.2023040443

Abstract

Abstract:

With the application of pre-trained language models in Natural Language Processing （NLP） tasks， joint modeling of Intent Detection （ID） and Slot Filling （SF） has improved the performance of Spoken Language Understanding （SLU）. Existing methods mostly focus on the interaction between intents and slots， neglecting the influence of modeling differential text sequences on SLU tasks. A joint method for Intent Detection and Slot Filling based on Multi-task Learning （IDSFML） was proposed. Firstly， differential texts were constructed using random mask strategy， and a neural network structure combining AutoEncoder and Attention mechanism （AEA） was designed to incorporate the features of differential text sequences into the SLU task. Secondly， a similarity distribution task was designed to make the representations of differential texts and original texts similar. Finally， three tasks of ID， SF and differential text sequence similarity distribution were jointly trained. Experimental results on Airline Travel Information Systems （ATIS） and SNIPS datasets show that， compared with the suboptimal baseline method SASGBC （Self-Attention and Slot-Gated on top of BERT with CRF）， IDSFML improves the F1 scores of slot filling by 1.9 and 1.6 percentage points respectively， and improves the accuracy of intent detection by 0.2 and 0.4 percentage points respectively， enhancing the accuracy of spoken language understanding tasks.

Key words: Intent Detection (ID), Slot Filling (SF), multi-task learning, Spoken Language Understanding (SLU), attention mechanism

摘要：

随着预训练语言模型在自然语言处理（NLP）任务上的应用，意图检测（ID）和槽位填充（SF）联合建模提高了口语理解的性能。现有方法大多关注意图和槽位的相互作用，忽略了差异文本序列建模对口语理解（SLU）任务的影响。因此，提出一种基于多任务学习的意图检测和槽位填充联合方法（IDSFML）。首先，使用随机掩盖mask策略构造差异文本，设计结合自编码器和注意力机制的神经网络（AEA）结构，为口语理解任务融入差异文本序列的特征；其次，设计相似性分布任务，使差异文本和原始文本的表征相似；最后，联合训练ID、SF和差异文本序列相似性分布三个任务。在航班旅行信息系统（ATIS）和SNIPS数据集上的实验结果表明，IDSFML与表现次优的基线方法SASGBC（Self-Attention and Slot-Gated on top of BERT with CRF）相比，槽位填充F1值分别提升了1.9和1.6个百分点，意图检测准确率分别提升了0.2和0.4个百分点，提高了口语理解任务的准确率。

关键词: 意图检测, 槽位填充, 多任务学习, 口语理解, 注意力机制

CLC Number:

TP391.1

Aiguo SHANG, Xinjuan ZHU. Joint approach of intent detection and slot filling based on multi-task learning[J]. Journal of Computer Applications, 2024, 44(3): 690-695.

尚爱国, 朱欣娟. 基于多任务学习的意图检测和槽位填充联合方法[J]. 《计算机应用》唯一官方网站, 2024, 44(3): 690-695.

Figures/Tables 7

References 31

1	C-W GOO， GAO G， HSU Y-K， et al. Slot-gated modeling for joint slot filling and intent prediction ［C］// Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics： Human Language Technologies （Volume 2： Short Papers）. Stroudsburg： ACL， 2018： 753-757. 10.18653/v1/n18-2118
2	BALDOMERO-NARANJO M， MARTINEZ-MERINO L I， RODRÍGUEZ-CHIA A M. A robust SVM-based approach with feature selection and outliers detection for classification problems［J］. Expert Systems with Applications， 2021， 178： 115017. 10.1016/j.eswa.2021.115017
3	SPEISER J L， MILLER M E， TOOZE J， et al. A comparison of random forest variable selection methods for classification prediction modeling ［J］. Expert Systems with Applications， 2019， 134： 93-101. 10.1016/j.eswa.2019.05.028
4	FERNÁNDEZ-MARTÍNEZ F， GRIOL D， CALLEJAS Z， et al. An approach to intent detection and classification based on attentive recurrent neural networks ［C］// Proceedings of the 2021 Interspeech. Grenoble： International Speech Communication Association， 2021： 46-50. 10.21437/iberspeech.2021-10
5	KUYUMCU B， AKSAKALLI C， DELIL S. An automated new approach in fast text classification （FastText）： a case study for Turkish text classification without pre-processing ［C］// Proceedings of the 2019 3rd International Conference on Natural Language Processing and Information Retrieval. New York： ACM， 2019： 1-4. 10.1145/3342827.3342828
6	DENG Q， SÖFFKER D. A review of the current HMM-based approaches of driving behaviors recognition and prediction ［J］. IEEE Transactions on Intelligent Vehicles， 2022，1（7）：21-31.
7	LIU S， HE T， DAI J. A survey of CRF algorithm based knowledge extraction of elementary mathematics in Chinese ［J］. Mobile Networks and Applications， 2021， 26： 1891-1903. 10.1007/s11036-020-01725-x
8	LINDEMANN B， MASCHLER B， SAHLAB N， et al. A survey on anomaly detection for technical systems using LSTM networks ［J］. Computers in Industry， 2021， 131： 103498. 10.1016/j.compind.2021.103498
9	DEVLIN J， CHANG M-W， LEE K， et al. BERT： pre-training of deep bidirectional transformers for language understanding ［EB/OL］. ［2023-04-23］.. 10.18653/v1/n18-2
10	CHEN Y-N， HAKANNI-TÜR D， TUR G， et al. Syntax or semantics？ Knowledge-guided joint semantic frame parsing ［C］// Proceedings of the 2016 IEEE Spoken Language Technology Workshop. Piscataway： IEEE， 2016： 348-355. 10.1109/slt.2016.7846288
11	WANG W， SUN D. The improved AdaBoost algorithms for imbalanced data classification ［J］. Information Sciences， 2021， 563： 358-374. 10.1016/j.ins.2021.03.042
12	刘睿珩，叶霞，岳增营.面向自然语言处理任务的预训练模型综述［J］.计算机应用，2021，41（5）：1236-1246.
	LIU R H， YE X， YUE Z Y. Review of pre-trained models for natural language processing tasks ［J］. Journal of Computer Applications， 2021，41（5）：1236-1246.
13	DEORAS A， SARIKAYA R. Deep belief network based semantic taggers for spoken language understanding ［C］// Proceedings of the Interspeech 2013. Lyon： ISCA， 2013： 2713-2717. 10.21437/interspeech.2013-623
14	YAO K， PENG B， ZWEIG G， et al. Recurrent conditional random field for language understanding ［C］// Proceedings of the 2014 IEEE International Conference on Acoustics， Speech and Signal Processing. Piscataway： IEEE， 2014： 4077-4081. 10.1109/icassp.2014.6854368
15	MESNIL G， HE X， DENG L， et al. Investigation of recurrent-neural-network architectures and learning methods for spoken language understanding ［C］// Proceedings of the 2013 Interspeech. Grenoble： International Speech Communication Association， 2013： 3771-3775. 10.21437/interspeech.2013-596
16	CHEN P-C， CHI T-C， SU S-Y， et al. Dynamic time-aware attention to speaker roles and contexts for spoken language understanding ［C］// Proceedings of the 2017 IEEE Automatic Speech Recognition and Understanding Workshop. Piscataway： IEEE， 2017： 554-560. 10.1109/asru.2017.8268985
17	ZUGARINI A， MORVAN J， MELACCI S， et al. Combining deep learning and symbolic processing for extracting knowledge from raw text ［C］// Proceedings of the 2018 International Conference on Artificial Neural Networks in Pattern Recognition： 8th IAPR TC3 Workshop. Cham： Springer， 2018： 90-101. 10.1007/978-3-319-99978-4_7
18	GUO D， TUR G， W-T YIH， et al. Joint semantic utterance classification and slot filling with recursive neural networks ［C］// Proceedings of the 2014 IEEE Spoken Language Technology Workshop. Piscataway： IEEE， 2014： 554-559. 10.1109/slt.2014.7078634
19	XU P， SARIKAYA R. Convolutional neural network based triangular CRF for joint intent detection and slot filling ［C］// Proceedings of the 2013 IEEE Workshop on Automatic Speech Recognition and Understanding. Piscataway： IEEE， 2013： 78-83. 10.1109/asru.2013.6707709
20	LIU B， LANE I. Attention-based recurrent neural network models for joint intent detection and slot filling ［C］// Proceedings of the 2016 Interspeech. Grenoble： International Speech Communication Association， 2016： 685-689. 10.21437/interspeech.2016-1352
21	LI C， LI L， QI J. A self-attentive model with gate mechanism for spoken language understanding ［C］// Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. Stroudsburg： ACL， 2018： 3824-3833. 10.18653/v1/d18-1417
22	ZHANG C， LI Y， DU N， et al. Joint slot filling and intent detection via capsule neural networks ［C］// Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. Stroudsburg： ACL， 2019： 5259-5267. 10.18653/v1/p19-1519
23	CHEN Q， ZHUO Z， WANG W. BERT for joint intent classification and slot filling ［EB/OL］. （2019-02-28）［2023-04-23］. .
24	E H， NIU P， CHEN Z， et al. A novel bi-directional interrelated model for joint intent detection and slot filling ［C］// Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. Stroudsburg： ACL， 2019： 5467-5471. 10.18653/v1/p19-1544
25	QIN L， CHE W， LI Y， et al. A stack-propagation framework with token-level intent detection for spoken language understanding［C］// Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing. Stroudsburg： ACL， 2019： 2078-2087. 10.18653/v1/d19-1214
26	HUI Y， WANG J， CHENG N， et al. Joint intent detection and slot filling based on continual learning model ［C］// Proceedings of the 2021 International Conference on Acoustics， Speech and Signal Processing. Piscataway： IEEE， 2021： 7643-7647. 10.1109/icassp39728.2021.9413360
27	GUO Y， XIE Z， CHEN X， et al. AWTE-BERT： attending to wordpiece tokenization explicitly on BERT for joint intent classification and slot filling ［EB/OL］. （2023-02-02）［2023-04-13］. .
28	COUCKE A， SAADE A， BALL A， et al. Snips voice platform： an embedded spoken language understanding system for private-by-design voice interfaces ［EB/OL］. （2018-11-06）［2023-04-13］. . 10.1109/emc2-nips53020.2019.00021
29	MA Z， SUN B， LI S. A two-stage selective fusion framework for joint intent detection and slot filling ［J］. IEEE Transactions on Neural Networks and Learning Systems， 2022 （Early Access）：1-12. 10.1109/tnnls.2022.3202562
30	WANG C， HUANG Z， HU M. SASGBC： improving sequence labeling performance for joint learning of slot filling and intent detection ［C］// Proceedings of the 2020 6th International Conference on Computing and Data Engineering. New York： ACM， 2020： 29-33. 10.1145/3379247.3379266
31	OpenAI. GPT-4 Technical Report ［EB/OL］. （2023-03-27）［2023-04-13］. .

方法	ATIS			SNIPS
方法	Slot F1	Intent acc	Sentence acc	Slot F1	Intent acc	Sentence acc
SF-ID Network	95.8	97.7	86.9	92.2	97.4	80.5
CAPSULE-NLU	95.2	95.0	83.4	91.8	97.3	80.9
Attention BiRNN	94.2	91.1	78.9	87.8	96.7	74.1
Slot-Gated	94.8	93.6	82.2	88.8	97.0	75.5
Joint BERT	96.1	97.5	88.2	96.4	98.8	92.5
SASGBC	96.6	98.2	91.6	96.4	98.9	92.5
IDSFML	98.5	98.4	92.4	98.0	99.3	92.4

方法	ATIS			SNIPS
方法	Slot F1	Intent acc	Sentence acc	Slot F1	Intent acc	Sentence acc
SF-ID Network	95.8	97.7	86.9	92.2	97.4	80.5
CAPSULE-NLU	95.2	95.0	83.4	91.8	97.3	80.9
Attention BiRNN	94.2	91.1	78.9	87.8	96.7	74.1
Slot-Gated	94.8	93.6	82.2	88.8	97.0	75.5
Joint BERT	96.1	97.5	88.2	96.4	98.8	92.5
SASGBC	96.6	98.2	91.6	96.4	98.9	92.5
IDSFML	98.5	98.4	92.4	98.0	99.3	92.4

方法	ATIS			SNIPS
方法	Slot F1	Intent acc	Sentence acc	Slot F1	Intent acc	Sentence acc
IDSFML-CRF	97.7	97.9	92.1	97.2	99.1	92.0
IDSFML-AEA	96.9	98.1	91.4	96.7	98.9	91.6
IDSFML-SD	97.9	98.0	91.9	97.8	99.0	91.7
IDSFML	98.5	98.4	92.4	98.0	99.3	92.4

方法	ATIS			SNIPS
方法	Slot F1	Intent acc	Sentence acc	Slot F1	Intent acc	Sentence acc
IDSFML-CRF	97.7	97.9	92.1	97.2	99.1	92.0
IDSFML-AEA	96.9	98.1	91.4	96.7	98.9	91.6
IDSFML-SD	97.9	98.0	91.9	97.8	99.0	91.7
IDSFML	98.5	98.4	92.4	98.0	99.3	92.4

[1]	Jie GUO, Jiayu LIN, Zuhong LIANG, Xiaobo LUO, Haitao SUN. Recommendation method based on knowledge‑awareness and cross-level contrastive learning [J]. Journal of Computer Applications, 2024, 44(4): 1121-1127.
[2]	Pengfei ZHANG, Litao HAN, Hengjian FENG, Hongmei LI. Point cloud semantic segmentation based on attention mechanism and global feature optimization [J]. Journal of Computer Applications, 2024, 44(4): 1086-1092.
[3]	Tianhua CHEN, Jiaxuan ZHU, Jie YIN. Bird recognition algorithm based on attention mechanism [J]. Journal of Computer Applications, 2024, 44(4): 1114-1120.
[4]	Lijun XU, Hui LI, Zuyang LIU, Kansong CHEN, Weixuan MA. 3D-GA-Unet： MRI image segmentation algorithm for glioma based on 3D-Ghost CNN [J]. Journal of Computer Applications, 2024, 44(4): 1294-1302.
[5]	Quan YUAN, Changping CHEN, Ze CHEN, Linfeng ZHAN. Twice attention mechanism distantly supervised relation extraction based on BERT [J]. Journal of Computer Applications, 2024, 44(4): 1080-1085.
[6]	Xinyuan YOU, Heng WANG. Monaural speech enhancement based on gated dilated convolutional recurrent network [J]. Journal of Computer Applications, 2024, 44(4): 1317-1324.
[7]	Rong HUANG, Junjie SONG, Shubo ZHOU, Hao LIU. Image aesthetic quality evaluation method based on self-supervised vision Transformer [J]. Journal of Computer Applications, 2024, 44(4): 1269-1276.
[8]	Bin XIAO, Yun GAN, Min WANG, Xingpeng ZHANG, Zhaoxing WANG. Network abnormal traffic detection based on port attention and convolutional block attention module [J]. Journal of Computer Applications, 2024, 44(4): 1027-1034.
[9]	Haihan WANG, Yan ZHU. Offensive speech detection with irony mechanism [J]. Journal of Computer Applications, 2024, 44(4): 1065-1071.
[10]	Wei LI, Ling CHEN, Xiuyuan XU, Min ZHU, Jixiang GUO, Kai ZHOU, Hao NIU, Yuchen ZHANG, Shanye YI, Yi ZHANG, Fengming LUO. Interstitial lung disease segmentation algorithm based on multi-task learning [J]. Journal of Computer Applications, 2024, 44(4): 1285-1293.
[11]	Zijie HUANG, Yang OU, Degang JIANG, Cailing GUO, Bailin LI. Lightweight deep learning algorithm for weld seam surface quality detection of traction seat [J]. Journal of Computer Applications, 2024, 44(3): 983-988.
[12]	Yuliang ZHENG, Yunhua CHEN, Weijie BAI, Pinghua CHEN. Vehicle target detection by fusing event data and image frames [J]. Journal of Computer Applications, 2024, 44(3): 931-937.
[13]	Kui ZHAO, Huiqi QIU, Xu LI, Zhifei XU. Real-time pulmonary nodule detection algorithm combining attention and multipath fusion [J]. Journal of Computer Applications, 2024, 44(3): 945-952.
[14]	Rui JIANG, Wei LIU, Cheng CHEN, Tao LU. Asymmetric unsupervised end-to-end image deraining network [J]. Journal of Computer Applications, 2024, 44(3): 922-930.
[15]	Tao SUN, Zhangtian DUAN, Haonan ZHU, Peihao GUO, Heli SUN. Social event recommendation method based on unexpectedness metric [J]. Journal of Computer Applications, 2024, 44(3): 760-766.