Joint approach of intent detection and slot filling based on multi-task learning

doi:10.11772/j.issn.1001-9081.2023040443

Abstract

Abstract:

With the application of pre-trained language models in Natural Language Processing （NLP） tasks， joint modeling of Intent Detection （ID） and Slot Filling （SF） has improved the performance of Spoken Language Understanding （SLU）. Existing methods mostly focus on the interaction between intents and slots， neglecting the influence of modeling differential text sequences on SLU tasks. A joint method for Intent Detection and Slot Filling based on Multi-task Learning （IDSFML） was proposed. Firstly， differential texts were constructed using random mask strategy， and a neural network structure combining AutoEncoder and Attention mechanism （AEA） was designed to incorporate the features of differential text sequences into the SLU task. Secondly， a similarity distribution task was designed to make the representations of differential texts and original texts similar. Finally， three tasks of ID， SF and differential text sequence similarity distribution were jointly trained. Experimental results on Airline Travel Information Systems （ATIS） and SNIPS datasets show that， compared with the suboptimal baseline method SASGBC （Self-Attention and Slot-Gated on top of BERT with CRF）， IDSFML improves the F1 scores of slot filling by 1.9 and 1.6 percentage points respectively， and improves the accuracy of intent detection by 0.2 and 0.4 percentage points respectively， enhancing the accuracy of spoken language understanding tasks.

Key words: Intent Detection (ID), Slot Filling (SF), multi-task learning, Spoken Language Understanding (SLU), attention mechanism

摘要：

随着预训练语言模型在自然语言处理（NLP）任务上的应用，意图检测（ID）和槽位填充（SF）联合建模提高了口语理解的性能。现有方法大多关注意图和槽位的相互作用，忽略了差异文本序列建模对口语理解（SLU）任务的影响。因此，提出一种基于多任务学习的意图检测和槽位填充联合方法（IDSFML）。首先，使用随机掩盖mask策略构造差异文本，设计结合自编码器和注意力机制的神经网络（AEA）结构，为口语理解任务融入差异文本序列的特征；其次，设计相似性分布任务，使差异文本和原始文本的表征相似；最后，联合训练ID、SF和差异文本序列相似性分布三个任务。在航班旅行信息系统（ATIS）和SNIPS数据集上的实验结果表明，IDSFML与表现次优的基线方法SASGBC（Self-Attention and Slot-Gated on top of BERT with CRF）相比，槽位填充F1值分别提升了1.9和1.6个百分点，意图检测准确率分别提升了0.2和0.4个百分点，提高了口语理解任务的准确率。

关键词: 意图检测, 槽位填充, 多任务学习, 口语理解, 注意力机制

CLC Number:

TP391.1

Aiguo SHANG, Xinjuan ZHU. Joint approach of intent detection and slot filling based on multi-task learning[J]. Journal of Computer Applications, 2024, 44(3): 690-695.

尚爱国, 朱欣娟. 基于多任务学习的意图检测和槽位填充联合方法[J]. 《计算机应用》唯一官方网站, 2024, 44(3): 690-695.

Figures/Tables 7

References 31

1	C-W GOO， GAO G， HSU Y-K， et al. Slot-gated modeling for joint slot filling and intent prediction ［C］// Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics： Human Language Technologies （Volume 2： Short Papers）. Stroudsburg： ACL， 2018： 753-757. 10.18653/v1/n18-2118
2	BALDOMERO-NARANJO M， MARTINEZ-MERINO L I， RODRÍGUEZ-CHIA A M. A robust SVM-based approach with feature selection and outliers detection for classification problems［J］. Expert Systems with Applications， 2021， 178： 115017. 10.1016/j.eswa.2021.115017
3	SPEISER J L， MILLER M E， TOOZE J， et al. A comparison of random forest variable selection methods for classification prediction modeling ［J］. Expert Systems with Applications， 2019， 134： 93-101. 10.1016/j.eswa.2019.05.028
4	FERNÁNDEZ-MARTÍNEZ F， GRIOL D， CALLEJAS Z， et al. An approach to intent detection and classification based on attentive recurrent neural networks ［C］// Proceedings of the 2021 Interspeech. Grenoble： International Speech Communication Association， 2021： 46-50. 10.21437/iberspeech.2021-10
5	KUYUMCU B， AKSAKALLI C， DELIL S. An automated new approach in fast text classification （FastText）： a case study for Turkish text classification without pre-processing ［C］// Proceedings of the 2019 3rd International Conference on Natural Language Processing and Information Retrieval. New York： ACM， 2019： 1-4. 10.1145/3342827.3342828
6	DENG Q， SÖFFKER D. A review of the current HMM-based approaches of driving behaviors recognition and prediction ［J］. IEEE Transactions on Intelligent Vehicles， 2022，1（7）：21-31.
7	LIU S， HE T， DAI J. A survey of CRF algorithm based knowledge extraction of elementary mathematics in Chinese ［J］. Mobile Networks and Applications， 2021， 26： 1891-1903. 10.1007/s11036-020-01725-x
8	LINDEMANN B， MASCHLER B， SAHLAB N， et al. A survey on anomaly detection for technical systems using LSTM networks ［J］. Computers in Industry， 2021， 131： 103498. 10.1016/j.compind.2021.103498
9	DEVLIN J， CHANG M-W， LEE K， et al. BERT： pre-training of deep bidirectional transformers for language understanding ［EB/OL］. ［2023-04-23］.. 10.18653/v1/n18-2
10	CHEN Y-N， HAKANNI-TÜR D， TUR G， et al. Syntax or semantics？ Knowledge-guided joint semantic frame parsing ［C］// Proceedings of the 2016 IEEE Spoken Language Technology Workshop. Piscataway： IEEE， 2016： 348-355. 10.1109/slt.2016.7846288
11	WANG W， SUN D. The improved AdaBoost algorithms for imbalanced data classification ［J］. Information Sciences， 2021， 563： 358-374. 10.1016/j.ins.2021.03.042
12	刘睿珩，叶霞，岳增营.面向自然语言处理任务的预训练模型综述［J］.计算机应用，2021，41（5）：1236-1246.
	LIU R H， YE X， YUE Z Y. Review of pre-trained models for natural language processing tasks ［J］. Journal of Computer Applications， 2021，41（5）：1236-1246.
13	DEORAS A， SARIKAYA R. Deep belief network based semantic taggers for spoken language understanding ［C］// Proceedings of the Interspeech 2013. Lyon： ISCA， 2013： 2713-2717. 10.21437/interspeech.2013-623
14	YAO K， PENG B， ZWEIG G， et al. Recurrent conditional random field for language understanding ［C］// Proceedings of the 2014 IEEE International Conference on Acoustics， Speech and Signal Processing. Piscataway： IEEE， 2014： 4077-4081. 10.1109/icassp.2014.6854368
15	MESNIL G， HE X， DENG L， et al. Investigation of recurrent-neural-network architectures and learning methods for spoken language understanding ［C］// Proceedings of the 2013 Interspeech. Grenoble： International Speech Communication Association， 2013： 3771-3775. 10.21437/interspeech.2013-596
16	CHEN P-C， CHI T-C， SU S-Y， et al. Dynamic time-aware attention to speaker roles and contexts for spoken language understanding ［C］// Proceedings of the 2017 IEEE Automatic Speech Recognition and Understanding Workshop. Piscataway： IEEE， 2017： 554-560. 10.1109/asru.2017.8268985
17	ZUGARINI A， MORVAN J， MELACCI S， et al. Combining deep learning and symbolic processing for extracting knowledge from raw text ［C］// Proceedings of the 2018 International Conference on Artificial Neural Networks in Pattern Recognition： 8th IAPR TC3 Workshop. Cham： Springer， 2018： 90-101. 10.1007/978-3-319-99978-4_7
18	GUO D， TUR G， W-T YIH， et al. Joint semantic utterance classification and slot filling with recursive neural networks ［C］// Proceedings of the 2014 IEEE Spoken Language Technology Workshop. Piscataway： IEEE， 2014： 554-559. 10.1109/slt.2014.7078634
19	XU P， SARIKAYA R. Convolutional neural network based triangular CRF for joint intent detection and slot filling ［C］// Proceedings of the 2013 IEEE Workshop on Automatic Speech Recognition and Understanding. Piscataway： IEEE， 2013： 78-83. 10.1109/asru.2013.6707709
20	LIU B， LANE I. Attention-based recurrent neural network models for joint intent detection and slot filling ［C］// Proceedings of the 2016 Interspeech. Grenoble： International Speech Communication Association， 2016： 685-689. 10.21437/interspeech.2016-1352
21	LI C， LI L， QI J. A self-attentive model with gate mechanism for spoken language understanding ［C］// Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. Stroudsburg： ACL， 2018： 3824-3833. 10.18653/v1/d18-1417
22	ZHANG C， LI Y， DU N， et al. Joint slot filling and intent detection via capsule neural networks ［C］// Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. Stroudsburg： ACL， 2019： 5259-5267. 10.18653/v1/p19-1519
23	CHEN Q， ZHUO Z， WANG W. BERT for joint intent classification and slot filling ［EB/OL］. （2019-02-28）［2023-04-23］. .
24	E H， NIU P， CHEN Z， et al. A novel bi-directional interrelated model for joint intent detection and slot filling ［C］// Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. Stroudsburg： ACL， 2019： 5467-5471. 10.18653/v1/p19-1544
25	QIN L， CHE W， LI Y， et al. A stack-propagation framework with token-level intent detection for spoken language understanding［C］// Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing. Stroudsburg： ACL， 2019： 2078-2087. 10.18653/v1/d19-1214
26	HUI Y， WANG J， CHENG N， et al. Joint intent detection and slot filling based on continual learning model ［C］// Proceedings of the 2021 International Conference on Acoustics， Speech and Signal Processing. Piscataway： IEEE， 2021： 7643-7647. 10.1109/icassp39728.2021.9413360
27	GUO Y， XIE Z， CHEN X， et al. AWTE-BERT： attending to wordpiece tokenization explicitly on BERT for joint intent classification and slot filling ［EB/OL］. （2023-02-02）［2023-04-13］. .
28	COUCKE A， SAADE A， BALL A， et al. Snips voice platform： an embedded spoken language understanding system for private-by-design voice interfaces ［EB/OL］. （2018-11-06）［2023-04-13］. . 10.1109/emc2-nips53020.2019.00021
29	MA Z， SUN B， LI S. A two-stage selective fusion framework for joint intent detection and slot filling ［J］. IEEE Transactions on Neural Networks and Learning Systems， 2022 （Early Access）：1-12. 10.1109/tnnls.2022.3202562
30	WANG C， HUANG Z， HU M. SASGBC： improving sequence labeling performance for joint learning of slot filling and intent detection ［C］// Proceedings of the 2020 6th International Conference on Computing and Data Engineering. New York： ACM， 2020： 29-33. 10.1145/3379247.3379266
31	OpenAI. GPT-4 Technical Report ［EB/OL］. （2023-03-27）［2023-04-13］. .

方法	ATIS			SNIPS
方法	Slot F1	Intent acc	Sentence acc	Slot F1	Intent acc	Sentence acc
SF-ID Network	95.8	97.7	86.9	92.2	97.4	80.5
CAPSULE-NLU	95.2	95.0	83.4	91.8	97.3	80.9
Attention BiRNN	94.2	91.1	78.9	87.8	96.7	74.1
Slot-Gated	94.8	93.6	82.2	88.8	97.0	75.5
Joint BERT	96.1	97.5	88.2	96.4	98.8	92.5
SASGBC	96.6	98.2	91.6	96.4	98.9	92.5
IDSFML	98.5	98.4	92.4	98.0	99.3	92.4

方法	ATIS			SNIPS
方法	Slot F1	Intent acc	Sentence acc	Slot F1	Intent acc	Sentence acc
SF-ID Network	95.8	97.7	86.9	92.2	97.4	80.5
CAPSULE-NLU	95.2	95.0	83.4	91.8	97.3	80.9
Attention BiRNN	94.2	91.1	78.9	87.8	96.7	74.1
Slot-Gated	94.8	93.6	82.2	88.8	97.0	75.5
Joint BERT	96.1	97.5	88.2	96.4	98.8	92.5
SASGBC	96.6	98.2	91.6	96.4	98.9	92.5
IDSFML	98.5	98.4	92.4	98.0	99.3	92.4

方法	ATIS			SNIPS
方法	Slot F1	Intent acc	Sentence acc	Slot F1	Intent acc	Sentence acc
IDSFML-CRF	97.7	97.9	92.1	97.2	99.1	92.0
IDSFML-AEA	96.9	98.1	91.4	96.7	98.9	91.6
IDSFML-SD	97.9	98.0	91.9	97.8	99.0	91.7
IDSFML	98.5	98.4	92.4	98.0	99.3	92.4

方法	ATIS			SNIPS
方法	Slot F1	Intent acc	Sentence acc	Slot F1	Intent acc	Sentence acc
IDSFML-CRF	97.7	97.9	92.1	97.2	99.1	92.0
IDSFML-AEA	96.9	98.1	91.4	96.7	98.9	91.6
IDSFML-SD	97.9	98.0	91.9	97.8	99.0	91.7
IDSFML	98.5	98.4	92.4	98.0	99.3	92.4

[1]	Zhiqiang ZHAO, Peihong MA, Xinhong HEI. Crowd counting method based on dual attention mechanism [J]. Journal of Computer Applications, 2024, 44(9): 2886-2892.
[2]	Jing QIN, Zhiguang QIN, Fali LI, Yueheng PENG. Diagnosis of major depressive disorder based on probabilistic sparse self-attention neural network [J]. Journal of Computer Applications, 2024, 44(9): 2970-2974.
[3]	Liting LI, Bei HUA, Ruozhou HE, Kuang XU. Multivariate time series prediction model based on decoupled attention mechanism [J]. Journal of Computer Applications, 2024, 44(9): 2732-2738.
[4]	Kaipeng XUE, Tao XU, Chunjie LIAO. Multimodal sentiment analysis network with self-supervision and multi-layer cross attention [J]. Journal of Computer Applications, 2024, 44(8): 2387-2392.
[5]	Pengqi GAO, Heming HUANG, Yonghong FAN. Fusion of coordinate and multi-head attention mechanisms for interactive speech emotion recognition [J]. Journal of Computer Applications, 2024, 44(8): 2400-2406.
[6]	Zhonghua LI, Yunqi BAI, Xuejin WANG, Leilei HUANG, Chujun LIN, Shiyu LIAO. Low illumination face detection based on image enhancement [J]. Journal of Computer Applications, 2024, 44(8): 2588-2594.
[7]	Shangbin MO, Wenjun WANG, Ling DONG, Shengxiang GAO, Zhengtao YU. Single-channel speech enhancement based on multi-channel information aggregation and collaborative decoding [J]. Journal of Computer Applications, 2024, 44(8): 2611-2617.
[8]	Li LIU, Haijin HOU, Anhong WANG, Tao ZHANG. Generative data hiding algorithm based on multi-scale attention [J]. Journal of Computer Applications, 2024, 44(7): 2102-2109.
[9]	Song XU, Wenbo ZHANG, Yifan WANG. Lightweight video salient object detection network based on spatiotemporal information [J]. Journal of Computer Applications, 2024, 44(7): 2192-2199.
[10]	Dahai LI, Zhonghua WANG, Zhendong WANG. Dual-branch low-light image enhancement network combining spatial and frequency domain information [J]. Journal of Computer Applications, 2024, 44(7): 2175-2182.
[11]	Wenliang WEI, Yangping WANG, Biao YUE, Anzheng WANG, Zhe ZHANG. Deep learning model for infrared and visible image fusion based on illumination weight allocation and attention [J]. Journal of Computer Applications, 2024, 44(7): 2183-2191.
[12]	Wu XIONG, Congjun CAO, Xuefang SONG, Yunlong SHAO, Xusheng WANG. Handwriting identification method based on multi-scale mixed domain attention mechanism [J]. Journal of Computer Applications, 2024, 44(7): 2225-2232.
[13]	Huanhuan LI, Tianqiang HUANG, Xuemei DING, Haifeng LUO, Liqing HUANG. Public traffic demand prediction based on multi-scale spatial-temporal graph convolutional network [J]. Journal of Computer Applications, 2024, 44(7): 2065-2072.
[14]	Dianhui MAO, Xuebo LI, Junling LIU, Denghui ZHANG, Wenjing YAN. Chinese entity and relation extraction model based on parallel heterogeneous graph and sequential attention mechanism [J]. Journal of Computer Applications, 2024, 44(7): 2018-2025.
[15]	Zexin XU, Lei YANG, Kangshun LI. Shorter long-sequence time series forecasting model [J]. Journal of Computer Applications, 2024, 44(6): 1824-1831.