Customs declaration good classification algorithm based on hierarchical multi-task BERT

doi:10.11772/j.issn.1001-9081.2021010122

Journal of Computer Applications ›› 2022, Vol. 42 ›› Issue (1): 71-77.DOI: 10.11772/j.issn.1001-9081.2021010122

Special Issue: 人工智能

• Artificial intelligence • Previous Articles Next Articles

Customs declaration good classification algorithm based on hierarchical multi-task BERT

Qiming RUAN¹, Yi GUO¹^,²^,³(), Nan ZHENG¹, Yexiang WANG¹

^1.School of Information Science and Engineering，East China University of Science and Technology，Shanghai 200237，China
^2.National Engineering Laboratory for Big Data Distribution and Exchange Technologies-Business Intelligence and Visualization Research Center，Shanghai 200436，China
^3.Shanghai Engineering Research Center of Big Data & Internet Audience，Shanghai 200072，China

Received:2021-01-22 Revised:2021-05-17 Accepted:2021-06-04 Online:2021-07-14 Published:2022-01-10
Contact: Yi GUO
About author:RUAN Qiming， born in 1997， M. S. candidate. His research interests include natural language processing， text classification， recommender system.
GUO Yi， born in 1975， Ph. D.， professor. His research interests include text mining， knowledge discovery， business intelligence analysis.
ZHENG Nan， born in 1997， M. S. candidate. Her research interests include data mining， recommender system.
WANG Yexiang， born in 1996， M. S. candidate. His research interests include natural language processing， dialogue system.
Supported by:
Scientific Research Program of Science & Technology Commission of Shanghai Municipality(17DZ1101003)

基于层级多任务BERT的海关报关商品分类算法

阮启铭¹, 过弋¹^,²^,³(), 郑楠¹, 王业相¹

^1.华东理工大学信息科学与工程学院，上海 200237
^2.大数据流通与交易技术国家工程实验室-商业智能与可视化技术研究中心，上海 200436
^3.上海大数据与互联网受众工程技术研究中心，上海 200072

通讯作者: 过弋
作者简介:阮启铭（1997—），男，浙江绍兴人，硕士研究生，CCF会员，主要研究方向：自然语言处理、文本分类、推荐系统
过弋（1975—），男，江苏无锡人，教授，博士，CCF高级会员，主要研究方向：文本挖掘、知识发现、商业智能分析
郑楠（1997—），女，山东威海人，硕士研究生，主要研究方向：数据挖掘、推荐系统
王业相（1996—），男，浙江温州人，硕士研究生，主要研究方向：自然语言处理、对话系统。
基金资助:
上海市科学技术委员会科研计划项目(17DZ1101003)

Abstract

Abstract:

In the customs good declaration scenarios， a classification model needs to be used to categorize the goods into uniform Harmonized System （HS） codes. However， the existing customs good classification models ignore the location information of words in the text to be classified， while the HS codes are in tens of thousands， which leads to problems such as class vector sparsity and slow convergence of the model.To address the above problems， a classification model based on Hierarchical Multi-task Bidirectional Encoder Representation from Transformers （HM-BERT） was proposed by combining the manual hierarchical classification strategy in real business scenarios and making full use of the hierarchical structure feature of HS codes. In one aspect， the dynamic word vector of Bidirectional Encoder Representation from Transformers （BERT） model was used to obtain the location information in the text of customs declaration goods. In other aspect， the accuracy and convergence of categorization were improved by making full use of the category information of different levels of HS codes to perform multi-task training of BERT model. In the effectiveness verification of the proposed model on the 2019 customs declaration dataset of a domestic customs service provider， HM-BERT model improves 2 percentage points in accuracy with faster training speed compared to BERT model， and improves 7.1 percentage points in accuracy compared with H （Hierarchical）-fastText. Experimental results show that HM-BERT model can effectively improve the classification effect of customs declaration goods.

Key words: Harmonized System (HS) code, Multi-Task Learning (MTL), text classification, Bidirectional Encoder Representation from Transformers (BERT), vector sparsity

摘要：

海关商品申报场景下，需采用分类模型将商品归类为统一的海关（HS）编码。然而现有海关商品分类模型忽略了待分类文本中词语的位置信息，同时HS编码数以万计，会导致类别向量稀疏、模型收敛速度慢等问题。针对上述问题，结合真实业务场景下人工逐层归类策略，充分利用HS编码的层次结构特点，提出了一种基于层级多任务BERT（HM-BERT）的分类模型。一方面通过BERT模型的动态词向量获取了报关商品文本中的位置信息，另一方面利用HS编码不同层级的类别信息对BERT模型进行多任务训练，以提高归类的准确性和收敛性。在国内某报关服务商2019年的报关数据集上进行的所提模型的有效性验证，相比BERT模型，HM-BERT模型的准确率提高了2个百分点，在模型训练速度上也有所提升；与同样分层级的H-fastText相比，准确率提高了7.1个百分点。实验结果表明，HM-BERT模型能有效改善海关报关商品的分类效果。

关键词: 海关编码, 多任务学习, 文本分类, BERT, 向量稀疏

CLC Number:

TP391.1

Qiming RUAN, Yi GUO, Nan ZHENG, Yexiang WANG. Customs declaration good classification algorithm based on hierarchical multi-task BERT[J]. Journal of Computer Applications, 2022, 42(1): 71-77.

阮启铭, 过弋, 郑楠, 王业相. 基于层级多任务BERT的海关报关商品分类算法[J]. 《计算机应用》唯一官方网站, 2022, 42(1): 71-77.

Figures/Tables 10

References 19

1	World Customs Organization. What is the Harmonized System （HS）？［EB/OL］. ［2020-08-03］.. 10.1201/9781315383026-1
2	3Technologies CE. Is HS classification difficult？［EB/OL］. ［2020-08-03］..
3	谢维，李银胜，邵永臻，等. HS编码查询知识库设计与实现［J］. 计算机应用与软件， 2008， 25（8）：143-146.
	XIE W， LI Y S， SHAO Y Z， et al. Design and implementation of HS code query knowledge base［J］. Computer Applications and Software， 2008， 25（8）：143-146.
4	DING L Y， FAN Z Z， CHEN D L. Auto-categorization of HS code using background net approach［J］. Procedia Computer Science， 2015， 60：1462-1471. 10.1016/j.procs.2015.08.224
5	LI Q， PENG H， LI J X， et al. A survey on text classification： from shallow to deep learning［EB/OL］. （2020-10-26）［2020-12-12］.. 10.1109/isci50694.2020.00020
6	MIKOLOV T， CHEN K， CORRADO G， et al. Efficient estimation of word representations in vector space［EB/OL］. （2013-09-07）［2020-12-12］.. 10.3126/jiee.v3i1.34327
7	JOULIN A， GRAVE E， BOJANOWSKI P， et al. Bag of tricks for efficient text classification［C］// Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics： Volume 2， Short Papers. Stroudsburg， PA： Association for Computational Linguistics， 2017： 427-431. 10.18653/v1/e17-2068
8	MIKOLOV T， SUTSKEVER I， CHEN K， et al. Distributed representations of words and phrases and their compositionality［C］// Proceedings of the 26th International Conference on Neural Information Processing Systems. Red Hook， NY： Curran Associates Inc.， 2013：3111-3119.
9	龚丽娟，王昊，张紫玄，等. Word2Vec对海关报关商品文本特征降维效果分析［J］. 数据分析与知识发现， 2020， 4（2/3）：89-100. 10.11925/infotech.2096-3467.2019.0613
	GONG L J， WANG H， ZHANG Z X， et al. Reducing dimensions of custom declaration texts with Word2Vec［J］. Data Analysis and Knowledge Discovery， 2020， 4（2/3）：89-100. 10.11925/infotech.2096-3467.2019.0613
10	PENNINGTON J， SOCHER R， MANNING C D. GloVe： global vectors for word representation［C］// Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing. Stroudsburg， PA： Association for Computational Linguistics， 2014：1532-1543. 10.3115/v1/d14-1162
11	KIM Y. Convolutional neural networks for sentence classification［C］// Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing. Stroudsburg， PA： Association for Computational Linguistics， 2014： 1746-1751. 10.3115/v1/d14-1181
12	PETERS M E， NEUMANN M， IYYER M， et al. Deep contextualized word representations［C］// Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics： Human Language Technologies， Volume 1 （Long Papers）. Stroudsburg， PA： Association for Computational Linguistics， 2018： 2227-2237. 10.18653/v1/n18-1202
13	RADFORD A， NARASIMHAN K， SALIMANS T， et al. Improving language understanding with unsupervised learning［EB/OL］. （2018-06-11）［2020-12-12］..
14	VASWANI A， SHAZEER N， PARMAR N， et al. Attention is all you need ［C］// Proceedings of the 2017 31st International Conference on Neural Information Processing Systems. Red Hook， NY： Curran Associates Inc.， 2017： 6000-6010. 10.1016/s0262-4079(17)32358-8
15	DEVLIN J， CHANG M W， LEE K， et al. BERT： pre-training of deep bidirectional transformers for language understanding［C］// Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics： Human Language Technologies， Volume 1 （Long and Short Papers）. Stroudsburg， PA： Association for Computational Linguistics， 2019： 4171-4186. 10.18653/v1/n19-1423
16	SØGAARD A， GOLDBERG Y. Deep multi-task learning with low level tasks supervised at lower layers［C］// Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics （Volume 2： Short Papers）. Stroudsburg， PA： Association for Computational Linguistics， 2016：231-235. 10.18653/v1/p16-2038
17	HASHIMOTO K， XIONG C M， TSURUOKA Y， et al. A joint many-task model： growing a neural network for multiple NLP tasks［C］// Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing. Stroudsburg， PA： Association for Computational Linguistics， 2017： 1923-1933. 10.18653/v1/d17-1206
18	SANH V， WOLF T， RUDER S. A hierarchical multi-task approach for learning embeddings from semantic tasks［C］// Proceedings of the 33rd AAAI Conference on Artificial Intelligence. Palo Alto， CA： AAAI Press， 2019：6949-6956. 10.1609/aaai.v33i01.33016949
19	LIU X D， HE P C， CHEN W Z， et al. Multi-task deep neural networks for natural language understanding［C］// Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. Stroudsburg， PA： Association for Computational Linguistics， 2019： 4487-4496. 10.18653/v1/p19-1441

章节	商品描述平均长度	总数据量	包含HS编码数
01	18.594 9	311	19
02	92.105 4	233 622	45
03	60.270 3	38 177	168
04	57.086 7	14 550	37
05	29.558 9	6 592	26
06	27.043 1	4 753	24
07	22.430 1	5 536	82
︙	︙	︙	︙
84	39.912 2	4 932 055	1 219
85	42.999 7	4 932 903	631

章节	商品描述平均长度	总数据量	包含HS编码数
01	18.594 9	311	19
02	92.105 4	233 622	45
03	60.270 3	38 177	168
04	57.086 7	14 550	37
05	29.558 9	6 592	26
06	27.043 1	4 753	24
07	22.430 1	5 536	82
︙	︙	︙	︙
84	39.912 2	4 932 055	1 219
85	42.999 7	4 932 903	631

模型	准确率	宏平均
TextCNN	0.701 2	0.678 9
fastText	0.675 0	0.623 7
H-fastText	0.729 7	0.703 4
BERT	0.780 1	0.746 0
HM-BERT	0.800 3	0.780 0

模型	准确率	宏平均
TextCNN	0.701 2	0.678 9
fastText	0.675 0	0.623 7
H-fastText	0.729 7	0.703 4
BERT	0.780 1	0.746 0
HM-BERT	0.800 3	0.780 0

模型	HS编码		目		章
模型	准确率	宏平均	准确率	宏平均	准确率	宏平均
BERT	0.780 1	0.746 0	0.954 6	0.941 3	0.981 5	0.977 7
HM-BERT	0.800 3	0.780 0	0.956 0	0.943 7	0.983 8	0.980 0

Customs declaration good classification algorithm based on hierarchical multi-task BERT

基于层级多任务BERT的海关报关商品分类算法

RichHTML

PDF

Knowledge

Abstract

Cite this article

share this article

Figures/Tables 10

References 19

Related Articles 15

Recommended Articles

Metrics

[1]	Xun YAO, Zhongzheng QIN, Jie YANG. Generative label adversarial text classification model [J]. Journal of Computer Applications, 2024, 44(6): 1781-1785.
[2]	Xinyan YU, Cheng ZENG, Qian WANG, Peng HE, Xiaoyu DING. Few-shot news topic classification method based on knowledge enhancement and prompt learning [J]. Journal of Computer Applications, 2024, 44(6): 1767-1774.
[3]	Hang YU, Yanling ZHOU, Mengxin ZHAI, Han LIU. Text classification based on pre-training model and label fusion [J]. Journal of Computer Applications, 2024, 44(3): 709-714.
[4]	Jiawei ZHANG, Guandong GAO, Ke XIAO, Shengzun SONG. Violent crime hierarchy algorithm by joint modeling of improved hierarchical attention network and TextCNN [J]. Journal of Computer Applications, 2024, 44(2): 403-410.
[5]	Kaitian WANG, Qing YE, Chunlei CHENG. Classification method for traditional Chinese medicine electronic medical records based on heterogeneous graph representation [J]. Journal of Computer Applications, 2024, 44(2): 411-417.
[6]	Bihui YU, Xingye CAI, Jingxuan WEI. Few-shot text classification method based on prompt learning [J]. Journal of Computer Applications, 2023, 43(9): 2735-2740.
[7]	Yumeng CUI, Jingya WANG, Xiaowen LIU, Shangyi YAN, Zhizhong TAO. General text classification model combining attention and cropping mechanism [J]. Journal of Computer Applications, 2023, 43(8): 2396-2405.
[8]	Senqi YANG, Xuliang DUAN, Zhan XIAO, Songsong LANG, Zhiyong LI. Text classification of agricultural news based on ERNIE+DPCNN+BiGRU [J]. Journal of Computer Applications, 2023, 43(5): 1461-1466.
[9]	Xu ZHANG, Long SHENG, Haifang ZHANG, Feng TIAN, Wei WANG. Pre-hospital emergency text classification model based on label confusion [J]. Journal of Computer Applications, 2023, 43(4): 1050-1055.
[10]	Chengyu LIN, Lei WANG, Cong XUE. Weakly-supervised text classification with label semantic enhancement [J]. Journal of Computer Applications, 2023, 43(2): 335-342.
[11]	Jing JIANG, Yu CHEN, Jieping SUN, Shenggen JU. Integrating posterior probability calibration training into text classification algorithm [J]. Journal of Computer Applications, 2022, 42(6): 1789-1795.
[12]	Xianfeng YANG, Jiahe ZHAO, Ziqiang LI. Text classification model combining word annotations [J]. Journal of Computer Applications, 2022, 42(5): 1317-1323.
[13]	Shigang YANG, Yongguo LIU. Short text classification method by fusing corpus features and graph attention network [J]. Journal of Computer Applications, 2022, 42(5): 1324-1329.
[14]	Haifeng ZHANG, Cheng ZENG, Lie PAN, Rusong HAO, Chaodong WEN, Peng HE. News topic text classification method based on BERT and feature projection network [J]. Journal of Computer Applications, 2022, 42(4): 1116-1124.
[15]	Wangjing TANG, Bin XU, Meihan TONG, Meihuan HAN, Liming WANG, Qi ZHONG. Popular science text classification model enhanced by knowledge graph [J]. Journal of Computer Applications, 2022, 42(4): 1072-1078.