基于改进Mask R-CNN的越南场景文字检测

doi:10.11772/j.issn.1001-9081.2021050821

《计算机应用》唯一官方网站 ›› 2021, Vol. 41 ›› Issue (12): 3551-3557.DOI: 10.11772/j.issn.1001-9081.2021050821

所属专题：第十八届中国机器学习会议(CCML 2021)

• 第十八届中国机器学习会议(CCML 2021) • 上一篇下一篇

基于改进Mask R-CNN的越南场景文字检测

俸亚特¹, 文益民²()

^1.桂林电子科技大学计算机与信息安全学院，广西桂林 541004
^2.广西图像图形与智能处理重点实验室（桂林电子科技大学），广西桂林 541004

收稿日期:2021-05-12 修回日期:2021-05-20 接受日期:2021-05-21 发布日期:2021-12-28 出版日期:2021-12-10
通讯作者: 文益民
作者简介:俸亚特（1996—），男，广西桂林人，硕士研究生，主要研究方向：机器学习、计算机视觉；
基金资助:
国家自然科学基金资助项目(61866007);广西自然科学基金资助项目(2018GXNSFDA138006);教育部人文社会科学研究项目(17JDGC022);广西学位与研究生教育改革项目(JGY2017055)

Vietnamese scene text detection based on modified Mask R-CNN

Yate FENG¹, Yimin WEN²()

^1.School of Computer Science and Information Security，Guilin University of Electronic Technology，Guilin Guangxi 541004，China
^2.Guangxi Key Laboratory of Image and Graphic Intelligent Processing （Guilin University of Electronic Technology），Guilin Guangxi 541004，China

Received:2021-05-12 Revised:2021-05-20 Accepted:2021-05-21 Online:2021-12-28 Published:2021-12-10
Contact: Yimin WEN
About author:FENG Yate， born in 1996， M. S. candidate. His research interests include machine learning， computer vision.
Supported by:
the National Natural Science Foundation of China(61866007);the Natural Science Foundation of Guangxi(2018GXNSFDA138006);the Humanities and Social Sciences Research Projects of the Ministry of Education(17JDGC022);the Guangxi Degree and Graduate Education Reform Project(JGY2017055)

摘要/Abstract

摘要：

针对越南场景文字检测训练数据缺乏及越南文字声调符号检测不全的问题，在改进的实例分割网络Mask R-CNN的基础上，提出一种针对越南场景文字的检测算法。为了准确地分割带声调符号的越南场景文字，该算法仅使用P2特征层来分割文字区域，并将文字区域的掩码矩阵大小从14×14调整为14×28以更好地适应文字区域。针对用常规非极大值抑制（NMS）算法不能剔除重复文字检测框的问题，设计了一个针对文字区域的文本区域过滤模块并添加在检测模块之后，以有效地剔除冗余检测框。使用模型联合训练的方法训练网络，训练过程包含两部分：第一部分为特征金字塔网络（FPN）和区域生成网络（RPN）的训练，训练使用的数据集为大规模公开的拉丁文字数据，目的是增强模型在不同场景下提取文字的泛化能力；第二部分为候选框坐标回归模块和区域分割模块的训练，此部分模型参数使用像素级标注的越南场景文字数据进行训练，使模型能对包括声调符号的越南文字区域进行分割。大量交叉验证实验和对比实验结果表明，与Mask R-CNN相比，所提算法在不同的交并比（IoU）阈值下都具有更好的准确率与召回率。

关键词: Mask R-CNN, 越南场景文字检测, 声调符号, 模型联合训练, 分割模型, 重复检测

Abstract:

In view of the lack of training data for Vietnamese scene text detection and the incomplete detection of Vietnamese tone marks in the detection， a text detection algorithm for Vietnamese scenes based on a modified instance segmentation method Mask R-CNN was proposed. In order to segment Vietnamese scene text with tone marks accurately， only P2 feature layer was utilized to segment the text area， and the mask matrix size of the text area was adjusted from 14 × 14 to 14 × 28 to adapt the shape of most texts. Aiming at the problem that duplicate text detection boxes cannot be eliminated by the conventional Non-Maximum Suppression （NMS） algorithm， a filter module for the text areas named Text region filtering branch was designed and added after the detection module to effectively eliminate duplicate detection boxes. A model joint training method was used to train the network. The training process consists of two parts： the first part is the training of the Feature Pyramid Network （FPN） and the Region Proposal Network （RPN） of the model， which used large-scale open Latin text data for training to enhance the generalization ability of the model to detect text in different scenes； the second part is the training of the candidate box coordinate regression module and the segmentation module named Box branch and Mask branch， which used pixel-level labelled Vietnamese scene text data for training to enable the model to segment the Vietnamese text area including tone marks. Many cross-validation experiments and comparison experiments verify that the proposed algorithm has better precision and recall under different Intersection over Union （IoU） thresholds compared with Mask R-CNN.

Key words: Mask R-CNN, Vietnamese scene text detection, tone mark, model joint training, segmentation model, duplicate detection

中图分类号:

TP391

俸亚特, 文益民. 基于改进Mask R-CNN的越南场景文字检测[J]. 计算机应用, 2021, 41(12): 3551-3557.

Yate FENG, Yimin WEN. Vietnamese scene text detection based on modified Mask R-CNN[J]. Journal of Computer Applications, 2021, 41(12): 3551-3557.

图/表 10

图1 使用四边形检测框标注越南文本区域时存在的问题

Fig. 1 Problems in labelling Vietnamese texts areas with quadrangular bonding boxes

图2 使用MLT中拉丁文字数据训练的模型对部分越南文字的检测结果

Fig. 2 Detection results of some Vietnamese texts using the model trained by Latin data in MLT dataset

图3 总体网络结构

Fig. 3 Overall network structure

图4 文本区域过滤模块效果

Fig. 4 Effect of Text region filtering branch

表1 使用与未使用模型联合训练方法在越南场景文本数据集上的对比

Tab. 1 Comparision between using and not using model joint training method on Vietnamese scene text dataset

交叉验证轮次	训练集图片为160张，测试集图片为40张						训练集图片为100张，测试集图片为100张						训练集图片为40张，测试集图片为160张
	使用像素级标注的数据训练			模型联合训练方法			使用像素级标注的数据训练			模型联合训练方法			使用像素级标注的数据训练			模型联合训练方法
	P	R	F₁	P	R	F₁	P	R	F₁	P	R	F₁	P	R	F₁	P	R	F₁
平均值	0.697	0.688	0.690	0.769	0.831	0.797	0.663	0.632	0.647	0.771	0.820	0.780	0.584	0.510	0.543	0.762	0.752	0.756
1	0.700	0.777	0.737	0.746	0.927	0.827	0.696	0.704	0.700	0.777	0.863	0.818	0.612	0.475	0.535	0.779	0.706	0.740
2	0.590	0.58	0.580	0.705	0.725	0.715	0.610	0.591	0.600	0.711	0.769	0.739	0.593	0.579	0.586	0.789	0.798	0.793
3	0.682	0.586	0.631	0.767	0.739	0.753	0.637	0.567	0.600	0.746	0.724	0.735	0.534	0.500	0.516	0.728	0.743	0.736
4	0.766	0.732	0.749	0.825	0.870	0.847	0.640	0.583	0.610	0.796	0.759	0.776	0.591	0.533	0.560	0.753	0.752	0.753
5	0.745	0.763	0.754	0.804	0.892	0.846	0.734	0.713	0.723	0.821	0.861	0.841	0.593	0.466	0.522	0.761	0.759	0.760

图5 模型联合训练方法对部分越南文字的检测结果

Fig. 5 Detection results of some Vietnamese texts using model joint training method

图6 四种实例分割算法在三种不同规格训练集、验证集上的F值对比

Fig. 6 F-mesure comparison of four instance segmentation algorithms on three different specifications of training sets and validation sets

图7 区域分割模块优化前后在三种不同规格训练集、验证集上的F值对比

Fig. 7 F-mesure comparison before and after optimization of Mask branch on three different specifications of training sets and validation sets

图8 区域分割模块优化前后的部分检测结果

Fig. 8 Partial detection results before and after optimization of Mask branch

表2 使用和不使用文本区域过滤模块的在越南场景文本数据集上的性能对比

Tab. 2 Performance comparison between using and not using the proposed text region filtering module on vietnamese scene text dataset

交叉验证轮次	测试集大小为160			测试集大小为100			测试集大小为40
交叉验证轮次	准确剔除的检测框数	总共检测框数量	提升/%	准确剔除的检测框数	总共检测框数量	提升/%	准确剔除的检测框数	总共检测框数量	提升/%
1	18	2 442	0.74	5	802	0.62	3	300	0.98
2	18	2 076	0.87	12	1 397	0.86	5	914	0.55
3	7	2 121	0.33	11	1 818	0.61	12	833	1.44
4	22	2 341	0.94	9	1 866	0.48	4	633	0.63
5	16	2 595	0.62	8	1 345	0.60	2	379	0.53

参考文献 31

1	刘瑞昌. 声调语言的特征及其分布［J］. 考试周刊， 2015（34）：19. 10.3969/j.issn.1673-8918.2015.34.016
	LIU R C. Characteristics and distribution of tone language［J］. Test Weekly， 2015（34）： 19. 10.3969/j.issn.1673-8918.2015.34.016
2	TIAN Z， HUANG W L， HE T， et al. Detecting text in natural image with connectionist text proposal network［C］// Proceedings of the 2016 European Conference on Computer Vision， LNCS9912. Cham： Springer， 2016： 56-72.
3	XIE E Z， ZANG Y H， SHAO S， et al. Scene text detection with supervised pyramid context network［C］// Proceedings of the 33rd AAAI Conference on Artificial Intelligence. Palo Alto， CA： AAAI Press， 2019： 9038-9045. 10.1609/aaai.v33i01.33019038
4	BAEK Y， LEE B， HAN D， et al. Character region awareness for text detection［C］// Proceedings of the 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2019： 9357-9366. 10.1109/cvpr.2019.00959
5	LIU W， ANGUELOV D， ERHAN D， et al. SSD： single shot multibox detector［C］// Proceedings of the 2016 European Conference on Computer Vision， LNCS9905. Cham： Springer， 2016： 21-37.
6	REDMON J， DIVVALA S， GIRSHICK R， et al. You only look once： unified， real-time object detection［C］// Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2016： 779-788. 10.1109/cvpr.2016.91
7	REN S Q， HE K M， GIRSHICK R， et al. Faster R-CNN： towards real-time object detection with region proposal networks［J］. IEEE Transactions on Pattern Analysis and Machine Intelligence， 2017， 39（6）： 1137-1149. 10.1109/tpami.2016.2577031
8	LIAO M H， SHI B G， BAI X， et al. TextBoxes： a fast text detector with a single deep neural network［C］// Proceedings of the 31st AAAI Conference on Artificial Intelligence. Palo Alto， CA： AAAI Press， 2017： 4161-4167. 10.1609/aaai.v34i07.6812
9	LIAO M H， SHI B G， BAI X. TextBoxes++： a single-shot oriented scene text detector［J］. IEEE Transactions on Image Processing， 2018， 27（8）： 3676-3690. 10.1109/tip.2018.2825107
10	LIAO M H， ZHU Z， SHI B G， et al. Rotation-sensitive regression for oriented scene text detection［C］// Proceedings of the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2018： 5909-5918. 10.1109/cvpr.2018.00619
11	ZHOU Y Z， YE Q X， QIU Q， et al. Oriented response networks［C］// Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2017： 4961-4970. 10.1109/cvpr.2017.527
12	ZHOU X Y， YAO C， WEN H， et al. EAST： an efficient and accurate scene text detector［C］// Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2017： 2642-2651. 10.1109/cvpr.2017.283
13	SHI B G， BAI X， BELONGIE S. Detecting oriented text in natural images by linking segments［C］// Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2017： 3482-3490. 10.1109/cvpr.2017.371
14	WANG W H， XIE E Z， LI X， et al. Shape robust text detection with progressive scale expansion network［C］// Proceedings of the 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2019： 9328-9337. 10.1109/cvpr.2019.00956
15	LIU J C， LIU X B， SHENG J， et al. Pyramid mask text detector［EB/OL］. （2019-03-28）［2020-12-03］. .
16	HE K M， GKIOXARI G， DOLLÁR P， et al. Mask R-CNN［C］// Proceedings of the 2017 IEEE International Conference on Computer Vision. Piscataway： IEEE， 2017： 2980-2988. 10.1109/iccv.2017.322
17	LONG S B， RUAN J Q， ZHANG W J， et al. TextSnake： a flexible representation for detecting text of arbitrary shapes［C］// Proceedings of the 2018 European Conference on Computer Vision， LNCS11206. Cham： Springer， 2018： 19-35.
18	LYU P Y， LIAO M H， YAO C， et al. Mask TextSpotter： an end-to-end trainable neural network for spotting text with arbitrary shapes［C］// Proceedings of the 2018 European Conference on Computer Vision， LNCS11218. Cham： Springer， 2018： 71-88.
19	LIU Y L， CHEN H， SHEN C H， et al. ABCNet： real-time scene text spotting with adaptive Bezier-curve network［C］// Proceedings of the 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2020： 9806-9815. 10.1109/cvpr42600.2020.00983
20	GIRSHICK R. Fast R-CNN［C］// Proceedings of the 2015 IEEE International Conference on Computer Vision. Piscataway： IEEE， 2015： 1440-1448. 10.1109/iccv.2015.169
21	LONG J， SHELHAMER E， DARRELL T. Fully convolutional networks for semantic segmentation［C］// Proceedings of the 2015 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2015： 3431-3440. 10.1109/cvpr.2015.7298965
22	LIN T Y， DOLLÁR P， GIRSHICK R， et al. Feature pyramid networks for object detection［C］// Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2017： 936-944. 10.1109/cvpr.2017.106
23	KRIZHEVSKY A， SUTSKEVER I， HINTON G E. ImageNet classification with deep convolutional neural networks［C］// Proceedings of the 25th International Conference on Neural Information Processing Systems. Red Hook， NY： Curran Associates Inc.， 2012： 1097-1105.
24	SIMONYAN K， ZISSERMAN A. Very deep convolutional networks for large-scale image recognition［EB/OL］. （2015-04-10）［2020-12-03］.. 10.5244/c.28.6
25	SZEGEDY C， LIU W， JIA Y Q， et al. Going deeper with convolutions［C］// Proceedings of the 2015 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2015： 1-9. 10.1109/cvpr.2015.7298594
26	IOFFE S， SZEGEDY C. Batch normalization： accelerating deep network training by reducing internal covariate shift［C］// Proceedings of the 32nd International Conference on Machine Learning. New York： JMLR.org， 2015： 448-456.
27	HE K M， ZHANG X Y， REN S Q， et al. Deep residual learning for image recognition［C］// Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2016： 770-778. 10.1109/cvpr.2016.90
28	NAYEF N， YIN F， BIZID I， et al. ICDAR2017 robust reading challenge on multi-lingual scene text detection and script identification — RRC-MLT［C］// Proceedings of the 14th IAPR International Conference on Document Analysis and Recognition. Piscataway： IEEE， 2017： 1454-1459. 10.1109/icdar.2017.237
29	NAYEF N， PATEL Y， BUSTA M， et al. ICDAR2019 robust reading challenge on multi-lingual scene text detection and recognition — RRC-MLT-2019［C］// Proceedings of the 15th IAPR International Conference on Document Analysis and Recognition. Piscataway： IEEE， 2019： 1582-1587. 10.1109/icdar.2019.00254
30	BOLYA D， ZHOU C， XIAO F Y， et al. YOLACT： real-time instance segmentation［C］// Proceedings of the 2019 IEEE/CVF International Conference on Computer Vision. Piscataway： IEEE， 2019： 9156-9165. 10.1109/iccv.2019.00925
31	BOLYA D， ZHOU C， XIAO F Y， et al. YOLACT++： better real-time instance segmentation［EB/OL］. （2020-09-24）［2020-12-03］.. 10.1109/tpami.2020.3014297

[1]	冯涛, 陈斌, 张跃飞. 基于改进的Mask R-CNN的染色体图像分割框架[J]. 计算机应用, 2020, 40(11): 3332-3339.
[2]	朱繁, 王洪元, 张继. 基于改进的Mask R-CNN的行人细粒度检测算法[J]. 计算机应用, 2019, 39(11): 3210-3215.

基于改进Mask R-CNN的越南场景文字检测

Vietnamese scene text detection based on modified Mask R-CNN

RichHTML

PDF

可视化

摘要/Abstract

引用本文

使用本文

图/表 10

参考文献 31

相关文章 2

编辑推荐

Metrics