Table structure recognition model integrating edge features and attention

doi:10.11772/j.issn.1001-9081.2022010053

Journal of Computer Applications ›› 2023, Vol. 43 ›› Issue (3): 752-758.DOI: 10.11772/j.issn.1001-9081.2022010053

• Artificial intelligence • Previous Articles

Table structure recognition model integrating edge features and attention

Xueqiang LYU¹, Yunan ZHANG¹, Jing HAN¹(), Yunpeng CUI², Huan LI²

^1.Beijing Key Laboratory of Internet Culture and Digital Dissemination Research （Beijing Information Science and Technology University），Beijing 100101，China
^2.Key Laboratory of Agricultural Big Data，Ministry of Agriculture （Agricultural Information Institute of Chinese Academy of Agricultural Science），Beijing 100081，China

Received:2022-01-17 Revised:2022-04-06 Accepted:2022-04-11 Online:2022-04-26 Published:2023-03-10
Contact: Jing HAN
About author:LYU Xueqiang， born in 1970， Ph. D.， professor. His research interests include multimedia information processing.
ZHANG Yunan， born in 1996， M. S. candidate. His research interests include computer vision.
CUI Yunpeng， born in 1972， Ph. D.， research fellow. His research interests include agricultural information technology.
LI Huan， born in 1992， M. S.， research fellow. Her research interests include data mining.
Supported by:
National Natural Science Foundation of China(62171043)

融合边特征与注意力的表格结构识别模型

吕学强¹, 张煜楠¹, 韩晶¹(), 崔运鹏², 李欢²

^1.网络文化与数字传播北京市重点实验室(北京信息科技大学), 北京 100101
^2.农业农村部农业大数据重点实验室(中国农业科学院农业信息研究所), 北京 100081

通讯作者: 韩晶
作者简介:吕学强（1970—），男，辽宁抚顺人，教授，博士，主要研究方向：多媒体信息处理
张煜楠（1996—），男，北京人，硕士研究生，主要研究方向：计算机视觉
韩晶（1990—），女，河北邯郸人，助理研究员，博士，主要研究方向：图像处理
崔运鹏（1972—），男，吉林和龙人，研究员，博士，主要研究方向：农业信息技术
李欢（1992—），女，湖北红安人，研究员，硕士，主要研究方向：数据挖掘。
基金资助:
国家自然科学基金资助项目(62171043)

Abstract

Abstract:

Aiming at the problems in the existing methods such as dependence on prior knowledge， insufficient robustness， and insufficient expression ability in table structure recognition， a new table structure recognition model integrating edge features and attention was proposed， namely Graph Edge-Attention Network based Table Structure Recognition model （GEAN-TSR）. Firstly， Graph Edge-Attention Network （GEAN） was proposed as the backbone network， and based on edge convolution structure， the graph attention mechanism was introduced and improved to aggregate graph node features， so as to solve the problem of information loss in the process of feature extraction of graph network， and improve the expression ability of graph network. Then， an edge feature fusion module was introduced to fuse the shallow graph node information with the graph network output to enhance the local information extraction and expression abilities of the graph network. Finally， the graph node text features extracted by Gated Recurrent Unit （GRU） were integrated into the text feature fusion module for edge’s classification and prediction. Comparative experiments on Scientific paper Table Structure Recognition-COMPlicated （SciTSR-COMP） dataset show that the recall and F1 score of GEAN-TSR are increased by 2.5 and 1.4 percentage points， respectively in comparison with the existing optimal model Split， Embed and Merge （SEM）. Ablation experiments show that all the indicators of GEAN-TSR have achieved the optimal values after using the feature fusion module， proving the effectiveness of the module. Experimental results show that GEAN-TSR can effectively improve the network performance and better complete the task of table structure recognition.

Key words: graph neural network, graph attention network, feature fusion, table structure recognition, table parsing

摘要：

针对现有方法在表格结构识别问题中存在的先验知识依赖、鲁棒性不足、表达能力不足等问题，提出一种新的融合边特征与注意力的表格结构识别模型——GEAN-TSR。首先，提出图边注意力网络（GEAN）并作为模型的主干网络，在边卷积结构的基础上引入并改进图注意力机制聚合图节点特征，解决图网络在特征提取过程中的信息损失的问题，提高图网络的表达能力；然后，引入边特征融合模块融合浅层图节点信息与图网络输出，增强图网络的局部信息提取能力与表达能力；最后，将门控循环单元（GRU）提取的图节点文本特征融入文本特征融合模块对边进行分类预测。在SciTSR-COMP数据集上的对比实验中，相较于目前最优的模型SEM，GEAN-TSR的召回率与F1值分别提升2.5与1.4个百分点。在消融实验中，GEAN-TSR采用特征融合模块后，所有指标都取得了最优值，验证了模块的有效性。实验结果表明，GEAN-TSR能够有效提升网络性能，更好地完成表格结构识别任务。

关键词: 图神经网络, 图注意力网络, 特征融合, 表格结构识别, 表格解析

CLC Number:

TP391.4

Xueqiang LYU, Yunan ZHANG, Jing HAN, Yunpeng CUI, Huan LI. Table structure recognition model integrating edge features and attention[J]. Journal of Computer Applications, 2023, 43(3): 752-758.

吕学强, 张煜楠, 韩晶, 崔运鹏, 李欢. 融合边特征与注意力的表格结构识别模型[J]. 《计算机应用》唯一官方网站, 2023, 43(3): 752-758.

Figures/Tables 11

References 19

1	ITONORI K. Table structure recognition based on textblock arrangement and ruled line position ［C］// Proceedings of the 2nd International Conference on Document Analysis and Recognition. Piscataway： IEEE， 1993： 765-768. 10.1109/icdar.1993.395796
2	SHIGAROV A O， FEDOROV R K. Simple algorithm for page layout analysis［J］. Pattern Recognition and Image Analysis， 2011， 21（2）：324-327. 10.1134/s1054661811021008
3	RAMEL J Y， CRUCIANU M， VINCENT N， et al. Detection， extraction and representation of tables ［C］// Proceedings of the 7th International Conference on Document Analysis and Recognition. Piscataway： IEEE， 2013： 374-378.
4	SHIGAROV A， MIKHAILOV A， ALTAEV A. Configurable table structure recognition in untagged PDF documents ［C］// Proceedings of the 2016 ACM Symposium on Document Engineering. New York： ACM， 2016：119-122. 10.1145/2960811.2967152
5	PALIWAL S S， VISHWANATH D， RAHUL R， et al. TableNet： deep learning model for end-to-end table detection and tabular data extraction from scanned document images ［C］// Proceedings of the 15th IAPR International Conference on Document Analysis and Recognition. Piscataway： IEEE， 2019：128-133. 10.1109/icdar.2019.00029
6	TENSMEYER C， MORARIU V I， PRICE B， et al. Deep splitting and merging for table structure decomposition ［C］// Proceedings of the 15th IAPR International Conference on Document Analysis and Recognition. Piscataway： IEEE， 2019： 114-121. 10.1109/icdar.2019.00027
7	SIDDIQUI S A， FATEH I A， RIZVI S T R， et al. DeepTabStR： deep learning based table structure recognition ［C］// Proceedings of the 2019 International Conference on Document Analysis and Recognition. Piscataway： IEEE， 2019： 1403-1409. 10.1109/icdar.2019.00226
8	QIAO L， LI Z S， CHENG Z Z， et al. LGPMA： complicated table structure recognition with local and global pyramid mask alignment［C］// Proceedings of the 2021 International Conference on Document Analysis and Recognition， LNCS 12821. Cham： Springer， 2021： 99-114. 10.48550/arXiv.2105.06224
9	QASIM S R， MAHMOOD H， SHAFAIT F. Rethinking table recognition using graph neural networks ［C］// Proceedings of the 15th IAPR International Conference on Document Analysis and Recognition. Piscataway： IEEE， 2019： 142-147. 10.1109/icdar.2019.00031
10	LI Y R， HUANG Z， YAN J C， et al. GFTE： Graph-based Financial Table Extraction［C］// Proceedings of the 2021 International Conference on Pattern Recognition， LNCS 12662. Cham： Springer， 2021： 644-658.
11	徐冰冰，岑科廷，黄俊杰，等. 图卷积神经网络综述［J］. 计算机学报， 2020， 43（5）： 755-780. 10.11897/SP.J.1016.2020.00755
	XU B B， CEN K T， HUANG J J， et al. A survey on graph convolutional neural network［J］. Chinese Journal of Computers， 2020， 43（5）： 755-780. 10.11897/SP.J.1016.2020.00755
12	WANG Y， SUN Y， LIU Z， et al. Dynamic graph CNN for learning on point clouds ［J］. ACM Transactions on Graphics， 2019， 38（5）：1-12. 10.1145/3326362
13	VELIČKOVIĆ P， CASANOVA A， LIO P， et al. Graph attention networks ［EB/OL］. （2018-02-04）［2021-08-14］. .
14	BRODY S， ALON U， YAHAV E. How attentive are graph attention networks？［EB/OL］. （2021-08-11）［2021-08-20］. .
15	GÖBEL M， HASSAN T， ORO E， et al. A methodology for evaluating algorithms for table understanding in PDF documents ［C］// Proceedings of the 2012 ACM symposium on Document engineering. New York： ACM， 2012： 45-48. 10.1145/2361354.2361365
16	SCHREIBER S， AGNE S， WOLF I， et al. DeepDeSRT： deep learning for detection and structure recognition of tables in document images ［C］// Proceedings of the14th IAPR International Conference on Document Analysis and Recognition. Piscataway： IEEE， 2017： 1162-1167. 10.1109/icdar.2017.192
17	CHI Z， HUANG H， XU H D， et al. Complicated table structure recognition ［EB/OL］. （2019-08-28）［2021-09-10］. .
18	ZHANG Z R， ZHANG J S， DU J， et al. Split， embed and merge： an accurate table structure recognizer［J］. Pattern Recognition， 2022， 126： No.108565. 10.1016/j.patcog.2022.108565
19	RAJA S， MONDAL A， JAWAHAR C V. Table structure recognition using top-down and bottom-up cues ［C］// Proceedings of the 2020 European Conference on Computer Vision. Cham： Springer， 2020： 70-86. 10.1007/978-3-030-58604-1_5

主干网模型	模型大小/MB	参数量/10⁶	平均预测时间/s
GCN	9.2	2.40	0.002 9
GAT	9.4	2.40	0.003 0
DGCNN	10.4	2.72	0.002 4
GEAN	10.4	2.72	0.002 9

主干网模型	模型大小/MB	参数量/10⁶	平均预测时间/s
GCN	9.2	2.40	0.002 9
GAT	9.4	2.40	0.003 0
DGCNN	10.4	2.72	0.002 4
GEAN	10.4	2.72	0.002 9

模型	测试集	精确率	召回率	F1值
Tabby	SciTSR	0.914	0.910	0.912
	SciTSR-COMP	0.869	0.841	0.855
	ICDAR 2013	0.846	0.862	0.854
DeepDeSRT	SciTSR	0.906	0.887	0.890
	SciTSR-COMP	0.863	0.831	0.846
	ICDAR 2013	0.632	0.617	0.854
GraphTSR	SciTSR	0.959	0.948	0.953
	SciTSR-COMP	0.964	0.945	0.955
	ICDAR 2013	0.885	0.860	0.872
SEM	SciTSR	0.977	0.965	0.971
SEM	SciTSR-COMP	0.968	0.947	0.957
TabStruct-Net	ICDAR 2013	0.915	0.897	0.906
GEAN-TSR	SciTSR	0.976	0.990	0.983
	SciTSR-COMP	0.970	0.972	0.971
	ICDAR 2013	0.900	0.936	0.913

模型	测试集	精确率	召回率	F1值
Tabby	SciTSR	0.914	0.910	0.912
	SciTSR-COMP	0.869	0.841	0.855
	ICDAR 2013	0.846	0.862	0.854
DeepDeSRT	SciTSR	0.906	0.887	0.890
	SciTSR-COMP	0.863	0.831	0.846
	ICDAR 2013	0.632	0.617	0.854
GraphTSR	SciTSR	0.959	0.948	0.953
	SciTSR-COMP	0.964	0.945	0.955
	ICDAR 2013	0.885	0.860	0.872
SEM	SciTSR	0.977	0.965	0.971
SEM	SciTSR-COMP	0.968	0.947	0.957
TabStruct-Net	ICDAR 2013	0.915	0.897	0.906
GEAN-TSR	SciTSR	0.976	0.990	0.983
	SciTSR-COMP	0.970	0.972	0.971
	ICDAR 2013	0.900	0.936	0.913

实验数据集	模型	精确率	召回率	F1值
SciTSR	DGCNN	0.966	0.983	0.974
	GEAN	0.971	0.985	0.977
	GEAN+文本	0.970	0.987	0.978
	GEAN+边	0.973	0.984	0.979
	本文模型	0.975	0.990	0.983
PubTabNet	DGCNN	0.973	0.952	0.962
	GEAN	0.976	0.963	0.969
	GEAN+文本	0.979	0.962	0.970
	GEAN+边	0.982	0.964	0.973
	本文模型	0.982	0.966	0.974

Table structure recognition model integrating edge features and attention

融合边特征与注意力的表格结构识别模型

RichHTML

PDF

Knowledge

Abstract

Cite this article

share this article

Figures/Tables 11

References 19

Related Articles 15

Recommended Articles

Metrics

[1]	Lubao LI, Tian CHEN, Fuji REN, Beibei LUO. Bimodal emotion recognition method based on graph neural network and attention [J]. Journal of Computer Applications, 2023, 43(3): 700-705.
[2]	Yu WANG, Yubo YUAN, Yi GUO, Jiajie ZHANG. Sentiment boosting model for emotion recognition in conversation text [J]. Journal of Computer Applications, 2023, 43(3): 706-712.
[3]	Ping WANG, Nan CHEN, Lei LU. Fall detection algorithm based on scene prior and attention guidance [J]. Journal of Computer Applications, 2023, 43(2): 529-535.
[4]	Gang CHEN, Yongwei LIAO, Zhenguo YANG, Wenying LIU. Image inpainting algorithm of multi-scale generative adversarial network based on multi-feature fusion [J]. Journal of Computer Applications, 2023, 43(2): 536-544.
[5]	Wenju LI, Gan ZHANG, Liu CUI, Wanghui CHU. Lightweight traffic sign recognition model based on coordinate attention [J]. Journal of Computer Applications, 2023, 43(2): 608-614.
[6]	Honggang YANG, Jiejie CHEN, Mengfei XU. Bilinear involution neural network for image classification of fundus diseases [J]. Journal of Computer Applications, 2023, 43(1): 259-264.
[7]	Yuchao LYU, Xi JIANG, Yinghao XU, Xijun ZHU. Improved brachial plexus nerve segmentation method based on multi-scale feature fusion [J]. Journal of Computer Applications, 2023, 43(1): 273-279.
[8]	Jun ZHANG, Pengli WU, Lukui SHI, Jin SHI, Bin PAN. Deep learning model for multi-station temperature prediction combined with MOD11A1 and surface meteorological station data [J]. Journal of Computer Applications, 2023, 43(1): 321-328.
[9]	Hangyuan DU, Sicong HAO, Wenjian WANG. Semi-supervised representation learning method combining graph auto-encoder and clustering [J]. Journal of Computer Applications, 2022, 42(9): 2643-2651.
[10]	Yuyu MENG, Jing GUO. Link prediction algorithm based on information entropy improved PCA model [J]. Journal of Computer Applications, 2022, 42(9): 2823-2829.
[11]	Zanxia QIANG, Xianfu BAO. Residual attention deraining network based on convolutional long short-term memory [J]. Journal of Computer Applications, 2022, 42(9): 2858-2864.
[12]	Xiaohan LI, Jun WANG, Huading JIA, Liu XIAO. Stock market volatility prediction method based on graph neural network with multi-attention mechanism [J]. Journal of Computer Applications, 2022, 42(7): 2265-2273.
[13]	Ruijie YANG, Guilin ZHENG. Face liveness detection based on InceptionV3 and feature fusion [J]. Journal of Computer Applications, 2022, 42(7): 2037-2042.
[14]	Dawei ZHANG, Xuchong LIU, Wei ZHOU, Zhuhui CHEN, Yao YU. Real-time traffic sign detection algorithm based on improved YOLOv3 [J]. Journal of Computer Applications, 2022, 42(7): 2219-2226.
[15]	Jiafan ZHOU, Yuefeng DU, Baoyan SONG, Xiaoguang LI, Azhu ZHAO, Xujie XIAO. MOOC video recommendation method based on meta-path attention mechanism [J]. Journal of Computer Applications, 2022, 42(6): 1808-1813.