Table structure recognition model integrating edge features and attention

doi:10.11772/j.issn.1001-9081.2022010053

Journal of Computer Applications ›› 2023, Vol. 43 ›› Issue (3): 752-758.DOI: 10.11772/j.issn.1001-9081.2022010053

Special Issue: 人工智能

• Artificial intelligence • Previous Articles Next Articles

Table structure recognition model integrating edge features and attention

Xueqiang LYU¹, Yunan ZHANG¹, Jing HAN¹(), Yunpeng CUI², Huan LI²

^1.Beijing Key Laboratory of Internet Culture and Digital Dissemination Research （Beijing Information Science and Technology University），Beijing 100101，China
^2.Key Laboratory of Agricultural Big Data，Ministry of Agriculture （Agricultural Information Institute of Chinese Academy of Agricultural Science），Beijing 100081，China

Received:2022-01-17 Revised:2022-04-06 Accepted:2022-04-11 Online:2022-04-26 Published:2023-03-10
Contact: Jing HAN
About author:LYU Xueqiang， born in 1970， Ph. D.， professor. His research interests include multimedia information processing.
ZHANG Yunan， born in 1996， M. S. candidate. His research interests include computer vision.
CUI Yunpeng， born in 1972， Ph. D.， research fellow. His research interests include agricultural information technology.
LI Huan， born in 1992， M. S.， research fellow. Her research interests include data mining.
Supported by:
National Natural Science Foundation of China(62171043)

融合边特征与注意力的表格结构识别模型

吕学强¹, 张煜楠¹, 韩晶¹(), 崔运鹏², 李欢²

^1.网络文化与数字传播北京市重点实验室(北京信息科技大学), 北京 100101
^2.农业农村部农业大数据重点实验室(中国农业科学院农业信息研究所), 北京 100081

通讯作者: 韩晶
作者简介:吕学强（1970—），男，辽宁抚顺人，教授，博士，主要研究方向：多媒体信息处理
张煜楠（1996—），男，北京人，硕士研究生，主要研究方向：计算机视觉
韩晶（1990—），女，河北邯郸人，助理研究员，博士，主要研究方向：图像处理
崔运鹏（1972—），男，吉林和龙人，研究员，博士，主要研究方向：农业信息技术
李欢（1992—），女，湖北红安人，研究员，硕士，主要研究方向：数据挖掘。
基金资助:
国家自然科学基金资助项目(62171043)

Abstract

Abstract:

Aiming at the problems in the existing methods such as dependence on prior knowledge， insufficient robustness， and insufficient expression ability in table structure recognition， a new table structure recognition model integrating edge features and attention was proposed， namely Graph Edge-Attention Network based Table Structure Recognition model （GEAN-TSR）. Firstly， Graph Edge-Attention Network （GEAN） was proposed as the backbone network， and based on edge convolution structure， the graph attention mechanism was introduced and improved to aggregate graph node features， so as to solve the problem of information loss in the process of feature extraction of graph network， and improve the expression ability of graph network. Then， an edge feature fusion module was introduced to fuse the shallow graph node information with the graph network output to enhance the local information extraction and expression abilities of the graph network. Finally， the graph node text features extracted by Gated Recurrent Unit （GRU） were integrated into the text feature fusion module for edge’s classification and prediction. Comparative experiments on Scientific paper Table Structure Recognition-COMPlicated （SciTSR-COMP） dataset show that the recall and F1 score of GEAN-TSR are increased by 2.5 and 1.4 percentage points， respectively in comparison with the existing optimal model Split， Embed and Merge （SEM）. Ablation experiments show that all the indicators of GEAN-TSR have achieved the optimal values after using the feature fusion module， proving the effectiveness of the module. Experimental results show that GEAN-TSR can effectively improve the network performance and better complete the task of table structure recognition.

Key words: graph neural network, graph attention network, feature fusion, table structure recognition, table parsing

摘要：

针对现有方法在表格结构识别问题中存在的先验知识依赖、鲁棒性不足、表达能力不足等问题，提出一种新的融合边特征与注意力的表格结构识别模型——GEAN-TSR。首先，提出图边注意力网络（GEAN）并作为模型的主干网络，在边卷积结构的基础上引入并改进图注意力机制聚合图节点特征，解决图网络在特征提取过程中的信息损失的问题，提高图网络的表达能力；然后，引入边特征融合模块融合浅层图节点信息与图网络输出，增强图网络的局部信息提取能力与表达能力；最后，将门控循环单元（GRU）提取的图节点文本特征融入文本特征融合模块对边进行分类预测。在SciTSR-COMP数据集上的对比实验中，相较于目前最优的模型SEM，GEAN-TSR的召回率与F1值分别提升2.5与1.4个百分点。在消融实验中，GEAN-TSR采用特征融合模块后，所有指标都取得了最优值，验证了模块的有效性。实验结果表明，GEAN-TSR能够有效提升网络性能，更好地完成表格结构识别任务。

关键词: 图神经网络, 图注意力网络, 特征融合, 表格结构识别, 表格解析

CLC Number:

TP391.4

Xueqiang LYU, Yunan ZHANG, Jing HAN, Yunpeng CUI, Huan LI. Table structure recognition model integrating edge features and attention[J]. Journal of Computer Applications, 2023, 43(3): 752-758.

吕学强, 张煜楠, 韩晶, 崔运鹏, 李欢. 融合边特征与注意力的表格结构识别模型[J]. 《计算机应用》唯一官方网站, 2023, 43(3): 752-758.

Figures/Tables 11

References 19

1	ITONORI K. Table structure recognition based on textblock arrangement and ruled line position ［C］// Proceedings of the 2nd International Conference on Document Analysis and Recognition. Piscataway： IEEE， 1993： 765-768. 10.1109/icdar.1993.395796
2	SHIGAROV A O， FEDOROV R K. Simple algorithm for page layout analysis［J］. Pattern Recognition and Image Analysis， 2011， 21（2）：324-327. 10.1134/s1054661811021008
3	RAMEL J Y， CRUCIANU M， VINCENT N， et al. Detection， extraction and representation of tables ［C］// Proceedings of the 7th International Conference on Document Analysis and Recognition. Piscataway： IEEE， 2013： 374-378.
4	SHIGAROV A， MIKHAILOV A， ALTAEV A. Configurable table structure recognition in untagged PDF documents ［C］// Proceedings of the 2016 ACM Symposium on Document Engineering. New York： ACM， 2016：119-122. 10.1145/2960811.2967152
5	PALIWAL S S， VISHWANATH D， RAHUL R， et al. TableNet： deep learning model for end-to-end table detection and tabular data extraction from scanned document images ［C］// Proceedings of the 15th IAPR International Conference on Document Analysis and Recognition. Piscataway： IEEE， 2019：128-133. 10.1109/icdar.2019.00029
6	TENSMEYER C， MORARIU V I， PRICE B， et al. Deep splitting and merging for table structure decomposition ［C］// Proceedings of the 15th IAPR International Conference on Document Analysis and Recognition. Piscataway： IEEE， 2019： 114-121. 10.1109/icdar.2019.00027
7	SIDDIQUI S A， FATEH I A， RIZVI S T R， et al. DeepTabStR： deep learning based table structure recognition ［C］// Proceedings of the 2019 International Conference on Document Analysis and Recognition. Piscataway： IEEE， 2019： 1403-1409. 10.1109/icdar.2019.00226
8	QIAO L， LI Z S， CHENG Z Z， et al. LGPMA： complicated table structure recognition with local and global pyramid mask alignment［C］// Proceedings of the 2021 International Conference on Document Analysis and Recognition， LNCS 12821. Cham： Springer， 2021： 99-114. 10.48550/arXiv.2105.06224
9	QASIM S R， MAHMOOD H， SHAFAIT F. Rethinking table recognition using graph neural networks ［C］// Proceedings of the 15th IAPR International Conference on Document Analysis and Recognition. Piscataway： IEEE， 2019： 142-147. 10.1109/icdar.2019.00031
10	LI Y R， HUANG Z， YAN J C， et al. GFTE： Graph-based Financial Table Extraction［C］// Proceedings of the 2021 International Conference on Pattern Recognition， LNCS 12662. Cham： Springer， 2021： 644-658.
11	徐冰冰，岑科廷，黄俊杰，等. 图卷积神经网络综述［J］. 计算机学报， 2020， 43（5）： 755-780. 10.11897/SP.J.1016.2020.00755
	XU B B， CEN K T， HUANG J J， et al. A survey on graph convolutional neural network［J］. Chinese Journal of Computers， 2020， 43（5）： 755-780. 10.11897/SP.J.1016.2020.00755
12	WANG Y， SUN Y， LIU Z， et al. Dynamic graph CNN for learning on point clouds ［J］. ACM Transactions on Graphics， 2019， 38（5）：1-12. 10.1145/3326362
13	VELIČKOVIĆ P， CASANOVA A， LIO P， et al. Graph attention networks ［EB/OL］. （2018-02-04）［2021-08-14］. .
14	BRODY S， ALON U， YAHAV E. How attentive are graph attention networks？［EB/OL］. （2021-08-11）［2021-08-20］. .
15	GÖBEL M， HASSAN T， ORO E， et al. A methodology for evaluating algorithms for table understanding in PDF documents ［C］// Proceedings of the 2012 ACM symposium on Document engineering. New York： ACM， 2012： 45-48. 10.1145/2361354.2361365
16	SCHREIBER S， AGNE S， WOLF I， et al. DeepDeSRT： deep learning for detection and structure recognition of tables in document images ［C］// Proceedings of the14th IAPR International Conference on Document Analysis and Recognition. Piscataway： IEEE， 2017： 1162-1167. 10.1109/icdar.2017.192
17	CHI Z， HUANG H， XU H D， et al. Complicated table structure recognition ［EB/OL］. （2019-08-28）［2021-09-10］. .
18	ZHANG Z R， ZHANG J S， DU J， et al. Split， embed and merge： an accurate table structure recognizer［J］. Pattern Recognition， 2022， 126： No.108565. 10.1016/j.patcog.2022.108565
19	RAJA S， MONDAL A， JAWAHAR C V. Table structure recognition using top-down and bottom-up cues ［C］// Proceedings of the 2020 European Conference on Computer Vision. Cham： Springer， 2020： 70-86. 10.1007/978-3-030-58604-1_5

主干网模型	模型大小/MB	参数量/10⁶	平均预测时间/s
GCN	9.2	2.40	0.002 9
GAT	9.4	2.40	0.003 0
DGCNN	10.4	2.72	0.002 4
GEAN	10.4	2.72	0.002 9

主干网模型	模型大小/MB	参数量/10⁶	平均预测时间/s
GCN	9.2	2.40	0.002 9
GAT	9.4	2.40	0.003 0
DGCNN	10.4	2.72	0.002 4
GEAN	10.4	2.72	0.002 9

模型	测试集	精确率	召回率	F1值
Tabby	SciTSR	0.914	0.910	0.912
	SciTSR-COMP	0.869	0.841	0.855
	ICDAR 2013	0.846	0.862	0.854
DeepDeSRT	SciTSR	0.906	0.887	0.890
	SciTSR-COMP	0.863	0.831	0.846
	ICDAR 2013	0.632	0.617	0.854
GraphTSR	SciTSR	0.959	0.948	0.953
	SciTSR-COMP	0.964	0.945	0.955
	ICDAR 2013	0.885	0.860	0.872
SEM	SciTSR	0.977	0.965	0.971
SEM	SciTSR-COMP	0.968	0.947	0.957
TabStruct-Net	ICDAR 2013	0.915	0.897	0.906
GEAN-TSR	SciTSR	0.976	0.990	0.983
	SciTSR-COMP	0.970	0.972	0.971
	ICDAR 2013	0.900	0.936	0.913

模型	测试集	精确率	召回率	F1值
Tabby	SciTSR	0.914	0.910	0.912
	SciTSR-COMP	0.869	0.841	0.855
	ICDAR 2013	0.846	0.862	0.854
DeepDeSRT	SciTSR	0.906	0.887	0.890
	SciTSR-COMP	0.863	0.831	0.846
	ICDAR 2013	0.632	0.617	0.854
GraphTSR	SciTSR	0.959	0.948	0.953
	SciTSR-COMP	0.964	0.945	0.955
	ICDAR 2013	0.885	0.860	0.872
SEM	SciTSR	0.977	0.965	0.971
SEM	SciTSR-COMP	0.968	0.947	0.957
TabStruct-Net	ICDAR 2013	0.915	0.897	0.906
GEAN-TSR	SciTSR	0.976	0.990	0.983
	SciTSR-COMP	0.970	0.972	0.971
	ICDAR 2013	0.900	0.936	0.913

实验数据集	模型	精确率	召回率	F1值
SciTSR	DGCNN	0.966	0.983	0.974
	GEAN	0.971	0.985	0.977
	GEAN+文本	0.970	0.987	0.978
	GEAN+边	0.973	0.984	0.979
	本文模型	0.975	0.990	0.983
PubTabNet	DGCNN	0.973	0.952	0.962
	GEAN	0.976	0.963	0.969
	GEAN+文本	0.979	0.962	0.970
	GEAN+边	0.982	0.964	0.973
	本文模型	0.982	0.966	0.974

Table structure recognition model integrating edge features and attention

融合边特征与注意力的表格结构识别模型

RichHTML

PDF

Knowledge

Abstract

Cite this article

share this article

Figures/Tables 11

References 19

Related Articles 15

Recommended Articles

Metrics

[1]	Hang YANG, Wanggen LI, Gensheng ZHANG, Zhige WANG, Xin KAI. Multi-layer information interactive fusion algorithm based on graph neural network for session-based recommendation [J]. Journal of Computer Applications, 2024, 44(9): 2719-2725.
[2]	Xingyao YANG, Yu CHEN, Jiong YU, Zulian ZHANG, Jiaying CHEN, Dongxiao WANG. Recommendation model combining self-features and contrastive learning [J]. Journal of Computer Applications, 2024, 44(9): 2704-2710.
[3]	Yexin PAN, Zhe YANG. Optimization model for small object detection based on multi-level feature bidirectional fusion [J]. Journal of Computer Applications, 2024, 44(9): 2871-2877.
[4]	Tingjie TANG, Jiajin HUANG, Jin QIN. Session-based recommendation with graph auxiliary learning [J]. Journal of Computer Applications, 2024, 44(9): 2711-2718.
[5]	Yu DU, Yan ZHU. Constructing pre-trained dynamic graph neural network to predict disappearance of academic cooperation behavior [J]. Journal of Computer Applications, 2024, 44(9): 2726-2731.
[6]	Fan YANG, Yao ZOU, Mingzhi ZHU, Zhenwei MA, Dawei CHENG, Changjun JIANG. Credit card fraud detection model based on graph attention Transformation neural network [J]. Journal of Computer Applications, 2024, 44(8): 2634-2642.
[7]	Ruihua LIU, Zihe HAO, Yangyang ZOU. Gait recognition algorithm based on multi-layer refined feature fusion [J]. Journal of Computer Applications, 2024, 44(7): 2250-2257.
[8]	Xinrui LIN, Xiaofei WANG, Yan ZHU. Academic anomaly citation group detection based on local extended community detection [J]. Journal of Computer Applications, 2024, 44(6): 1855-1861.
[9]	Tianci KE, Jianhua LIU, Shuihua SUN, Zhixiong ZHENG, Zijie CAI. Aspect-level sentiment analysis model combining strong association dependency and concise syntax [J]. Journal of Computer Applications, 2024, 44(6): 1786-1795.
[10]	Mengyuan HUANG, Kan CHANG, Mingyang LING, Xinjie WEI, Tuanfa QIN. Progressive enhancement algorithm for low-light images based on layer guidance [J]. Journal of Computer Applications, 2024, 44(6): 1911-1919.
[11]	Yue LIU, Fang LIU, Aoyun WU, Qiuyue CHAI, Tianxiao WANG. 3D object detection network based on self-attention mechanism and graph convolution [J]. Journal of Computer Applications, 2024, 44(6): 1972-1977.
[12]	Guijin HAN, Xinyuan ZHANG, Wentao ZHANG, Ya HUANG. Self-supervised image registration algorithm based on multi-feature fusion [J]. Journal of Computer Applications, 2024, 44(5): 1597-1604.
[13]	Hongtian LI, Xinhao SHI, Weiguo PAN, Cheng XU, Bingxin XU, Jiazheng YUAN. Few-shot object detection via fusing multi-scale and attention mechanism [J]. Journal of Computer Applications, 2024, 44(5): 1437-1444.
[14]	Jiong WANG, Taotao TANG, Caiyan JIA. PAGCL： positive augmentation graph contrastive learning recommendation method without negative sampling [J]. Journal of Computer Applications, 2024, 44(5): 1485-1492.
[15]	Xin LI, Qiao MENG, Junyi HUANGFU, Lingchen MENG. YOLOv5 multi-attribute classification based on separable label collaborative learning [J]. Journal of Computer Applications, 2024, 44(5): 1619-1628.