Cross-modal person re-identification relation network based on dual-stream structure

doi:10.11772/j.issn.1001-9081.2022050665

Journal of Computer Applications ›› 2023, Vol. 43 ›› Issue (6): 1803-1810.DOI: 10.11772/j.issn.1001-9081.2022050665

Special Issue: 人工智能

• Artificial intelligence • Previous Articles Next Articles

Cross-modal person re-identification relation network based on dual-stream structure

Yubin GUO¹^,², Xiang WEN¹, Pan LIU¹, Ximing LI¹^,²()

^1.College of Mathematics and Informatics，South China Agricultural University，Guangzhou Guangdong 510642，China
^2.Guangzhou Key Laboratory of Intelligent Agriculture （South China Agricultural University），Guangzhou Guangdong 510642，China

Received:2022-05-08 Revised:2022-08-09 Accepted:2022-08-11 Online:2023-06-08 Published:2023-06-10
Contact: Ximing LI
About author:GUO Yubin， born in 1973， Ph. D.， associate professor. Her research interests include database， big data， data mining， deep learning.
WEN Xiang， born in 1998， M. S. candidate. His research interests include deep learning， data mining， computer vision.
LIU Pan， born in 1992， M. S. candidate. His research interests include deep learning， data mining， computer vision.
Supported by:
National Natural Science Foundation of China(61872152);Science and Technology Program of Guangzhou(201902010081)

基于双流结构的跨模态行人重识别关系网络

郭玉彬¹^,², 文向¹, 刘攀¹, 李西明¹^,²()

^1.华南农业大学数学与信息学院，广州 510642
^2.广州市智慧农业重点实验室（华南农业大学），广州 510642

通讯作者: 李西明
作者简介:郭玉彬（1973—），女，山东高唐人，副教授，博士，主要研究方向：数据库、大数据、数据挖掘、深度学习
文向（1998—），男，湖南长沙人，硕士研究生，主要研究方向：深度学习、数据挖掘、计算机视觉
刘攀（1992—），男，湖南耒阳人，硕士研究生，主要研究方向：深度学习、数据挖掘、计算机视觉
李西明（1974—），男，山东临清人，高级工程师，博士，主要研究方向：深度学习、计算机视觉、信息安全Email：liximing@scau.edu.cn。
基金资助:
国家自然科学基金资助项目(61872152);广州市科技计划项目(201902010081)

Abstract

Abstract:

In visible-infrared cross-modal person re-identification， the modal differences will lead to low identification accuracy. Therefore， a dual-stream structure based cross-modal person re-identification relation network， named IVRNBDS （Infrared and Visible Relation Network Based on Dual-stream Structure）， was proposed. Firstly， the dual-stream structure was used to extract the features of the visible light modal and the infrared modal person images respectively. Then， the feature map of the person image was divided into six segments horizontally to extract relationships between the local features of each segment and the features of other segments of the person and the relationship between the core features and average features of the person. Finally， when designing loss function， the Hetero-Center triplet Loss （HC Loss） function was introduced to relax the strict constraints of the ordinary triplet loss function， so that image features of different modals were able to be better mapped into the same feature space. Experimental results on public datasets SYSU-MM01 （SunYat-Sen University MultiModal re-identification） and RegDB （Dongguk Body-based person Recognition） show that the computational cost of IVRNBDS is slightly higher than those of the mainstream cross-modal person re-identification algorithms， but the proposed network has the Rank-1 （similarity Rank 1） and mAP （mean Average Precision） improved compared to the mainstream algorithms， increasing the recognition accuracy of the cross-modal people re-identification algorithm.

Key words: person re-identification, visible-infrared cross-modal, dual-stream structure, hetero-center triplet loss, local feature

摘要：

针对可见光-红外跨模态行人重识别中模态差异导致的识别精确率低的问题，提出了一种基于双流结构的跨模态行人重识别关系网络（IVRNBDS）。首先，利用双流结构分别提取可见光模态和红外模态行人图像的特征；然后，将行人图像的特征图水平切分为6个片段，以提取行人的每个片段的局部特征和其他片段的特征之间的关系，以及行人的核心特征和平均特征之间的关系；最后，在设计损失函数时，引入异质中心三元组损失（HC Loss）函数放松普通三元组损失函数的严格约束，从而使不同模态的图像特征可以更好地映射到同一特征空间中。在公开数据集SYSU-MM01（SunYat-Sen University MultiModal re-identification）和RegDB（Dongguk Body-based person Recognition）上的实验结果表明，虽然IVRNBDS的计算量略高于当前主流的跨模态行人重识别算法，但所提网络在相似度排名第1（Rank-1）指标和平均精度均值（mAP）指标上都有所提高，提高了跨模态行人重识别算法的识别精确率。

关键词: 行人重识别, 可见光-红外跨模态, 双流结构, 异质中心三元组损失, 局部特征

CLC Number:

TP391.4

Yubin GUO, Xiang WEN, Pan LIU, Ximing LI. Cross-modal person re-identification relation network based on dual-stream structure[J]. Journal of Computer Applications, 2023, 43(6): 1803-1810.

郭玉彬, 文向, 刘攀, 李西明. 基于双流结构的跨模态行人重识别关系网络[J]. 《计算机应用》唯一官方网站, 2023, 43(6): 1803-1810.

Figures/Tables 11

References 34

1	ZAJDEL W， ZIVKOVIC Z， KROSE B J A. Keeping track of humans： have I seen this person before？［C］// Proceedings of the 2005 IEEE International Conference on Robotics and Automation. Piscataway： IEEE， 2005： 2081-2086.
2	GHEISSARI N， SEBASTIAN T B， HARTLEY R. Person reidentification using spatiotemporal appearance［C］// Proceedings of the 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2006： 1528-1535. 10.1109/cvpr.2006.4
3	WU A C， ZHENG W S， YU H X， et al. RGB-infrared cross-modality person re-identification［C］// Proceedings of the 2017 IEEE International Conference on Computer Vision. Piscataway： IEEE， 2017： 5390-5399. 10.1109/iccv.2017.575
4	YE M， LAN X Y， LI J W， et al. Hierarchical discriminative learning for visible thermal person re-identification［C］// Proceedings of the 32nd AAAI Conference on Artificial Intelligence. Palo Alto， CA： AAAI Press， 2018： 7501-7508. 10.1609/aaai.v32i1.12293
5	SCHROFF F， KALENICHENKO D， PHILBIN J. FaceNet： a unified embedding for face recognition and clustering［C］// Proceedings of the 2015 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2015： 815-823. 10.1109/cvpr.2015.7298682
6	FENG Y J， CHEN F， JI Y M， et al. Efficient cross-modality graph reasoning for RGB-infrared person re-identification［J］. IEEE Signal Processing Letters， 2021， 28： 1425-1429. 10.1109/lsp.2021.3093865
7	WANG G A， ZHANG T Z， YANG Y， et al. Cross-modality paired-images generation for RGB-infrared person re-identification［C］// Proceedings of the 34th AAAI Conference on Artificial Intelligence. Palo Alto， CA： AAAI Press， 2020： 12144-12151. 10.1609/aaai.v34i07.6894
8	LIU H J， CHENG J， WANG W， et al. Enhancing the discriminative feature learning for visible-thermal cross-modality person re-identification ［J］. Neurocomputing， 2020， 398： 11-19. 10.1016/j.neucom.2020.01.089
9	YE M， SHEN J B， LIN G J， et al. Deep learning for person re-identification： a survey and outlook［J］. IEEE Transactions on Pattern Analysis and Machine Intelligence， 2022， 44（6）：2872-2893. 10.1109/tpami.2021.3054775
10	LIU H J， TAN X H， ZHOU X C. Parameter sharing exploration and hetero-center triplet loss for visible-thermal person re-identification ［J］. IEEE Transactions on Multimedia， 2021， 23： 4414-4425. 10.1109/tmm.2020.3042080
11	NGUYEN D T， HONG H G， KIM K W， et al. Person recognition system based on a combination of body images from visible light and thermal cameras ［J］. Sensors， 2017， 17（3）： No.605. 10.3390/s17030605
12	XIANG X Z， LV N， YU Z T， et al. Cross-modality person re-identification based on dual-path multi-branch network ［J］. IEEE Sensors Journal， 2019， 19（23）：11706-11713. 10.1109/jsen.2019.2936916
13	WANG G S， YUAN Y F， CHEN X， et al. Learning discriminative features with multiple granularities for person re-identification［C］// Proceedings of the 26th ACM International Conference on Multimedia. New York： ACM， 2018： 274-282. 10.1145/3240508.3240552
14	LU Y， WU Y， LIU B， et al. Cross-modality person re-identification with shared-specific feature transfer［C］// Proceedings of the 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2020：13376-13386. 10.1109/cvpr42600.2020.01339
15	ZHAO Y B， LIN J W， XUAN Q， et al. HPILN： a feature learning framework for cross-modality person re-identification［J］. IET Image Processing， 2019， 13（14）：2897-2904. 10.1049/iet-ipr.2019.0699
16	ZHANG S Z， YANG Y F， WANG P， et al. Attend to the difference： cross-modality person re-identification via contrastive correlation［J］. IEEE Transactions on Image Processing， 2021， 30： 8861-8872. 10.1109/tip.2021.3120881
17	YE M， WANG Z， LAN X Y， et al. Visible thermal person re-identification via dual-constrained top-ranking ［C］// Proceedings of the 27th International Joint Conference on Artificial Intelligence. California： ijcai.org， 2018： 1092-1099. 10.24963/ijcai.2018/152
18	ZHU Y X， YANG Z， WANG L， et al. Hetero-center loss for cross-modality person re-identification［J］. Neurocomputing， 2020， 386： 97-109. 10.1016/j.neucom.2019.12.100
19	HAO Y， WANG N N， LI J， et al. HSME： hypersphere manifold embedding for visible thermal person re-identification［C］// Proceedings of the 33rd AAAI Conference on Artificial Intelligence. Palo Alto， CA： AAAI Press， 2019： 8385-8392. 10.1609/aaai.v33i01.33018385
20	WANG Z X， WANG Z， ZHENG Y Q， et al. Learning to reduce dual-level discrepancy for infrared-visible person re-identification［C］// Proceedings of the 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2019： 618-626. 10.1109/cvpr.2019.00071
21	WANG G A， ZHANG T Z， CHENG J， et al. RGB-infrared cross-modality person re-identification via joint pixel and feature alignment［C］// Proceedings of the 2019 IEEE/CVF International Conference on Computer Vision. Piscataway： IEEE， 2019： 3622-3631. 10.1109/iccv.2019.00372
22	ZHANG Z Y， JIANG S， HUANG C Z T， et al. RGB-IR cross-modality person ReID based on teacher-student GAN model ［J］. Pattern Recognition Letters， 2021， 150： 155-161. 10.1016/j.patrec.2021.07.006
23	FAN X， LUO H， ZHANG C， et al. Cross-spectrum dual-subspace pairing for RGB-infrared cross-modality person re-identification［EB/OL］. （2020-02-29）［2022-02-22］.. 10.24963/ijcai.2020/143
24	CHOI S， LEE S， KIM Y， et al. Hi-CMD： hierarchical cross-modality disentanglement for visible-infrared person re-identification ［C］// Proceedings of the 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2020： 10254-10263. 10.1109/cvpr42600.2020.01027
25	SUN Y F， ZHENG L， YANG Y， et al. Beyond part models： person retrieval with refined part pooling （and a strong convolutional baseline）［C］// Proceedings of the 2018 European Conference on Computer Vision， LNCS 11208. Cham： Springer， 2018： 501-518.
26	袁铭阳，周长胜，黄宏博，等.卷积神经网络池化方法综述［J］. 软件工程与应用， 2020， 9（5）： 360-372.
	YUAN M Y， ZHOU C S， HUANG H B， et al. Survey on convolutional neural network pooling methods ［J］. Software Engineering and Applications， 2020， 9（5）： 360-372.
27	FU Y， WEI Y C， ZHOU Y Q， et al. Horizontal pyramid matching for person re-identification［C］// Proceedings of the 33rd AAAI Conference on Artificial Intelligence. Palo Alto， CA： AAAI Press， 2019： 8295-8302. 10.1609/aaai.v33i01.33018295
28	LI D G， WEI X， HONG X P， et al. Infrared-visible cross-modal person re-identification with an X modality ［C］// Proceedings of the 34th AAAI Conference on Artificial Intelligence. Palo Alto， CA： AAAI Press， 2020： 4610-4617. 10.1609/aaai.v34i04.5891
29	YE M， SHEN J B， CRANDALL D J， et al. Dynamic dual-attentive aggregation learning for visible-infrared person re-identification ［C］// Proceedings of the 2020 European Conference on Computer Vision， LNCS 12362. Cham： Springer， 2020： 229-247.
30	ZHANG L Y， DU G D， LIU F， et al. Global-local multiple granularity learning for cross-modality visible-infrared person reidentification ［J］. IEEE Transactions on Neural Networks and Learning Systems， 2021（Early Access）： 1-11. 10.1109/tnnls.2021.3085978
31	WU A C， ZHENG W S， GONG S G， et al. RGB-IR person re-identification by cross-modality similarity preservation［J］. International Journal of Computer Vision， 2020， 128（6）： 1765-1785. 10.1007/s11263-019-01290-1
32	ZHAO Z W， LIU B， CHU Q， et al. Joint color-irrelevant consistency learning and identity-aware modality adaptation for visible-infrared cross modality person re-identification［C］// Proceedings of the 35th AAAI Conference on Artificial Intelligence. Palo Alto， CA： AAAI Press， 2021： 3520-3528. 10.1609/aaai.v35i4.16466
33	CHEN Y， WAN L， LI Z， et al. Neural feature search for RGB-infrared person re-identification ［C］// Proceedings of the 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2021： 587-597. 10.1109/cvpr46437.2021.00065
34	HERMANS A， BEYER L， LEIBE B. In defense of the triplet loss for person re-identification［EB/OL］. （2017-11-21）［2022-03-11］.. 10.21203/rs.3.rs-1501673/v1

方法	来源	全局搜索		室内搜索
方法	来源	Rank1	mAP	Rank1	mAP
TONE	AAAI2018	12.52	14.42	—	—
BDTR	IJCAI2018	17.01	19.66	—	—
JSIA	AAAI2020	38.10	36.90	43.80	52.90
AlignGAN	ICCV2019	42.40	40.70	45.90	54.30
CMSP	IJCV2020	43.56	44.98	48.62	57.50
AGW	TPAMI2021	47.50	47.65	54.17	62.97
XIV	AAAI2020	49.92	50.73	—	—
DDAG	ECCV2020	54.75	53.02	61.02	67.98
NFS	CVPR2021	56.91	55.45	62.79	69.79
CICL	AAAI2021	57.20	59.30	66.60	74.70
cm-SSFT	CVPR2020	61.60	63.20	70.50	72.60
GLMC	TNNLS2021	64.37	63.43	67.35	74.02
IVRNBDS	—	70.13	65.33	70.36	73.15

方法	来源	全局搜索		室内搜索
方法	来源	Rank1	mAP	Rank1	mAP
TONE	AAAI2018	12.52	14.42	—	—
BDTR	IJCAI2018	17.01	19.66	—	—
JSIA	AAAI2020	38.10	36.90	43.80	52.90
AlignGAN	ICCV2019	42.40	40.70	45.90	54.30
CMSP	IJCV2020	43.56	44.98	48.62	57.50
AGW	TPAMI2021	47.50	47.65	54.17	62.97
XIV	AAAI2020	49.92	50.73	—	—
DDAG	ECCV2020	54.75	53.02	61.02	67.98
NFS	CVPR2021	56.91	55.45	62.79	69.79
CICL	AAAI2021	57.20	59.30	66.60	74.70
cm-SSFT	CVPR2020	61.60	63.20	70.50	72.60
GLMC	TNNLS2021	64.37	63.43	67.35	74.02
IVRNBDS	—	70.13	65.33	70.36	73.15

方法	来源	全局搜索		室内搜索
方法	来源	Rank1	mAP	Rank1	mAP
TONE	AAAI2018	16.87	14.92	13.86	16.98
BDTR	IJCAI2018	33.47	31.83	32.72	31.10
JSIA	AAAI2020	48.50	49.30	48.10	48.90
AlignGAN	ICCV2019	57.90	53.60	56.30	53.40
CMSP	IJCV2020	65.07	64.50	—	—
AGW	TPAMI2021	70.05	66.37	—	—
XIV	AAAI2020	62.21	60.18	—	—
DDAG	ECCV2020	69.34	63.46	68.06	61.80
NFS	CVPR2021	80.54	72.10	77.95	69.79
CICL	AAAI2021	78.80	69.40	77.90	69.40
cm-SSFT	CVPR2020	72.30	72.90	71.00	71.70
GLMC	TNNLS2021	91.84	81.42	91.12	81.06
IVRNBDS	—	92.34	92.58	91.35	91.78

方法	来源	全局搜索		室内搜索
方法	来源	Rank1	mAP	Rank1	mAP
TONE	AAAI2018	16.87	14.92	13.86	16.98
BDTR	IJCAI2018	33.47	31.83	32.72	31.10
JSIA	AAAI2020	48.50	49.30	48.10	48.90
AlignGAN	ICCV2019	57.90	53.60	56.30	53.40
CMSP	IJCV2020	65.07	64.50	—	—
AGW	TPAMI2021	70.05	66.37	—	—
XIV	AAAI2020	62.21	60.18	—	—
DDAG	ECCV2020	69.34	63.46	68.06	61.80
NFS	CVPR2021	80.54	72.10	77.95	69.79
CICL	AAAI2021	78.80	69.40	77.90	69.40
cm-SSFT	CVPR2020	72.30	72.90	71.00	71.70
GLMC	TNNLS2021	91.84	81.42	91.12	81.06
IVRNBDS	—	92.34	92.58	91.35	91.78

方法配置	全局搜索		室内搜索
方法配置	Rank-1	mAP	Rank-1	mAP
B	47.50	47.65	54.17	62.97
B+ORRM	62.68	57.51	62.41	67.17
B+GCRM	62.00	59.44	67.37	70.14
B+HC_Tri Loss	61.52	58.52	65.41	69.30
IVRNBDS	70.13	65.33	70.36	73.15

Cross-modal person re-identification relation network based on dual-stream structure

基于双流结构的跨模态行人重识别关系网络

RichHTML

PDF

Knowledge

Abstract

Cite this article

share this article

Figures/Tables 11

References 34

Related Articles 15

Recommended Articles

Metrics

方法	训练时间
A+三元组损失函数	172.24
A+批量难样本三元组损失函数	235.36
IVRNBDS	164.98

[1]	Jieru JIA, Jianchao YANG, Shuorui ZHANG, Tao YAN, Bin CHEN. Unsupervised person re-identification based on self-distilled vision Transformer [J]. Journal of Computer Applications, 2024, 44(9): 2893-2902.
[2]	Cui WANG, Miaolei DENG, Dexian ZHANG, Lei LI, Xiaoyan YANG. Review of end-to-end person search algorithms based on images [J]. Journal of Computer Applications, 2024, 44(8): 2544-2550.
[3]	Yuwei DING, Hongbo SHI, Jie LI, Min LIANG. Image denoising network based on local and global feature decoupling [J]. Journal of Computer Applications, 2024, 44(8): 2571-2579.
[4]	Jiaming HE, Jucheng YANG, Chao WU, Xiaoning YAN, Nenghua XU. Person re-identification method based on multi-modal graph convolutional neural network [J]. Journal of Computer Applications, 2023, 43(7): 2182-2189.
[5]	Zhixiong ZHENG, Jianhua LIU, Shuihua SUN, Ge XU, Honghui LIN. Aspect-based sentiment analysis model fused with multi-window local information [J]. Journal of Computer Applications, 2023, 43(6): 1796-1802.
[6]	Yingmao YAO, Xiaoyan JIANG. Video-based person re-identification method based on graph convolution network and self-attention graph pooling [J]. Journal of Computer Applications, 2023, 43(3): 728-735.
[7]	Jie SUN, Shaoxin WU, Xuejun WANG, Jing HUA. Efficient person search algorithm and optimization with Sophon SC5+ chip architecture [J]. Journal of Computer Applications, 2023, 43(3): 744-751.
[8]	Jianhua ZHONG, Chuangyi QIU, Jianshu CHAO, Ruicheng MING, Jianfeng ZHONG. Cloth-changing person re-identification model based on semantic-guided self-attention network [J]. Journal of Computer Applications, 2023, 43(12): 3719-3726.
[9]	Feiyu LIAN, Liang ZHANG, Jiedong WANG, Yukang JIN, Yu CHAI. Outdoor scene point cloud segmentation model based on graph model and attention mechanism [J]. Journal of Computer Applications, 2023, 43(12): 3911-3917.
[10]	Tianhao QIU, Shurong CHEN. EfficientNet based dual-branch multi-scale integrated learning for pedestrian re-identification [J]. Journal of Computer Applications, 2022, 42(7): 2065-2071.
[11]	Daili CHEN, Guoliang XU. Cross-domain person re-identification method based on attention mechanism with learning intra-domain variance [J]. Journal of Computer Applications, 2022, 42(5): 1391-1397.
[12]	Yuchang YIN, Hongyuan WANG, Li CHEN, Zundeng FENG, Yu XIAO. One-shot video-based person re-identification with multi-loss learning and joint metric [J]. Journal of Computer Applications, 2022, 42(3): 764-769.
[13]	Dawei LI, Zhiyong ZENG. Cross-modal person re-identification model based on dynamic dual-attention mechanism [J]. Journal of Computer Applications, 2022, 42(10): 3200-3208.
[14]	LIU Ziyan, ZHU Mingcheng, YUAN Lei, MA Shanshan, CHEN Lingzhouting. Video person re-identification based on non-local attention and multi-feature fusion [J]. Journal of Computer Applications, 2021, 41(2): 530-536.
[15]	Yunpeng GONG, Zhiyong ZENG, Feng YE. Person re-identification method based on grayscale feature enhancement [J]. Journal of Computer Applications, 2021, 41(12): 3590-3595.

方法	计算量/10⁹	推理时间/ms
AWG	10.81	4.49
DDAG	12.93	4.58
IVRNBDS	12.96	5.15

方法	计算量/10⁹	推理时间/ms
AWG	10.81	4.49
DDAG	12.93	4.58
IVRNBDS	12.96	5.15