基于双流结构的跨模态行人重识别关系网络

doi:10.11772/j.issn.1001-9081.2022050665

《计算机应用》唯一官方网站 ›› 2023, Vol. 43 ›› Issue (6): 1803-1810.DOI: 10.11772/j.issn.1001-9081.2022050665

基于双流结构的跨模态行人重识别关系网络

郭玉彬¹^,², 文向¹, 刘攀¹, 李西明¹^,²()

^1.华南农业大学数学与信息学院，广州 510642
^2.广州市智慧农业重点实验室（华南农业大学），广州 510642

收稿日期:2022-05-08 修回日期:2022-08-09 接受日期:2022-08-11 发布日期:2023-06-08 出版日期:2023-06-10
通讯作者: 李西明
作者简介:郭玉彬（1973—），女，山东高唐人，副教授，博士，主要研究方向：数据库、大数据、数据挖掘、深度学习
文向（1998—），男，湖南长沙人，硕士研究生，主要研究方向：深度学习、数据挖掘、计算机视觉
刘攀（1992—），男，湖南耒阳人，硕士研究生，主要研究方向：深度学习、数据挖掘、计算机视觉
李西明（1974—），男，山东临清人，高级工程师，博士，主要研究方向：深度学习、计算机视觉、信息安全Email：liximing@scau.edu.cn。
基金资助:
国家自然科学基金资助项目(61872152);广州市科技计划项目(201902010081)

Cross-modal person re-identification relation network based on dual-stream structure

Yubin GUO¹^,², Xiang WEN¹, Pan LIU¹, Ximing LI¹^,²()

^1.College of Mathematics and Informatics，South China Agricultural University，Guangzhou Guangdong 510642，China
^2.Guangzhou Key Laboratory of Intelligent Agriculture （South China Agricultural University），Guangzhou Guangdong 510642，China

Received:2022-05-08 Revised:2022-08-09 Accepted:2022-08-11 Online:2023-06-08 Published:2023-06-10
Contact: Ximing LI
About author:GUO Yubin， born in 1973， Ph. D.， associate professor. Her research interests include database， big data， data mining， deep learning.
WEN Xiang， born in 1998， M. S. candidate. His research interests include deep learning， data mining， computer vision.
LIU Pan， born in 1992， M. S. candidate. His research interests include deep learning， data mining， computer vision.
Supported by:
National Natural Science Foundation of China(61872152);Science and Technology Program of Guangzhou(201902010081)

摘要/Abstract

摘要：

针对可见光-红外跨模态行人重识别中模态差异导致的识别精确率低的问题，提出了一种基于双流结构的跨模态行人重识别关系网络（IVRNBDS）。首先，利用双流结构分别提取可见光模态和红外模态行人图像的特征；然后，将行人图像的特征图水平切分为6个片段，以提取行人的每个片段的局部特征和其他片段的特征之间的关系，以及行人的核心特征和平均特征之间的关系；最后，在设计损失函数时，引入异质中心三元组损失（HC Loss）函数放松普通三元组损失函数的严格约束，从而使不同模态的图像特征可以更好地映射到同一特征空间中。在公开数据集SYSU-MM01（SunYat-Sen University MultiModal re-identification）和RegDB（Dongguk Body-based person Recognition）上的实验结果表明，虽然IVRNBDS的计算量略高于当前主流的跨模态行人重识别算法，但所提网络在相似度排名第1（Rank-1）指标和平均精度均值（mAP）指标上都有所提高，提高了跨模态行人重识别算法的识别精确率。

关键词: 行人重识别, 可见光-红外跨模态, 双流结构, 异质中心三元组损失, 局部特征

Abstract:

In visible-infrared cross-modal person re-identification， the modal differences will lead to low identification accuracy. Therefore， a dual-stream structure based cross-modal person re-identification relation network， named IVRNBDS （Infrared and Visible Relation Network Based on Dual-stream Structure）， was proposed. Firstly， the dual-stream structure was used to extract the features of the visible light modal and the infrared modal person images respectively. Then， the feature map of the person image was divided into six segments horizontally to extract relationships between the local features of each segment and the features of other segments of the person and the relationship between the core features and average features of the person. Finally， when designing loss function， the Hetero-Center triplet Loss （HC Loss） function was introduced to relax the strict constraints of the ordinary triplet loss function， so that image features of different modals were able to be better mapped into the same feature space. Experimental results on public datasets SYSU-MM01 （SunYat-Sen University MultiModal re-identification） and RegDB （Dongguk Body-based person Recognition） show that the computational cost of IVRNBDS is slightly higher than those of the mainstream cross-modal person re-identification algorithms， but the proposed network has the Rank-1 （similarity Rank 1） and mAP （mean Average Precision） improved compared to the mainstream algorithms， increasing the recognition accuracy of the cross-modal people re-identification algorithm.

Key words: person re-identification, visible-infrared cross-modal, dual-stream structure, hetero-center triplet loss, local feature

中图分类号:

TP391.4

郭玉彬, 文向, 刘攀, 李西明. 基于双流结构的跨模态行人重识别关系网络[J]. 计算机应用, 2023, 43(6): 1803-1810.

Yubin GUO, Xiang WEN, Pan LIU, Ximing LI. Cross-modal person re-identification relation network based on dual-stream structure[J]. Journal of Computer Applications, 2023, 43(6): 1803-1810.

图/表 11

参考文献 34

1	ZAJDEL W， ZIVKOVIC Z， KROSE B J A. Keeping track of humans： have I seen this person before？［C］// Proceedings of the 2005 IEEE International Conference on Robotics and Automation. Piscataway： IEEE， 2005： 2081-2086.
2	GHEISSARI N， SEBASTIAN T B， HARTLEY R. Person reidentification using spatiotemporal appearance［C］// Proceedings of the 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2006： 1528-1535. 10.1109/cvpr.2006.4
3	WU A C， ZHENG W S， YU H X， et al. RGB-infrared cross-modality person re-identification［C］// Proceedings of the 2017 IEEE International Conference on Computer Vision. Piscataway： IEEE， 2017： 5390-5399. 10.1109/iccv.2017.575
4	YE M， LAN X Y， LI J W， et al. Hierarchical discriminative learning for visible thermal person re-identification［C］// Proceedings of the 32nd AAAI Conference on Artificial Intelligence. Palo Alto， CA： AAAI Press， 2018： 7501-7508. 10.1609/aaai.v32i1.12293
5	SCHROFF F， KALENICHENKO D， PHILBIN J. FaceNet： a unified embedding for face recognition and clustering［C］// Proceedings of the 2015 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2015： 815-823. 10.1109/cvpr.2015.7298682
6	FENG Y J， CHEN F， JI Y M， et al. Efficient cross-modality graph reasoning for RGB-infrared person re-identification［J］. IEEE Signal Processing Letters， 2021， 28： 1425-1429. 10.1109/lsp.2021.3093865
7	WANG G A， ZHANG T Z， YANG Y， et al. Cross-modality paired-images generation for RGB-infrared person re-identification［C］// Proceedings of the 34th AAAI Conference on Artificial Intelligence. Palo Alto， CA： AAAI Press， 2020： 12144-12151. 10.1609/aaai.v34i07.6894
8	LIU H J， CHENG J， WANG W， et al. Enhancing the discriminative feature learning for visible-thermal cross-modality person re-identification ［J］. Neurocomputing， 2020， 398： 11-19. 10.1016/j.neucom.2020.01.089
9	YE M， SHEN J B， LIN G J， et al. Deep learning for person re-identification： a survey and outlook［J］. IEEE Transactions on Pattern Analysis and Machine Intelligence， 2022， 44（6）：2872-2893. 10.1109/tpami.2021.3054775
10	LIU H J， TAN X H， ZHOU X C. Parameter sharing exploration and hetero-center triplet loss for visible-thermal person re-identification ［J］. IEEE Transactions on Multimedia， 2021， 23： 4414-4425. 10.1109/tmm.2020.3042080
11	NGUYEN D T， HONG H G， KIM K W， et al. Person recognition system based on a combination of body images from visible light and thermal cameras ［J］. Sensors， 2017， 17（3）： No.605. 10.3390/s17030605
12	XIANG X Z， LV N， YU Z T， et al. Cross-modality person re-identification based on dual-path multi-branch network ［J］. IEEE Sensors Journal， 2019， 19（23）：11706-11713. 10.1109/jsen.2019.2936916
13	WANG G S， YUAN Y F， CHEN X， et al. Learning discriminative features with multiple granularities for person re-identification［C］// Proceedings of the 26th ACM International Conference on Multimedia. New York： ACM， 2018： 274-282. 10.1145/3240508.3240552
14	LU Y， WU Y， LIU B， et al. Cross-modality person re-identification with shared-specific feature transfer［C］// Proceedings of the 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2020：13376-13386. 10.1109/cvpr42600.2020.01339
15	ZHAO Y B， LIN J W， XUAN Q， et al. HPILN： a feature learning framework for cross-modality person re-identification［J］. IET Image Processing， 2019， 13（14）：2897-2904. 10.1049/iet-ipr.2019.0699
16	ZHANG S Z， YANG Y F， WANG P， et al. Attend to the difference： cross-modality person re-identification via contrastive correlation［J］. IEEE Transactions on Image Processing， 2021， 30： 8861-8872. 10.1109/tip.2021.3120881
17	YE M， WANG Z， LAN X Y， et al. Visible thermal person re-identification via dual-constrained top-ranking ［C］// Proceedings of the 27th International Joint Conference on Artificial Intelligence. California： ijcai.org， 2018： 1092-1099. 10.24963/ijcai.2018/152
18	ZHU Y X， YANG Z， WANG L， et al. Hetero-center loss for cross-modality person re-identification［J］. Neurocomputing， 2020， 386： 97-109. 10.1016/j.neucom.2019.12.100
19	HAO Y， WANG N N， LI J， et al. HSME： hypersphere manifold embedding for visible thermal person re-identification［C］// Proceedings of the 33rd AAAI Conference on Artificial Intelligence. Palo Alto， CA： AAAI Press， 2019： 8385-8392. 10.1609/aaai.v33i01.33018385
20	WANG Z X， WANG Z， ZHENG Y Q， et al. Learning to reduce dual-level discrepancy for infrared-visible person re-identification［C］// Proceedings of the 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2019： 618-626. 10.1109/cvpr.2019.00071
21	WANG G A， ZHANG T Z， CHENG J， et al. RGB-infrared cross-modality person re-identification via joint pixel and feature alignment［C］// Proceedings of the 2019 IEEE/CVF International Conference on Computer Vision. Piscataway： IEEE， 2019： 3622-3631. 10.1109/iccv.2019.00372
22	ZHANG Z Y， JIANG S， HUANG C Z T， et al. RGB-IR cross-modality person ReID based on teacher-student GAN model ［J］. Pattern Recognition Letters， 2021， 150： 155-161. 10.1016/j.patrec.2021.07.006
23	FAN X， LUO H， ZHANG C， et al. Cross-spectrum dual-subspace pairing for RGB-infrared cross-modality person re-identification［EB/OL］. （2020-02-29）［2022-02-22］.. 10.24963/ijcai.2020/143
24	CHOI S， LEE S， KIM Y， et al. Hi-CMD： hierarchical cross-modality disentanglement for visible-infrared person re-identification ［C］// Proceedings of the 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2020： 10254-10263. 10.1109/cvpr42600.2020.01027
25	SUN Y F， ZHENG L， YANG Y， et al. Beyond part models： person retrieval with refined part pooling （and a strong convolutional baseline）［C］// Proceedings of the 2018 European Conference on Computer Vision， LNCS 11208. Cham： Springer， 2018： 501-518.
26	袁铭阳，周长胜，黄宏博，等.卷积神经网络池化方法综述［J］. 软件工程与应用， 2020， 9（5）： 360-372.
	YUAN M Y， ZHOU C S， HUANG H B， et al. Survey on convolutional neural network pooling methods ［J］. Software Engineering and Applications， 2020， 9（5）： 360-372.
27	FU Y， WEI Y C， ZHOU Y Q， et al. Horizontal pyramid matching for person re-identification［C］// Proceedings of the 33rd AAAI Conference on Artificial Intelligence. Palo Alto， CA： AAAI Press， 2019： 8295-8302. 10.1609/aaai.v33i01.33018295
28	LI D G， WEI X， HONG X P， et al. Infrared-visible cross-modal person re-identification with an X modality ［C］// Proceedings of the 34th AAAI Conference on Artificial Intelligence. Palo Alto， CA： AAAI Press， 2020： 4610-4617. 10.1609/aaai.v34i04.5891
29	YE M， SHEN J B， CRANDALL D J， et al. Dynamic dual-attentive aggregation learning for visible-infrared person re-identification ［C］// Proceedings of the 2020 European Conference on Computer Vision， LNCS 12362. Cham： Springer， 2020： 229-247.
30	ZHANG L Y， DU G D， LIU F， et al. Global-local multiple granularity learning for cross-modality visible-infrared person reidentification ［J］. IEEE Transactions on Neural Networks and Learning Systems， 2021（Early Access）： 1-11. 10.1109/tnnls.2021.3085978
31	WU A C， ZHENG W S， GONG S G， et al. RGB-IR person re-identification by cross-modality similarity preservation［J］. International Journal of Computer Vision， 2020， 128（6）： 1765-1785. 10.1007/s11263-019-01290-1
32	ZHAO Z W， LIU B， CHU Q， et al. Joint color-irrelevant consistency learning and identity-aware modality adaptation for visible-infrared cross modality person re-identification［C］// Proceedings of the 35th AAAI Conference on Artificial Intelligence. Palo Alto， CA： AAAI Press， 2021： 3520-3528. 10.1609/aaai.v35i4.16466
33	CHEN Y， WAN L， LI Z， et al. Neural feature search for RGB-infrared person re-identification ［C］// Proceedings of the 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2021： 587-597. 10.1109/cvpr46437.2021.00065
34	HERMANS A， BEYER L， LEIBE B. In defense of the triplet loss for person re-identification［EB/OL］. （2017-11-21）［2022-03-11］.. 10.21203/rs.3.rs-1501673/v1

方法	来源	全局搜索		室内搜索
方法	来源	Rank1	mAP	Rank1	mAP
TONE	AAAI2018	12.52	14.42	—	—
BDTR	IJCAI2018	17.01	19.66	—	—
JSIA	AAAI2020	38.10	36.90	43.80	52.90
AlignGAN	ICCV2019	42.40	40.70	45.90	54.30
CMSP	IJCV2020	43.56	44.98	48.62	57.50
AGW	TPAMI2021	47.50	47.65	54.17	62.97
XIV	AAAI2020	49.92	50.73	—	—
DDAG	ECCV2020	54.75	53.02	61.02	67.98
NFS	CVPR2021	56.91	55.45	62.79	69.79
CICL	AAAI2021	57.20	59.30	66.60	74.70
cm-SSFT	CVPR2020	61.60	63.20	70.50	72.60
GLMC	TNNLS2021	64.37	63.43	67.35	74.02
IVRNBDS	—	70.13	65.33	70.36	73.15

方法	来源	全局搜索		室内搜索
方法	来源	Rank1	mAP	Rank1	mAP
TONE	AAAI2018	12.52	14.42	—	—
BDTR	IJCAI2018	17.01	19.66	—	—
JSIA	AAAI2020	38.10	36.90	43.80	52.90
AlignGAN	ICCV2019	42.40	40.70	45.90	54.30
CMSP	IJCV2020	43.56	44.98	48.62	57.50
AGW	TPAMI2021	47.50	47.65	54.17	62.97
XIV	AAAI2020	49.92	50.73	—	—
DDAG	ECCV2020	54.75	53.02	61.02	67.98
NFS	CVPR2021	56.91	55.45	62.79	69.79
CICL	AAAI2021	57.20	59.30	66.60	74.70
cm-SSFT	CVPR2020	61.60	63.20	70.50	72.60
GLMC	TNNLS2021	64.37	63.43	67.35	74.02
IVRNBDS	—	70.13	65.33	70.36	73.15

方法	来源	全局搜索		室内搜索
方法	来源	Rank1	mAP	Rank1	mAP
TONE	AAAI2018	16.87	14.92	13.86	16.98
BDTR	IJCAI2018	33.47	31.83	32.72	31.10
JSIA	AAAI2020	48.50	49.30	48.10	48.90
AlignGAN	ICCV2019	57.90	53.60	56.30	53.40
CMSP	IJCV2020	65.07	64.50	—	—
AGW	TPAMI2021	70.05	66.37	—	—
XIV	AAAI2020	62.21	60.18	—	—
DDAG	ECCV2020	69.34	63.46	68.06	61.80
NFS	CVPR2021	80.54	72.10	77.95	69.79
CICL	AAAI2021	78.80	69.40	77.90	69.40
cm-SSFT	CVPR2020	72.30	72.90	71.00	71.70
GLMC	TNNLS2021	91.84	81.42	91.12	81.06
IVRNBDS	—	92.34	92.58	91.35	91.78

方法	来源	全局搜索		室内搜索
方法	来源	Rank1	mAP	Rank1	mAP
TONE	AAAI2018	16.87	14.92	13.86	16.98
BDTR	IJCAI2018	33.47	31.83	32.72	31.10
JSIA	AAAI2020	48.50	49.30	48.10	48.90
AlignGAN	ICCV2019	57.90	53.60	56.30	53.40
CMSP	IJCV2020	65.07	64.50	—	—
AGW	TPAMI2021	70.05	66.37	—	—
XIV	AAAI2020	62.21	60.18	—	—
DDAG	ECCV2020	69.34	63.46	68.06	61.80
NFS	CVPR2021	80.54	72.10	77.95	69.79
CICL	AAAI2021	78.80	69.40	77.90	69.40
cm-SSFT	CVPR2020	72.30	72.90	71.00	71.70
GLMC	TNNLS2021	91.84	81.42	91.12	81.06
IVRNBDS	—	92.34	92.58	91.35	91.78

方法配置	全局搜索		室内搜索
方法配置	Rank-1	mAP	Rank-1	mAP
B	47.50	47.65	54.17	62.97
B+ORRM	62.68	57.51	62.41	67.17
B+GCRM	62.00	59.44	67.37	70.14
B+HC_Tri Loss	61.52	58.52	65.41	69.30
IVRNBDS	70.13	65.33	70.36	73.15

基于双流结构的跨模态行人重识别关系网络

Cross-modal person re-identification relation network based on dual-stream structure

RichHTML

PDF

可视化

摘要/Abstract

引用本文

使用本文

图/表 11

参考文献 34

相关文章 15

编辑推荐

Metrics

方法	训练时间
A+三元组损失函数	172.24
A+批量难样本三元组损失函数	235.36
IVRNBDS	164.98

[1]	郑智雄, 刘建华, 孙水华, 徐戈, 林鸿辉. 融合多窗口局部信息的方面级情感分析模型[J]. 《计算机应用》唯一官方网站, 2023, 43(6): 1796-1802.
[2]	张广耀, 宋纯锋. 融合人体全身表观特征的行人头部跟踪模型[J]. 《计算机应用》唯一官方网站, 2023, 43(5): 1372-1377.
[3]	姚英茂, 姜晓燕. 基于图卷积网络与自注意力图池化的视频行人重识别方法[J]. 《计算机应用》唯一官方网站, 2023, 43(3): 728-735.
[4]	孙杰, 吴绍鑫, 王学军, 华璟. 基于Sophon SC5+芯片构架的行人搜索算法与优化[J]. 《计算机应用》唯一官方网站, 2023, 43(3): 744-751.
[5]	仇天昊, 陈淑荣. 基于EfficientNet的双分路多尺度联合学习行人再识别[J]. 《计算机应用》唯一官方网站, 2022, 42(7): 2065-2071.
[6]	陈代丽, 许国良. 基于注意力机制学习域内变化的跨域行人重识别方法[J]. 《计算机应用》唯一官方网站, 2022, 42(5): 1391-1397.
[7]	殷雨昌, 王洪元, 陈莉, 冯尊登, 肖宇. 基于单标注样本的多损失学习与联合度量视频行人重识别[J]. 《计算机应用》唯一官方网站, 2022, 42(3): 764-769.
[8]	耿艳兵, 廉永健. 基于多粒度特征生成对抗网络的跨分辨率行人重识别[J]. 《计算机应用》唯一官方网站, 2022, 42(11): 3573-3579.
[9]	李大伟, 曾智勇. 基于动态双注意力机制的跨模态行人重识别模型[J]. 《计算机应用》唯一官方网站, 2022, 42(10): 3200-3208.
[10]	刘紫燕, 朱明成, 袁磊, 马珊珊, 陈霖周廷. 基于非局部关注和多重特征融合的视频行人重识别[J]. 计算机应用, 2021, 41(2): 530-536.
[11]	龚云鹏, 曾智勇, 叶锋. 基于灰度域特征增强的行人重识别方法[J]. 《计算机应用》唯一官方网站, 2021, 41(12): 3590-3595.
[12]	刘乾, 王洪元, 曹亮, 孙博言, 肖宇, 张继. 基于联合损失胶囊网络的换衣行人重识别[J]. 《计算机应用》唯一官方网站, 2021, 41(12): 3596-3601.
[13]	韩建栋, 李晓宇. 基于多尺度特征融合的行人重识别方法[J]. 计算机应用, 2021, 41(10): 2991-2996.
[14]	陈莉, 王洪元, 张云鹏, 曹亮, 殷雨昌. 联合均等采样随机擦除和全局时间特征池化的视频行人重识别方法[J]. 计算机应用, 2021, 41(1): 164-169.
[15]	邱耀儒, 孙为军, 黄永慧, 唐瑜祺, 张浩川, 吴俊鹏. 基于生成对抗网络联合时空模型的行人重识别方法[J]. 计算机应用, 2020, 40(9): 2493-2498.

方法	计算量/10⁹	推理时间/ms
AWG	10.81	4.49
DDAG	12.93	4.58
IVRNBDS	12.96	5.15

方法	计算量/10⁹	推理时间/ms
AWG	10.81	4.49
DDAG	12.93	4.58
IVRNBDS	12.96	5.15