Cross‑resolution person re‑identification by generative adversarial network based on multi‑granularity features

doi:10.11772/j.issn.1001-9081.2021122124

Abstract

Abstract:

Existing Super Resolution （SR） reconstruction methods based on Generative Adversarial Network （GAN） for cross?resolution person Re?IDentification （ReID） suffer from deficiencies in both texture structure content recovery and feature consistency maintenance of the reconstructed images. To solve these problems， a cross?resolution pedestrian re?identification method based on multi?granularity information generation network was proposed. Firstly， a self?attention mechanism was introduced into multiple layers of generator to focus on multi?granularity stable regions with structural correlation， focusing on recovering the texture and structure information of the Low Resolution （LR） person image. At the same time， an identifier was added at the end of the generator to minimize the loss in different granularity features between the generated image and the real image during the training process， improving the feature consistency between the generated image and the real image in terms of features. Secondly， the self?attention generator and identifier were jointed， then they were optimized alternately with the discriminator to improve the generated image on content and features. Finally， the improved GAN and person re?identification network were combined to train the model parameters of the optimized network alternately until the model converged. Comparison Experimental results on several cross?resolution person re?identification datasets show that the proposed algorithm improves rank?1 accuracy on Cumulative Match Characteristic（CMC） by 10 percentage points on average， and has better performance in enhancing both content consistency and feature expression consistency of SR images.

Key words: cross?resolution, person Re?IDentification (ReID), Generative Adversarial Network (GAN), self?attention mechanism, multi?granularity feature

摘要：

现有基于生成对抗网络（GAN）的超分辨率（SR）重建方法用于跨分辨率行人重识别（ReID）时，重建图像在纹理结构内容的恢复和特征一致性保持方面均存在不足。针对上述问题，提出基于多粒度信息生成网络的跨分辨率行人ReID方法。首先，在生成器的多层网络上均引入自注意力机制，聚焦多粒度稳定的结构关联区域，重点恢复低分辨率（LR）行人图像的纹理结构信息；同时，在生成器后增加一个识别器，在训练过程中最小化生成图像与真实图像在不同粒度特征上的损失，提升生成图像与真实图像在特征上的一致性。然后，联合自注意力生成器和识别器，与判别器交替优化，在内容和特征上改进生成图像。最后，联合改进的GAN和行人ReID网络交替训练优化网络的模型参数，直至模型收敛。在多个跨分辨率行人数据集上的实验结果表明，所提算法的累计匹配曲线（CMC）在其首选识别率（rank?1）上的准确率较现有同类算法平均提升10个百分点，在提升SR图像内容一致性和特征表达一致性方面均表现更优。

关键词: 跨分辨率, 行人重识别, 生成对抗网络, 自注意力机制, 多粒度特征

CLC Number:

TP391.41

Yanbing GENG, Yongjian LIAN. Cross‑resolution person re‑identification by generative adversarial network based on multi‑granularity features[J]. Journal of Computer Applications, 2022, 42(11): 3573-3579.

耿艳兵, 廉永健. 基于多粒度特征生成对抗网络的跨分辨率行人重识别[J]. 《计算机应用》唯一官方网站, 2022, 42(11): 3573-3579.

Figures/Tables 6

References 32

1	魏文钰，杨文忠，马国祥，等. 基于深度学习的行人再识别技术研究综述［J］. 计算机应用， 2020， 40（9）：2479-2492. 10.11772/j.issn.1001-9081.2020010038
	WEI W Y， YANG W Z， MA G X， et al. Survey of person re‑identification technology based on deep learning［J］. Journal of Computer Applications， 2020， 40（9）：2479-2492. 10.11772/j.issn.1001-9081.2020010038
2	CHANG X B， HOSPEDALES T M， XIANG T. Multi‑level factorisation net for person re‑identification［C］// Proceedings of the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2018： 2109-2118. 10.1109/cvpr.2018.00225
3	CHEN D P， XU D， LI H S， et al. Group consistent similarity learning via deep CRF for person re‑identification［C］// Proceedings of the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2018： 8649-8658. 10.1109/cvpr.2018.00902
4	FARENZENA M， BAZZANI L， PERINA A， et al. Person re‑identification by symmetry‑driven accumulation of local features［C］// Proceedings of the 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2010：2360-2367. 10.1109/cvpr.2010.5539926
5	CHEN Y C， LI Y J， DU X F， et al. Learning resolution‑invariant deep representations for person re‑identification［C］// Proceedings of the 33rd AAAI Conference on Artificial Intelligence. Palo Alto， CA： AAAI Press， 2019：8215-8222. 10.1609/aaai.v33i01.33018215
6	LI X， ZHENG W S， WANG X J， et al. Multi‑scale learning for low‑resolution person re‑identification［C］// Proceedings of the 2015 IEEE International Conference on Computer Vision. Piscataway： IEEE， 2015： 3765‑3773. 10.1109/iccv.2015.429
7	JING X Y， ZHU X K， WU F， et al. Super‑resolution person re‑identification with semi‑coupled low‑rank discriminant dictionary learning［C］// Proceedings of the 2015 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2015： 695-704. 10.1109/cvpr.2015.7298669
8	胡雪影，郭海儒，朱蓉. 基于混合深度卷积网络的图像超分辨率重建［J］. 计算机应用， 2020， 40（7）： 2069-2076. 10.11772/j.issn.1001-9081.2019122149
	HU X Y， GUO H R， ZHU R. Image super‑resolution reconstruction based on hybrid deep convolutional network［J］. Journal of Computer Applications， 2020， 40（7）：2069-2076. 10.11772/j.issn.1001-9081.2019122149
9	WANG S L， ZHANG L， LIANG Y， et al. Semi‑coupled dictionary learning with applications to image super‑resolution and photo‑sketch synthesis［C］// Proceedings of the 2012 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2012： 2216-2223. 10.1109/cvpr.2012.6247930
10	DONG C， LOY C C， HE K M， et al. Image super‑resolution using deep convolutional networks［J］. IEEE Transactions on Pattern Analysis and Machine Intelligence， 2016， 38（2）：295-307. 10.1109/tpami.2015.2439281
11	LEDIG C， THEIS L， HUSZÁR F， et al. Photo‑realistic single image super‑resolution using a generative adversarial network［C］// Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2017： 105-114. 10.1109/cvpr.2017.19
12	欧阳宁，梁婷，林乐平. 基于自注意力网络的图像超分辨率重建［J］. 计算机应用， 2019， 39（8）：2391-2395. 10.11772/j.issn.1001-9081.2019010158
	OUYANG N， LIANG T， LIN L P. Self‑attention network based image super‑resolution［J］. Journal of Computer Applications， 2019， 39（8）：2391-2395. 10.11772/j.issn.1001-9081.2019010158
13	焦云清，王世新，周艺，等.基于神经网络的遥感影像超高分辨率目标识别［J］.系统仿真学报，2007，19（14）：3223-3225. 10.3969/j.issn.1004-731X.2007.14.025
	JIAO Y Q， WANG S X， ZHOU Y， et al. Super‑resolution target identification from remotely sensed imagery using Hopfield Neural Network［J］. Journal of System Simulation， 2007， 19（14）：3223-3225. 10.3969/j.issn.1004-731X.2007.14.025
14	潘志庚，郑星，张明敏. 多分辨率模型生成中颜色和纹理属性的处理［J］. 系统仿真学报， 2002， 14（11）：1506-1508， 1530. 10.3969/j.issn.1004-731X.2002.11.027
	PAN Z G， ZHENG X， ZHANG M M. Processing of color and texture attributes in multi‑resolution modeling［J］. Journal of System Simulation， 2002， 14（11）：1506-1508， 1530. 10.3969/j.issn.1004-731X.2002.11.027
15	陈佛计，朱枫，吴清潇，等. 生成对抗网络及其在图像生成中的应用研究综述［J］. 计算机学报， 2021， 44（2）：347-369. 10.11897/SP.J.1016.2021.00347
	CHEN F J， ZHU F， WU Q X， et al. A survey about image generation with generative adversarial nets［J］. Chinese Journal of Computers， 2021， 44（2）：347-369. 10.11897/SP.J.1016.2021.00347
16	HAN K， HUANG Y， SONG C F， et al. Adaptive super‑resolution for person re‑identification with low‑resolution images［J］. Pattern Recognition， 2021， 114： No.107682. 10.1016/j.patcog.2020.107682
17	JIAO J N， ZHENG W S， WU A C， et al. Deep low‑resolution person re‑identification［C］// Proceedings of the 32nd AAAI Conference on Artificial Intelligence. Palo Alto， CA： AAAI Press， 2018：6967-6974. 10.1609/aaai.v32i1.12284
18	WANG Z， YE M， YANG F， et al. Cascaded SR‑GAN for scale‑ adaptive low resolution person re‑identification［C］// Proceedings of the 27th International Joint Conference on Artificial Intelligence. California： ijcai.org， 2018： 3891-3897. 10.24963/ijcai.2018/541
19	LI Y J， CHEN Y C， LIN Y Y， et al. Recover and identify： a generative dual model for cross‑resolution person re‑identification［C］// Proceedings of the 2019 IEEE/CVF International Conference on Computer Vision. Piscataway： IEEE， 2019： 8089-8098. 10.1109/iccv.2019.00818
20	CHENG Z Y， DONG Q， GONG S G， et al. Inter‑task association critic for cross‑resolution person re‑identification［C］// Proceedings of the 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2020： 2602-2612. 10.1109/cvpr42600.2020.00268
21	丁明航，邓然然，邵恒. 基于注意力生成对抗网络的图像超分辨率重建方法［J］. 计算机系统应用， 2020， 29（2）：209-211.
	DING M H， DENG R R， SHAO H. Image super‑resolution reconstruction method based on attentive generative adversarial network［J］. Computer Systems and Applications， 2020， 29（2）：209-211.
22	许一宁，何小海，张津，等. 基于多层次分辨率递进生成对抗网络的文本生成图像方法［J］. 计算机应用， 2020， 40（12）：3612-3617. 10.11772/j.issn.1001-9081.2020040575
	XU Y N， HE X H， ZHANG J， et al. Text‑to‑image synthesis method based on multi‑level progressive resolution generative adversarial networks［J］. Journal of Computer Applications， 2020， 40（12）：3612-3617. 10.11772/j.issn.1001-9081.2020040575
23	王雪松，晁杰，程玉虎. 基于自注意力生成对抗网络的图像超分辨率重建［J］. 控制与决策， 2021， 36（6）：1324-1332. 10.13195/j.kzyjc.2019.1290
	WANG X S， CHAO J， CHENG Y H. Image super‑resolution reconstruction based on self‑attention GAN［J］. Control and Decision， 2021， 36（6）：1324-1332. 10.13195/j.kzyjc.2019.1290
24	杨婉香，严严，陈思，等. 基于多尺度生成对抗网络的遮挡行人重识别方法［J］. 软件学报， 2020， 31（7）：1943-1958.
	YANG W X， YAN Y， CHEN S， et al. Multi‑scale generative adversarial network for person re‑identification under occlusion［J］. Journal of Software， 2020， 31（7）：1943-1958.
25	DENG J， DONG W， SOCHER R， et al. ImageNet： a large‑scale hierarchical image database［C］// Proceedings of the 2009 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2009：248-255. 10.1109/cvpr.2009.5206848
26	GULRAJANI I， AHMED F， ARJOVSKY M， et al. Improved training of Wasserstein GANs［C］// Proceedings of the 31st International Conference on Neural Information Processing Systems. Red Hook， NY： Curran Associates Inc.， 2017： 5769-5779.
27	LI W， ZHU X T， GONG S G. Person re‑identification by deep joint learning of multi‑loss classification［C］// Proceedings of the 26th International Joint Conference on Artificial Intelligence. California： ijcai.org， 2017： 2194-2200. 10.24963/ijcai.2017/305
28	YUAN Y， CHEN W Y， YANG Y， et al. In defense of the triplet loss again： learning robust person re‑identification with fast approximated triplet loss and label distillation［C］// Proceedings of the 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops. Piscataway： IEEE， 2020： 1454-1463. 10.1109/cvprw50498.2020.00185
29	ZHONG Z， ZHENG L， ZHENG Z D， et al. Camera style adaptation for person re‑identification［C］// Proceedings of the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2018： 5157-5166. 10.1109/cvpr.2018.00541
30	GE Y X， LI Z W， ZHAO H Y， et al. FD‑GAN： pose‑guided feature distilling GAN for robust person re‑identification［C］// Proceedings of the 32nd International Conference on Neural Information Processing Systems. Red Hook， NY： Curran Associates Inc.， 2018： 1230-1241.
31	WANG Z， HU R M， YU Y， et al. Scale‑adaptive low‑resolution person re‑identification via learning a discriminating surface［C］// Proceedings of the 25th International Joint Conference on Artificial Intelligence. California： ijcai.org， 2016： 2669-2675.
32	SUN Y F， ZHENG L， YANG Y， et al. Beyond part models： person retrieval with refined part pooling （and a strong convolutional baseline）［C］// Proceedings of the 2018 European Conference on Computer Vision， LNCS 11208. Cham： Springer， 2018： 501-518.

方法	MLR‑CUHK03
方法	SSIM	PSNR/dB	LPIPS	rank‑1/%
GAN	0.30	13.9	0.42	47.8
GAN+自注意力	0.61	19.2	0.19	65.3
GAN+识别网络（高层损失）	0.41	18.1	0.23	76.9
GAN+识别网络（中低层损失）	0.57	20.9	0.16	78.1
GAN+识别网络（高层&中低层损失）	0.61	21.3	0.13	80.5
GAN+自注意力+识别网络（高层损失）	0.69	22.5	0.12	85.5
GAN+自注意力+识别网络（中低层损失）	0.71	22.8	0.08	85.3
GAN+自注意力+识别网络（高层&中低层损失）	0.89	23.5	0.06	87.1

方法	MLR‑CUHK03
方法	SSIM	PSNR/dB	LPIPS	rank‑1/%
GAN	0.30	13.9	0.42	47.8
GAN+自注意力	0.61	19.2	0.19	65.3
GAN+识别网络（高层损失）	0.41	18.1	0.23	76.9
GAN+识别网络（中低层损失）	0.57	20.9	0.16	78.1
GAN+识别网络（高层&中低层损失）	0.61	21.3	0.13	80.5
GAN+自注意力+识别网络（高层损失）	0.69	22.5	0.12	85.5
GAN+自注意力+识别网络（中低层损失）	0.71	22.8	0.08	85.3
GAN+自注意力+识别网络（高层&中低层损失）	0.89	23.5	0.06	87.1

方法	MLR‑Market1501		MLR‑CUHK03		MLR‑VIPeR		MLR‑DukeMTMC‑reID		CAVIAR
方法	rank‑1	rank‑5	rank‑1	rank‑5	rank‑1	rank‑5	rank‑1	rank‑5	rank‑1	rank‑5
CamStyle^［29］	74.5	88.6	69.1	89.6	34.4	56.8	64.0	78.1	32.1	72.3
FD‑GAN^［30］	79.6	91.6	73.4	92.8	39.1	62.1	67.5	82.0	33.5	71.4
SLD²L^［7］	―	―	―	―	20.3	44.0	―	―	18.4	44.8
SING^［17］	74.4	87.8	67.7	90.7	33.5	57.0	65.2	80.1	33.5	72.7
CSR‑GAN^［18］	76.4	88.5	71.3	92.1	37.2	62.3	67.6	81.4	34.7	72.5
JUDEA^［6］	―	―	26.2	58.0	26.0	55.1	―	―	22.0	60.1
SDF^［31］	―	―	22.2	48.0	9.3	38.1	―	―	14.3	37.5
RAIN^［5］	―	―	78.9	97.3	42.5	68.3	―	―	42.0	77.3
CAD^［19］	83.7	92.7	82.1	97.4	43.1	68.2	75.6	86.7	42.8	76.2
INTACT^［20］	88.1	95.0	86.4	97.4	46.2	73.1	81.2	90.1	44.0	81.8
本文方法	88.9	96.3	87.1	97.5	46.9	73.6	82.1	91.8	44.9	83.2

方法	MLR‑Market1501		MLR‑CUHK03		MLR‑VIPeR		MLR‑DukeMTMC‑reID		CAVIAR
方法	rank‑1	rank‑5	rank‑1	rank‑5	rank‑1	rank‑5	rank‑1	rank‑5	rank‑1	rank‑5
CamStyle^［29］	74.5	88.6	69.1	89.6	34.4	56.8	64.0	78.1	32.1	72.3
FD‑GAN^［30］	79.6	91.6	73.4	92.8	39.1	62.1	67.5	82.0	33.5	71.4
SLD²L^［7］	―	―	―	―	20.3	44.0	―	―	18.4	44.8
SING^［17］	74.4	87.8	67.7	90.7	33.5	57.0	65.2	80.1	33.5	72.7
CSR‑GAN^［18］	76.4	88.5	71.3	92.1	37.2	62.3	67.6	81.4	34.7	72.5
JUDEA^［6］	―	―	26.2	58.0	26.0	55.1	―	―	22.0	60.1
SDF^［31］	―	―	22.2	48.0	9.3	38.1	―	―	14.3	37.5
RAIN^［5］	―	―	78.9	97.3	42.5	68.3	―	―	42.0	77.3
CAD^［19］	83.7	92.7	82.1	97.4	43.1	68.2	75.6	86.7	42.8	76.2
INTACT^［20］	88.1	95.0	86.4	97.4	46.2	73.1	81.2	90.1	44.0	81.8
本文方法	88.9	96.3	87.1	97.5	46.9	73.6	82.1	91.8	44.9	83.2

[1]	Li LIU, Haijin HOU, Anhong WANG, Tao ZHANG. Generative data hiding algorithm based on multi-scale attention [J]. Journal of Computer Applications, 2024, 44(7): 2102-2109.
[2]	Haoran WANG, Dan YU, Yuli YANG, Yao MA, Yongle CHEN. Domain transfer intrusion detection method for unknown attacks on industrial control systems [J]. Journal of Computer Applications, 2024, 44(4): 1158-1165.
[3]	Sunjie YU, Hui ZENG, Shiyu XIONG, Hongzhou SHI. Incentive mechanism for federated learning based on generative adversarial network [J]. Journal of Computer Applications, 2024, 44(2): 344-352.
[4]	Hui ZHOU, Yuling CHEN, Xuewei WANG, Yangwen ZHANG, Jianjiang HE. Deep shadow defense scheme of federated learning based on generative adversarial network [J]. Journal of Computer Applications, 2024, 44(1): 223-232.
[5]	Anyang LIU, Huaici ZHAO, Wenlong CAI, Zechao XU, Ruideng XIE. Adaptive image deblurring generative adversarial network algorithm based on active discrimination mechanism [J]. Journal of Computer Applications, 2023, 43(7): 2288-2294.
[6]	Shaoquan CHEN, Jianping CAI, Lan SUN. Differential privacy generative adversarial network algorithm with dynamic gradient threshold clipping [J]. Journal of Computer Applications, 2023, 43(7): 2065-2072.
[7]	Xin JIN, Yangchuan LIU, Yechen ZHU, Zijian ZHANG, Xin GAO. Sinogram inpainting for sparse-view cone-beam computed tomography image reconstruction based on residual encoder-decoder generative adversarial network [J]. Journal of Computer Applications, 2023, 43(6): 1950-1957.
[8]	Jinwen GUO, Xinghua MA, Gongning LUO, Wei WANG, Yang CAO, Kuanquan WANG. Guidewire artifact removal method of structure-enhanced IVOCT based on Transformer [J]. Journal of Computer Applications, 2023, 43(5): 1596-1605.
[9]	Jiagao WU, Shiwen ZHANG, Yudong JIANG, Linfeng LIU. Social-interaction GAN for pedestrian trajectory prediction based on state-refinement long short-term memory and attention mechanism [J]. Journal of Computer Applications, 2023, 43(5): 1565-1570.
[10]	Xiaoyu FAN, Suzhen LIN, Yanbo WANG, Feng LIU, Dawei LI. Reconstruction algorithm for highly undersampled magnetic resonance images based on residual graph convolutional neural network [J]. Journal of Computer Applications, 2023, 43(4): 1261-1268.
[11]	Hao WANG, Zicheng WANG, Chao ZHANG, Yunsheng MA. Generative adversarial network based data uncertainty quantification method [J]. Journal of Computer Applications, 2023, 43(4): 1094-1101.
[12]	Chunyong YIN, Liwen ZHOU. Unsupervised time series anomaly detection model based on re-encoding [J]. Journal of Computer Applications, 2023, 43(3): 804-811.
[13]	Lingling TAO, Bo LIU, Wenbo LI, Xiping HE. Controllable face editing algorithm with closed-form solution [J]. Journal of Computer Applications, 2023, 43(2): 601-607.
[14]	Gang CHEN, Yongwei LIAO, Zhenguo YANG, Wenying LIU. Image inpainting algorithm of multi-scale generative adversarial network based on multi-feature fusion [J]. Journal of Computer Applications, 2023, 43(2): 536-544.
[15]	Ruoying WANG, Fan LYU, Liuqing ZHAO, Fuyuan HU. Floorplan generation algorithm integrating user requirements and boundary constraints [J]. Journal of Computer Applications, 2023, 43(2): 575-582.