Cross‑resolution person re‑identification by generative adversarial network based on multi‑granularity features

doi:10.11772/j.issn.1001-9081.2021122124

Abstract

Abstract:

Existing Super Resolution （SR） reconstruction methods based on Generative Adversarial Network （GAN） for cross?resolution person Re?IDentification （ReID） suffer from deficiencies in both texture structure content recovery and feature consistency maintenance of the reconstructed images. To solve these problems， a cross?resolution pedestrian re?identification method based on multi?granularity information generation network was proposed. Firstly， a self?attention mechanism was introduced into multiple layers of generator to focus on multi?granularity stable regions with structural correlation， focusing on recovering the texture and structure information of the Low Resolution （LR） person image. At the same time， an identifier was added at the end of the generator to minimize the loss in different granularity features between the generated image and the real image during the training process， improving the feature consistency between the generated image and the real image in terms of features. Secondly， the self?attention generator and identifier were jointed， then they were optimized alternately with the discriminator to improve the generated image on content and features. Finally， the improved GAN and person re?identification network were combined to train the model parameters of the optimized network alternately until the model converged. Comparison Experimental results on several cross?resolution person re?identification datasets show that the proposed algorithm improves rank?1 accuracy on Cumulative Match Characteristic（CMC） by 10 percentage points on average， and has better performance in enhancing both content consistency and feature expression consistency of SR images.

Key words: cross?resolution, person Re?IDentification (ReID), Generative Adversarial Network (GAN), self?attention mechanism, multi?granularity feature

摘要：

现有基于生成对抗网络（GAN）的超分辨率（SR）重建方法用于跨分辨率行人重识别（ReID）时，重建图像在纹理结构内容的恢复和特征一致性保持方面均存在不足。针对上述问题，提出基于多粒度信息生成网络的跨分辨率行人ReID方法。首先，在生成器的多层网络上均引入自注意力机制，聚焦多粒度稳定的结构关联区域，重点恢复低分辨率（LR）行人图像的纹理结构信息；同时，在生成器后增加一个识别器，在训练过程中最小化生成图像与真实图像在不同粒度特征上的损失，提升生成图像与真实图像在特征上的一致性。然后，联合自注意力生成器和识别器，与判别器交替优化，在内容和特征上改进生成图像。最后，联合改进的GAN和行人ReID网络交替训练优化网络的模型参数，直至模型收敛。在多个跨分辨率行人数据集上的实验结果表明，所提算法的累计匹配曲线（CMC）在其首选识别率（rank?1）上的准确率较现有同类算法平均提升10个百分点，在提升SR图像内容一致性和特征表达一致性方面均表现更优。

关键词: 跨分辨率, 行人重识别, 生成对抗网络, 自注意力机制, 多粒度特征

CLC Number:

TP391.41

Yanbing GENG, Yongjian LIAN. Cross‑resolution person re‑identification by generative adversarial network based on multi‑granularity features[J]. Journal of Computer Applications, 2022, 42(11): 3573-3579.

耿艳兵, 廉永健. 基于多粒度特征生成对抗网络的跨分辨率行人重识别[J]. 《计算机应用》唯一官方网站, 2022, 42(11): 3573-3579.

Figures/Tables 6

References 32

1	魏文钰，杨文忠，马国祥，等. 基于深度学习的行人再识别技术研究综述［J］. 计算机应用， 2020， 40（9）：2479-2492. 10.11772/j.issn.1001-9081.2020010038
	WEI W Y， YANG W Z， MA G X， et al. Survey of person re‑identification technology based on deep learning［J］. Journal of Computer Applications， 2020， 40（9）：2479-2492. 10.11772/j.issn.1001-9081.2020010038
2	CHANG X B， HOSPEDALES T M， XIANG T. Multi‑level factorisation net for person re‑identification［C］// Proceedings of the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2018： 2109-2118. 10.1109/cvpr.2018.00225
3	CHEN D P， XU D， LI H S， et al. Group consistent similarity learning via deep CRF for person re‑identification［C］// Proceedings of the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2018： 8649-8658. 10.1109/cvpr.2018.00902
4	FARENZENA M， BAZZANI L， PERINA A， et al. Person re‑identification by symmetry‑driven accumulation of local features［C］// Proceedings of the 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2010：2360-2367. 10.1109/cvpr.2010.5539926
5	CHEN Y C， LI Y J， DU X F， et al. Learning resolution‑invariant deep representations for person re‑identification［C］// Proceedings of the 33rd AAAI Conference on Artificial Intelligence. Palo Alto， CA： AAAI Press， 2019：8215-8222. 10.1609/aaai.v33i01.33018215
6	LI X， ZHENG W S， WANG X J， et al. Multi‑scale learning for low‑resolution person re‑identification［C］// Proceedings of the 2015 IEEE International Conference on Computer Vision. Piscataway： IEEE， 2015： 3765‑3773. 10.1109/iccv.2015.429
7	JING X Y， ZHU X K， WU F， et al. Super‑resolution person re‑identification with semi‑coupled low‑rank discriminant dictionary learning［C］// Proceedings of the 2015 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2015： 695-704. 10.1109/cvpr.2015.7298669
8	胡雪影，郭海儒，朱蓉. 基于混合深度卷积网络的图像超分辨率重建［J］. 计算机应用， 2020， 40（7）： 2069-2076. 10.11772/j.issn.1001-9081.2019122149
	HU X Y， GUO H R， ZHU R. Image super‑resolution reconstruction based on hybrid deep convolutional network［J］. Journal of Computer Applications， 2020， 40（7）：2069-2076. 10.11772/j.issn.1001-9081.2019122149
9	WANG S L， ZHANG L， LIANG Y， et al. Semi‑coupled dictionary learning with applications to image super‑resolution and photo‑sketch synthesis［C］// Proceedings of the 2012 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2012： 2216-2223. 10.1109/cvpr.2012.6247930
10	DONG C， LOY C C， HE K M， et al. Image super‑resolution using deep convolutional networks［J］. IEEE Transactions on Pattern Analysis and Machine Intelligence， 2016， 38（2）：295-307. 10.1109/tpami.2015.2439281
11	LEDIG C， THEIS L， HUSZÁR F， et al. Photo‑realistic single image super‑resolution using a generative adversarial network［C］// Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2017： 105-114. 10.1109/cvpr.2017.19
12	欧阳宁，梁婷，林乐平. 基于自注意力网络的图像超分辨率重建［J］. 计算机应用， 2019， 39（8）：2391-2395. 10.11772/j.issn.1001-9081.2019010158
	OUYANG N， LIANG T， LIN L P. Self‑attention network based image super‑resolution［J］. Journal of Computer Applications， 2019， 39（8）：2391-2395. 10.11772/j.issn.1001-9081.2019010158
13	焦云清，王世新，周艺，等.基于神经网络的遥感影像超高分辨率目标识别［J］.系统仿真学报，2007，19（14）：3223-3225. 10.3969/j.issn.1004-731X.2007.14.025
	JIAO Y Q， WANG S X， ZHOU Y， et al. Super‑resolution target identification from remotely sensed imagery using Hopfield Neural Network［J］. Journal of System Simulation， 2007， 19（14）：3223-3225. 10.3969/j.issn.1004-731X.2007.14.025
14	潘志庚，郑星，张明敏. 多分辨率模型生成中颜色和纹理属性的处理［J］. 系统仿真学报， 2002， 14（11）：1506-1508， 1530. 10.3969/j.issn.1004-731X.2002.11.027
	PAN Z G， ZHENG X， ZHANG M M. Processing of color and texture attributes in multi‑resolution modeling［J］. Journal of System Simulation， 2002， 14（11）：1506-1508， 1530. 10.3969/j.issn.1004-731X.2002.11.027
15	陈佛计，朱枫，吴清潇，等. 生成对抗网络及其在图像生成中的应用研究综述［J］. 计算机学报， 2021， 44（2）：347-369. 10.11897/SP.J.1016.2021.00347
	CHEN F J， ZHU F， WU Q X， et al. A survey about image generation with generative adversarial nets［J］. Chinese Journal of Computers， 2021， 44（2）：347-369. 10.11897/SP.J.1016.2021.00347
16	HAN K， HUANG Y， SONG C F， et al. Adaptive super‑resolution for person re‑identification with low‑resolution images［J］. Pattern Recognition， 2021， 114： No.107682. 10.1016/j.patcog.2020.107682
17	JIAO J N， ZHENG W S， WU A C， et al. Deep low‑resolution person re‑identification［C］// Proceedings of the 32nd AAAI Conference on Artificial Intelligence. Palo Alto， CA： AAAI Press， 2018：6967-6974. 10.1609/aaai.v32i1.12284
18	WANG Z， YE M， YANG F， et al. Cascaded SR‑GAN for scale‑ adaptive low resolution person re‑identification［C］// Proceedings of the 27th International Joint Conference on Artificial Intelligence. California： ijcai.org， 2018： 3891-3897. 10.24963/ijcai.2018/541
19	LI Y J， CHEN Y C， LIN Y Y， et al. Recover and identify： a generative dual model for cross‑resolution person re‑identification［C］// Proceedings of the 2019 IEEE/CVF International Conference on Computer Vision. Piscataway： IEEE， 2019： 8089-8098. 10.1109/iccv.2019.00818
20	CHENG Z Y， DONG Q， GONG S G， et al. Inter‑task association critic for cross‑resolution person re‑identification［C］// Proceedings of the 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2020： 2602-2612. 10.1109/cvpr42600.2020.00268
21	丁明航，邓然然，邵恒. 基于注意力生成对抗网络的图像超分辨率重建方法［J］. 计算机系统应用， 2020， 29（2）：209-211.
	DING M H， DENG R R， SHAO H. Image super‑resolution reconstruction method based on attentive generative adversarial network［J］. Computer Systems and Applications， 2020， 29（2）：209-211.
22	许一宁，何小海，张津，等. 基于多层次分辨率递进生成对抗网络的文本生成图像方法［J］. 计算机应用， 2020， 40（12）：3612-3617. 10.11772/j.issn.1001-9081.2020040575
	XU Y N， HE X H， ZHANG J， et al. Text‑to‑image synthesis method based on multi‑level progressive resolution generative adversarial networks［J］. Journal of Computer Applications， 2020， 40（12）：3612-3617. 10.11772/j.issn.1001-9081.2020040575
23	王雪松，晁杰，程玉虎. 基于自注意力生成对抗网络的图像超分辨率重建［J］. 控制与决策， 2021， 36（6）：1324-1332. 10.13195/j.kzyjc.2019.1290
	WANG X S， CHAO J， CHENG Y H. Image super‑resolution reconstruction based on self‑attention GAN［J］. Control and Decision， 2021， 36（6）：1324-1332. 10.13195/j.kzyjc.2019.1290
24	杨婉香，严严，陈思，等. 基于多尺度生成对抗网络的遮挡行人重识别方法［J］. 软件学报， 2020， 31（7）：1943-1958.
	YANG W X， YAN Y， CHEN S， et al. Multi‑scale generative adversarial network for person re‑identification under occlusion［J］. Journal of Software， 2020， 31（7）：1943-1958.
25	DENG J， DONG W， SOCHER R， et al. ImageNet： a large‑scale hierarchical image database［C］// Proceedings of the 2009 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2009：248-255. 10.1109/cvpr.2009.5206848
26	GULRAJANI I， AHMED F， ARJOVSKY M， et al. Improved training of Wasserstein GANs［C］// Proceedings of the 31st International Conference on Neural Information Processing Systems. Red Hook， NY： Curran Associates Inc.， 2017： 5769-5779.
27	LI W， ZHU X T， GONG S G. Person re‑identification by deep joint learning of multi‑loss classification［C］// Proceedings of the 26th International Joint Conference on Artificial Intelligence. California： ijcai.org， 2017： 2194-2200. 10.24963/ijcai.2017/305
28	YUAN Y， CHEN W Y， YANG Y， et al. In defense of the triplet loss again： learning robust person re‑identification with fast approximated triplet loss and label distillation［C］// Proceedings of the 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops. Piscataway： IEEE， 2020： 1454-1463. 10.1109/cvprw50498.2020.00185
29	ZHONG Z， ZHENG L， ZHENG Z D， et al. Camera style adaptation for person re‑identification［C］// Proceedings of the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2018： 5157-5166. 10.1109/cvpr.2018.00541
30	GE Y X， LI Z W， ZHAO H Y， et al. FD‑GAN： pose‑guided feature distilling GAN for robust person re‑identification［C］// Proceedings of the 32nd International Conference on Neural Information Processing Systems. Red Hook， NY： Curran Associates Inc.， 2018： 1230-1241.
31	WANG Z， HU R M， YU Y， et al. Scale‑adaptive low‑resolution person re‑identification via learning a discriminating surface［C］// Proceedings of the 25th International Joint Conference on Artificial Intelligence. California： ijcai.org， 2016： 2669-2675.
32	SUN Y F， ZHENG L， YANG Y， et al. Beyond part models： person retrieval with refined part pooling （and a strong convolutional baseline）［C］// Proceedings of the 2018 European Conference on Computer Vision， LNCS 11208. Cham： Springer， 2018： 501-518.

方法	MLR‑CUHK03
方法	SSIM	PSNR/dB	LPIPS	rank‑1/%
GAN	0.30	13.9	0.42	47.8
GAN+自注意力	0.61	19.2	0.19	65.3
GAN+识别网络（高层损失）	0.41	18.1	0.23	76.9
GAN+识别网络（中低层损失）	0.57	20.9	0.16	78.1
GAN+识别网络（高层&中低层损失）	0.61	21.3	0.13	80.5
GAN+自注意力+识别网络（高层损失）	0.69	22.5	0.12	85.5
GAN+自注意力+识别网络（中低层损失）	0.71	22.8	0.08	85.3
GAN+自注意力+识别网络（高层&中低层损失）	0.89	23.5	0.06	87.1

方法	MLR‑CUHK03
方法	SSIM	PSNR/dB	LPIPS	rank‑1/%
GAN	0.30	13.9	0.42	47.8
GAN+自注意力	0.61	19.2	0.19	65.3
GAN+识别网络（高层损失）	0.41	18.1	0.23	76.9
GAN+识别网络（中低层损失）	0.57	20.9	0.16	78.1
GAN+识别网络（高层&中低层损失）	0.61	21.3	0.13	80.5
GAN+自注意力+识别网络（高层损失）	0.69	22.5	0.12	85.5
GAN+自注意力+识别网络（中低层损失）	0.71	22.8	0.08	85.3
GAN+自注意力+识别网络（高层&中低层损失）	0.89	23.5	0.06	87.1

方法	MLR‑Market1501		MLR‑CUHK03		MLR‑VIPeR		MLR‑DukeMTMC‑reID		CAVIAR
方法	rank‑1	rank‑5	rank‑1	rank‑5	rank‑1	rank‑5	rank‑1	rank‑5	rank‑1	rank‑5
CamStyle^［29］	74.5	88.6	69.1	89.6	34.4	56.8	64.0	78.1	32.1	72.3
FD‑GAN^［30］	79.6	91.6	73.4	92.8	39.1	62.1	67.5	82.0	33.5	71.4
SLD²L^［7］	―	―	―	―	20.3	44.0	―	―	18.4	44.8
SING^［17］	74.4	87.8	67.7	90.7	33.5	57.0	65.2	80.1	33.5	72.7
CSR‑GAN^［18］	76.4	88.5	71.3	92.1	37.2	62.3	67.6	81.4	34.7	72.5
JUDEA^［6］	―	―	26.2	58.0	26.0	55.1	―	―	22.0	60.1
SDF^［31］	―	―	22.2	48.0	9.3	38.1	―	―	14.3	37.5
RAIN^［5］	―	―	78.9	97.3	42.5	68.3	―	―	42.0	77.3
CAD^［19］	83.7	92.7	82.1	97.4	43.1	68.2	75.6	86.7	42.8	76.2
INTACT^［20］	88.1	95.0	86.4	97.4	46.2	73.1	81.2	90.1	44.0	81.8
本文方法	88.9	96.3	87.1	97.5	46.9	73.6	82.1	91.8	44.9	83.2

方法	MLR‑Market1501		MLR‑CUHK03		MLR‑VIPeR		MLR‑DukeMTMC‑reID		CAVIAR
方法	rank‑1	rank‑5	rank‑1	rank‑5	rank‑1	rank‑5	rank‑1	rank‑5	rank‑1	rank‑5
CamStyle^［29］	74.5	88.6	69.1	89.6	34.4	56.8	64.0	78.1	32.1	72.3
FD‑GAN^［30］	79.6	91.6	73.4	92.8	39.1	62.1	67.5	82.0	33.5	71.4
SLD²L^［7］	―	―	―	―	20.3	44.0	―	―	18.4	44.8
SING^［17］	74.4	87.8	67.7	90.7	33.5	57.0	65.2	80.1	33.5	72.7
CSR‑GAN^［18］	76.4	88.5	71.3	92.1	37.2	62.3	67.6	81.4	34.7	72.5
JUDEA^［6］	―	―	26.2	58.0	26.0	55.1	―	―	22.0	60.1
SDF^［31］	―	―	22.2	48.0	9.3	38.1	―	―	14.3	37.5
RAIN^［5］	―	―	78.9	97.3	42.5	68.3	―	―	42.0	77.3
CAD^［19］	83.7	92.7	82.1	97.4	43.1	68.2	75.6	86.7	42.8	76.2
INTACT^［20］	88.1	95.0	86.4	97.4	46.2	73.1	81.2	90.1	44.0	81.8
本文方法	88.9	96.3	87.1	97.5	46.9	73.6	82.1	91.8	44.9	83.2

[1]	Zanxia QIANG, Xianfu BAO. Residual attention deraining network based on convolutional long short-term memory [J]. Journal of Computer Applications, 2022, 42(9): 2858-2864.
[2]	Zefang HAN, Xiong ZHANG, Hong SHANGGUAN, Xinglong HAN, Jing HAN, Gang FENG, Xueying CUI. Artifacts sensing generative adversarial network for low-dose CT denoising [J]. Journal of Computer Applications, 2022, 42(7): 2301-2310.
[3]	Wentao MAO, Guifang WU, Chao WU, Zhi DOU. Animation video generation model based on Chinese impressionistic style transfer [J]. Journal of Computer Applications, 2022, 42(7): 2162-2169.
[4]	Jia LI, Yuanlin ZHENG, Kaiyang LIAO, Haojie LOU, Shiyu LI, Zehao CHEN. No-reference image quality assessment algorithm based on saliency deep features [J]. Journal of Computer Applications, 2022, 42(6): 1957-1964.
[5]	Yuming ZHAO, Shenkai GU. Adversarial attack defense model with residual dense block self-attention mechanism and generative adversarial network [J]. Journal of Computer Applications, 2022, 42(3): 921-929.
[6]	Yimin CAO, Lei CAI, Jingyang GAO. Gene data generation method based on generative adversarial network [J]. Journal of Computer Applications, 2022, 42(3): 783-790.
[7]	Dingkang YANG, Shuai HUANG, Shunli WANG, Peng ZHAI, Yidan LI, Lihua ZHANG. EE-GAN：facial expression recognition method based on generative adversarial network and network integration [J]. Journal of Computer Applications, 2022, 42(3): 750-756.
[8]	Yuli CHEN, Qiang TONG, Tongtong CHEN, Shoulu HOU, Xiulei LIU. Short-term trajectory prediction model of aircraft based on attention mechanism and generative adversarial network [J]. Journal of Computer Applications, 2022, 42(10): 3292-3299.
[9]	GUAN Qijie, ZHANG Ting, LI Deya, ZHOU Shaojing, DU Yi. Indefinite reconstruction method of spatial data based on multi-resolution generative adversarial network [J]. Journal of Computer Applications, 2021, 41(8): 2306-2311.
[10]	SUN Xiao, XU Jindong. Remote sensing image dehazing method based on cascaded generative adversarial network [J]. Journal of Computer Applications, 2021, 41(8): 2440-2444.
[11]	TANG Guihua, SUN Lei, MAO Xiuqing, DAI Leyu, HU Yongjin. Generative adversarial network synthesized face detection based on deep alignment network [J]. Journal of Computer Applications, 2021, 41(7): 1922-1927.
[12]	WANG Xianwu, ZHANG Ting, JI Xin, DU Yi. 3D shale digital core reconstruction method based on deep convolutional generative adversarial network with gradient penalty [J]. Journal of Computer Applications, 2021, 41(6): 1805-1811.
[13]	LI Yanzhi, FAN Yong, GAO Lin. Anomaly detection of oil drilling water flow based on shape flow [J]. Journal of Computer Applications, 2021, 41(6): 1842-1848.
[14]	SUN Heli, SUN Yuzhu, ZHANG Xiaoyun. Event description generation based on generative adversarial network [J]. Journal of Computer Applications, 2021, 41(5): 1256-1261.
[15]	GUO Maozu, YANG Qiannan, ZHAO Lingling. Image generation based on conditional-Wassertein generative adversarial network [J]. Journal of Computer Applications, 2021, 41(5): 1432-1437.