Crowd counting network based on multi-scale spatial attention feature fusion

doi:10.11772/j.issn.1001-9081.2020060793

Abstract

Abstract: Concerning the poor performance problem of crowd counting tasks in different dense scenes caused by severe scale changes and occlusions, a new Multi-scale spatial Attention Feature fusion Network (MAFNet) was proposed based on the Congested Scene Recognition Network (CSRNet) by combining the multi-scale feature fusion structure and the spatial attention module. Before extracting features with MAFNet, the scene images with head markers were processed with the Gaussian filter to obtain the ground truth density maps of images. In addition, the method of jointly using two basic loss functions was proposed to constrain the consistency of the density estimation map and the ground truth density map. Next, with the multi-scale feature fusion structure as the backbone of MAFNet, the strategy of extracting and fusing multi-scale features simultaneously was used to obtain the multi-scale fusion feature map, then the feature maps were calibrated and refused by the spatial attention module. After that, an estimated density image was generated through dilated convolution, and the number of people in the scene was obtained by integrating the estimated density image pixel by pixel. To verify the effectiveness of the proposed model, evaluations were conducted on four datasets (ShanghaiTech, UCF_CC_50, UCF_QRNF and World-Expo'10). Experimental results on ShanghaiTech dataset PartB show that, compared with CSRNet, MAFNet has a Mean Absolute Error (MAE) reduction of 34.9% and a Mean Square Error (MSE) reduction of 29.4%. Furthermore, experimental results on multiple datasets show that by using the attention mechanism and multi-scale feature fusion strategy, MAFNet can extract more detailed information and reduce the impact of scale changes and occlusions.

Key words: dense crowd counting, Convolutional Neural Network (CNN), feature fusion, attention mechanism, multi-scale

摘要： 针对严重的尺度变化和遮挡导致在不同密集场景人群计数任务中性能差的问题，在密集场景识别网络（CSRNet）的基础上通过增加多尺度特征融合结构并引入空间注意力机制，提出了一种多尺度空间注意力特征融合网络（MAFNet）。在MAFNet进行特征提取之前，需要对添加了人头标记的场景图进行高斯滤波生成真实密度图；此外，MAFNet还通过联合使用两种基本损失函数的方法来约束密度估计图与真实密度图的一致性。接着，MAFNet以多尺度特征融合结构为主干，首先采用边提取多尺度特征边融合的策略得到多尺度融合特征图，然后使用空间注意力模块对特征图进行校准和再融合，之后通过扩张卷积生成密度估计图，最后对密度估计图逐像素积分得到场景中的人数。为了验证所提出模型的有效性，在四个人群计数数据集（ShanghaiTech、UCF_CC_50、UCF_QRNF和World-Expo’10）上进行了评估。其中ShanghaiTech数据集PartB的实验结果显示，MAFNet与CSRNet相比，平均绝对误差（MAE）降低了34.9%，均方误差（MSE）降低了29.4%。在多个数据集上的实验结果表明，采用注意力机制和多尺度特征融合策略使MAFNet可以提取更多细节信息，减少尺度变化和遮挡带来的影响。

关键词: 密集人群计数, 卷积神经网络, 特征融合, 注意力机制, 多尺度

CLC Number:

TP391.4

DU Peide, YAN Hua. Crowd counting network based on multi-scale spatial attention feature fusion[J]. Journal of Computer Applications, 2021, 41(2): 537-543.

杜培德, 严华. 基于多尺度空间注意力特征融合的人群计数网络[J]. 计算机应用, 2021, 41(2): 537-543.

References

[1] FRENCH G,FISHER M,MACKIEWICZ M,et al. Convolutional neural networks for counting fish in fisheries surveillance video[C]//Proceedings of the 2015 Workshop on Machine Vision of Animals and Their Behavior. Durham:BMVA Press,2015:7.
[2] LEMPITSKY V,ZISSERMAN A. Learning to count objects in images[C]//Proceedings of the 23rd International Conference on Neural Information Processing. Red Hook:Curran Associates Inc., 2010:1324-1332.
[3] GUERRERO-GÓMEZ-OLMEDO R,TORRE-JIMÉNEZ B,LÓPEZSASTRE R,et al. Extremely overlapping vehicle counting[C]//Proceeding of the 2015 Iberian Conference on Pattern Recognition and Image Analysis,LNCS 9117. Cham:Springer,2015:423-431.
[4] IDREES H,SOOMRO K,SHAH M. Detecting humans in dense crowds using locally-consistent scale prior and global occlusion reasoning[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence,2015,37(10):1986-1998.
[5] CHEN K,LOY C C,GONG S,et al. Feature mining for localised crowd counting[C]//Proceeding of the 2012 British Machine Vision Conference. Durham:BMVA Press,2012:No. 21.
[6] CHAN A B,VASCONCELOS N. Bayesian Poisson regression for crowd counting[C]//Proceedings of the IEEE 12th International Conference on Computer Vision. Piscataway:IEEE, 2009:545-551.
[7] RYAN D,DENMAN S,FOOKES C,et al. Crowd counting using multiple local features[C]//Proceeding of the 2009 Digital Image Computing:Techniques and Applications. Piscataway:IEEE, 2009:81-88.
[8] SIMONYAN K,ZISSERMAN A. Very deep convolutional networks for large-scale image recognition[EB/OL].[2019-12-01]. https://arxiv.org/pdf/1409.1556.pdf.
[9] CHOLLET F. Xception:deep learning with depthwise separable convolutions[C]//Proceeding of the 2017 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway:IEEE, 2017:1800-1807.
[10] KRIZHEVSKY A, SUTSKEVER I, HINTON G. ImageNet classification with deep convolutional neural networks[J]. Communications of the ACM,2017,60(6):84-90.
[11] ZHANG Y, ZHOU D, CHEN S, et al. Single-image crowd counting via multi-column convolutional neural network[C]//Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway:IEEE,2016:589-597.
[12] ZHANG C,LI H,WANG X,et al. Cross-scene crowd counting via deep convolutional neural networks[C]//Proceedings of the 2015 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway:IEEE,2015:833-841.
[13] IDREES H,SALEEMI I,SEIBERT C,et al. Multi-source multiscale counting in extremely dense crowd images[C]//Proceedings of the 2013 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway:IEEE,2013:2547-2554.
[14] IDREES H,TAYYAB M,ATHREY K,et al. Composition loss for counting,density map estimation and localization in dense crowds[C]//Proceedings of the 2018 European Conference on Computer Vision. Berlin:Springer,2018:544-559.
[15] SAM D B,SURYA S,BABU R V. Switching convolutional neural network for crowd counting[C]//Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway:IEEE,2017:4031-4039.
[16] SINDAGI V A,PATEL V M. Generating high-quality crowd density maps using contextual pyramid CNNs[C]//Proceedings of the 2017 IEEE International Conference on Computer Vision. Piscataway:IEEE,2017:1879-1888.
[17] 陆金刚, 张莉. 基于多尺度多列卷积神经网络的密集人群计数模型[J]. 计算机应用,2019,39(12):3445-3449.(LU J G, ZHANG L. Crowd counting model based on multi-scale multicolumn convolutional neural network[J]. Journal of Computer Applications,2019,39(12):3445-3449.)
[18] LI Y, ZHANG X, CHEN D. CSRNet:dilated convolutional neural networks for understanding the highly congested scenes[C]//Proceedings of the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway:IEEE,2018:1091-1100.
[19] 陈美云, 王必胜, 曹国, 等. 基于像素级注意力机制的人群计数方法[J]. 计算机应用,2020,40(1):56-61.(CHEN M Y, WANG B S,CAO G,et al. Crowd counting method based on pixel-level attention mechanism[J]. Journal of Computer Applications,2020,40(1):56-61.)
[20] ZHU L,ZHAO Z,LU C,et al. Dual path multi-scale fusion networks with attention for crowd counting[EB/OL].[2019-12-10]. https://arxiv.org/pdf/1902.01115.pdf.
[21] ZHANG Y,ZHOU C,CHANG F,et al. Multi-resolution attention convolutional neural network for crowd counting[J]. Neurocomputing,2018,329:144-152.
[22] ZOU Z,LIU Y,XU S,et al. Crowd counting via hierarchical scale recalibration network[EB/OL].[2020-02-01]. https://arxiv.org/pdf/2003.03545.pdf.
[23] VARIOR R R,SHUAI B,TIGHE J,et al. Multi-scale attention network for crowd counting[EB/OL].[2020-02-01]. https://arxiv.org/pdf/1901.06026.pdf.
[24] WOO S,PARK J,LEE J Y,et al. CBAM:convolutional block attention module[C]//Proceedings of the 2018 European Conference on Computer Vision,LNCS 11211. Cham:Springer, 2018:3-19.
[25] CAO X,WANG Z,ZHAO Y,et al. Scale aggregation network for accurate and efficient crowd counting[C]//Proceedings of the 2018 European Conference on Computer Vision,LNCS 11209. Cham:Springer,2018:757-773.
[26] ZHANG A,SHEN J,XIAO Z,et al. Relational attention network for crowd counting[C]//Proceedings of the 2019 IEEE/CVF International Conference on Computer Vision. Piscataway:IEEE, 2019:6787-6796.