基于深度神经网络的移动端人像分割

doi:10.11772/j.issn.1001-9081.2020050699

计算机应用 ›› 2020, Vol. 40 ›› Issue (12): 3644-3650.DOI: 10.11772/j.issn.1001-9081.2020050699

• 虚拟现实与多媒体计算 • 上一篇下一篇

基于深度神经网络的移动端人像分割

杨坚伟¹, 严群^1,2, 姚剑敏^1,2, 林志贤¹

1. 福州大学物理与信息工程学院, 福州 350108;
2. 晋江市博感电子科技有限公司, 福建晋江 362200

收稿日期:2020-05-25 修回日期:2020-07-13 出版日期:2020-12-10 发布日期:2020-08-14
通讯作者: 严群(1965-),男,美籍华人,教授,博士,主要研究方向:Micro-LED、人工智能、信息显示。qunfyan@gmail.com
作者简介:杨坚伟(1995-),男,福建泉州人,硕士研究生,主要研究方向:深度学习、图像语义分割;姚剑敏(1978-),男,福建莆田人,副研究员,博士,主要研究方向:人工智能、图像处理、信息显示;林志贤(1975-),男,福建泉州人,教授,博士,主要研究方向:信息显示、平板显示驱动系统、图像处理
基金资助:
国家重点研发计划项目（2016YFB0401503）；广东省科技重大专项（2016B090906001）；福建省科技重大专项（2014HZ0003-1）；广东省光信息材料与技术重点实验室开放基金资助项目（2017B030301007）。

Portrait segmentation on mobile devices based on deep neural network

YANG Jianwei¹, YAN Qun^1,2, YAO Jianmin^1,2, LIN Zhixian¹

1. College of Physics and Information Engineering, Fuzhou University, Fuzhou Fujian 350108, China;
2. Jinjiang RichSense Electronic Technology Company Limited, Jinjiang Fujian 362200, China

Received:2020-05-25 Revised:2020-07-13 Online:2020-12-10 Published:2020-08-14
Supported by:
This work is partially supported by National Key Research and Development Program of China （2016YFB0401503）， the Science and Technology Major Program of Guangdong Province （2016B090906001）， the Science and Technology Major Program of Fujian Province （2014HZ0003-1）， the Open Foundation of Guangdong Provincial Key Laboratory of Optical Information Materials and Technology （2017B030301007）.

摘要/Abstract

摘要： 针对现有的人像分割算法大多忽略移动设备的硬件限制，盲目追求效果，以致无法满足移动端对于分割速度要求的问题，提出了一种可在移动设备上高效运行的人像分割网络。首先，基于编码器-解码器的轻量级U型架构来构建网络；其次，为了克服全卷积网络（FCN）受制于较小的感受域，无法充分捕获长距离信息的缺陷，引入期望最大化注意力块（EMAU）置于编码器之后、解码器之前；然后，在训练阶段添加多层边界辅助损失，有助于提高人物边界轮廓的准确度；最后，对模型进行量化和压缩。在Veer数据集上将所提网络与PortraitFCN+、ENet和BiSeNet等网络进行对比实验。实验结果表明，所提网络可以提高图像推理速度和分割效果，并能够以95.57%的准确率处理分辨率为224×224的RGB图像。

关键词: 深度神经网络, 人像分割, 期望最大化, 辅助损失, 注意力

Abstract: Most of the existing portrait segmentation algorithms ignore the hardware limitation of mobile devices and blindly pursue the effect, so that they cannot meet the segmentation speed requirement of mobile terminals. Therefore, a portrait segmentation network which could run efficiently on mobile devices was proposed. Firstly, the network was constructed based on the lightweight U-shaped architecture of encoder-decoder. Secondly, in order to make up for the fact that the Fully Convolutional Network (FCN) was limited by a small sensing domain, so that it was not able to fully capture the long-distance information, an Expectation Maximization Attention Unit (EMAU) was introduced after the encoder and before the decoder. Thirdly, for improving the accuracy of portrait boundary contour, a multi-layer boundary auxiliary loss was added at the training stage. Finally, the model was quantized and compressed. The proposed network was compared with other networks such as PortraitFCN+, ENet and BiSeNet on Veer dataset. Experimental results show that, the proposed network can improve the image reasoning speed and segmentation effect, as well as process the RGB images with the resolution of 224×224 at the accuracy of 95.57%.

Key words: deep neural network, portrait segmentation, Expectation Maximization (EM), auxiliary loss, attention

中图分类号:

TP389.1

杨坚伟, 严群, 姚剑敏, 林志贤. 基于深度神经网络的移动端人像分割[J]. 计算机应用, 2020, 40(12): 3644-3650.

YANG Jianwei, YAN Qun, YAO Jianmin, LIN Zhixian. Portrait segmentation on mobile devices based on deep neural network[J]. Journal of Computer Applications, 2020, 40(12): 3644-3650.

参考文献

[1] LONG J,SHELHAMER E,DARRELL T. Fully convolutionalnetworks for semantic segmentation[C]//Proceedings of the 2015 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway:IEEE,2015:3431-3440.
[2] RONNEBERGER O,FISCHER P,BROX T. U-Net:convolutionalnetworks for biomedical image segmentation[C]//Proceedings of the 2015 International Conference on Medical Image Computing and Computer-Assisted Intervention, LNCS 9351. Cham:Springer, 2015:234-241.
[3] CHEN L C,PAPANDREOU G,KOKKINOS I,et al. DeepLab:semantic image segmentation with deep convolutionalnets,atrous convolution,and fully connected CRFs[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2018, 40(4):834-848.
[4] CHEN L C,PAPANDREOU G,KOKKINOS I,et al. Semantic image segmentation with deep convolutionalnets and fully connected CRFs[EB/OL].[2019-12-29]. https://arxiv.org/pdf/1412.7062.pdf.
[5] CHEN L C,PAPANDREOU G,SCHROFF F,et al. Rethinking atrous convolution for semantic image segmentation[EB/OL].[2019-12-29]. https://arxiv.org/pdf/1706.05587.pdf.
[6] DU X,WANG X,LI D,et al. Boundary-sensitivenetwork for portrait segmentation[EB/OL].[2019-12-29]. https://arxiv.org/pdf/1712.08675.pdf.
[7] SANDLER M, HOWARD A, ZHU M, et al. MobileNetV2:inverted residuals and linear bottlenecks[C]//Proceedings of the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway:IEEE,2018:4510-4520.
[8] HE K,ZHANG X,REN S,et al. Deep residual learning for image recognition[C]//Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway:IEEE, 2016:770-778.
[9] BADRINARAYANAN V,KENDALL A,CIPOLLA R. SegNet:a deep convolutional encoder-decoder architecture for image segmentation[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence,2017,39(12):2481-2495.
[10] ZHANG Z,ZHANG X,CHAO P,et al. ExFuse:enhancing feature fusion for semantic segmentation[C]//Proceedings of the 2018 European Conference on Computer Vision,LNCS 11214. Cham:Springer,2018:273-288.
[11] ZHAO H,SHI J,QI X,et al. Pyramid scene parsingnetwork[C]//Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway:IEEE,2017:6230-6239.
[12] WANG X,GIRSHICK R,GUPTA A,et al. Non-local neuralnetworks[C]//Proceedings of the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway:IEEE, 2018:7794-7803.
[13] DEMPSTER A P,LAIRD N M,RUBIN D B. Maximum likelihood from incomplete data via the EM algorithm[J]. Journal of the Royal Statistical Society:Series B(Methodological),1977,39(1):1-22.
[14] HOWARD A G,ZHU M,CHEN B,et al. MobileNets:efficient convolutional neuralnetworks for mobile vision applications[EB/OL].[2019-12-29]. https://arxiv.org/pdf/1704.04861.pdf.
[15] 王本杰, 农丽萍, 张文辉, 等. 基于Spider卷积的三维点云分类与分割网络[J]. 计算机应用, 2020, 40(6):1607-1612.(WANG B J, NONG L P, ZHANG W H, et al. 3D point cloud classification and segmentationnetwork based on Spider convolution[J]. Journal of Computer Applications,2020,40(6):1607-1612.)
[16] LI X, ZHONG Z, WU J, et al. Expectation-maximization attentionnetworks for semantic segmentation[C]//Proceedings of the 2019 IEEE/CVF International Conference on Computer Vision. Piscataway:IEEE,2019:9166-9175.
[17] 赵杨璐, 段丹丹, 胡饶敏, 等. 基于EM算法的混合模型中子总体个数的研究[J]. 数理统计与管理, 2020, 39(1):35-50. (ZHAO Y L,DUAN D D,HU R M,et al. On the number ofcomponents in mixture model based on EM algorithm[J]. Journal of Applied Statistics and Management,2020,39(1):35-50.)
[18] HE K,ZHANG X,REN S,et al. Delving deep into rectifiers:surpassing human-level performance on ImageNet classification[C]//Proceedings of the 2015 IEEE International Conference on Computer Vision. Piscataway:IEEE,2015:1026-1034.
[19] ZHANG S,DONG X,LI H,et al. PortraitNet:real-time portrait segmentationnetwork for mobile device[J]. Computers and Graphics,2019,80:104-113.
[20] LIN T Y,GOYAL P,GIRSHICK R,et al. Focal loss for dense object detection[C]//Proceedings of the 2017 IEEE International Conference on Computer Vision. Piscataway:IEEE,2017:2999-3007.
[21] 杨威, 张建林, 徐智勇, 等. 一种改进的Focal Loss在语义分割上的应用[J]. 半导体光电, 2019, 40(4):555-559.(YANG W, ZHANG J L,XU Z Y,et al. An improved focal loss function for semantic segmentation[J]. Journal of Semiconductor Optoelectronics,2019,40(4):555-559.)
[22] XU N,PRICE B,COHEN S,et al. Deep image matting[C]//Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway:IEEE,2017:311-320
[23] SHEN X,HERTZMANN A,JIA J,et al. Automatic portrait segmentation for image stylization[J]. Computer Graphics Forum,2016,35(2):93-102.
[24] PASZKE A,CHAURASIA A,KIM S,et al. ENet:a deep neuralnetwork architecture for real-time semantic segmentation[EB/OL].[2020-03-16]. https://arxiv.org/pdf/1606.02147.pdf.
[25] YU C,WANG J,PENG C,et al. BiSeNet:bilateral segmentationnetwork for real-time semantic segmentation[C]//Proceedings of the 2018 European Conference on Computer Vision, LNCS 11217. Cham:Springer,2018:334-349.

基于深度神经网络的移动端人像分割

Portrait segmentation on mobile devices based on deep neural network

PDF

可视化

摘要/Abstract

引用本文

使用本文

参考文献

相关文章 15

编辑推荐

Metrics

[1]	代雨柔, 杨庆, 张凤荔, 周帆. 基于自监督学习的社交网络用户轨迹预测模型[J]. 计算机应用, 2021, 41(9): 2545-2551.
[2]	刘雅璇, 钟勇. 基于头实体注意力的实体关系联合抽取方法[J]. 计算机应用, 2021, 41(9): 2517-2522.
[3]	李康康, 张静. 基于注意力机制的多层次编码和解码的图像描述模型[J]. 计算机应用, 2021, 41(9): 2504-2509.
[4]	赵宏, 孔东一. 图像特征注意力与自适应注意力融合的图像内容中文描述[J]. 计算机应用, 2021, 41(9): 2496-2503.
[5]	牛瑞华, 杨俊, 邢斓馨, 吴仁彪. 基于卷积注意力模块和双通道网络的微表情识别算法[J]. 计算机应用, 2021, 41(9): 2552-2559.
[6]	党伟超, 李涛, 白尚旺, 高改梅, 刘春霞. 基于自注意力长短期记忆网络的Web软件系统实时剩余寿命预测方法[J]. 计算机应用, 2021, 41(8): 2346-2351.
[7]	何正海, 线岩团, 王蒙, 余正涛. 融合句法指导与字符注意力机制的案情阅读理解方法[J]. 计算机应用, 2021, 41(8): 2427-2431.
[8]	李扬志, 袁家政, 刘宏哲. 基于时空注意力图卷积网络模型的人体骨架动作识别算法[J]. 计算机应用, 2021, 41(7): 1915-1921.
[9]	张洋, 江铭虎. 基于注意力机制的文本作者识别[J]. 计算机应用, 2021, 41(7): 1897-1901.
[10]	高钦泉, 黄炳城, 刘文哲, 童同. 基于改进CenterNet的竹条表面缺陷检测方法[J]. 计算机应用, 2021, 41(7): 1933-1938.
[11]	王曙燕, 侯则昱, 孙家泽. 面向深度学习的对抗样本差异性检测方法[J]. 计算机应用, 2021, 41(7): 1849-1856.
[12]	武维, 李泽平, 杨华蔚, 林川, 王忠德. 融合内容特征和时序信息的深度注意力视频流行度预测模型[J]. 计算机应用, 2021, 41(7): 1878-1884.
[13]	谭道强, 曾诚, 乔金霞, 张俊. 基于混合注意力模型的阴影检测方法[J]. 计算机应用, 2021, 41(7): 2076-2081.
[14]	李朝, 兰海, 魏宪. 基于注意力的毫米波-激光雷达融合目标检测[J]. 计算机应用, 2021, 41(7): 2137-2144.
[15]	刘世泽, 朱奕达, 陈润泽, 罗海勇, 赵方, 孙艺, 王宝会. 基于残差时域注意力神经网络的交通模式识别算法[J]. 计算机应用, 2021, 41(6): 1557-1565.