Face sketch-photo synthesis based on locality-constrained neighbor embedding

doi:10.11772/j.issn.1001-9081.2015.02.0535

Journal of Computer Applications ›› 2015, Vol. 35 ›› Issue (2): 535-539.DOI: 10.11772/j.issn.1001-9081.2015.02.0535

Previous Articles Next Articles

Face sketch-photo synthesis based on locality-constrained neighbor embedding

HU Yanting¹, WANG Nannan², CHEN Jianjun^1,3, MURAT Hamit¹, ABDUGHRNI Kutluk¹

1. School of Medical Engineering Technology, Xinjiang Medical University, Urumqi Xinjiang 830011, China;
2. School of Telecommunications Engineering, Xidian University, Xi'an Shaanxi 710071, China;
3. School of Physical Science and Technology, Southwest University, Chongqing 400715, China

Received:2014-08-04 Revised:2014-09-16 Online:2015-02-12 Published:2015-02-10

基于局部约束邻域嵌入的人脸画像照片合成

胡彦婷¹, 王楠楠², 陈建军^1,3, 木拉提·哈米提¹, 阿布都艾尼·库吐鲁克¹

1. 新疆医科大学医学工程技术学院, 乌鲁木齐 830011;
2. 西安电子科技大学通信工程学院, 西安 710071;
3. 西南大学物理科学与技术学院, 重庆 400715

通讯作者: 木拉提·哈米提
作者简介:胡彦婷(1980-),女,湖北孝感人,讲师,硕士,主要研究方向:图像超分辨重建、计算机视觉; 王楠楠(1986-),男,山东潍坊人,博士,主要研究方向: 异质图像转换、计算机视觉、机器学习; 陈建军(1977-),男,河南开封人,讲师,博士研究生,主要研究方向:光电信号处理; 木拉提·哈米提(1957-),男(维族),新疆乌鲁木齐人,教授,主要研究方向:图像处理及分析; 阿布都艾尼·库吐鲁克(1977-),男(维族),新疆喀什人,副教授,博士,主要研究方向:信息检测与处理。
基金资助:
国家自然科学基金资助项目(81160182,61201125)。

Abstract

Abstract:

The neighboring relationship of sketch patches and photo patches on the manifold cannot always reflect their intrinsic data structure. To resolve this problem, a Locality-Constrained Neighbor Embedding (LCNE) based face sketch-photo synthesis algorithm was proposed. The Neighbor Embedding (NE) based synthesis method was first applied to estimate initial sketches or photos. Then, the weight coefficients were constrained according to the similarity between the estimated sketch patches or photo patches and the training sketch patches or training photo patches. Subsequently, alternative optimization was deployed to determine the weight coefficients, select K candidate image patches and update the target synthesis patch. Finally, the synthesized image was generated by merging all the estimated sketch patches or photo patches. In the contrast experiments, the proposed method outperformed the NE based synthesis method by 0.0503 in terms of Structural SIMilarity (SSIM) index and by 14% in terms of face recognition accuracy. The experimental results illustrate that the proposed method resolves the problem of weak compatibility among neighbor patches in the NE based method and greatly alleviates the noises and deformations in the synthetic image.

Key words: face sketch-photo synthesis, local constraint, Neighbor Embedding (NE), manifold learning, face recognition

摘要：

针对画像块和照片块在流形上的邻域关系并不能完全反映彼此内在数据结构的问题,提出一种基于局部约束邻域嵌入(LCNE)的画像-照片合成算法。首先,利用基于邻域嵌入(NE)的合成方法得到待合成照片或画像的初始估计;其次,根据待合成的照片块或画像块与训练集中的照片块或画像块的相似性来约束合成权值;然后,通过交替优化方法进行权值的确定和K-近邻的选择,并更新待合成目标块;最后,合并所有估计的照片块或画像块合成目标图像。与基于邻域嵌入的画像照片合成方法相比,所提方法合成图像的结构相似度提高0.0503,脸识别准确率提高14%。实验结果表明,该方法解决了基于NE方法导致的邻域之间兼容性不强的问题,能大大减少合成图像上的噪声及变形。

关键词: 人脸画像-照片合成, 局部约束, 邻域嵌入, 流形学习, 人脸识别

CLC Number:

TP391.41

HU Yanting, WANG Nannan, CHEN Jianjun, MURAT Hamit, ABDUGHRNI Kutluk. Face sketch-photo synthesis based on locality-constrained neighbor embedding[J]. Journal of Computer Applications, 2015, 35(2): 535-539.

胡彦婷, 王楠楠, 陈建军, 木拉提·哈米提, 阿布都艾尼·库吐鲁克. 基于局部约束邻域嵌入的人脸画像照片合成[J]. 计算机应用, 2015, 35(2): 535-539.

References

[1] TANG X, WANG X. Face sketch recognition [J]. IEEE Transactions on Circuits and Systems for Video Technology, 2004, 14(1): 50-57.
[2] WANG N, TAO D, GAO X, et al. A comprehensive survey to face hallucination [J]. International Journal of Computer Vision, 2014, 106(1): 9-30.
[3] GAO X, WANG N. Heterogeneous face image synthesis [M]//Machine Learning and Its Applications. Beijing: Tsinghua University Press, 2013: 79-94. (高新波,王楠楠. 异质人脸图像合成 [M]//机器学习及其应用. 北京: 清华大学出版社, 2013: 79-94.)
[4] WANG N, TAO D, GAO X, et al. Transductive face sketch-photo synthesis [J]. IEEE Transactions on Neural Networks and Learning Systems, 2013, 24(9): 1364-1376.
[5] TANG X, WANG X. Face photo recognition using sketch [C]//Proceedings of the 2002 IEEE International Conference on Image Processing. Piscataway: IEEE, 2002, 1: 257-260.
[6] TANG X, WANG X. Face sketch synthesis and recognition [C]//Proceedings of the 9th IEEE International Conference on Computer Vision. Piscataway: IEEE, 2003, 1: 687-694.
[7] LI Y-H, SAVVIDES M, BHAGAVATULA V. Illumination tolerance face recognition using a novel face from sketch synthesis approach and advanced correlation filters [C]//ICASSP 2006: Proceedings of the 2006 IEEE International Conference on Acoustics, Speech and Signal Processing. Piscataway: IEEE, 2006: 357-360.
[8] LIU Q, TANG X, JIN H, et al. A nonlinear approach for face sketch synthesis and recognition [C]//CVPR 2005: Proceedings of the 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. Washington, DC: IEEE Computer Society, 2005, 1: 1005-1010.
[9] LIU W, TANG X, LIU J. Bayesian tensor inference for sketch-based face photo hallucination [C]//IJCAI'07: Proceedings of the 20th International Joint Conference on Artificial Intelligence. San Francisco: Morgan Kaufmann, 2007: 2141-2146.
[10] CHANG M, ZHOU L, HAN Y, et al. Face sketch synthesis via sparse representation [C]//Proceedings of the 20th International Conference on Pattern Recognition. Piscataway: IEEE, 2010: 2146-2149.
[11] GAO X, WANG N, TAO D, et al. Face sketch-photo synthesis and retrieval using sparse representation [J].IEEE Transactions on Circuits and Systems for Video Technology,2012,22(8):1213-1226.
[12] WANG S, ZHANG L, LIANG Y, et al. Semi-coupled dictionary learning with applications to image super-resolution and photo-sketch synthesis [C]//Proceedings of the 2012 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE, 2012: 2216-2223.
[13] GAO X, ZHONG J, LI J, et al. Face sketch synthesis algorithm based on E-HMM and selective ensemble [J]. IEEE Transactions on Circuits and Systems for Video Technology, 2008, 18(4): 487-496.
[14] GAO X, ZHONG J, TAO D, et al. Local face sketch synthesis learning [J]. Neurocomputing, 2008, 71(10/11/12): 1921-1930.
[15] XIAO B, GAO X, LI X, et al. A new approach for face recognition by sketches in photos [J]. Signal Processing. 2009, 89(8): 1531-1539.
[16] XIAO B, GAO X, TAO D, et al. Photo-sketch synthesis and recognition based on subspace learning [J]. Neurocomputing. 2010, 73(4/5/6): 840-852.
[17] WANG X, TANG X. Face photo-sketch synthesis and recognition [J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2009, 31(11): 1955-1967.
[18] ZHANG W, WANG X, TANG X. Lighting and pose robust face sketch synthesis [C]//ECCV 2010: Proceedings of the 11th European Conference on Computer Vision, LNCS 6316. Berlin: Springer, 2010: 420-423.
[19] ZHOU H, KUANG Z, WONG K-Y K. Markov weight fields for face sketch synthesis [C]//Proceedings of the 2012 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE, 2012: 1091-1097.
[20] WANG J, YANG J, YU K, et al. Locality-constrained linear coding for image classification [C] //Proceedings of the 2010 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE, 2010: 3360-3367.
[21] WANG X, TANG X. Face photo-sketch synthesis and recognition [J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2009, 31(11): 1955-1967.
[22] MARTINEE A M, BENAVENTE R. The AR face database, CVC Technical Report #24 [R]. Columbus: Ohio State University, 1998.
[23] MESSER K, MATAS J, KITTLER J, et al. XM2VTSDB: the extended M2VTS database [C]//AVBPA '99: Proceedings of the 2nd International Conference on Audio Video-Based Biometric Person Authentication. Berlin: Springer, 1999: 72-77.
[24] WANG Z, BOVIK A C, SHEIKH H R, et al. Image quality assessment: from error visibility to structural similarity [J]. IEEE Transactions on Image Processing, 2004, 13(4): 600-612.
[25] TURK M, PENTLAND A. Face recognition using eigenfaces [C] //CVPR '91: Proceedings of the 1991 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. Washington, DC: IEEE Computer Society, 1991: 586-591.
[26] WRIGHT J, YANG A Y, GANESH A, et al. Robust face recognition via sparse representation [J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2009, 31(2): 210-227.
[27] REN X, LIU X, HU T, et al. Objective quality evaluation of image fusion based on visual attention mechanism and regional structural similarity [J]. Journal of Computer Applications, 2014, 31(11): 3022-3026. (任仙怡, 刘秀坚, 胡涛, 等. 基于视觉注意机制与区域结构相似度的图像融合质量评价 [J]. 计算机应用, 2014, 31(11): 3022-3026.)

Face sketch-photo synthesis based on locality-constrained neighbor embedding

基于局部约束邻域嵌入的人脸画像照片合成

PDF

Knowledge

Abstract

Cite this article

share this article

References

Related Articles 15

Recommended Articles

Metrics

[1]	Kansong CHEN, Yuan ZHENG, Lijun XU, Zhouyu WANG, Zhe ZHANG, Fujuan YAO. ShuffaceNet： face recognition neural network based on ThetaMEX global pooling [J]. Journal of Computer Applications, 2023, 43(8): 2572-2580.
[2]	Chungang HAN, Yonghui LIU. Face liveness detection algorithm based on GhostNet and feature fusion [J]. Journal of Computer Applications, 2023, 43(8): 2588-2592.
[3]	Mingyue WU, Dong ZHOU, Wenyu ZHAO, Wei QU. Sentence embedding optimization based on manifold learning [J]. Journal of Computer Applications, 2023, 43(10): 3062-3069.
[4]	Ruijie YANG, Guilin ZHENG. Face liveness detection based on InceptionV3 and feature fusion [J]. Journal of Computer Applications, 2022, 42(7): 2037-2042.
[5]	Huaiqing HE, Jianqing YAN, Kanghua HUI. Lightweight face recognition method based on deep residual network [J]. Journal of Computer Applications, 2022, 42(7): 2030-2036.
[6]	Lili FAN, Guifu LU, Ganyi TANG, Dan YANG. Low-rank representation subspace clustering method based on Hessian regularization and non-negative constraint [J]. Journal of Computer Applications, 2022, 42(1): 115-122.
[7]	GAO Gong, YANG Hongyu, LIU Hong. 3D point cloud face recognition based on deep learning [J]. Journal of Computer Applications, 2021, 41(9): 2736-2740.
[8]	CHEN Zhengtao, HUANG Can, YANG Bo, ZHAO Li, LIAO Yong. Yak face recognition algorithm of parallel convolutional neural network based on transfer learning [J]. Journal of Computer Applications, 2021, 41(5): 1332-1336.
[9]	LI Hongxia, QIN Pinle, YAN Hanmei, ZENG Jianchao, BAO Qianyue, CHAI Rui. Face frontalization generative adversarial network algorithm based on face feature map symmetry [J]. Journal of Computer Applications, 2021, 41(3): 714-720.
[10]	Han DU, Xianzhong LONG, Yun LI. Graph learning regularized discriminative non-negative matrix factorization based face recognition [J]. Journal of Computer Applications, 2021, 41(12): 3455-3461.
[11]	ZHAO Qing, YU Yuanhui. 3D face recognition based on hierarchical feature network [J]. Journal of Computer Applications, 2020, 40(9): 2514-2518.
[12]	FANG Shuya, LIU Shouyin. Non-perception class attendance method based on student body detection [J]. Journal of Computer Applications, 2020, 40(9): 2519-2524.
[13]	WANG Jinkai, JIA Xu. Vehicle face recognition algorithm based on NMF with weighted and orthogonal constraints [J]. Journal of Computer Applications, 2020, 40(4): 1050-1055.
[14]	LI Dongbo, HUANG Lyuwen. Reweighted sparse principal component analysis algorithm and its application in face recognition [J]. Journal of Computer Applications, 2020, 40(3): 717-722.
[15]	GONG Rui, DING Sheng, ZHANG Chaohua, SU Hao. Lightweight and multi-pose face recognition method based on deep learning [J]. Journal of Computer Applications, 2020, 40(3): 704-709.