Clothing retrieval based on landmarks

doi:10.11772/j.issn.1001-9081.2017.11.3249

Abstract

Abstract: At present, the same or similar style clothing retrieval is mainly text-based or content-based. The text-based algorithms often require massive labled samples, and the shortages of exist label missing and annotation difference caused by artificial subjectivity. The content-based algorithms usually extract image features, such as color, shape, texture, and then measured the similarity, but it is difficult to deal with background color interference, and clothing deformation due to different angles, attitude, etc. Aiming at these problems, clothing retrieval based on landmarks was proposed. The proposed method used cascaded deep convolutional neural network to locate the key points and combined the low-level visual information of the key point region as well as the high-level semantic information of the whole image. Compared with traditional methods, the proposed method can effectively deal with the distortion of clothing and complex background interference due to angle of view and attitude. Meanwhile, it does not need huge labeled samples, and is robust to background and deformation. Experiments on two large scale datasets Fashion Landmark and BDAT-Clothes show that the proposed algorithm can effectively improve the precision and recall.

Key words: landmark, deep convolution neural network, cascade, clothing retrieval

摘要： 目前，同款或近似款式服装检索主要分为基于文本和基于内容两类。基于文本算法往往需要海量标注样本，且存在人工主观性带来的标注缺失和标注差异等问题；基于内容算法一般对服装图像的颜色、形状、纹理提取特征，进行相似性度量，但难以应对背景颜色干扰，以及视角、姿态引起的服装形变等问题。针对上述问题，提出一种基于关键点的服装检索方法。利用级联深度卷积神经网络为基础，定位服装关键点，融合关键点区域低层视觉信息以及整幅图像的高层语义信息。对比传统检索方法，所提算法能有效处理视角、姿态引起的服装形变和复杂背景的干扰；同时不需大量样本标定，且对背景、形变鲁棒。在Fashion Landmark数据集和BDAT-Clothes数据集上与常用算法进行对比实验。实验结果表明所提算法能有效提升检索的查准率和查全率。

关键词: 关键点, 深度卷积神经网络, 级联, 服装检索

CLC Number:

TP311

CHEN Aiai, LI Lai, LIU Guangcan, LIU Qingshan. Clothing retrieval based on landmarks[J]. Journal of Computer Applications, 2017, 37(11): 3249-3255.

陈嫒嫒, 李来, 刘光灿, 刘青山. 基于关键点的服装检索[J]. 计算机应用, 2017, 37(11): 3249-3255.

References

[1] 薛培培, 邬延辉. 基于图像内容和支持向量机的服装图像检索方法研究[J]. 移动通信, 2016(2):79-82.(XUE P P, WU Y H. A clothing image retrieval method based on image content and support vector machine[J]. Mobile Communication, 2016(2):79-82.)
[2] 兰丽, 耿增民. 服装图像检索研究综述[J]. 电脑知识与技术, 2015, 11(12):184-187.(LAN L, GENG Z M. The clothing image research review[J]. Computer Knowledge and Technology, 2015, 11(12):184-187.)
[3] LIU S, SONG Z, LIU G, et al. Street-to-shop:cross-scenario clothing retrieval via parts alignment and auxiliary set[C]//Proceedings of the 20th ACM International Conference on Multimedia. New York:ACM, 2012:1335-1336.
[4] WEI D, WAH C, BHARDWAJ A, et al. Style finder:fine-grained clothing style detection and retrieval[C]//Proceedings of the 2013 IEEE Conference on Computer Vision and Pattern Recognition Workshops. Piscataway, NJ:IEEE, 2013:8-13.
[5] HUANG J, FERIS R S, CHEN Q, et al. Cross-domain image retrieval with a dual attribute-aware ranking network[C]//Proceedings of the 2015 IEEE International Conference on Computer Vision. Piscataway, NJ:IEEE, 2015:1062-1070.
[6] VEIT A, KOVACS B, BELL S, et al. Learning visual clothing style with heterogeneous dyadic co-occurrences[C]//Proceedings of the 2015 IEEE International Conference on Computer Vision. Piscataway, NJ:IEEE, 2015:4642-4650.
[7] LIANG X, LIN L, YANG W, et al. Clothes co-parsing via joint image segmentation and labeling with application to clothing retrieval[J]. IEEE Transactions on Multimedia, 2016, 18(6):1175-1186.
[8] 应玉龙. 基于内容的服装图像检索系统研究[J]. 福建电脑, 2016, 32(3):25-26. (YING Y L. Research on clothing image retrieval system based on content[J]. Fujian Computer, 2016, 32(3):25-26.)
[9] KIAPOUR M H, HAN X, LAZEBNIK S, et al. Where to buy it:matching street clothing photos in online shops[C]//Proceedings of the 2015 IEEE International Conference on Computer Vision. Piscataway, NJ:IEEE, 2015:3343-3351.
[10] LI L, LIU G, LIU Q. Advancing iterative quantization hashing using isotropic prior[C]//Proceedings of the 22nd International Conference on Multimedia Modeling. Berlin:Springer, 2016:174-184.
[11] 张腾, 郭清宇. 基于HSV颜色直方图的服装检索[J]. 信息与电脑, 2011(10):28-29.(ZHANG T, GUO Q Y. Clothing retrieval based on HSV color histogram[J]. China Computer and Communication, 2011(10):28-29.)
[12] 陈倩, 潘中良. 基于内容的服装检索系统中颜色特征提取算法的研究和改进[J]. 激光杂志, 2016, 37(4):62-68.(CHEN Q, PAN Z L. Research and improvement of color feature extraction algorithms in the content-based clothing images retrieval system[J]. Laser Journal, 2016, 37(4):62-68.)
[13] 李克磊, 刘正东. 基于SURF算法的服装视觉图像检索研究[J]. 北京服装学院学报(自然科学版), 2014, 34(3):57-62. (LI K L, LIU Z D. Visual clothing image retrieval research based on the SURF algorithm[J]. Journal of Beijing Institute of Clothing Technology (Natural Science Edition), 2014, 34(3):57-62.)
[14] 侯阿临, 赵柳青, 桃敏, 等. 基于多特征的服装图像检索[J]. 现代电子技术, 2010, 33(6):171-175. (HOU A L, ZHAO L Q, TAO M, et al. Clothing image retrieval based on multi-features[J]. Modern Electronics Technique, 2010, 33(6):171-175.)
[15] 王海龙, 杜俊俐, 郭清宇. 基于内容的图像检索技术在服装检索中的应用[J]. 计算技术与自动化, 2009, 28(2):88-91.(WANG H L, DU J L, GUO Q Y. The application of content based image retrieval technology in clothing retrieval system[J]. Computer Technology and Automation, 2009, 28(2):88-91.)
[16] LIU H, WANG R, SHAN S, et al. Deep supervised hashing for fast image retrieval[C]//Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway, NJ:IEEE, 2016:2064-2072.
[17] KRIZHEVSKY A, SUTSKEVER I, HINTON G E. ImageNet classification with deep convolutional neural networks[C]//Proceedings of the 25th International Conference on Neural Information Processing Systems. Lake Tahoe:Curran Associates Inc., 2012:1097-1105.
[18] SZEGEDY C, LIU W, JIA Y, et al. Going deeper with convolutions[C]//Proceedings of the 2015 IEEE Conference on Computer Vision and Pattern Recognition. Washington, DC:IEEE Computer Society, 2015:1-9.
[19] HE K, ZHANG X, REN S, et al. Delving deep into rectifiers:surpassing human-level performance on ImageNet classification[C]//Proceedings of the 2015 IEEE International Conference on Computer Vision. Piscataway, NJ:IEEE, 2015:1026-1034.
[20] REN S, HE K, GIRSHICK R, et al. Faster R-CNN:towards real-time object detection with region proposal networks[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2016, 39(6):1137-1149.
[21] LIU W, ANGUELOV D, ERHAN D, et al. SSD:single shot multibox detector[C]//Proceedings of the 14th European Conference on Computer Vision. Berlin:Springer, 2015:21-37.
[22] REDMON J, DIVVALA S, GIRSHICK R, et al. You only look once:unified, real-time object detection[C]//Proceedings of the 2015 IEEE Conference on Computer Vision and Pattern Recognition. Washington, DC:IEEE Computer Society, 2015:779-788.
[23] SUN Y, WANG X, TANG X. Deep convolutional network cascade for facial point detection[C]//Proceedings of the 2013 IEEE Conference on Computer Vision and Pattern Recognition. Washington, DC:IEEE Computer Society, 2013:3476-3483.
[24] SUN Y, WANG X, TANG X. Deep learning face representation by joint identification-verification[EB/OL].[2016-11-20]. https://core.ac.uk/download/pdf/25035739.pdf.
[25] DENG J, DING N, JIA Y, et al. Large-scale object classification using label relation graphs[M]//Proceedings of the 13th European Conference on Computer Vision. Berlin:Springer, 2014:48-64.
[26] SHELHAMER E, LONG J, DARRELL T. Fully convolutional networks for semantic segmentation[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2016, 79(10):1337-1342.
[27] LIU Z, YAN S, LUO P, et al. Fashion landmark detection in the wild[C]//Proceedings of the 14th European Conference on Computer Vision. Berlin:Springer, 2016:229-245.
[28] LIU Z, LUO P, QIU S, et al. DeepFashion:powering robust clothes recognition and retrieval with rich annotations[C]//Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition. Washington, DC:IEEE Computer Society, 2016:1096-1104.
[29] OLIVA A, TORRALBA A. Modeling the shape of the scene:a holistic representation of the spatial envelope[J]. International Journal of Computer Vision, 2001, 42(3):145-175.