Advances in automatic image annotation

doi:10.11772/j.issn.1001-9081.2016.08.2274

Journal of Computer Applications ›› 2016, Vol. 36 ›› Issue (8): 2274-2281.DOI: 10.11772/j.issn.1001-9081.2016.08.2274

Previous Articles Next Articles

Advances in automatic image annotation

LIU Mengdi¹, CHEN Yanli¹, CHEN Lei^1,2,3

1. College of Computer, Nanjing University of Posts and Telecommunications, Nanjing Jiangsu 210003, China;
2. Jiangsu High Technology Research Key Laboratory for Wireless Sensor Networks(Nanjing University of Posts and Telecommunications), Nanjing Jiangsu 210003, China;
3. College of Computer Science and Technology, Nanjing University of Aeronautics and Astronautics, Nanjing Jiangsu 210016, China

Received:2015-12-14 Revised:2016-03-21 Online:2016-08-10 Published:2016-08-10
Supported by:
This work is partially supported by the Fundamental Research Funds for the Central Universities (JD1413).

图像自动标注技术研究进展

刘梦迪¹, 陈燕俐¹, 陈蕾^1,2,3

1. 南京邮电大学计算机学院, 南京 210003;
2. 江苏省无线传感网高技术研究重点实验室(南京邮电大学), 南京 210003;
3. 南京航空航天大学计算机科学与技术学院, 南京 210016

通讯作者: 刘梦迪
作者简介:刘梦迪(1993-),女,河南通许人,硕士研究生,CCF会员,主要研究方向:大规模机器学习;陈燕俐(1969-),女,江苏苏州人,教授,博士,主要研究方向:智能信息处理、网络信息安全;陈蕾(1975-),男,江西宜春人,副教授,博士,CCF会员,主要研究方向:大规模机器学习、模式识别、数据挖掘。
基金资助:
国家自然科学基金资助项目（61572263）；中国博士后科学基金资助项目（2015M581794）；江苏省高校自然科学研究面上项目（15KJB520027）；江苏省博士后科研资助计划项目（1501023C）；南京邮电大学校级科研基金资助项目（NY214127,NY215096）。

Abstract

Abstract: Existing image annotation algorithms can be roughly divided into four categories:the semantics based methods, the probability based methods, the matrix decomposition based methods and the graph learning based methods. Some representative algorithms for every category were introduced and the problem models and characteristics of these algorithms were analyzed. Then the main optimization methods of these algorithms were induced, and the common image datasets and the evaluation metrics of these algorithms were introduced. Finally, the main problems of automatic image annotation were pointed out, and the solutions to these problems were put forward. The analytical results show that the full use of complementary advantages of the current algorithms, or taking multi-disciplinary advantages may provide more efficient algorithm for automatic image annotation.

Key words: image retrieval, automatic image annotation, tag completion, tag denoising, tag prediction

摘要： 现有图像自动标注技术算法可以大致划分为基于语义的标注算法、基于矩阵分解的标注算法、基于概率的标注算法以及基于图学习的标注算法等四大类。介绍了各类别中具有代表性的图像自动标注算法，分析了这些算法的问题模型及其功能特点，并归纳了图像自动标注算法中主要的优化求解方法及算法评价中常用的图像数据集和性能评价指标。最后，指出了图像自动标注技术目前存在的主要问题，并且提出了这些问题的解决思路。分析结果表明，对于图像自动标注技术的研究，可充分利用现有算法的优势互补，或借助多学科交叉的优势，寻找更有效的算法。

关键词: 图像检索, 图像自动标注, 标签填补, 标签去噪, 标签预测

CLC Number:

LIU Mengdi, CHEN Yanli, CHEN Lei. Advances in automatic image annotation[J]. Journal of Computer Applications, 2016, 36(8): 2274-2281.

刘梦迪, 陈燕俐, 陈蕾. 图像自动标注技术研究进展[J]. 计算机应用, 2016, 36(8): 2274-2281.

References

[1] DATTA R,JOSHI D,LI J,et al.Image retrieval:ideas,influ-ences,and trends of the new age[J].ACM Computing Surveys,2008,40(2):Article No.5.
[2] SMEULDERS A W M,WORRING M,SANTINI S,et al.Content-based image retrieval at the end of the early years[J].IEEE Transactions on Pattern Analysis and Machine Intelligence,2000,22(12):1349-1380.
[3] DESELAERS T,KEYSERS D,NEY H.Features for image retrieval:an experimental comparison[J].Information Retrieval,2008,11(2):77-107.
[4] KULKARNI A.Association-based image retrieval[C]//Proceedings of the 201042nd Southeastern Symposium on System Theory.Piscataway,NJ:IEEE,2010:30-34.
[5] CHEN L,YANG G,ZHANG Y,et al.Asymptotically stable multi-valued many-to-many associative memory neural network and its application in image retrieval[J].Neural Network World,2013,2(13):169-189.
[6] 鲍泓,徐光美,冯松鹤,等.自动图像标注技术研究进展[J].计算机科学,2011,38(7):35-40.(BAO H,XU G M,FENG S H,et al.Advances in automatic image annotation[J].Computer Science,2011,38(7):35-40.)
[7] MILLER G A.WordNet:a lexical database for English[J].Communications of the ACM,1995,38(11):39-41.
[8] JIN Y,KHAN L,WANG L,et al.Image annotations by combining multiple evidence&WordNet[C]//Proceedings of the 13th Annual ACM International Conference on Multimedia.New York:ACM,2005:706-715.
[9] LEE S,DE NEVE W,RO Y M.Image tag refinement along the' what' dimension using tag categorization and neighbor voting[C]//Proceedings of the 2010 IEEE International Conference on Multimedia and Expo.Piscataway,NJ:IEEE,2010:48-53.
[10] LIU D,HUA X S,YANG L,et al.Tag ranking[C]//Proceedings of the 18th International Conference on World Wide Web.New York:ACM,2009:351-360.
[11] LIU X,YAN S,CHUA T S,et al.Image label completion by pursuing contextual decomposability[J].ACM Transactions on Multimedia Computing,Communications,and Applications,2012,8(2):Article No.21.
[12] ZHU G,YAN S,MA Y.Image tag refinement towards low-rank,content-tag prior and error sparsity[C]//Proceedings of the 2010 International Conference on Multimedia.New York:ACM,2010:461-470.
[13] XIA Z,FENG X,PENG J,et al.A regularized optimization framework for tag completion and image retrieval[J].Neurocomputing,2015,147:500-508.
[14] LI X,ZHANG Y J,SHEN B,et al.Image tag completion by low-rank factorization with dual reconstruction structure preserved[C]//Proceedings of the 2014 IEEE International Conference on Image Processing.Piscataway,NJ:IEEE,2014:3062-3066.
[15] XU H,WANG J,HUA X S,et al.Tag refinement by regularized LDA[C]//Proceedings of the 17th ACM International Conference on Multimedia.New York:ACM,2009:573-576.
[16] JORDAN M I,BLEI D M,NG A Y.Latent Dirichlet allocation[J].Journal of Machine Learning Research,2003,3:465-473.
[17] LIU D,HUA X S,WANG M,et al.Image retagging[C]//Proceedings of the 2010 International Conference on Multimedia.New York:ACM,2010:491-500.
[18] LEE S,DE NEVE W,RO Y M.Tag refinement in an image folksonomy using visual similarity and tag co-occurrence statistics[J].Signal Processing:Image Communication,2010,25(10):761-773.
[19] XIA Z,FENG X,PENG J,et al.Content-irrelevant tag cleansing via bi-layer clustering and peer cooperation[J].Journal of Signal Processing Systems,2014,81(1):29-44.
[20] FREY B J,DUECK D.Clustering by passing messages between data points[J].Science,2007,315(5814):972-976.
[21] 卢汉清,刘静.基于图学习的自动图像标注[J].计算机学报,2008,31(9):1629-1639.(LU H Q,LIU J.Image annotation based on graph learning[J].Chinese Journal of Computers,2008,31(9):1629-1639.)
[22] LIU J,LI M,LIU Q,et al.Image annotation via graph learning[J].Pattern Recognition,2009,42(2):218-228.
[23] LIU D,YAN S,HUA X S,et al.Image retagging using collaborative tag propagation[J].IEEE Transactions on Multimedia,2011,13(4):702-712.
[24] FENG Z,FENG S,JIN R,et al.Image tag completion by noisy matrix recovery[M]//ECCV 2014:Proceedings of the 13th European Conference on Computer Vision.Berlin:Springer,2014:424-438.
[25] LIN Z,DING G,HU M,et al.Image tag completion via image-specific and tag-specific linear sparse reconstructions[C]//Proceedings of the 2013 IEEE Conference on Computer Vision and Pattern Recognition.Piscataway,NJ:IEEE,2013:1618-1625.
[26] LIN Z,DING G,HU M,et al.Image tag completion via dual-view linear sparse reconstructions[J].Computer Vision and Image Understanding,2014,124:42-60.
[27] WANG Q,RUAN L,ZHANG Z,et al.Learning compact hashing codes for efficient tag completion and prediction[C]//Proceedings of the 22nd ACM International Conference on Conference on Information&Knowledge Management.New York:ACM,2013:1789-1794.
[28] WU L,JIN R,JAIN A K.Tag completion for image retrieval[J].IEEE Transactions on Pattern Analysis and Machine Intelligence,2013,35(3):716-727.
[29] ZNAIDIA A,LE BORGNE H,HUDELOT C.Tag completion based on belief theory and neighbor voting[C]//Proceedings of the 3rd ACM Conference on International Conference on Multimedia Retrieval.New York:ACM,2013:49-56.
[30] SHAFER G.A Mathematical Theory of Evidence[M].Princeton:Princeton University Press,1976:35-46.
[31] KRIZHEVSKY A,SUTSKEVER I,HINTON G E.ImageNet classification with deep convolutional neural networks[C]//NIPS 2012:Advances in Neural Information Processing Systems.Cambridge:MIT Press,2012:1106-1114.
[32] CIRESAN D,MEIER U,SCHMIDHUBER J.Multi-column deep neural networks for image classification[C]//Proceedings of the 25th IEEE Conference on Computer Vision and Pattern Recognition.Piscataway,NJ:IEEE,2012:3642-3649.
[33] SRIVASTAVA N,SALAKHUTDINOV R.Learning representations for multimodal data with deep belief nets[C]//Proceedings of the 29th International Conference on Machine Learning Workshop.New York:ACM,2012:1-8.
[34] FENG F X,LI R F,WANG X J.Deep correspondence restricted Boltzmann machine for cross-modal retrieval[J].Neurocomputing,2015,154:50-60.
[35] 杨阳,张文生.基于深度学习的图像自动标注算法[J].数据采集与处理,2015,30(1):88-98.(YANG Y,ZHANG W S.Image auto-annotation based on deep learning[J].Journal of Data Acquisition and Processing,2015,30(1):88-98.)
[36] DUYGULU P,BARNARD K,DE FREITAS J F G,et al.Object recognition as machine translation:Learning a lexicon for a fixed image vocabulary[C]//Proceedings of the 2002 European Conference on Computer Vision.Berlin:Springer,2002:97-112.
[37] RUSSELL B C,TORRALBA A,MURPHY K P,et al.LabelMe:a database and Web-based tool for image annotation[J].International Journal of Computer Vision,2008,77(1/2/3):157-173.
[38] SHOTTON J,WINN J,ROTHER C,et al.TextonBoost:joint appearance,shape and context modeling for multi-class object recognition and segmentation[C]//ECCV 2006:Proceedings of the 9th European Conference on Computer Vision.Berlin:Springer,2006:1-15.
[39] HUISKES M J,LEW M S.The MIR flickr retrieval evaluation[C]//Proceedings of the 1st ACM International Conference on Multimedia Information Retrieval.New York:ACM,2008:39-43.
[40] CHUA T S,TANG J,HONG R,et al.NUS-WIDE:a real-world Web image database from National University of Singapore[C]//Proceedings of the 2009 ACM International Conference on Image and Video Retrieval.New York:ACM,2009:Article No.48.
[41] GRUBINGER M,CLOUGH P,MVLLER H,et al.The IAPR TC-12 benchmark:a new evaluation resource for visual information systems[C]//Proceedings of the 2006 International Workshop OntoImage Language Resources for Content-Based Image Retrieval.Genoa,Italy:[s.n.],2006:13-23.
[42] MAKADIA A,PAVLOVIC V,KUMAR S.Baselines for image annotation[J].International Journal of Computer Vision,2010,90(1):88-105.
[43] JÄRVELIN K,KEKÄLÄINEN J.Cumulated gain-based evaluation of IR techniques[J].ACM Transactions on Information Systems,2002,20(4):422-44.

Advances in automatic image annotation

图像自动标注技术研究进展

PDF

Knowledge

Abstract

Cite this article

share this article

References

Related Articles 15

Recommended Articles

Metrics

[1]	YANG Su, OUYANG Zhi, DU Nisuo. Unsupervised parallel hash image retrieval based on correlation distance [J]. Journal of Computer Applications, 2021, 41(7): 1902-1907.
[2]	LU Rongxiu, CHEN Mingming, YANG Hui, ZHU Jianyong. Element component content dynamic monitoring system based on time sequence characteristics of solution images [J]. Journal of Computer Applications, 2021, 41(10): 3075-3081.
[3]	CAO Jianfang, ZHAO Aidi, ZHANG Zibang. Application of convolutional neural network with threshold optimization in image annotation [J]. Journal of Computer Applications, 2020, 40(6): 1587-1592.
[4]	GU Junhua, WANG Feng, QI Yongjun, SUN Zheran, TIAN Zepei, ZHANG Yajuan. Retrieval method of pulmonary nodule images based on multi-scale convolution feature fusion [J]. Journal of Computer Applications, 2020, 40(2): 561-565.
[5]	SHUI Liucheng, LIU Weizhong, FENG Zhuoming. Automatic image annotation based on generative adversarial network [J]. Journal of Computer Applications, 2019, 39(7): 2129-2133.
[6]	ZHANG Meiling, WU Junfeng, YU Hong, CUI Zhen, DONG Wanting. Fish image retrieval algorithm based on color four channels and spatial pyramid [J]. Journal of Computer Applications, 2019, 39(5): 1466-1472.
[7]	QIN Pingle, LI Qi, ZENG Jianchao, ZHANG Na, SONG Yulong. Image retrieval algorithm for pulmonary nodules based on multi-scale dense network [J]. Journal of Computer Applications, 2019, 39(2): 392-397.
[8]	CHEN Hongyu, DENG Dexiang, YAN Jia, FAN Ci'en. Image retrieval algorithm based on saliency semantic region weighting [J]. Journal of Computer Applications, 2019, 39(1): 136-142.
[9]	YANG Xiaoling, LI Zhiqing, LIU Yutong. Automatic image annotation based on multi-label discriminative dictionary learning [J]. Journal of Computer Applications, 2018, 38(5): 1294-1298.
[10]	WANG Peng, ZHANG Aofan, WANG Liqin, DONG Yongfeng. Image automatic annotation based on transfer learning and multi-label smoothing strategy [J]. Journal of Computer Applications, 2018, 38(11): 3199-3203.
[11]	ZHU Jie, ZHANG Junsan, WU Shufang, DONG Yukun, LYU Lin. Multi-center convolutional feature weighting based image retrieval [J]. Journal of Computer Applications, 2018, 38(10): 2778-2781.
[12]	GAO Yaodong, HOU Lingyan, YANG Dali. Automatic image annotation method using multi-label learning convolutional neural network [J]. Journal of Computer Applications, 2017, 37(1): 228-232.
[13]	HUANG Xiaodong, SUN Liang. Retrieval method of images based on robust Cosine-Euclidean metric dimensionality reduction [J]. Journal of Computer Applications, 2016, 36(8): 2292-2295.
[14]	GUO Qian, YANG Hongju, LIANG Xinyan. Image retrieval method based on new space relationship feature [J]. Journal of Computer Applications, 2016, 36(7): 1918-1922.
[15]	ZHAI Minghan, GAO Ling. New image retrieval method based on weighted color stratification and texture unit [J]. Journal of Computer Applications, 2016, 36(6): 1668-1672.