Zero-shot image classification based on visual error and semantic attributes

doi:10.11772/j.issn.1001-9081.2019081475

Journal of Computer Applications ›› 2020, Vol. 40 ›› Issue (4): 1016-1022.DOI: 10.11772/j.issn.1001-9081.2019081475

• Artificial intelligence • Previous Articles Next Articles

Zero-shot image classification based on visual error and semantic attributes

XU Ge¹, XIAO Yongqiang^2,3,4, WANG Tao^1,2, CHEN Kaizhi^2,3,4, LIAO Xiangwen^2,3,4, WU Yunbing^2,3,4

1. College of Computer and Control Engineering, Minjiang University, Fuzhou Fujian 350108, China;
2. College of Mathematics and Computer Science, Fuzhou University, Fuzhou Fujian 350116, China;
3. Fujian Provincial Key Laboratory of Networking Computing and Intelligent Information Processing(Fuzhou University), Fuzhou Fujian 350116, China;
4. Digital Fujian Financial Big Data Institute, Fuzhou Fujian 350116, China

Received:2019-09-03 Revised:2019-10-23 Online:2020-04-10 Published:2019-11-18
Supported by:
This work is partially supported by the National Natural Science Foundation of China(61772135,U1605251,61703195),the Open Fund of the Key Laboratory of Network Data Science and Technology of the Chinese Academy of Sciences(CASNDST201708,CASNDST 201606), the Open Fund of the State Key Laboratory of Pattern Recognition (201900041),the Surface Program of Fujian Natural Science Foundation (2017J01755), the CERNET Innovation Project(NGII20160501).

基于视觉误差与语义属性的零样本图像分类

徐戈¹, 肖永强^2,3,4, 汪涛^1,2, 陈开志^2,3,4, 廖祥文^2,3,4, 吴运兵^2,3,4

1. 闽江学院计算机与控制工程学院, 福州 350108;
2. 福州大学数学与计算机科学学院, 福州 350116;
3. 福建省网络计算与智能信息处理重点实验室(福州大学), 福州 350116;
4. 数字福建金融大数据研究所, 福州 350116

通讯作者: 廖祥文
作者简介:徐戈(1978-),男,福建福州人,副教授,博士,主要研究方向:人工智能、自然语言处理;肖永强(1994-),男,福建龙岩人,硕士研究生,主要研究方向:多模态、人工智能;汪涛(1987-),男,福建福州人,讲师,博士,主要研究方向:场景理解、目标检测与分割;陈开志(1983-),男,福建福州人,讲师,博士,主要研究方向:自然语言处理、图像识别、深度学习算法;廖祥文(1980-),男,福建福州人,教授,博士,主要研究方向:观点挖掘、情感分析、自然语言处理;吴运兵(1976-),男,福建福州人,副教授,硕士,主要研究方向:知识表示与知识发现。
基金资助:
国家自然科学基金资助项目（61772135，U1605251，61703195）；中国科学院网络数据科学与技术重点实验室开放课题基金资助项目（CASNDST201708，CASNDST201606）；模式识别国家重点实验室开放课题基金资助项目（201900041）；福建省自然科学基金面上项目（2017J01755）；赛尔网络下一代互联网技术创新项目（NGII20160501）。

Abstract

Abstract: In the practical applications of image classification,some categories may have no labeled training data at all. The purpose of Zero-Shot Learning(ZSL)is to transfer knowledge such as image features of labeled categories to unlabeled categories and to correctly classify the unlabeled categories. However,the existing state-of-the-art methods cannot explicitly distinguish the input image belonging to the known categories or unknown categories,which leads to a large performance gap for unlabeled categories between the traditional ZSL prediction and the Generalized ZSL(GZSL)prediction. Therefore,a method of fusing of visual error and semantic attributes was proposed to alleviate the prediction bias problem in zero-shot image classification. Firstly,a semi-supervised learning based generative adversarial network framework was designed to obtain visual error information,so as to predict whether the image belongs to the known categories. Then,a zero-shot image classification network combining semantic attributes was proposed to achieve zero-shot image classification. Finally,the performance of zero-shot image classification algorithm combining visual error and semantic attributes was tested on AwA2 (Animal with Attributes) and CUB (Caltech-UCSD-Birds-200-2011) datasets. The experimental results show that, compared to the baseline models,the proposed method can effectively alleviate the prediction bias problem,and has the harmonic index H increased by 31. 7 percentage points on AwA2 dataset and 8. 7 percentage points on CUB dataset.

Key words: Zero-Shot Learning (ZSL), image classification, generative adversarial network, visual error, semantic attribute

摘要： 在图像分类的实际应用过程中，部分类别可能完全没有带标签的训练数据。零样本学习（ZSL）的目的是将带标签类别的图像特征等知识迁移到无标签的类别上，实现无标签类别的正确分类。现有方法在测试时无法显式地区分输入图像属于已知类还是未知类，很大程度上导致未知类在传统设定下的ZSL和广义设定下的ZSL（GZSL）上的预测效果相差甚远。为此，提出一种融合视觉误差与属性语义信息的方法来缓解零样本图像分类中的预测偏置问题。首先，设计一种半监督学习方式的生成对抗网络架构来获取视觉误差信息，由此预测图像是否属于已知类；然后，提出融合属性语义信息的零样本图像分类网络来实现零样本图像分类；最后，测试融合视觉误差与属性语义的零样本图像分类方法在数据集AwA2和CUB上的效果。实验结果表明，与对比模型相比，所提方法有效缓解了预测偏置问题，其调和指标H在AwA2（Animal with Attributes）上提升了31.7个百分点，在CUB（Caltech-UCSD-Birds-200-2011）上提升了8.7个百分点。

关键词: 零样本学习, 图像分类, 生成对抗网络, 视觉误差, 属性语义

CLC Number:

TP391

XU Ge, XIAO Yongqiang, WANG Tao, CHEN Kaizhi, LIAO Xiangwen, WU Yunbing. Zero-shot image classification based on visual error and semantic attributes[J]. Journal of Computer Applications, 2020, 40(4): 1016-1022.

徐戈, 肖永强, 汪涛, 陈开志, 廖祥文, 吴运兵. 基于视觉误差与语义属性的零样本图像分类[J]. 计算机应用, 2020, 40(4): 1016-1022.

References

[1] LAROCHELLE H,ERHAN D,BENGIO Y. Zero-data learning of new tasks[C]//Proceedings of the 23rd AAAI Conference on Artificial Intelligence. Palo Alto, CA:AAAI Press, 2008:646-651.
[2] LAMPERT C H,NICKISCH H,HARMELING S. Learning to detect unseen object classes by between-class attribute transfer[C]//Proceedings of the 2009 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway:IEEE,2009:951-958.
[3] LAMPERT C H,NICKISCH H,HARMELING S. Attribute-based classification for zero-shot visual object categorization[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence,2014, 36(3):453-465.
[4] ROMERA-PAREDES B,TORR P H S. An embarrassingly simple approach to zero-shot learning[C]//Proceedings of the 32nd International Conference on Machine Learning. New York:JMLR. org, 2015:2152-2161.
[5] LIU M,ZHANG D,CHEN S. Attribute relation learning for zero-shot classification[J]. Neurocomputing,2014,139:34-46.
[6] 巩萍, 程玉虎, 王雪松. 基于属性关系图正则化特征选择的零样本分类[J]. 中国矿业大学学报,2015,44(6):1097-1104. (GONG P,CHEN Y H,WANG X S. Zero-shot classification based on attribute correlation graph regularized feature selection[J]. Journal of China University of Mining and Technology,2015,44(6):1097-1104.)
[7] AKATA Z,REED S,WALTER D,et al. Evaluation of output embeddings for fine-grained image classification[C]//Proceedings of the 2015 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway:IEEE,2015:2927-2936.
[8] XIAN Y,AKATA Z,SHARMA G,et al. Latent embeddings for zero-shot classification[C]//Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway:IEEE,2016:69-77.
[9] AKATA Z, MALINOWSKI M, FRITZ M, et al. Multi-cue zero-shot learning with strong supervision[C]//Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway:IEEE,2016:59-68.
[10] ZHANG L,XIANG T,GONG S. Learning a deep embedding model for zero-shot learning[C]//Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway:IEEE,2017:3010-3019.
[11] ZHU Y,ELHOSEINY M,LIU B,et al. A generative adversarial approach for zero-shot learning from noisy texts[C]//Proceedings of the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway:IEEE,2018:1004-1013.
[12] SUNG F,YANG Y,ZHANG L,et al. Learning to compare:relation network for few-shot learning[C]//Proceedings of the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway:IEEE,2018:1199-1208.
[13] LI Y,ZHANG J,ZHANG J,et al. Discriminative learning of latent features for zero-shot recognition[C]//Proceedings of the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway:IEEE,2018:7463-7471.
[14] FU J,ZHENG H,MEI T. Look closer to see better:recurrent attention convolutional neural network for fine-grained image recognition[C]//Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway:IEEE, 2017:4476-4484.
[15] AKCAY S, ATAPOUR-ABARGHOUEI A, BRECKON T P. GANomaly:semi-supervised anomaly detection via adversarial training[C]//Proceedings of the 2018 Asian Conference on Computer Vision,LNCS 11363. Cham:Springer,2018:622-637.
[16] RUSSAKOVSKY O,DENG J,SU H,et al. ImageNet large scale visual recognition challenge[J]. International Journal of Computer Vision,2015,115(3):211-252.
[17] HE K,ZHANG X,REN S,et al. Deep residual learning for image recognition[C]//Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway:IEEE, 2016:770-778.

Zero-shot image classification based on visual error and semantic attributes

基于视觉误差与语义属性的零样本图像分类

PDF

Knowledge

Abstract

Cite this article

share this article

References

Related Articles 15

Recommended Articles

Metrics

[1]	GUAN Qijie, ZHANG Ting, LI Deya, ZHOU Shaojing, DU Yi. Indefinite reconstruction method of spatial data based on multi-resolution generative adversarial network [J]. Journal of Computer Applications, 2021, 41(8): 2306-2311.
[2]	SUN Xiao, XU Jindong. Remote sensing image dehazing method based on cascaded generative adversarial network [J]. Journal of Computer Applications, 2021, 41(8): 2440-2444.
[3]	TANG Guihua, SUN Lei, MAO Xiuqing, DAI Leyu, HU Yongjin. Generative adversarial network synthesized face detection based on deep alignment network [J]. Journal of Computer Applications, 2021, 41(7): 1922-1927.
[4]	WANG Xianwu, ZHANG Ting, JI Xin, DU Yi. 3D shale digital core reconstruction method based on deep convolutional generative adversarial network with gradient penalty [J]. Journal of Computer Applications, 2021, 41(6): 1805-1811.
[5]	LI Yanzhi, FAN Yong, GAO Lin. Anomaly detection of oil drilling water flow based on shape flow [J]. Journal of Computer Applications, 2021, 41(6): 1842-1848.
[6]	JIA Heming, LANG Chunbo, JIANG Zichao. Plant leaf disease recognition method based on lightweight convolutional neural network [J]. Journal of Computer Applications, 2021, 41(6): 1812-1819.
[7]	GAN Lan, SHEN Hongfei, WANG Yao, ZHANG Yuejin. Data augmentation method based on improved deep convolutional generative adversarial networks [J]. Journal of Computer Applications, 2021, 41(5): 1305-1313.
[8]	SUN Heli, SUN Yuzhu, ZHANG Xiaoyun. Event description generation based on generative adversarial network [J]. Journal of Computer Applications, 2021, 41(5): 1256-1261.
[9]	GUO Maozu, YANG Qiannan, ZHAO Lingling. Image generation based on conditional-Wassertein generative adversarial network [J]. Journal of Computer Applications, 2021, 41(5): 1432-1437.
[10]	DUAN Youxiang, ZHANG Hanxiao, SUN Qifeng, SUN Youkai. Image super-resolution reconstruction algorithm based on Laplacian pyramid generative adversarial network [J]. Journal of Computer Applications, 2021, 41(4): 1020-1026.
[11]	OU Lili, SHAO Fengjing, SUN Rencheng, SUI Yi. Cerebral infarction image recognition based on semi-supervised method [J]. Journal of Computer Applications, 2021, 41(4): 1221-1226.
[12]	LI Hongxia, QIN Pinle, YAN Hanmei, ZENG Jianchao, BAO Qianyue, CHAI Rui. Face frontalization generative adversarial network algorithm based on face feature map symmetry [J]. Journal of Computer Applications, 2021, 41(3): 714-720.
[13]	XU Xuebin, ZHANG Jiada, LIU Wei, LU Longbin, ZHAO Yuqing. High-precision classification method for breast cancer fusing spatial features and channel features [J]. Journal of Computer Applications, 2021, 41(10): 3025-3032.
[14]	ZHANG Ya, JIN Xin, JIANG Qian, LEE Shin-jye, DONG Yunyun, YAO Shaowen. Deepfake image detection method based on autoencoder [J]. Journal of Computer Applications, 2021, 41(10): 2985-2990.
[15]	QIU Yaoru, SUN Weijun, HUANG Yonghui, TANG Yuqi, ZHANG Haochuan, WU Junpeng. Person re-identification method based on GAN uniting with spatial-temporal pattern [J]. Journal of Computer Applications, 2020, 40(9): 2493-2498.