Traffic image semantic retrieval method based on specific object self-recognition

doi:10.11772/j.issn.1001-9081.2019101795

Journal of Computer Applications ›› 2020, Vol. 40 ›› Issue (2): 553-560.DOI: 10.11772/j.issn.1001-9081.2019101795

• CCF Bigdata 2019 • Previous Articles Next Articles

Traffic image semantic retrieval method based on specific object self-recognition

Yi ZHAO¹^,², Xing DUAN¹(), Shiyi XIE¹, Chunlin LIANG¹^,²

^1.College of Mathematics and Computer Science，Guangdong Ocean University，Zhanjiang Guangdong 524000，China
^2.Guangdong Zhanjiang Bay Laboratory，Fisheries Big Data Center of South China Sea，Zhanjiang Guangdong 524000，China

Received:2019-08-30 Revised:2019-10-24 Accepted:2019-10-24 Online:2019-10-31 Published:2020-02-10
Contact: Xing DUAN
About author:ZHAO Yi， born in 1984， Ph. D.， lecturer. His research interests include software engineering， artificial intelligence.
XIE Shiyi， born in 1963， M. S.， professor. His research interests include software engineering.
LIANG Chunlin， born in 1975， M. S.， associate professor. His research interests include software engineering.
Supported by:
the Guangdong Laboratory Autonomous Set Up Major Project for Southern Marine Science and Engineering(ZJW-2019-08);the Major Scientific Research Projects of Innovative and Strong University of Guangdong Ocean University(GDOU2017052501)

面向特定目标自识别的交通图像语义检索方法

赵一¹^,², 段兴¹(), 谢仕义¹, 梁春林¹^,²

^1.广东海洋大学数学与计算机学院，广东湛江 524000
^2.湛江湾实验室南海渔业大数据中心，广东湛江 524000

通讯作者: 段兴
作者简介:赵一（1984—），男，湖北武汉人，讲师，博士，CCF会员，主要研究方向：软件工程、人工智能
谢仕义（l963—），男，广东湛江人，教授，硕士，主要研究方向：软件工程
梁春林（l975—），男，广东湛江人，副教授，硕士，主要研究方向：软件工程。
基金资助:
南方海洋科学与工程广东省实验室自主立项重大项目(ZJW-2019-08);广东海洋大学创新强校重大科研项目(GDOU2017052501)

Abstract

Abstract:

In order to retrieve images of traffic violations from a large number of road traffic images， a semantic retrieval method based on specific object self-recognition was proposed. Firstly， road traffic domain ontology as well as road traffic rule description were established by experts in traffic domain. Secondly， traffic image features were extracted by Convolutional Neural Network （CNN）， and combined with the strategy for classifying image features which is based on the proposed improved Support Vector Machine based Decision Tree （SVM-DT） algorithm， the specific objects and the spatial positional relationship between the objects in the traffic images were automatically recognized and mapped into the association relationship （rule instance） between the corresponding ontology instance and its objects. Finally， the image semantic retrieval result was obtained by reasoning based on ontology instances and rule instances. Experimental results show that the proposed method has higher accuracy， recall and retrieval efficiency compared to keyword and ontology traffic image semantic retrieval methods.

Key words: traffic ontology, image semantic retrieval, semantic reasoning, Support Vector Machine Decision Tree (SVM-DT) classification, object recognition

摘要：

为了从海量的道路交通图像中检索出违反交通法规的图像，提出了一种特定目标自识别的语义图像检索方法。首先，通过交通领域专家建立交通领域本体及道路交通规则描述；然后，通过卷积神经网络（CNN）对交通图像的特征进行提取，并结合改进的支持向量机决策树（SVM-DT）算法对图像特征进行分类的策略，对交通图像中的特定目标及目标间空间位置关系进行自动识别，并映射成为相应的本体实例及其对象之间的关联关系（规则实例）；最后，利用本体实例和规则实例，通过推理得到语义检索结果。实验结果表明，相比关键字和本体交通图像语义检索方法，所提方法具有更高的准确率、召回率和检索效率。

关键词: 交通领域本体, 图像语义检索, 语义推理, 支持向量机决策树分类, 目标识别

CLC Number:

TP751

Yi ZHAO, Xing DUAN, Shiyi XIE, Chunlin LIANG. Traffic image semantic retrieval method based on specific object self-recognition[J]. Journal of Computer Applications, 2020, 40(2): 553-560.

赵一, 段兴, 谢仕义, 梁春林. 面向特定目标自识别的交通图像语义检索方法[J]. 《计算机应用》唯一官方网站, 2020, 40(2): 553-560.

Figures/Tables 11

Fig. 1 Traffic image semantic retrieval framework based on domain knowledge

Tab. 1 Rules in the domain of road traffic

法规#	法规描述
Rule1	当斑马线前方红灯亮起时，如果行人还处在斑马线上，则判定行人违规
Rule2	当机动车道前方红灯亮起时，如果机动车压在停车线上，则判定机动车违规
Rule3	非机动车驶入机动车道行驶，则判定非机动车违规
Rule4	驾驶摩托车时后座乘坐不满12岁小孩，则判定摩托车违规
Rule5	机动车压在双黄线上行驶，则判定机动车违规

Fig. 2 Road traffic domain knowledge

Fig. 3 Local relationship of traffic graph instance

Fig. 4 Mapping of image objects and their associations with domain ontologies

Fig. 5 Example of positional relationship between cars and double solid lines

Tab. 2 Mapping of image object instances and ontology-related attributes

序号	图像目标实例	本体实例间关联属性
1	$R 1$ 处于 $R 2$ 上面	在上面（？A，？B）
2	$R 1$ 处于 $R 2$ 下面	在下面（？A，？B）
3	$R 1$ 处于 $R 2$ 左边	在左边（？A，？B）
4	$R 1$ 处于 $R 2$ 右边	在右边（？A，？B）
5	$R 1$ 处于 $R 2$ 中	相交（？A，？B）
6	$R 1$ 与 $R 2$ 平行	平行（？A，？B）
7	$R 1$ 处于 $R 2$ 左上	在左上（？A，？B）
8	$R 1$ 处于 $R 2$ 右上	在右上（？A，？B）
9	$R 1$ 处于 $R 2$ 左下	在左下（？A，？B）
10	$R 1$ 处于 $R 2$ 右下	在右下（？A，？B）

Tab. 2 Mapping of image object instances and ontology-related attributes

序号	图像目标实例	本体实例间关联属性
1	$R 1$ 处于 $R 2$ 上面	在上面（？A，？B）
2	$R 1$ 处于 $R 2$ 下面	在下面（？A，？B）
3	$R 1$ 处于 $R 2$ 左边	在左边（？A，？B）
4	$R 1$ 处于 $R 2$ 右边	在右边（？A，？B）
5	$R 1$ 处于 $R 2$ 中	相交（？A，？B）
6	$R 1$ 与 $R 2$ 平行	平行（？A，？B）
7	$R 1$ 处于 $R 2$ 左上	在左上（？A，？B）
8	$R 1$ 处于 $R 2$ 右上	在右上（？A，？B）
9	$R 1$ 处于 $R 2$ 左下	在左下（？A，？B）
10	$R 1$ 处于 $R 2$ 右下	在右下（？A，？B）

Fig. 6 Traffic image semantic retrieval tool framework based on target self-recognition

Fig. 7 Results of different methods for searching traffic violation images

Fig. 8 Result comparison of each algorithm on different indicators

Fig. 9 Results of retrieval by four rules

References 22

1	郭雪梅. 中国工程院院士高文：城市多媒体大数据高效存储与处理技术［EB/OL］. ［2019-02-20］.［EB/OL］. ［2019-02-20］..
2	高隽，谢昭，张骏，等. 图像语义分析与理解综述［J］. 模式识别与人工智能， 2010， 23（2）：191-202. 10.3969/j.issn.1003-6059.2010.02.010
	GAO J， XIE Z， ZHANG J. et al. Image semantic analysis and understanding： a review［J］. Pattern Recognition and Artificial Intelligence， 2010， 23（2）：191-202. 10.3969/j.issn.1003-6059.2010.02.010
3	DESEMO T M， ANTANI S， LONG R. Ontology of gaps in content-based image retrieval［J］. Journal of Digital Imaging， 2009， 22（2）：202-215. 10.1007/s10278-007-9092-x
4	SCHOBER J P， HERMES T， HERZOG O. Content-based image retrieval by ontology-based object recognition［C/OL］// Ki-2004 Workshop on Applications of Description Logics. ［2019-02-20］.. 10.1109/icme.2005.1521735
5	ZHOU W， LI H， SUN J， et al. Collaborative index embedding for image retrieval［J］. IEEE Transactions on Pattern Analysis and Machine Intelligence， 2018， 40（5）：1154-1166. 10.1109/tpami.2017.2676779
6	MEERSMAN R A. Semantic ontology tools in IS design［C］// Proceedings of the 1999 International Symposium on Methodologies for Intelligent Systems， LNCS1609. Berlin： Springer， 1999： 30-45.
7	KANIMOZHI T， CHRISTY A. Applications of ontology and semantic Web in image retrieval and research issues［J］. International Journal of Computer Science and Information Technologies， 2014， 5（1）：763-769
8	SCHREIBER A T， DUBBELDAM B， WIELEMAKER J， et al. Ontology-based photo annotation［J］. IEEE Intelligent Systems， 2001， 16（3）：66-74. 10.1109/5254.940028
9	HOLLINK L， SCHREIBER G， WIELEMAKER J， et al. Semantic annotation of image collections ［EB/OL］. ［2019-02-20］. . 10.2139/ssrn.3199367
10	POSLAD S， KESORN K. A Multi-Modal Incompleteness Ontology model （MMIO） to enhance information fusion for image retrieval［J］. Information Fusion， 2014， 20：225-241. 10.1016/j.inffus.2014.02.003
11	HAUBOLD A， NATSEV A， NAPHADE M R. Semantic multimedia retrieval using lexical query expansion and model-based reranking［C］// Proceedings of the 2006 IEEE International Conference on Multimedia and Expo. Piscataway： IEEE， 2006： 1761-1764. 10.1109/icme.2006.262892
12	DASIOPOULOU S， MEZARIS V， KOMPATSIARIS I， et al. An ontology-based framework for semantic image analysis and retrieval［M］// Semantic-Based Visual Information Retrieval. Hershey， PA： IRMA Press. 2007： 208-246. 10.4018/978-1-59904-370-8.ch010
13	KESORN K， POSLAD S. An enhanced bag-of-visual word vector space model to represent visual content in athletics images［J］. IEEE Transactions on Multimedia， 2012， 14（1）： 211-222. 10.1109/TMM.2011.2170665
14	NATSEV A， HAUBOLD A， TEŠIĆ J， et al. Semantic concept-based query expansion and re-ranking for multimedia retrieval［C］// Proceedings of the 15th ACM International Conference on Multimedia. New York： ACM， 2007： 991-1000. 10.1145/1291233.1291448
15	万华林， CHOWDHURY M U. 基于支持向量机的图像语义分类［J］. 软件学报， 2003， 14（11）：1891-1899.
	WAN H L， CHOWDHURY M U. Image semantic classification by using SVM［J］. Journal of Software， 2003， 14（11）：1891-1899.
16	DENG L. The MNIST database of handwritten digit images for machine learning research ［best of the Web］［J］. IEEE Signal Processing Magazine， 2012， 29（6）： 141-142. 10.1109/MSP.2012.2211477
17	MEZARIS V， KOMPATSIARIS I， STRINTZIS M G. An ontology approach to object-based image retrieval［C］// Proceedings of the 2003 International Conference on Image Processing. Piscataway： IEEE， 2003， 2： 511-514. 10.1109/icip.2003.1246729
18	AL-SMADI M， TALAFHA B， AL-AYYOUB M， et al. Using long short-term memory deep neural networks for aspect-based sentiment analysis of Arabic reviews［J］. International Journal of Machine Learning and Cybernetics， 2019， 10（8）：2163-2175. 10.1007/s13042-018-0799-4
19	REN S， HE K， GIRSHICK R， et al. Faster R-CNN： towards real-time object detection with region proposal networks［J］. IEEE Transactions on Pattern Analysis and Machine Intelligence， 2017， 39（6）： 1137-1149. 10.1109/tpami.2016.2577031
20	KANG J， Y-J PARK， LEE J. Novel leakage detection by ensemble CNN-SVM and graph-based localization in water distribution systems［J］. IEEE Transactions on Industrial Electronics， 2018， 65（5）： 4279-4289. 10.1109/tie.2017.2764861
21	WANG X， WU C， QI Y， et al. HRRP classification by using improved SVM decision tree［C］// Proceedings of the 6th World Congress on Intelligent Control and Automation. Piscataway： IEEE， 2006： 10096-10100. 10.1109/wcica.2006.1713975
22	连可，黄建国，王厚军，等. 一种基于遗传算法的SVM决策树多分类策略研究［J］. 电子学报， 2008， 36（8）：1502-1507. 10.3321/j.issn:0372-2112.2008.08.006
	LIAN K， HUANG J G， WANG H J， et al. Study on a GA-based SVM decision-tree multi-classification strategy［J］. Acta Electronica Sinica， 2008， 36（8）： 1502-1507. 10.3321/j.issn:0372-2112.2008.08.006

[1]	LIU Shuangyuan, ZHENG Wangli, LIN Yunhan. Adaptive binary simplification method for 3D feature descriptor [J]. Journal of Computer Applications, 2021, 41(7): 2062-2069.
[2]	HUANG Xueyu, XU Haote, TAO Jianwen. Multi-source adaptation classification framework with feature selection [J]. Journal of Computer Applications, 2020, 40(9): 2499-2506.
[3]	WANG Yiting, ZHANG Ke, LI Jie, HAO Zongbo, DUAN Chang, ZHU Ce. Recognition and localization method of super-large-scale variance objects in the same scene [J]. Journal of Computer Applications, 2020, 40(12): 3520-3525.
[4]	ZHU Jie, WU Shufang, XIE Bojun, MA Liyan. Color based compact hierarchical image representation [J]. Journal of Computer Applications, 2017, 37(11): 3238-3243.
[5]	LUO Jian, JIANG Min. Object recognition method based on RGB-D image kernel descriptor [J]. Journal of Computer Applications, 2017, 37(1): 255-261.
[6]	BAN Xiaokun, HAN Jun, LU Dongming, WANG Wanguo, LIU Liang. Similar circular object recognition method based on local contour feature in natural scenario [J]. Journal of Computer Applications, 2016, 36(5): 1399-1403.
[7]	HUANG Bin, LU Jinjin, WANG Jianhua, WU Xingming, CHEN Weihai. Object recognition algorithm based on deep convolution neural networks [J]. Journal of Computer Applications, 2016, 36(12): 3333-3340.
[8]	SHAO Mingzheng, QI Jianfeng, WANG Xiwu, WANG Lu. Object localization method based on fusion of visual saliency and superpixels [J]. Journal of Computer Applications, 2015, 35(1): 215-219.
[9]	Wei CHEN Xu-chu YU Peng-qiang ZHANG Zhi-chao WANG He WANG. Object recognition based on one-class support vector machine in hyperspectral image [J]. Journal of Computer Applications, 2011, 31(08): 2092-2096.
[10]	Fang CHEN Yun-liang JIANG Yun-xi XU. Improved CenSurE detector and a new rapid descriptor based on gradient of summed image patch [J]. Journal of Computer Applications, 2011, 31(07): 1818-1821.
[11]	QIU Liu-dong WANG Niu LI Zu-shu. Fast component amending and labeling algorithm for convex objects [J]. Journal of Computer Applications, 2011, 31(05): 1214-1216.
[12]	. Recognition of 3D points cloud object based on local fingerprint patch [J]. Journal of Computer Applications, 2010, 30(10): 2718-2722.
[13]	. Effective method for dam recognition from visible images [J]. Journal of Computer Applications, 2006, 26(8): 1972-1974.

Traffic image semantic retrieval method based on specific object self-recognition

面向特定目标自识别的交通图像语义检索方法

RichHTML

PDF

Knowledge

Abstract

Cite this article

share this article

Figures/Tables 11

References 22

Related Articles 13

Recommended Articles

Metrics