New dish recognition network based on lightweight YOLOv5

doi:10.11772/j.issn.1001-9081.2023030271

Journal of Computer Applications ›› 2024, Vol. 44 ›› Issue (2): 638-644.DOI: 10.11772/j.issn.1001-9081.2023030271

• Frontier and comprehensive applications • Previous Articles

New dish recognition network based on lightweight YOLOv5

Chenghanyu ZHANG¹, Yuzhe LIN¹, Chengke TAN¹, Junfan WANG¹^,², Yeting GU¹^,², Zhekang DONG¹^,², Mingyu GAO¹^,²()

^1.School of Electronics and Information Engineering，Hangzhou Dianzi University，Hangzhou Zhejiang 310018，China
^2.Zhejiang Provincial Key Laboratory of Equipment Electronics （Hangzhou Dianzi University），Hangzhou Zhejiang 310018，China

Received:2023-03-16 Revised:2023-04-19 Accepted:2023-04-23 Online:2023-05-18 Published:2024-02-10
Contact: Mingyu GAO
About author:ZHANG Chenghanyu， born in 2002， M. S. candidate. His research interests include computer vision， network lightweight.
LIN Yuzhe， born in 2002， M. S. candidate. His research interests include embedded development.
TAN Chengke， born in 2002， M. S. candidate. His research interests include robot arm cooperative control.
WANG Junfan， born in 1998， Ph. D. candidate. Her research interests include intelligent transportation， computer vision.
GU Yeting， born in 1998， M. S. candidate. Her research interests include computer vision， image detection.
DONG Zhekang， born in 1989， Ph. D.， associate professor. His research interests include neuromorphic computing， object detection.
Supported by:
National Key Research and Development Program of China(2020YFB1710600);National Natural Science Foundation of China(62001149);Key Research and Development Program of Zhejiang Province(2020C01110)

基于轻量化YOLOv5的新型菜品识别网络

张成涵宇¹, 林钰哲¹, 谭程珂¹, 王俊帆¹^,², 顾烨婷¹^,², 董哲康¹^,², 高明煜¹^,²()

^1.杭州电子科技大学电子信息学院，杭州 310018
^2.浙江省装备电子研究重点实验室（杭州电子科技大学），杭州 310018

通讯作者: 高明煜
作者简介:张成涵宇（2002—），男，浙江温州人，硕士研究生，主要研究方向：计算机视觉、网络轻量化
林钰哲（2002—），男，浙江金华人，硕士研究生，主要研究方向：嵌入式开发
谭程珂（2002—），男，浙江绍兴人，硕士研究生，主要研究方向：机械臂协同控制
王俊帆（1998—），女，浙江绍兴人，博士研究生，主要研究方向：智能交通、计算机视觉
顾烨婷（1998—），女，浙江嘉兴人，硕士研究生，主要研究方向：计算机视觉、图像检测
董哲康（1989—），男，浙江杭州人，副教授，博士，CCF会员，主要研究方向：神经形态计算、目标检测；
基金资助:
国家重点研发计划项目(2020YFB1710600);国家自然科学基金资助项目(62001149);浙江省重点研发计划项目(2020C01110)

Abstract

Abstract:

In order to better meet the accuracy and timeliness requirements of Chinese food dish recognition， a new type of dish recognition network was designed. The original YOLOv5 model was pruned by combining Supermask method and structured channel pruning method， and lightweighted finally by Int8 quantization technology. This ensured that the proposed model could balance accuracy and speed in dish recognition， achieving a good trade-off while improving the model portability. Experimental results show that the proposed model achieves a mean Average Precision （mAP） of 99.00% and an average recognition speed of 59.54 ms /frame at an Intersection over Union （IoU） of 0.5， which is 20 ms/frame faster than that of the original YOLOv5 model while maintaining the same level of accuracy. In addition， the new dish recognition network was ported to the Renesas RZ/G2L board by Qt. Based on this， an intelligent service system was constructed to realize the whole process of ordering， generating orders， and automatic meal distribution. A theoretical and practical foundation was provided for the future construction and application of truly intelligent service systems in restaurants.

Key words: Supermask method, YOLOv5, lightweight, network porting, Chinese food dish recognition

摘要：

为了更好地满足中餐菜品识别对准确性和时效性的应用需求，设计一种新型的菜品识别网络。在原YOLOv5模型的基础上，结合Supermask方法与结构化通道剪枝对模型进行剪枝操作，并利用Int8量化技术最终实现对模型的轻量化处理，保证模型在菜品识别中兼顾准确率和速度，同时提高模型的可移植性。实验结果表明，所提模型在交并比（IoU）为0.5时，平均精度均值（mAP）达到99.00%，平均每帧识别时间达到59.54 ms，相较于原始YOLOv5降低了20 ms，且准确率基本保持一致。此外，利用Qt软件将新型菜品识别网络移植到瑞萨RZ/G2L开发板，构建智能出餐系统，可实现点餐、生成订单、自动出餐全流程，为未来真正的餐厅智能出餐系统的构建应用提供了理论与实践基础。

关键词: Supermask方法, YOLOv5, 轻量化, 网络移植, 中餐菜品识别

CLC Number:

TP183

Chenghanyu ZHANG, Yuzhe LIN, Chengke TAN, Junfan WANG, Yeting GU, Zhekang DONG, Mingyu GAO. New dish recognition network based on lightweight YOLOv5[J]. Journal of Computer Applications, 2024, 44(2): 638-644.

张成涵宇, 林钰哲, 谭程珂, 王俊帆, 顾烨婷, 董哲康, 高明煜. 基于轻量化YOLOv5的新型菜品识别网络[J]. 《计算机应用》唯一官方网站, 2024, 44(2): 638-644.

Figures/Tables 9

Fig.1 Construction scheme of intelligent service system

Fig. 2 Architecture optimized YOLOv5 model

Fig. 3 Model training results

Tab.1 Average accuracy comparison among different models to recognize 85 kinds of real dishes

模型	mAP@0.5	mAP@0.5：0.95
RetinaNet	98.70	93.60
FasterRCNN	96.60	92.70
YOLOv3	95.40	90.10
YOLOv5	99.50	96.30
YOLOv5-Lite	99.20	86.20
YOLOv6	98.70	94.40
NanoDet-Plus	93.50	91.90
本文模型	99.00	93.90

Tab.2 Time comparison among different models to recognize 85 kinds of real dishes from video stream on PC

模型	每帧识别时间	模型	每帧识别时间
RetinaNet	156.58	YOLOv5-Lite	134.63
FasterRCNN	232.38	YOLOv6	60.31
YOLOv3	126.24	NanoDet-Plus	68.19
YOLOv5	79.53	本文模型	59.54

Tab.3 Dish recognition speed comparison between YOLOv5 model and optimized YOLOv5 model

神经网络模型	每帧的识别时间/s
YOLOv5模型	8.263
优化YOLOv5模型	2.313

Fig. 4 Demonstration of real dish recognition on PC

Fig. 5 UI interface of intelligent service system

Fig. 6 Robot arm cooking process

References 28

1	GAO M， SHI J， DONG Z， et al. A Chinese dish detector with modified YOLOv3［C］// Intelligent Equipment， Robots， and Vehicles： Proceedings of the 7th International Conference on Life System Modeling and Simulation， and 7th International Conference on Intelligent Computing for Sustainable Energy and Environment， Part III. Cham： Springer， 2021： 174-183. 10.1007/978-981-16-7213-2_17
2	KAWANO Y， YANAI K. Automatic expansion of a food image dataset leveraging existing categories with domain adaptation［C］// Proceedings of the 2014 European Conference on Computer Vision. Cham： Springer， 2015： 3-17. 10.1007/978-3-319-16199-0_1
3	苏国炀. 基于图像的中餐菜品分割与识别［D］.杭州：浙江大学， 2019：36-58.
	SU G Y. Segmentation and recognition of Chinese dishes based on image ［D］. Hangzhou： Zhejiang University， 2019：36-58.
4	边竞，王艺璇，代宇晖，等.基于卷积神经网络的中餐菜品名称成分识别［J］.智能计算机与应用，2020，10（6）：55-58. 10.3969/j.issn.2095-2163.2020.06.012
	BIAN J， WANG Y X， DAI Y H， et al. Recognition of ingredients and dish names based on convolutional neural network ［J］. Intelligent Computer and Applications， 2020， 10（6）： 55-58. 10.3969/j.issn.2095-2163.2020.06.012
5	王晓朋. 精细粒度的菜品识别方法研究［D］. 成都：电子科技大学， 2020：39-63.
	WANG X P. Research on fine-grained food image recognition ［D］. Chengdu： University of Electronic Science and Technology of China， 2020： 39-63.
6	吴正东. 基于深度学习的中餐菜品图像分类算法研究［D］.成都：电子科技大学，2020：41-58.
	WU Z D. Research on Chinese food dishes image classification algorithm based on deep learning ［D］. Chengdu： University of Electronic Science and Technology of China， 2020： 41-58.
7	何志洋. 基于深度学习的菜品识别算法研究与系统实现［D］.杭州：浙江工商大学， 2022：19-28.
	HE Z Y. Research and system implementation of dish recognition algorithm based on deep learning［D］. Hangzhou： Zhejiang Gongshang University， 2022：19-28.
8	朱瑶. 基于卷积神经网络的菜品识别系统研究［D］.南京：南京信息工程大学，2020：15-42.
	ZHU Y. Research on dish recognition system based on convolutional neural network ［D］. Nanjing： Nanjing University of Information Science and Technology， 2020：15-42.
9	朱凌云. 基于少样本学习的菜品识别算法研究［D］.金华：浙江师范大学，2021：15-29.
	ZHU L Y. Research on dishes recognition algorithm based on few sample learning［D］. Jinhua： Zhejiang Normal University， 2021：15-29.
10	姚华莹，彭亚雄.基于轻量型卷积神经网络的菜品图像识别［J］.软件工程， 2021，24（10）：23-27.
	YAO H Y， PENG Y X. Dishes image recognition based on lightweight convolutional neural network［J］. Software Engineering， 2021， 24（10）：23-27.
11	邓志良，李磊.基于改进残差网络的中式菜品识别模型［J］.激光与光电子学进展，2021，58（6）：256-264. 10.3788/lop202158.0610019
	DENG Z L， LI L.Chinese food recognition model based on improved residual network［J］. Laser & Optoelectronics Progress，2021，58（6）：256-264. 10.3788/lop202158.0610019
12	张锦，屈佩琪，孙程，等.基于改进YOLOv5的安全帽佩戴检测算法［J］.计算机应用，2022，42（4）：1292-1300.
	ZHANG J， QU P Q， SUN C， et al. Safety helmet wearing detection algorithm based on improved YOLOv5 ［J］. Journal of Computer Applications， 2022， 42（4）： 1292-1300.
13	尹靖涵，瞿绍军，姚泽楷，等.基于YOLOv5的雾霾天气下交通标志识别模型［J］.计算机应用，2022，42（9）：2876-2884. 10.11772/j.issn.1001-9081.2021071305
	YIN J H， QU S J， YAO Z K， et al. Traffic sign recognition model in hazy weather based on YOLOv5［J］. Journal of Computer Applications， 2022，42（9）： 2876-2884. 10.11772/j.issn.1001-9081.2021071305
14	WANG J， CHEN Y， DONG Z， et al. Improved YOLOv5 network for real-time multi-scale traffic sign detection［J］. Neural Computing and Applications， 2023， 35： 7853-7865. 10.1007/s00521-022-08077-5
15	张利红，蔡敬菊.基于轻量化YOLOv5算法的目标检测系统［J］.计算机技术与发展，2022，32（11）：134-139. 10.3969/j.issn.1673-629X.2022.11.020
	ZHANG L H， CAI J J. Target detection system based on lightweight YOLOv5 algorithm［J］. Computer Technology and Development， 2022，32（11）：134-139. 10.3969/j.issn.1673-629X.2022.11.020
16	钟志峰，夏一帆，周冬平，等.基于改进YOLOv4的轻量化目标检测算法［J］.计算机应用，2022，42（7）：2201-2209.
	ZHONG Z F， XIA Y F， ZHOU D P， et al. Lightweight target detection algorithm based on improved YOLOv4［J］. Journal of Computer Applications， 2022，42（7）： 2201-2209.
17	WEN W， WU C， WANG Y， et al. Learning structured sparsity in deep neural networks［C］// Proceedings of the 30th International Conference on Neural Information Processing Systems. Red Hook Curran Associates Inc.， 2016： 2082-2090.
18	WANG W， CHEN M， ZHAO S， et al. Accelerate CNNs from three dimensions： a comprehensive pruning framework［C］// Proceedings of the 38th International Conference on Machine Learning. New York： PMLR， 2021， 139： 10717-10726.
19	LIU Z， LI J， SHEN Z， et al. Learning efficient convolutional networks through network slimming［C］// Proceedings of the 2017 IEEE International Conference on Computer Vision. Piscataway： IEEE， 2017： 2755-2763. 10.1109/iccv.2017.298
20	ZHOU H， LAN J， LIU R， et al. Deconstructing lottery tickets： Zeros， signs， and the supermask［EB/OL］. ［2023-03-01］. .
21	TAN J H， CHAN C S， CHUAH J H. End-to-end supermask pruning： Learning to prune image captioning models［J］. Pattern Recognition， 2022， 122： 108366. 10.1016/j.patcog.2021.108366
22	GHOLAMI A， KIM S， DONG Z， et al. A survey of quantization methods for efficient neural network inference ［EB/OL］. ［2023-02-14］. . 10.1201/9781003162810-13
23	BANNER R， NAHSHAN Y， HOFFER E， et al. Post training 4-bit quantization of convolutional networks for rapid-deployment［EB/OL］. ［2023-03-01］. .
24	WU B， WANG Y， ZHANG P， et al. Mixed precision quantization of ConvNets via differentiable neural architecture search ［EB/OL］. ［2022-12-21］. .
25	LIN T-Y， GOYAL P， GIRSHICK R， et al. Focal loss for dense object detection［C］// Proceedings of the 2017 IEEE International Conference on Computer Vision.Piscataway： IEEE， 2017： 2999-3007. 10.1109/iccv.2017.324
26	REN S， HE K， GIRSHICK R， et al. Faster R-CNN： towards real-time object detection with region proposal networks［J］. IEEE Transactions on Pattern Analysis and Machine Intelligence， 2017，39（6）： 1137-1149. 10.1109/tpami.2016.2577031
27	REDMON J， FARHADI A. YOLOv3： an incremental improvement ［EB/OL］. ［2023-02-15］. . 10.1109/cvpr.2017.690
28	LI C， LI L， JIANG H， et al. YOLOv6： a single-stage object detection framework for industrial applications ［EB/OL］. ［2022-12-06］. .

[1]	Hao CHEN, Zhenping XIA, Cheng CHENG, Xing LIN-LI, Bowen ZHANG. Lightweight image super-resolution reconstruction network based on Transformer-CNN [J]. Journal of Computer Applications, 2024, 44(1): 292-299.
[2]	Hong WANG, Qing QIAN, Huan WANG, Yong LONG. Lightweight image tamper localization algorithm based on large kernel attention convolution [J]. Journal of Computer Applications, 2023, 43(9): 2692-2699.
[3]	Guangyao ZHAO, Xuan SHEN, Bo YU, Chenhui YI, Zhen LI. Impossible differential cryptanalysis of reduced-round ultra-lightweight block cipher PFP [J]. Journal of Computer Applications, 2023, 43(9): 2784-2788.
[4]	Shengwei DUAN, Xinyu CHENG, Haozhou WANG, Fei WANG. Dam surface disease detection algorithm based on improved YOLOv5 [J]. Journal of Computer Applications, 2023, 43(8): 2619-2629.
[5]	Kansong CHEN, Yuan ZHENG, Lijun XU, Zhouyu WANG, Zhe ZHANG, Fujuan YAO. ShuffaceNet： face recognition neural network based on ThetaMEX global pooling [J]. Journal of Computer Applications, 2023, 43(8): 2572-2580.
[6]	Zongzhe LYU, Hui XU, Xiao YANG, Yong WANG, Weijian WANG. Small object detection algorithm of YOLOv5 for safety helmet [J]. Journal of Computer Applications, 2023, 43(6): 1943-1949.
[7]	Bin WANG, Tian XIANG, Yidong LYU, Xiaofan WANG. Adaptive multi-scale feature channel grouping optimization algorithm based on NSGA‑Ⅱ [J]. Journal of Computer Applications, 2023, 43(5): 1401-1408.
[8]	Juming HAO, Jingyu YANG, Shumei HAN, Yangping WANG. YOLOv4 highway pavement crack detection method using Ghost module and ECA [J]. Journal of Computer Applications, 2023, 43(4): 1284-1290.
[9]	Zhouhua ZHU, Qi QI. Automatic detection and recognition of electric vehicle helmet based on improved YOLOv5s [J]. Journal of Computer Applications, 2023, 43(4): 1291-1296.
[10]	Yun ZHANG, Shuying WANG, Qing ZHENG, Haizhu ZHANG. Lightweight algorithm of 3D mesh model for preserving detailed geometric features [J]. Journal of Computer Applications, 2023, 43(4): 1226-1232.
[11]	Jiadong LI, Danpu ZHANG, Yaqiong FAN, Jianfeng YANG. Lightweight ship target detection algorithm based on improved YOLOv5 [J]. Journal of Computer Applications, 2023, 43(3): 923-929.
[12]	Yongxiang GU, Xin LAN, Boyi FU, Xiaolin QIN. Object detection algorithm for remote sensing images based on geometric adaptation and global perception [J]. Journal of Computer Applications, 2023, 43(3): 916-922.
[13]	Wenju LI, Gan ZHANG, Liu CUI, Wanghui CHU. Lightweight traffic sign recognition model based on coordinate attention [J]. Journal of Computer Applications, 2023, 43(2): 608-614.
[14]	SUN Zeqiang, CHEN Bingcai, CUI Xiaobo, WANG Lei, LU Yanuo. Strip steel surface defect detection by YOLOv5 algorithm fusing frequency domain attention mechanism and decoupled head [J]. Journal of Computer Applications, 2023, 43(1): 242-249.
[15]	CHEN Haiyan, JIA Mingming, ZHAO Wenli, WANG Chanfei. Target detection of Ochotona curzoniae based on embedded Jetson TX2 [J]. Journal of Computer Applications, 2023, 43(1): 98-103.

New dish recognition network based on lightweight YOLOv5

基于轻量化YOLOv5的新型菜品识别网络

RichHTML

PDF

Knowledge

Abstract

Cite this article

share this article

Figures/Tables 9

References 28

Related Articles 15

Recommended Articles

Metrics