Real-time reconstruction method of visual information for manipulator operation

doi:10.11772/j.issn.1001-9081.2022020262

Journal of Computer Applications ›› 2023, Vol. 43 ›› Issue (4): 1255-1260.DOI: 10.11772/j.issn.1001-9081.2022020262

• Multimedia computing and computer simulation • Previous Articles

Real-time reconstruction method of visual information for manipulator operation

Qingyu JIA¹, Liang CHANG¹, Xianyi YANG²(), Baohua QIANG², Shihao ZHANG¹, Wu XIE², Minghao YANG²

^1.School of Computer Science and Information Security，Guilin University of Electronic Technology，Guilin Guangxi 541004，China
^2.Guangxi Key Laboratory of Image and Graphic Intelligent Processing （Guilin University of Electronic Technology），Guilin Guangxi 541004，China

Received:2022-03-08 Revised:2022-05-24 Accepted:2022-05-26 Online:2022-08-16 Published:2023-04-10
Contact: Xianyi YANG
About author:JIA Qingyu， born in 1995， M. S. candidate. Her research interests include machine learning， artificial intelligence.
CHANG Liang， born in 1980， Ph. D.， professor. His research interests include data and knowledge engineering， formal methods， trusted software.
QIANG Baohua， born in 1972， Ph. D.， professor. His research interests include big data analysis， image processing.
ZHANG Shihao， born in 1991， Ph. D. candidate. His research interests include human skeleton key point detection， image processing.
XIE Wu， born in 1979， Ph. D.， associate professor. His research interests include data mining， information processing.
YANG Minghao， born in 1977， Ph. D.， associate research fellow. His research interests include multimodal information fusion， man-machine cooperation.
Supported by:
Natural Science Foundation of Guangxi(2019GXNSFDA185006);Guangxi Science and Technology Base and Talent Project(Guike AD19110137)

面向机械臂操作的视觉信息实时重建方法

贾清玉¹, 常亮¹, 杨先一²(), 强保华², 张世豪¹, 谢武², 杨明浩²

^1.桂林电子科技大学计算机与信息安全学院, 广西桂林 541004
^2.广西图像图形与智能处理重点实验室(桂林电子科技大学), 广西桂林 541004

通讯作者: 杨先一
作者简介:贾清玉（1995—），女，山西大同人，硕士研究生，主要研究方向：机器学习、人工智能；
常亮（1980—），男，贵州毕节人，教授，博士，CCF高级会员，主要研究方向：数据和知识工程、形式化方法、可信软件；
强保华（1972—），男，河南南阳人，教授，博士，CCF会员，主要研究方向：大数据分析、图像处理；
张世豪（1991—），男，河南许昌人，博士研究生，主要研究方向：人体骨骼关键点检测、图像处理；
谢武（1979—），男，江西宜春人，副教授，博士，CCF会员，主要研究方向：数据挖掘、信息处理；
杨明浩（1977—），男，四川达州人，副研究员，博士，CCF会员，主要研究方向：多模态信息融合、人机协同。
基金资助:
广西自然科学基金资助项目(2019GXNSFDA185006);广西科技基地和人才专项(桂科AD19110137)

Abstract

Abstract:

Current skill teaching methods of manipulator mainly construct a virtual space through three-dimensional reconstruction technology for manipulator to simulate and train. However， due to the different visual angles between human and manipulator， the traditional visual information reconstruction methods have large reconstruction errors， long time， and need harsh experimental environment and many sensors， so that the skills learned by manipulator in virtual space can not be well transferred to the real environment. To solve the above problems， a visual information real-time reconstruction method for manipulator operation was proposed. Firstly， information was extracted from real-time RGB images through Mask-Region Convolutional Neural Network（Mask-RCNN）. Then， the extracted RGB images and other visual information were jointly encoded， and the visual information was mapped to the three-dimensional position information of the manipulator operation space through Residual Neural Network-18 （ResNet-18）. Finally， an outlier adjustment method based on Cluster Center DIStance constrained （CC-DIS） was proposed to reduce the reconstruction error， and the adjusted position information was visualized by Open Graphics Library （OpenGL）. In this way， the three-dimensional real-time reconstruction of the manipulator operation space was completed. Experimental results show that the proposed method has high reconstruction speed and reconstruction accuracy. It only takes 62.92 milliseconds to complete a three-dimensional reconstruction with a reconstruction speed of up to 16 frames per second and a reconstruction relative error of about 5.23%. Therefore， it can be effectively applied to the manipulator skill teaching tasks.

Key words: skill teaching, Mask-Region Convolutional Neural Network (Mask-RCNN), Residual Neural Network-18 (ResNet-18), three-dimensional real-time reconstruction, manipulator

摘要：

现阶段的机械臂技能传授方法主要通过三维实时重建技术搭建虚拟空间进行模拟训练。然而人与机械臂视角不同，传统视觉信息重建方法由于重建误差大、时间长，而且实验环境苛刻、所需传感器较多等原因，导致机械臂在虚拟空间内习得的技能不能很好地迁移于现实环境。针对以上问题，提出了一种面向机械臂操作的视觉信息实时重建方法。首先，通过Mask-RCNN（Mask-Region Convolutional Neural Network）对实时采集到的RGB图像提取信息；然后，将提取后的RGB图像及其他视觉信息联合编码，并通过ResNet-18将视觉信息映射为机械臂操作空间的三维位置信息；最后，为减小重建误差，提出了一种聚类簇中心距离受限离群值调整方法（CC-DIS），并利用OpenGL（Open Graphics Library）将调整后的位置信息可视化，完成机械臂操作空间三维实时重建。实验结果表明，所提的实时重建方法具有较快的重建速度和较高的重建精度，完成一次三维重建仅需62.92 ms，重建速度高达每秒16帧，重建相对误差约为5.23%，能有效用于机械臂技能传授任务。

关键词: 技能传授, Mask-RCNN, ResNet-18, 三维实时重建, 机械臂

CLC Number:

TP391.41

Qingyu JIA, Liang CHANG, Xianyi YANG, Baohua QIANG, Shihao ZHANG, Wu XIE, Minghao YANG. Real-time reconstruction method of visual information for manipulator operation[J]. Journal of Computer Applications, 2023, 43(4): 1255-1260.

贾清玉, 常亮, 杨先一, 强保华, 张世豪, 谢武, 杨明浩. 面向机械臂操作的视觉信息实时重建方法[J]. 《计算机应用》唯一官方网站, 2023, 43(4): 1255-1260.

Figures/Tables 11

Fig. 1 Flow of the proposed method

Fig. 2 Flow of visual information extraction

Fig. 3 Error of 2DM image generated with different mask sizes

Fig. 4 Flow of DL-VOM

Fig. 5 Experimental environment

Fig. 6 Schematic diagram of method accuracy measurement

Tab. 1 Reconstruction accuracy of some samples

重建物体	$A m e N N$ /mm	$A m e D P$ /mm	$E r r N N$ /%	$E r r D P$ /%
平均值	9.13	5.73	8.60	5.23
白色盒子	11.34	7.06	6.71	3.65
蓝色盒子	7.51	5.21	6.89	4.57
灰色茶杯	7.80	5.83	8.46	6.15
鼠标	10.71	5.54	11.12	5.71
白色茶杯	8.28	5.02	9.85	6.06

Tab. 1 Reconstruction accuracy of some samples

重建物体	$A m e N N$ /mm	$A m e D P$ /mm	$E r r N N$ /%	$E r r D P$ /%
平均值	9.13	5.73	8.60	5.23
白色盒子	11.34	7.06	6.71	3.65
蓝色盒子	7.51	5.21	6.89	4.57
灰色茶杯	7.80	5.83	8.46	6.15
鼠标	10.71	5.54	11.12	5.71
白色茶杯	8.28	5.02	9.85	6.06

Fig. 7 Reconstruction results of some samples

Tab. 2 Reconstruction time of some samples

重建物体	$T M a s k ‑ R C N N$	$T D L ‑ V O M$	$T C C ‑ D I S$
平均值	60.20	2.08	0.64
白色盒子	60.14	3.90	1.50
蓝色盒子	60.06	2.70	0.90
灰色茶杯	60.09	0.70	0.10
鼠标	60.46	1.70	0.40
白色茶杯	60.25	1.40	0.30

Tab. 2 Reconstruction time of some samples

重建物体	$T M a s k ‑ R C N N$	$T D L ‑ V O M$	$T C C ‑ D I S$
平均值	60.20	2.08	0.64
白色盒子	60.14	3.90	1.50
蓝色盒子	60.06	2.70	0.90
灰色茶杯	60.09	0.70	0.10
鼠标	60.46	1.70	0.40
白色茶杯	60.25	1.40	0.30

Tab. 3 Comparison of experimental results of different methods

方法	$E r r$ /%	$T$ /ms	帧率/（frame·s^-1）
uvd	16.98	57.49	17
NN	8.60	62.32	15
CC-DIS-NN	5.23	62.92	16

Tab. 3 Comparison of experimental results of different methods

方法	$E r r$ /%	$T$ /ms	帧率/（frame·s^-1）
uvd	16.98	57.49	17
NN	8.60	62.32	15
CC-DIS-NN	5.23	62.92	16

Fig. 8 Reconstruction results in custer environment

References 25

1	GOODFELLOW I， BENGIO Y， COURVILLE A. Chapter 5： Machine learning basics［M］// Deep Learning. Cambridge： MIT Press， 2016： 96-161.
2	LeCUN Y， BENGIO Y， HINTON G. Deep learning［J］. Nature， 2015， 521（7553）： 436-444. 10.1038/nature14539
3	RAINA A， McCOMB C， CAGAN J. Learning to design from humans： imitating human designers through deep learning［J］. Journal of Mechanical Design， 2019， 141（11）： No.111102. 10.1115/1.4044256
4	HUA J， ZENG L C， LI G F， et al. Learning for a robot： deep reinforcement learning， imitation learning， transfer learning［J］. Sensors， 2021， 21（4）： No.1278. 10.3390/s21041278
5	DOERING M， GLAS D F， ISHIGURO H. Modeling interaction structure for robot imitation learning of human social behavior［J］. IEEE Transactions on Human-Machine Systems， 2019， 49（3）： 219-231. 10.1109/thms.2019.2895753
6	SCHERZINGER S， ROENNAU A， DILLMANN R. Contact skill imitation learning for robot-independent assembly programming［C］// Proceedings of the 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems. Piscataway： IEEE， 2019： 4309-4316. 10.1109/iros40897.2019.8967523
7	FERI L E， AHN J， LUTFILLOHONOV S， et al. A three-dimensional microstructure reconstruction framework for permeable pavement analysis based on 3D-IWGAN with enhanced gradient penalty［J］. Sensors， 2021， 21（11）： No.3603. 10.3390/s21113603
8	LI H Q， WANG R H. Method of real-time wellbore surface reconstruction based on spiral contour［J］. Energies， 2021， 14（2）： No.291. 10.3390/en14020291
9	刘东生，陈建林，费点，等. 基于深度相机的大场景三维重建［J］. 光学精密工程， 2020， 28（1）：234-243. 10.3788/ope.20202801.0234
	LIU D S， CHEN J L， FEI D， et al. Three-dimensional reconstruction of large-scale scene based on depth camera［J］. Optics Precision Engineering， 2020， 28（1）：234-243. 10.3788/ope.20202801.0234
10	HENRY P， KRAININ M， HERBST E， et al. RGB-D mapping： using Kinect-style depth cameras for dense 3D modeling of indoor environments［J］. The International Journal of Robotics Research， 2012， 31（5）： 647-663. 10.1177/0278364911434148
11	NEWCOMBE R A， IZADI S， HILLIGES O， et al. KinectFusion： real-time dense surface mapping and tracking［C］// Proceedings of the 10th IEEE International Symposium on Mixed and Augmented Reality. Piscataway： IEEE， 2011： 127-136. 10.1109/ismar.2011.6092378
12	FURUKAWA Y， Accurate PONCE J.， dense， and robust multiview stereopsis ［J］. IEEE Transactions on Pattern Analysis and Machine Intelligence， 2010， 32（8）： 1362-1376. 10.1109/tpami.2009.161
13	TORKINGTON J， SMITH S G T， REES B I， et al. Skill transfer from virtual reality to a real laparoscopic task［J］. Surgical Endoscopy， 2001， 15（10）： 1076-1079. 10.1007/s004640000233
14	LIU Y Y， LI Z J， LIU H P， et al. Skill transfer learning for autonomous robots and human-robot cooperation： a survey［J］. Robotics and Autonomous Systems， 2020， 128： No.103515. 10.1016/j.robot.2020.103515
15	SUN Y， GUO Q Q， ZHAO S M， et al. Context-aware augmented reality using human-computer interaction models［J/OL］. Journal of Control and Decision （2022-01-24）［2022-02-10］.. 10.1080/23307706.2022.2026260
16	马子玉，何明，刘祖均，等. 无人机协同控制研究综述［J］. 计算机应用， 2021， 41（5）： 1477-1483. 10.11772/j.issn.1001-9081.2020081314
	MA Z Y， HE M， LIU Z J， et al. Survey of unmanned aerial vehicle cooperative control［J］. Journal of Computer Applications， 2021， 41（5）： 1477-1483. 10.11772/j.issn.1001-9081.2020081314
17	BREAZEAL C， SCASSELLATI B. Robots that imitate humans［J］. Trends in Cognitive Sciences， 2002， 6（11）： 481-487. 10.1016/s1364-6613(02)02016-8
18	EDSINGER A， KEMP C C. Human-robot interaction for cooperative manipulation： handing objects to one another［C］// Proceedings of the 16th IEEE International Symposium on Robot and Human Interactive Communication. Piscataway： IEEE， 2007： 1167-1172. 10.1109/roman.2007.4415256
19	ZHU Z Y， HU H S. Robot learning from demonstration in robotic assembly： a survey［J］. Robotics， 2018， 7（2）： No.17. 10.3390/robotics7020017
20	RAVICHANDAR H， POLYDOROS A S， CHERNOVA S， et al. Recent advances in robot learning from demonstration［J］. Annual Review of Control， Robotics， and Autonomous Systems， 2020， 3： 297-330. 10.1146/annurev-control-100819-063206
21	HAVOUTIS I， CALINON S. Learning from demonstration for semi-autonomous teleoperation［J］. Autonomous Robots， 2019， 43（3）： 713-726. 10.1007/s10514-018-9745-2
22	张继凯，赵君，张然，等. 深度学习的图像实例分割方法综述［J］. 小型微型计算机系统， 2021， 42（1）： 161-71. 10.3969/j.issn.1000-1220.2021.01.028
	ZHANG J K， ZHAO J， ZHANG R， et al. Survey of image instance segmentation methods based on deep learning［J］. Journal of Chinese Computer Systems， 2021， 42（1）： 161-71. 10.3969/j.issn.1000-1220.2021.01.028
23	REN S Q， HE K M， GIRSHICK R. Faster R-CNN： towards real-time object detection with region proposal networks［C］// Proceedings of the 28th International Conference on Neural Information Processing Systems - Volume 1. Cambridge： MIT Press， 2015：91-99.
24	LIN T Y， DOLLÁR P， GIRSHICK R. Feature pyramid networks for object detection［C］// Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2017： 936-944. 10.1109/cvpr.2017.106
25	中国科学院自动化研究所. 面向执行器操作空间的RGBD视觉实时重建方法及系统： 202110642486.9［P］. 2021-08-17.
	Institute of Automation of Chinese Academy of Sciences. RGBD visual real-time reconstruction method and system for actuator operation space： 202110642486.9［P］. 2021-08-17.

[1]	Xianbojun FAN, Lijia CHEN, Shen LI, Chenlu WANG, Min WANG, Zan WANG, Mingguo LIU. Robust joint modeling and optimization method for visual manipulators [J]. Journal of Computer Applications, 2023, 43(3): 962-971.
[2]	Bao CHEN, Zupeng ZHOU, Huan WEI, Yanzhao LYU, Zhicheng SUI. Control method of quadrotor UAV with manipulator based on expert PID [J]. Journal of Computer Applications, 2022, 42(8): 2637-2642.
[3]	WANG Jianping, WANG Gang, MAO Xiaobin, MA Enqi. Motion control method of two-link manipulator based on deep reinforcement learning [J]. Journal of Computer Applications, 2021, 41(6): 1799-1804.
[4]	LI Kene, ZHANG Zeng, WANG Wenxin. Pseudoinverse-based motion planning scheme for deviation correction of rail manipulator joint velocity [J]. Journal of Computer Applications, 2020, 40(12): 3695-3700.
[5]	YUAN Meng'en, CHEN Lijia, FENG Zikai. Path planning algorithm of multi-population particle swarm manipulator based on monocular vision [J]. Journal of Computer Applications, 2020, 40(10): 2863-2871.
[6]	ZHOU Feng, LIN Nan, CHEN Xiaoping. Inverse kinematics equation solving method for six degrees of freedom manipulator based on six dimensional linear interpolation [J]. Journal of Computer Applications, 2018, 38(2): 563-567.
[7]	ZOU Sifan, WU Guoqing, MAO Jingfeng, ZHU Weinan, WANG Yurong, WANG Jian. Adaptive backstepping sliding mode control for robotic manipulator with the improved nonlinear disturbance observer [J]. Journal of Computer Applications, 2018, 38(10): 2827-2832.
[8]	DU Shanshan, ZHOU Xiang. Robot tool calibration method based on camera space point constraint [J]. Journal of Computer Applications, 2015, 35(9): 2678-2681.
[9]	CAI Zhuang ZHANG Guoliang TIAN Qi. Trajectory tracking control of manipulator based on FSMC [J]. Journal of Computer Applications, 2014, 34(1): 232-235.
[10]	LI Gui-qiu CHEN Zhi-wang. Indirect adaptive robust predictive control of robotic manipulators based on uncertain parameter approximation [J]. Journal of Computer Applications, 2012, 32(06): 1707-1712.

Real-time reconstruction method of visual information for manipulator operation

面向机械臂操作的视觉信息实时重建方法

RichHTML

PDF

PDF (Mobile)

Knowledge

Abstract

Cite this article

share this article

Figures/Tables 11

References 25

Related Articles 10

Recommended Articles

Metrics