Decoupled visual servoing control method based on point and line features

doi:10.11772/j.issn.1001-9081.2021071178

Journal of Computer Applications ›› 2022, Vol. 42 ›› Issue (8): 2556-2563.DOI: 10.11772/j.issn.1001-9081.2021071178

Special Issue: 多媒体计算与计算机仿真

• Multimedia computing and computer simulation • Previous Articles Next Articles

Decoupled visual servoing control method based on point and line features

Jinyan LU¹, Xiaoke QI²()

^1.School of Electrical and Information Engineering，Henan University of Engineering，Zhengzhou Henan 451191，China
^2.School of Information Management for Law，China University of Political Science and Law，Beijing 102249，China

Received:2021-07-07 Revised:2021-10-11 Accepted:2021-10-13 Online:2021-11-09 Published:2022-08-10
Contact: Xiaoke QI
About author:LU Jinyan， born in 1985， Ph. D.， lecturer. Her research interests include robot control， intelligent control， signal processing， machine learning.
QI Xiaoke， born in 1985， Ph. D.， associate professor. Her research interests include multimedia， natural language processing， machine learning， wireless communication.
Supported by:
National Natural Science Foundation of China(62173126);Key Research and Development and Promotion Project of Henan Province （Key Problem of Science and Technology）(202102210187);Doctoral Foundation of Henan University of Engineering(Dkj2018003)

基于点线特征的解耦视觉伺服控制方法

卢金燕¹, 戚肖克²()

^1.河南工程学院电气信息工程学院，郑州 451191
^2.中国政法大学法治信息管理学院，北京 102249

通讯作者: 戚肖克
作者简介:卢金燕（1985—），女，河南信阳人，讲师，博士，CCF会员，主要研究方向：机器人控制、智能控制、信号处理、机器学习；
戚肖克（1985—），女，山东菏泽人，副教授，博士，CCF会员，主要研究方向：多媒体、自然语言处理、机器学习、无线通信。
基金资助:
国家自然科学基金资助项目(62173126);河南省重点研发与推广专项（科技攻关）(202102210187);河南工程学院博士基金资助项目(Dkj2018003)

Abstract

Abstract:

Aiming at the problem of automatic alignment for robot， a decoupled visual servoing control method based on point and line features was proposed. In the method， the points and lines were used as image features， and the interactive matrix of image features was used to decouple attitude control and position control， so as to realize six degrees of freedom alignment. Firstly， the attitude control law was designed according to the lines and their interactive matrix to eliminate the rotational deviation. Then， the position control law was designed according to the points and their interactive matrix to eliminate the positional deviation. Finally， the automatic alignment between the robot end-effector and the target was realized. In the alignment control process， based on the amount of camera motion and the variation of features before and after camera motion， the online estimation of depth was able to be realized. In addition， a monitor was designed to adjust the motion speed of the camera， thereby ensuring that the features were always in the field of view of the camera. The six degrees of freedom alignment of the robot on the Eye-in-Hand platform was completed by the proposed method and the traditional image based visual servoing method， respectively. The proposed method realizes the automatic alignment of the robot in 16 steps， and has the maximum translation error of 3.26 mm and the maximum rotation error of 0.72° of the robot end-effector after alignment. Compared with the comparison method， the proposed method has more efficient control process， faster convergence of control error and less alignment error. Experimental results show that the proposed method can realize fast and high-precision automatic alignment， improving the autonomy and intelligent level of robot operation， and is expected to be appied in the fields of target tracking， picking and positioning， automatic assembly， welding， service robot and so on.

Key words: visual servoing, visual control, interactive matrix, decouped control, six degrees of freedom alignment

摘要：

针对机器人的自动对准问题，提出一种基于点线特征的解耦视觉伺服控制方法。所提方法以点和直线作为图像特征，并利用图像特征的交互矩阵解耦姿态控制和位置控制，实现六自由度对准。首先利用直线及其交互矩阵设计姿态控制律，以消除旋转偏差；然后利用点及其交互矩阵设计位置控制律，以消除位置偏差；最后实现机器人末端目标的自动对准。在对准控制过程中，基于执行的相机运动量以及相机运动前后特征的变化量，可实现对深度的在线估计。另外，还设计了监督器对相机的运动速度进行调节，从而确保特征一直处于相机视野当中。在Eye-in-Hand机器人平台上，分别用所提方法和传统的基于图像的视觉伺服方法实现了机器人的六自由度对准。所提方法经过16步实现了机器人的自动对准，对准结束时机器人末端位姿的最大平移误差为3.26 mm，最大旋转误差为0.72°。相较于对比方法，该方法的控制过程更加高效，控制误差收敛更快，对准误差更小。实验结果表明，所提方法可以实现快速高精度的自动对准，能够提高机器人操作的自主性和智能化水平，有望应用于目标跟踪、拾取和定位、自动化装配、焊接、服务机器人等领域。

关键词: 视觉伺服, 视觉控制, 交互矩阵, 解耦控制, 六自由度对准

CLC Number:

TP242.6

Jinyan LU, Xiaoke QI. Decoupled visual servoing control method based on point and line features[J]. Journal of Computer Applications, 2022, 42(8): 2556-2563.

卢金燕, 戚肖克. 基于点线特征的解耦视觉伺服控制方法[J]. 《计算机应用》唯一官方网站, 2022, 42(8): 2556-2563.

Figures/Tables 14

Fig. 1 Block diagram of the proposed control system

Fig. 2 Eye-in-Hand robot platform

Tab. 1 Expectedd values for lines

直线编号	ρ/mm	θ/（°）
直线1	0.216 5	-90.96
直线2	0.333 0	-1.23
直线3	0.140 1	88.78
直线4	0.251 7	179.43

Tab. 2 Expected values for points

角点编号	x₁_c /mm	y₁_c /mm
点A	-0.253 8	-0.212 3
点B	0.328 3	-0.222 0
点C	0.335 9	0.133 0
点D	-0.250 3	0.145 5

Fig. 3 Angle errors of four edges

Fig. 4 Errors of four points on focal length normalization plane

Fig. 5 Errors of four points on focal length normalization plane for traditional IBVS method

Fig. 6 Feature point trajectories by using the proposed method

Fig. 7 Feature point trajectories by using traditional IBVS method

Fig. 8 Position and pose changes of end-effector of robot by using the proposed method

Fig. 9 Position and pose changes of end-effector of robot by using traditional IBVS method

Fig. 10 End-effector trajectory of robot by using the proposed method

Fig. 11 End-effector trajectory of robot by using traditional IBVS method

Tab. 3 Attitude deviation of end-effector of robot after alignment

视觉伺服方法	x/mm	y/mm	z/mm	rx/（°）	ry/（°）	rz/（°）
本文方法	3.26	1.82	2.05	-0.72	0.37	-0.52
传统IBVS方法	-4.72	5.06	-7.35	-1.26	0.53	-0.78

References 23

1	CHEN S Y， LI Y F， KWOK N M. Active vision in robotic systems： a survey of recent developments［J］. The International Journal of Robotics Research， 2011， 30（11）： 1343-1377. 10.1177/0278364911410755
2	SHI H B， SUN G， WANG Y P， et al. Adaptive image-based visual servoing with temporary loss of the visual signal［J］. IEEE Transactions on Industrial Informatics， 2019， 15（4）： 1956-1965. 10.1109/tii.2018.2865004
3	CAO Z Q， CHEN X C， YU Y Y， et al. Image dynamics-based visual servoing for quadrotors tracking a target with a nonlinear trajectory observer［J］. IEEE Transactions on System， Man， and Cybernetics： Systems， 2020， 50（1）： 376-384. 10.1109/tsmc.2017.2720173
4	SALEHI I， ROTITHOR G， SALTUS R， et al. Constrained image-based visual servoing using barrier functions ［C］// Proceedings of the 2021 IEEE International Conference on Robotics and Automation. Piscataway： IEEE， 2021： 14254-14260. 10.1109/icra48506.2021.9560983
5	LIANG X W， WANG H S， LIU Y H， et al. Formation control of nonholonomic mobile robots without position and velocity measurements［J］. IEEE Transactions on Robotics， 2018， 34（2）： 434-446. 10.1109/tro.2017.2776304
6	WANG H S， YANG B H， LIU Y T， et al. Visual servoing of soft robot manipulator in constrained environments with an adaptive controller［J］. IEEE/ASME Transactions on Mechatronics， 2017， 22（1）： 41-50. 10.1109/tmech.2016.2613410
7	ZHENG D L， WANG H S， WANG J C， et al. Image-based visual servoing of a quadrotor using virtual camera approach［J］. IEEE/ASME Transactions on Mechatronics， 2017， 22（2）： 972-982. 10.1109/tmech.2016.2639531
8	AGRAVANTE D J， CHAUMETTE F. Active vision for pose estimation applied to singularity avoidance in visual servoing ［C］// Proceedings of the 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems. Piscataway： IEEE， 2017： 2947-2952. 10.1109/iros.2017.8206129
9	ZHANG K X， CHEN J， LI Y， et al. Visual tracking and depth estimation of mobile robots without desired velocity information［J］. IEEE Transactions on Cybernetics， 2020， 50（1）： 361-373. 10.1109/tcyb.2018.2869623
10	LI J N， XIE H， LOW K H， et al. Image-based visual servoing of rotorcrafts to planar visual targets of arbitrary orientation［J］. IEEE Robotics and Automation Letters， 2021， 6（4）： 7861-7868. 10.1109/lra.2021.3101878
11	ESPIAU B， CHAUMETTE F， RIVES P. A new approach to visual servoing in robotics［J］. IEEE Transactions on Robotics and Automation， 1992， 8（3）： 313-326. 10.1109/70.143350
12	徐德，卢金燕.直线特征的交互矩阵求取［J］.自动化学报， 2015， 41（10）： 1762-1771. 10.16383/j.aas.2015.c150097
	XU D， LU J Y. Determination for interactive matrix of line feature［J］. Acta Automatica Sinica， 2015， 41（10）： 1762-1771. 10.16383/j.aas.2015.c150097
13	XU C， ZHANG L L， CHENG L， et al. Pose estimation from line correspondences： a complete analysis and a series of solutions［J］. IEEE Transactions on Pattern Analysis and Machine Intelligence， 2017， 39（6）： 1209-1222. 10.1109/tpami.2016.2582162
14	ZHANG G X， LEE J H， LIM J， et al. Building a 3-D line-based map using stereo SLAM［J］. IEEE Transaction on Robotics， 2015， 31（6）： 1364-1377. 10.1109/tro.2015.2489498
15	ZHOU H Z， ZOU D P， PEI L， et al. StructSLAM： visual SLAM with building structure lines［J］. IEEE Transactions on Vehicular Technology， 2015， 64（4）： 1364-1375. 10.1109/tvt.2015.2388780
16	MATEUS A， TAHRI O， MIRALDO P. Active estimation of 3D lines in spherical coordinates ［C］// Proceedings of the 2019 American Control Conference. Piscataway： IEEE， 2019： 3950-3955. 10.23919/acc.2019.8814938
17	DE FARIAS C， ADJIGBLE M， TAMADAZTE B， et al. Dual quaternion-based visual servoing for grasping moving objects ［C］// Proceedings of the IEEE 17th International Conference on Automation Science and Engineering. Piscataway： IEEE， 2021： 151-158. 10.1109/case49439.2021.9551631
18	FU Q， YU H S， LAI L H， et al. A robust RGB-D SLAM system with points and lines for low texture indoor environments［J］. IEEE Sensors Journal， 2019， 19（21）： 9908-9920. 10.1109/jsen.2019.2927405
19	PUMAROLA A， VAKHITOV A， AGUDO A， et al. PL-SLAM： real-time monocular visual SLAM with points and lines ［C］// Proceedings of the 2017 IEEE International Conference on Robotics and Automation. Piscataway： IEEE， 2017： 4503-4508. 10.1109/icra.2017.7989522
20	HE Y J， ZHAO J， GUO Y， et al. PL-VIO： tightly-coupled monocular visual-inertial odometry using point and line features［J］. Sensors， 2018， 18（4）： No.1159. 10.3390/s18041159
21	WANG R Z， DI K C， WAN W H， et al. Improved point-line feature based visual SLAM method for indoor scenes［J］. Sensors， 2018， 18（10）： No.3559. 10.3390/s18103559
22	HUANG S S， MA Z Y， MU T J， et al. Lidar-monocular visual odometry using point and line features ［C］// Proceedings of the 2020 IEEE International Conference on Robotics and Automation. Piscataway： IEEE， 2020： 1091-1097. 10.1109/icra40945.2020.9196613
23	WANG R H， ZHANG X B， FANG Y C， et al. Virtual-goal-guided RRT for visual servoing of mobile robots with FOV constraint［J］. IEEE Transactions on Systems， Man， and Cybernetics： Systems， 2022， 52（4）： 2073-2083. 10.1109/tsmc.2020.3044347

[1]	. The SLAM Algorithm Based on Point-Line Features and Multi-IMU Fusion [J]. Journal of Computer Applications, 0, (): 0-0.
[2]	Enbao QIAO, Xiangyang GAO, Jun CHENG. Self-recovery adaptive Monte Carlo localization algorithm based on support vector machine [J]. Journal of Computer Applications, 2024, 44(10): 3246-3251.
[3]	Haodong HE, Hao FU, Qiang WANG, Shuai ZHOU, Wei LIU. Multi-robot path following and formation based on deep reinforcement learning [J]. Journal of Computer Applications, 2024, 44(8): 2626-2633.
[4]	WANG Caiqi, CUI Xining, XIONG Yi, WU Shiqian. Adaptive extended RRT^* path planning algorithm based on node-to-obstacle distance [J]. Journal of Computer Applications, 0, (): 0-0.
[5]	Chao GE, Jiabin ZHANG, Lei WANG, Zhixin LUN. Trajectory planning for autonomous vehicles based on model predictive control [J]. Journal of Computer Applications, 2024, 44(6): 1959-1964.
[6]	. Dynamic VSLAM algorithm incorporating object detection and feature point association [J]. Journal of Computer Applications, 0, (): 0-0.
[7]	Yuanchao LI, Chongben TAO, Chen WANG. Gait control method based on maximum entropy deep reinforcement learning for biped robot [J]. Journal of Computer Applications, 2024, 44(2): 445-451.
[8]	Tianzouzi XIAO, Xiaobo ZHOU, Xin LUO, Qipeng TANG. Robust RGB-D SLAM system incorporating instance segmentation and clustering in dynamic environment [J]. Journal of Computer Applications, 2023, 43(4): 1220-1225.
[9]	Jinyue LIU, Huiyu LI, Xiaohui JIA, Jiarui LI. Dynamic gait recognition method based on human model constraints [J]. Journal of Computer Applications, 2023, 43(3): 972-977.
[10]	Zhenghong QIN, Ran LIU, Yufeng XIAO, Kaixiang CHEN, Zhongyuan DENG, Tianrui DENG. Simultaneous localization and mapping for mobile robots based on WiFi fingerprint sequence matching [J]. Journal of Computer Applications, 2022, 42(10): 3268-3274.
[11]	Zhihong XI, Jiaxu WEN. Indoor dynamic scene localization and mapping based on target detection [J]. Journal of Computer Applications, 2022, 42(9): 2853-2857.
[12]	Gaofeng PAN, Yuan FAN, Yu RU, Yuchao GUO. Low-texture monocular visual simultaneous localization and mapping algorithm based on point-line feature fusion [J]. Journal of Computer Applications, 2022, 42(7): 2170-2176.
[13]	Rui ZHANG, Qizhi ZHANG, Yali ZHOU. Starting and walking human-like control of semi-passive bipedal robot with variable length telescopic legs [J]. Journal of Computer Applications, 2022, 42(1): 252-257.
[14]	ZHENG Sicheng, KONG Linghua, YOU Tongfei, YI Dingrong. Semantic SLAM algorithm based on deep learning in dynamic environment [J]. Journal of Computer Applications, 2021, 41(10): 2945-2951.
[15]	HUANG Juting, GAO Hongli, DAI Zhikun. Semantic segmentation method of power line on mobile terminals based on encoder-decoder structure [J]. Journal of Computer Applications, 2021, 41(10): 2952-2958.

Decoupled visual servoing control method based on point and line features

基于点线特征的解耦视觉伺服控制方法

RichHTML

PDF

Knowledge

Abstract

Cite this article

share this article

Figures/Tables 14

References 23

Related Articles 15

Recommended Articles

Metrics