Construction and application of 3D dataset of human grasping objects

doi:10.11772/j.issn.1001-9081.2023010009

Journal of Computer Applications ›› 2024, Vol. 44 ›› Issue (1): 278-284.DOI: 10.11772/j.issn.1001-9081.2023010009

• Multimedia computing and computer simulation • Previous Articles

Construction and application of 3D dataset of human grasping objects

Jian LIU¹^,², Chenchen YOU¹, Jinming CAO¹, Qiong ZENG¹, Changhe TU¹()

^1.College of Computer Science and Technology，Shandong University，Qingdao Shandong 266237，China
^2.School of Software，Tsinghua University，Beijing 100084，China

Received:2023-01-06 Revised:2023-03-18 Accepted:2023-03-20 Online:2023-06-06 Published:2024-01-10
Contact: Changhe TU
About author:LIU Jian， born in 1987， Ph. D.， assistant research fellow. His research interests include computer graphics， robot.
YOU Chenchen， born in 1994， M. S. Her research interests include computer graphics.
CAO Jinming， born in 1990， Ph. D. Her research interests include computer vision.
ZENG Qiong， born in 1987， Ph. D.， associate research fellow. Her research interests include visual data analysis， computer graphics.
Supported by:
National Natural Science Foundation of China(62072284)

人手抓取物体的三维数据集的建立及应用

刘健¹^,², 尤晨晨¹, 曹金明¹, 曾琼¹, 屠长河¹()

^1.山东大学计算机科学与技术学院，山东青岛 266237
^2.清华大学软件学院，北京 100084

通讯作者: 屠长河
作者简介:刘健（1987—），男，山东潍坊人，助理研究员，博士，CCF会员，主要研究方向：计算机图形学、机器人；
尤晨晨（1994—），女，山东济南人，硕士，主要研究方向：计算机图形学；
曹金明（1990—），女，山东临沂人，博士，CCF会员，主要研究方向：计算机视觉；
曾琼（1987—），女，湖南新化人，副研究员，博士，CCF会员，主要研究方向：数据可视分析、计算机图形学；
第一联系人：屠长河（1968—），男，山东济南人，教授，博士，CCF会员，主要研究方向：计算机图形学、机器人。
基金资助:
国家自然科学基金资助项目(62072284)

Abstract

Abstract:

Realistic human grasping data is of vital importance in the research of human grasping behavior analysis and human-like robotic grasping. A grasping dataset should include object shape information， contact points， and hand shapes and poses. However， related works often capture images or videos to estimate the human grasping behavior， which leads to the inaccuracy of joint degrees of freedom. Virtual Reality （VR） technology was used to establish a virtual environment， and digital gloves were used to directly capture 3D objects and hand poses in the virtual environment as capturing data. The proposed dataset contains 91 objects with various shapes （each with 108 poses） from 49 object categories， and 52 173 3D hand grasps， which scale and richness are far more than existing dataset used to study human grasping behavior and human-centered grasp technology. In addition， the collected dataset was used for grasp saliency analysis and human-like grasping calculation， and the experimental results demonstrate the practical value of this dataset.

Key words: human grasping dataset, virtual reality, joint degree of freedom, grasp saliency, human-like grasping

摘要：

真实人手抓取数据在人类抓取行为分析和机器人类人抓取等研究中起到至关重要的作用。抓取数据集中应包含复杂形状的三维物体信息、抓取点的信息以及手的姿态和形状，然而目前普遍方法是采集视频图像并从中估计人的抓取行为，导致不能准确记录手部各个关节的自由度信息。利用虚拟现实技术建立虚拟环境，利用数据手套直接捕捉在虚拟环境中三维物体和手部姿态信息作为抓取数据。提出的数据集包含生活中常见的49类物体中的91个不同形状的物体（每个有108个姿态）以及共52 173人次的抓取记录，规模和丰富性都远远超过了已有的用于研究人类的抓取行为和研究以人为核心的抓取技术的数据集。此外，使用采集的数据集进行抓取显著性分析和类人抓取计算，实验结果验证了数据集的应用价值。

关键词: 人手抓取数据集, 虚拟现实, 关节自由度, 抓取显著性, 类人抓取

CLC Number:

TP391

Jian LIU, Chenchen YOU, Jinming CAO, Qiong ZENG, Changhe TU. Construction and application of 3D dataset of human grasping objects[J]. Journal of Computer Applications, 2024, 44(1): 278-284.

刘健, 尤晨晨, 曹金明, 曾琼, 屠长河. 人手抓取物体的三维数据集的建立及应用[J]. 《计算机应用》唯一官方网站, 2024, 44(1): 278-284.

Figures/Tables 9

Fig. 1 Objects to be grasped by hand

Fig. 2 Virtual platform for grasping

Fig. 3 Grasping data capture system built based on Unity

Fig. 4 Grasping with different heights and poses

Fig. 5 Results after grasping data processing

Fig. 6 Object grasping saliency analysis via deep neural network learning

Fig. 7 Human-like grasping computation based on simulated annealing algorithm

Fig. 8 Human-like calculation grasping results for various objects

Fig. 9 Various human-like candidate grasping results for same object

References 42

1	TEGIN J， EKVALL S， KRAGIC D， et al. Demonstration-based learning and control for automatic grasping ［J］. Intelligent Service Robotics， 2009， 2： 23-30. 10.1007/s11370-008-0026-3
2	YUN L， SUN Y. Grasp planning based on strategy extracted from demonstration ［C］// Proceedings of the 2014 IEEE/RSJ International Conference on Intelligent Robots and Systems. Piscataway： IEEE， 2014： 4458-4463. 10.1109/iros.2014.6943193
3	BRAHMBHATT S， HAM C， KEMP C C， et al. ContactDB： Analyzing and predicting grasp contact via thermal imaging ［C］// Proceedings of the 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2019： 8701-8711. 10.1109/cvpr.2019.00891
4	HUANG D-A， MA M， MA W-C， et al. How do we use our hands？ Discovering a diverse set of common grasps ［C］// Proceedings of the 2015 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2015： 666-675. 10.1109/cvpr.2015.7298666
5	BULLOCK I M， FEIX T， DOLLAR A M. The Yale human grasping dataset： Grasp， object and task data in household and machine shop environments ［J］. The International Journal of Robotics Research， 2015， 34（3）： 251-255. 10.1177/0278364914555720
6	SAUDABAYEV A， RYSBEK Z， KHASSENOVA R， et al. Human grasping database for activities of daily living with depth， color and kinematic data streams ［J］. Scientific Data， 2018， 5： 180101. 10.1038/sdata.2018.101
7	HEUMER G， AMOR H B， WEBER M， et al. Grasp recognition with uncalibrated data gloves — a comparison of classification methods ［C］// Proceedings of the 2007 IEEE Virtual Reality Conference. Piscataway： IEEE， 2007： 19-26. 10.1109/vr.2007.352459
8	YUAN S， YE Q， STENGER B， et al. BigHand2.2M benchmark： Hand pose dataset and state of the art analysis ［C］// Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2017： 4866-4874. 10.1109/cvpr.2017.279
9	GARCIA-HERNANDO G， YUAN S， BAEK S， et al. First-person hand action benchmark with RGB-D videos and 3D hand pose annotations ［C］// Proceedings of the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2018： 409-419. 10.1109/cvpr.2018.00050
10	CHANGE A X， FUNKHOUSER T， GUIBAS L， et al. ShapeNet： An information-rich 3D model repository ［EB/OL］. （2015-12-09）［2023-01-05］. .
11	CAI M， KITANI K M， SATO Y. An ego-vision system for hand grasp analysis ［J］. IEEE Transactions on Human-Machine Systems， 2017， 47（4）： 524-535. 10.1109/thms.2017.2681423
12	ABBASI B， NOOHI E， PARASTEGARI S， et al. Grasp taxonomy based on force distribution ［C］// Proceedings of the 2016 25th IEEE International Symposium on Robot and Human Interactive Communication. Piscataway： IEEE， 2016： 1098-1103. 10.1109/roman.2016.7745245
13	BETTI S， ZANI G， GUERRA S， et al. Reach-to-grasp movements： A multimodal techniques study ［J］. Frontiers in Psychology， 2018， 9： 990. 10.3389/fpsyg.2018.00990
14	PUHLMANN S， HEINEMANN F， BROCK O， et al. A compact representation of human single-object grasping ［C］// Proceedings of the 2016 IEEE/RJS International Conference on Intelligent Robots and Systems. Piscataway： IEEE， 2016： 1954-1959. 10.1109/iros.2016.7759308
15	HEINEMANN F， PUHIMANN S， EPPNER C， et al. A taxonomy of human grasping behavior suitable for transfer to robotic hands ［C］// Proceedings of the 2015 IEEE International Conference on Robotics and Automation. Piscataway： IEEE， 2015： 4286-4291. 10.1109/icra.2015.7139790
16	KUMAR V， TODOROV E. MuJoCo HAPTIX： A virtual reality system for hand manipulation ［C］// Proceedings of the 2015 IEEE-RAS 15th International Conference on Humanoid Robots. Piscataway： IEEE， 2015： 657-663. 10.1109/humanoids.2015.7363441
17	AMOR H BEN， KROEMER O， HILLENBRAND U， et al. Generalization of human grasping for multi-fingered robot hands ［C］// Proceedings of the 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems. Piscataway： IEEE， 2012： 2043-2050. 10.1109/iros.2012.6386072
18	ROMERO J， KJELLSTROM H， KRAGIC D. Modeling and evaluation of human-to-robot mapping of grasps ［C］// Proceedings of the 2009 IEEE International Conference on Advanced Robotics. Piscataway： IEEE， 2009： 1-6.
19	HERZOG A， PASTOR P， KALAKRISHNAN M， et al. Template-based learning of grasp selection ［C］// Proceedings of the 2012 IEEE International Conference on Robotics and Automation. Piscataway： IEEE， 2012： 2379-2384. 10.1109/icra.2012.6225271
20	YANG Y， FERMÜLLER C， LI Y， et al. Grasp type revisited： A modern perspective on a classical feature vision ［C］// Proceedings of the 2015 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2015： 400-408. 10.1109/cvpr.2015.7298637
21	NAKAMURA Y C， TRONIAK D M， RODRIGUEZ A， et al. The complexities of grasping in the wild ［C］// Proceedings of the 2017 IEEE-RAS International Conference on Humanoid Robotics. Piscataway： IEEE， 2017： 233-240. 10.1109/humanoids.2017.8246880
22	CUTKOSKY M R. On grasp choice， grasp models， and the design of hands for manufacturing tasks ［J］. IEEE Transactions on Robotics and Automation， 1989， 5（3）： 269-279. 10.1109/70.34763
23	KAMAKURA N， MATSUO M， ISHII H， et al. Patterns of static prehension in normal hands ［J］. The American Journal of Occupational Therapy， 1980， 34（7）： 437-445. 10.5014/ajot.34.7.437
24	TANG D， CHANG H J， TEJANI A， et al. Latent regression forest： Structured estimation of 3D articulated hand posture ［C］// Proceedings of the 2014 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2014： 3786-3793. 10.1109/cvpr.2014.490
25	BALASUBRAMANIAN R， XU L， BROOK P D， et al. Physical human interactive guidance： Identifying grasping principles from human-planned grasps ［J］. IEEE Transactions on Robotics， 2012， 28（4）： 899-910. 10.1109/tro.2012.2189498
26	TOMPSON J， STEIN M， LeCUN Y， et al. Real-time continuous pose recovery of human hands using convolutional networks ［J］. ACM Transactions on Graphics， 2014， 33（5）： No. 169. 10.1145/2629500
27	BERNARDIN K， OGAWARA K， IKEUCHI K， et al. A sensor fusion approach for recognizing continuous human grasping sequences using hidden Markov models ［J］. IEEE Transactions on Robotics and Automation， 2005， 21（1）： 47-57. 10.1109/tro.2004.833816
28	T-H PHAM， KYRIAZIS N， ARGYROS A A， et al. Hand-object contact force estimation from marker less visual tracking ［J］. IEEE Transactions on Pattern Analysis and Machine Intelligence， 2018， 40（12）： 2883-2896. 10.1109/tpami.2017.2759736
29	ROGEZ G， SUPANCIC J S， RAMANAN D， et al. Understanding everyday hands in action from RGB-D images ［C］// Proceedings of the 2015 IEEE International Conference on Computer Vision. Piscataway： IEEE， 2015： 3889-3897. 10.1109/iccv.2015.443
30	尤晨晨.抓取数据集构建及未知物体抓取显著性预测［D］.济南：山东大学， 2021.
	YOU C C. Construction if grasping data set and prediction of grasping salience of unknown objects ［D］. Jinan： Shandong University， 2021.
31	BRAHMBHATT S， TANG C， TWIGG C D， et al. ContactPose： A dataset of grasps with object contact and hand pose ［C］// Proceedings of the 2020 European Conference on Computer Vision. Cham： Springer， 2020： 361-378. 10.1007/978-3-030-58601-0_22
32	LAU M， DEV K， SHI W， et al. Tactile mesh saliency ［J］. ACM Transactions on Graphics， 2016， 35（4）： No. 52. 10.1145/2897824.2925927
33	ZHANG L-N， WANG Y， ZHOU J， et al. 3D grasp saliency analysis via deep shape correspondence ［J］. Computer Aided Geometric Design， 2020， 81： 101901. 10.1016/j.cagd.2020.101901
34	MILLER A T， ALLEN P K. Graspit！ A versatile simulator for robotic grasping ［J］. IEEE Robotics & Automation Magazine， 2004， 11（4）： 110-122. 10.1109/mra.2004.1371616
35	HAMER H， GALL J， WEISE T， et al. An object-dependent hand pose prior from sparse training data ［C］// Proceedings of the 2010 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2010： 671-678. 10.1109/cvpr.2010.5540150
36	JIANG Y， MOSESON S， SAXENA A. Efficient grasping from RGBD images： Learning using a new rectangle representation ［C］// Proceedings of the 2011 IEEE International Conference on Robotics and Automation. Piscataway： IEEE， 2011： 3304-3311. 10.1109/icra.2011.5980145
37	PINTO L， GUPTA A， et al. Supersizing self-supervision： Learning to grasp from 50K tries and 700 robot hours ［C］// Proceedings of the 2016 IEEE International Conference on Robotics and Automation. Piscataway： IEEE， 2016： 3406-3413. 10.1109/icra.2016.7487517
38	YAN X， HSU J， KHANSARI M， et al. Learning 6-DOF grasping interaction via deep geometry aware 3D representations ［C］// Proceedings of the 2018 IEEE International Conference on Robotics and Automation. Piscataway： IEEE， 2018： 3766-3773. 10.1109/icra.2018.8460609
39	LWOWSKI J， MAJUMDAT A， BENAVIDEZ P， et al. HTC vive tracker： Accuracy for indoor location ［J］. IEEE Systems， Man， and Cybernetics Magazine， 2020， 6（4）： 15-22. 10.1109/msmc.2020.2969031
40	QI C R， YI L， SU H， et al. PointNet++： Deep hierarchical feature learning on point sets in a metric space ［C］// Proceedings of the 31st International Conference on Neural Information Processing Systems. Red Hook： Curran Associates Inc.， 2017： 5105-5114.
41	WELLING M， THE Y W. Bayesian learning via stochastic gradient Langevin dynamics ［C］// Proceedings of the 28 th International Conference on Machine Learning. Madison： Omnipress， 2011： 681-688.
42	KIRKPATRICK S， GELATT C D， Jr， VECCHI M P. Optimization by simulated annealing ［J］. Science， 1983， 220（4598）： 671-680. 10.1126/science.220.4598.671

[1]	Jieke WANG, Lin LI, Hailong ZHANG, Liping ZHENG. Passive haptic interaction method for multiple virtual targets in vast virtual reality space [J]. Journal of Computer Applications, 2022, 42(11): 3544-3550.
[2]	Shouming HOU, Chaolan JIA, Mingmin ZHANG. Review of eye movement‑based interaction techniques for virtual reality systems [J]. Journal of Computer Applications, 2022, 42(11): 3534-3543.
[3]	WANG Yi, LYU Jian, YOU Qian, ZHAO Zeyu, YAN Baoming, ZHU Shuman. Virtual reality arbitrary shape selection model based on Fitts' law [J]. Journal of Computer Applications, 2020, 40(11): 3320-3326.
[4]	HU Min, LI Chong, LU Rongrong, HUANG Hongcheng. Performance analysis of motor imagery training based on 3D visual guidance [J]. Journal of Computer Applications, 2018, 38(3): 836-841.
[5]	LIN Chang, LI Guoping, ZHAO Haiwu, WANG Guozhong, GU Xiao. Double ring mapping projection for panoramic video [J]. Journal of Computer Applications, 2017, 37(9): 2631-2635.
[6]	WU Saiwen, CHEN Jian, SUN Xiaoying. Electrostatic force tactile rendering method for video perception [J]. Journal of Computer Applications, 2016, 36(4): 1137-1140.
[7]	LI Danni, LIU Qi, TIAN Qi, ZHAO Leiyu, HE Ling, HUANG Yunzhi, ZHANG Jing. Design of virtual surgery system in reduction of maxillary fracture [J]. Journal of Computer Applications, 2015, 35(6): 1730-1733.
[8]	CAO Yanjue, AN Bowen, LI Qiming. Real-time simulation and application of depth-of-field based on post-processing [J]. Journal of Computer Applications, 2015, 35(5): 1439-1443.
[9]	YANG Xiaowen, ZHANG Zhichun, KUANG liqun, HAN Xie. Key technologies of human-computer interaction based on virtual hand [J]. Journal of Computer Applications, 2015, 35(10): 2945-2949.
[10]	. Automatic path finding method for real-time rendering of 3D scene [J]. Journal of Computer Applications, 2010, 30(1): 85-89.
[11]	. Realistic simulation of plant based on geometry parameter [J]. Journal of Computer Applications, 2009, 29(1): 97-100.
[12]	. Curved surface modeling and texture acquisition from three images [J]. Journal of Computer Applications, 2007, 27(2): 470-472.
[13]	. Study on model transformation and computer demonstration of virtual prototype [J]. Journal of Computer Applications, 2006, 26(9): 2057-2058.
[14]	. Level-of-detail technology of animation and collision in virtual reality [J]. Journal of Computer Applications, 2006, 26(11): 2730-2732.
[15]	WANG Yi-yuan,CHEN Fu-min. Real-time synchronization algorithm based on blending of motions in VR [J]. Journal of Computer Applications, 2005, 25(08): 1951-1953.

Construction and application of 3D dataset of human grasping objects

人手抓取物体的三维数据集的建立及应用

RichHTML

PDF

Knowledge

Abstract

Cite this article

share this article

Figures/Tables 9

References 42

Related Articles 15

Recommended Articles

Metrics