3D part assembly method based on line drawing segmentation

doi:10.11772/j.issn.1001-9081.2025050711

Journal of Computer Applications ›› 2026, Vol. 46 ›› Issue (5): 1545-1550.DOI: 10.11772/j.issn.1001-9081.2025050711

• Multimedia computing and computer simulation • Previous Articles

3D part assembly method based on line drawing segmentation

Huaze ZHU, Weihao WANG, Mingyu YOU(), Hongjun ZHOU

College of Electronic and Information Engineering，Tongji University，Shanghai 201804，China

Received:2025-06-27 Revised:2025-08-22 Accepted:2025-08-26 Online:2025-09-05 Published:2026-05-10
Contact: Mingyu YOU
About author:ZHU Huaze， born in 2001， M. S. candidate. His research interests include 3D assembly.
WANG Weihao， born in 1997， Ph. D. candidate. His research interests include 3D assembly and generation.
ZHOU Hongjun， born in 1972， Ph. D.， associate professor. His research interests include robot positioning， navigation and path planning， human-computer interaction， industrial image detection.
Supported by:
National Natural Science Foundation of China(62473290)

基于线条图分割的三维部件装配方法

朱华泽, 王伟昊, 尤鸣宇(), 周洪钧

同济大学电子与信息工程学院，上海 201804

通讯作者: 尤鸣宇
作者简介:朱华泽（2001—），男，黑龙江鹤岗人，硕士研究生，主要研究方向：三维装配
王伟昊（1997—），男，河北唐山人，博士研究生，主要研究方向：三维装配与生成
周洪钧（1972—），男，上海人，副教授，博士，主要研究方向：机器人定位导航及路径规划、人机交互、工业图像检测。
基金资助:
国家自然科学基金资助项目(62473290)

Abstract

Abstract:

Three-dimensional （3D） part assembly is an important task in 3D computer vision. It aims to estimate the poses of a set of 3D parts and accurately combine them into a target structure. However， existing methods mainly rely on large-scale data for training and learn from past experience to complete assembly， resulting in weak generalization and poor adaptability to new assembly tasks. To address the problem of insufficient generalization in 3D part assembly， assembly instructions with line drawings were introduced as auxiliary information， with the expectation that robots could establish correspondence between 3D parts and regions in 2D line drawings. Nevertheless， establishing such a correspondence faced many challenges. Firstly， multiple identical 3D parts often existed in the assembly， but their corresponding 2D regions had different shapes and positions， which posed difficulties for neural networks in establishing such 3D-2D correspondence. Secondly， occlusions among parts in the line drawings further complicated the establishment of these correspondences. Therefore， a 3D part assembly method based on line-drawing segmentation was proposed， consisting of two main stages. In the first stage， point cloud information was used to perform part instance segmentation on the line drawings， effectively establishing the 3D-2D correspondence of the parts； in the second stage， a graph convolutional network was used to integrate the image information with the segmentation results for component pose estimation， thereby completing the assembly task. On the PartNet dataset， the proposed method was compared with three baseline methods： single-stage， layer-by-layer assembly， and two-stage approaches， demonstrating that it consistently improves component assembly accuracy and validating its effectiveness.

Key words: 3D part assembly, instance segmentation, pose estimation, line drawing, 3D point cloud

摘要：

三维部件装配是三维计算机视觉领域的重要任务，旨在对一组三维部件进行位姿估计，并将它们准确地组合成目标结构；但现有方法主要依赖大规模数据训练以及学习过往经验完成装配，泛化能力较弱，难以适应新的装配任务。为解决三维部件装配泛化性不足的问题，引入线条图形式的装配说明书作为辅助信息，希望机器人能够建立三维部件与二维线条图中区域的对应关系。建立这种对应关系面临诸多挑战：首先，装配体中通常有多个相同三维部件，而它们在二维线条图中的位置与姿态却各不相同，建立这种三维-二维对应关系对神经网络而言很困难；其次，线条图中部件存在相互遮挡，使三维-二维的对应关系更难建立。因此，提出一种基于线条图分割的三维部件装配方法，主要分为2个阶段：第一阶段利用点云信息对线条图进行部件实例分割，有效建立部件三维-二维的对应关系；第二阶段利用图卷积网络整合图像信息与分割结果，进行部件的位姿估计，完成装配任务。在PartNet数据集上与涵盖了单阶段、逐层组装、双阶段的3种基线方法开展对比，结果显示所提方法的部件装配准确率得到了普遍提升，验证了该方法的有效性。

关键词: 三维部件装配, 实例分割, 位姿估计, 线条图, 三维点云

CLC Number:

TP399

Huaze ZHU, Weihao WANG, Mingyu YOU, Hongjun ZHOU. 3D part assembly method based on line drawing segmentation[J]. Journal of Computer Applications, 2026, 46(5): 1545-1550.

朱华泽, 王伟昊, 尤鸣宇, 周洪钧. 基于线条图分割的三维部件装配方法[J]. 《计算机应用》唯一官方网站, 2026, 46(5): 1545-1550.

Figures/Tables 5

References 25

[1]	FUNKHOUSER T， KAZHDAN M， SHILANE P， et al. Modeling by example［J］. ACM Transactions on Graphics， 2004， 23（3）： 652-663.
[2]	HUANG J， ZHAN G， FAN Q， et al. Generative 3D part assembly via dynamic graph learning［C］// Proceedings of the 34th International Conference on Neural Information Processing Systems. Red Hook： Curran Associates Inc.， 2020： 6315-6326.
[3]	IZADINIA H， SHAN Q， SEITZ S M. IM2CAD［C］// Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2017： 2422-2431.
[4]	DU B， GAO X， HU W， et al. Generative 3D part assembly via part-whole-hierarchy message passing［C］// Proceedings of the 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2024： 20850-20859.
[5]	ZHANG R， KONG T， WANG W， et al. 3D part assembly generation with instance encoded Transformer［J］. IEEE Robotics and Automation Letters， 2022， 7（4）： 9051-9058.
[6]	SHAO L， MIGIMATSU T， BOHG J. Learning to scaffold the development of robotic manipulation skills［C］// Proceedings of the 2020 IEEE International Conference on Robotics and Automation. Piscataway： IEEE， 2020： 5671-5677.
[7]	ZACHARES P A， LEE M A， LIAN W， et al. Interpreting contact interactions to overcome failure in robot assembly tasks［C］// Proceedings of the 2021 IEEE International Conference on Robotics and Automation. Piscataway： IEEE， 2021： 3410-3417.
[8]	WANG H， SRIDHAR S， HUANG J， et al. Normalized object coordinate space for category-level 6D object pose and size estimation［C］// Proceedings of the 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2019： 2637-2646.
[9]	YOON Y， DESOUZA G N， KAK A C. Real-time tracking and pose estimation for industrial objects using geometric features［C］// Proceedings of the 2003 IEEE International Conference on Robotics and Automation — Volume 3. Piscataway： IEEE， 2003： 3473-3478.
[10]	LI Y， MO K， SHAO L， et al. Learning 3D part assembly from a single image［C］// Proceedings of the 2020 European Conference on Computer Vision， LNCS 12351. Cham： Springer， 2020： 664-682.
[11]	LI J， XU K， CHAUDHURI S， et al. GRASS： generative recursive autoencoders for shape structures［J］. ACM Transactions on Graphics， 2017， 36（4）： No.52.
[12]	MO K， GUERRERO P， YI L， et al. StructureNet： hierarchical graph networks for 3D shape generation［J］. ACM Transactions on Graphics， 2019， 38（6）： No.242.
[13]	HARIHARAN B， ARBELÁEZ P， GIRSHICK R， et al. Simultaneous detection and segmentation［C］// Proceedings of the 2014 European Conference on Computer Vision， LNCS 8695. Cham： Springer， 2014： 297-312.
[14]	BOLYA D， ZHOU C， XIAO F Y， et al. YOLACT： real-time instance segmentation［C］// Proceedings of the 17th IEEE International Conference on Computer Vision. Piscataway： IEEE， 2019： 9156-9165.
[15]	GIRSHICK R， DONAHUE J， DARRELL T， et al. Rich feature hierarchies for accurate object detection and semantic segmentation［C］// Proceedings of the 2014 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2014： 580-587.
[16]	HE K， GKIOXARI G， DOLLÁR P， et al. Mask R-CNN［C］// Proceedings of the 2017 IEEE International Conference on Computer Vision. Piscataway： IEEE， 2017： 2980-2988.
[17]	郭琪周，袁春. 基于空间语义信息特征融合的目标检测与分割［J］. 软件学报， 2023， 34（6）： 2776-2788.
	GUO Q Z， YUAN C. Leveraging spatial-semantic information in object detection and segmentation［J］. Journal of Software， 2023， 34（6）： 2776-2788.
[18]	毛琳，任凤至，杨大伟，等. 基于卷积神经网络的全景分割 Transformer 模型［J］. 软件学报， 2023， 34（7）： 3408-3421.
	MAO L， REN F Z， YANG D W， et al. CNN based Transformer for panoptic segmentation［J］. Journal of Software， 2023， 34（7）： 3408-3421.
[19]	KIRILLOV A， MINTUN E， RAVI N， et al. Segment anything［C］// Proceedings of the 2023 IEEE/CVF International Conference on Computer Vision. Piscataway： IEEE， 2023： 3992-4003.
[20]	ZHANG C， HAN D， QIAO Y， et al. Faster segment anything： towards lightweight SAM for mobile applications［EB/OL］. ［2025-07-31］..
[21]	ZHAO X， DING W， AN Y， et al. Fast segment anything［EB/OL］. ［2025-07-31］..
[22]	LÜDDECKE T， ECKER A. Image segmentation using text and image prompts［C］// Proceedings of the 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2022： 7076-7086.
[23]	陈浩楠，朱映映，赵骏骐，等. 基于多模态关系建模的三维形状识别方法［J］. 软件学报， 2024， 35（5）： 2208-2219.
	CHEN H N， ZHU Y Y， ZHAO J Q， et al. 3D shape recognition method based on multi-modal relationship modeling［J］. Journal of Software， 2024， 35（5）： 2208-2219.
[24]	WANG B， CHEN C， CUI Z， et al. P2-Net： joint description and detection of local features for pixel and point matching［C］// Proceedings of the 2021 IEEE/CVF International Conference on Computer Vision. Piscataway： IEEE， 2021： 15984-15993.
[25]	BARROSO-LAGUNA A， MUNUKUTLA S， PRISACARIU V A， et al. Matching 2D images in 3D： metric relative pose from metric correspondences［C］// Proceedings of the 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2024： 4852-4863.

方法	CD			PA/%
方法	椅子	桌子	柜子	椅子	桌子	柜子
3DHPA^†	0.005 1	0.002 8	0.003 7	64.13	64.83	55.13
IET^†	0.005 4	0.003 5	0.003 9	62.80	61.67	54.95
3DPA	0.008 7	0.003 6	0.004 3	54.86	51.40	30.04
本文方法	0.004 8	0.002 7	0.003 9	64.24	66.45	58.52

方法	CD			PA/%
方法	椅子	桌子	柜子	椅子	桌子	柜子
3DHPA^†	0.005 1	0.002 8	0.003 7	64.13	64.83	55.13
IET^†	0.005 4	0.003 5	0.003 9	62.80	61.67	54.95
3DPA	0.008 7	0.003 6	0.004 3	54.86	51.40	30.04
本文方法	0.004 8	0.002 7	0.003 9	64.24	66.45	58.52

方法	MIoU	方法	MIoU
无实例编码	0.519 8	无自注意力	0.540 8
无L₁	0.473 3	本文方法	0.660 9
无L₂	0.618 6

方法	MIoU	方法	MIoU
无实例编码	0.519 8	无自注意力	0.540 8
无L₁	0.473 3	本文方法	0.660 9
无L₂	0.618 6

3D part assembly method based on line drawing segmentation

基于线条图分割的三维部件装配方法

RichHTML

PDF

Knowledge

Abstract

Cite this article

share this article

Figures/Tables 5

References 25

Related Articles 15

Recommended Articles

Metrics

[1]	Shiwei LI, Yufeng ZHOU, Pengfei SUN, Weisong LIU, Zhuxuan MENG, Haojie LIAN. Point cloud data augmentation method based on scattering and absorption effects of coal dust on LiDAR electromagnetic waves [J]. Journal of Computer Applications, 2026, 46(1): 331-340.
[2]	Yi XIONG, Caiqi WANG, Ling MEI, Shiqian WU. Global feature pose estimation method based on keypoint distance [J]. Journal of Computer Applications, 2026, 46(1): 260-269.
[3]	Zimo ZHANG, Xuezhuan ZHAO. Multi-scale sparse graph guided vision graph neural networks [J]. Journal of Computer Applications, 2025, 45(7): 2188-2194.
[4]	Shiyue GUO, Jianwu DANG, Yangping WANG, Jiu YONG. 3D hand pose estimation combining attention mechanism and multi-scale feature fusion [J]. Journal of Computer Applications, 2025, 45(4): 1293-1299.
[5]	Benjie SHE, Shuzhi SU, Yanmin ZHU, Jian HUA, Chao WANG. Lightweight pose estimation network based on non-globally dependent integral regression [J]. Journal of Computer Applications, 2025, 45(3): 972-977.
[6]	Jianhua REN, Jiahui CAO, Di JIA. Hand pose estimation based on mask prompts and attention [J]. Journal of Computer Applications, 2025, 45(12): 4012-4020.
[7]	Zhuoran LI, Hua LI, Tong WANG, Chaozhe JIANG. Lightweight human pose estimation based on merge state space model [J]. Journal of Computer Applications, 2025, 45(10): 3179-3186.
[8]	Shixiong KUANG, Junbo YAO, Jiawei LU, Qibing WANG, Gang XIAO. Data augmentation method for abnormal elevator passenger behaviors based on dynamic graph convolutional network [J]. Journal of Computer Applications, 2025, 45(10): 3187-3194.
[9]	Zhengdong ZENG, Ming ZHAO. Spatio-temporal context network for 3D human pose estimation based on graph attention [J]. Journal of Computer Applications, 2025, 45(10): 3161-3169.
[10]	Junying CHEN, Shijie GUO, Lingling CHEN. Lightweight human pose estimation based on decoupled attention and ghost convolution [J]. Journal of Computer Applications, 2025, 45(1): 223-233.
[11]	Xun SUN, Ruifeng FENG, Yanru CHEN. Monocular 3D object detection method integrating depth and instance segmentation [J]. Journal of Computer Applications, 2024, 44(7): 2208-2215.
[12]	Zheng WU, Zhiyou CHENG, Zhentian WANG, Chuanjian WANG, Sheng WANG, Hui XU. Deep learning-based classification of head movement amplitude during patient anaesthesia resuscitation [J]. Journal of Computer Applications, 2024, 44(7): 2258-2263.
[13]	Yaxing BING, Yangping WANG, Jiu YONG, Haomou BAI. Six degrees of freedom object pose estimation algorithm based on filter learning network [J]. Journal of Computer Applications, 2024, 44(6): 1920-1926.
[14]	Wenshuo GAO, Xiaoyun CHEN. Point cloud classification network based on node structure [J]. Journal of Computer Applications, 2024, 44(5): 1471-1478.
[15]	Pengfei ZHANG, Litao HAN, Hengjian FENG, Hongmei LI. Point cloud semantic segmentation based on attention mechanism and global feature optimization [J]. Journal of Computer Applications, 2024, 44(4): 1086-1092.