Heterogeneous multi-agent reinforcement learning enabled co-optimization of UAV 3D obstacle avoidance and edge computing

doi:10.11772/j.issn.1001-9081.2025080956

Journal of Computer Applications ›› 2026, Vol. 46 ›› Issue (9): 2898-2909.DOI: 10.11772/j.issn.1001-9081.2025080956

• Advanced computing • Previous Articles

Heterogeneous multi-agent reinforcement learning enabled co-optimization of UAV 3D obstacle avoidance and edge computing

Guanliang CHEN¹, Yi LIU¹(), Yi YU²

^1.School of Automation，Guangdong University of Technology，Guangzhou Guangdong 510006，China
^2.Changsha Electronic Industry School，Changsha Hunan 410116，China

Received:2025-08-19 Revised:2025-11-30 Accepted:2025-12-05 Online:2026-02-12 Published:2026-09-10
Contact: Yi LIU
About author:CHEN Guanliang， born in 1999， M. S. candidate. His research interests include edge computing.
LIU Yi， born in 1981， Ph. D.， professor. His research interests include edge computing， smart grid.
YU Yi， born in 1984， M. S.， lecturer. Her research interests include computer networks， computer graphics and image processing.
Supported by:
National Key R&D Program of China(2020YFB1807805)

异构多智能体强化学习驱动的无人机三维避障与边缘计算协同优化

陈冠良¹, 刘义¹(), 余意²

^1.广东工业大学自动化学院，广州 510006
^2.长沙市电子工业学校，长沙 410116

通讯作者: 刘义
作者简介:陈冠良（1999—），男，广东广州人，硕士研究生，主要研究方向：边缘计算
刘义（1981—），男，广东广州人，教授，博士，主要研究方向：边缘计算、智能电网
余意（1984—），女，湖南长沙人，讲师，硕士，主要研究方向：计算机网络、计算机图形图像处理。
基金资助:
国家重点研发计划项目(2020YFB1807805)

Abstract

Abstract:

Severe challenges in terms of real-time processing and low-energy transmission brought by the development of Internet of Things （IoT） and the proliferation of mobile terminal devices are face by compute-intensive tasks. Particularly in multi-Unmanned Aerial Vehicle （UAV） -assisted Mobile Edge Computing （MEC） scenarios， communication links are constrained by obstacle blockages and UAV trajectories in complex 3D environments， and latency and energy consumption pressures are further exacerbated. For the scenario of multiple UAVs providing computational offloading services to ground users， an optimization model was established to minimize the weighted sum of the system's maximum task completion latency and total energy consumption， so as to optimize the users' discrete offloading decisions and the UAVs' continuous 3D trajectories jointly. To solve the problem of mixed （discrete-continuous） action space and strong decision coupling， a heterogeneous multi-agent algorithm UOUM （User Offloading and UAV Mobility co-optimization） was proposed. In the algorithm， under a heterogeneous multi-agent deep reinforcement learning framework， dedicated network architectures were designed for the two types of heterogeneous agents： users and UAVs. And differential reward mechanism was introduced to quantify marginal contributions of the agents， thereby solving the multi-agent credit allocation problem. Concurrently， an Artificial Potential Field （APF） was innovatively integrated as a differentiable physical constraint into the agent learning framework， so as to ensure safe obstacle avoidance for the UAVs. Simulation results show that compared to three benchmark methods （Only User Offloading optimization （OUO）， Only UAV Trajectory optimization （OUT）， and a standard heterogeneous multi-agent reinforcement learning （H-MARL） using only a global reward mechanism））， UOUM has advantages in various scenarios with different numbers of users， UAVs， and obstacle densities. Compared with H-MARL， UOUM has the final convergence reward improved by approximately 28.6% on average， and achieves strong environmental adaptability in terms of latency control， energy optimization， and safe obstacle avoidance.

Key words: Mobile Edge Computing (MEC), multi-agent reinforcement learning, Unmanned Aerial Vehicle （UAV) obstacle avoidance, task offloading, 3D trajectory optimization

摘要：

随着物联网（IoT）的发展与移动终端设备的激增，计算密集型任务在实时处理与低能耗传输方面面临严峻挑战。尤其在多无人机（UAV）辅助的移动边缘计算（MEC）场景中，复杂三维环境中的通信链路受障碍物遮挡与UAV轨迹限制，进一步增大了时延与能耗压力。本文针对多UAV为地面用户提供计算卸载服务的场景，建立最小化系统最大任务完成时延与总能耗加权和的优化模型，以联合优化用户的离散卸载决策与UAV的连续三维轨迹。为了解决混合（离散-连续）动作空间与强决策耦合的问题，提出异构多智能体算法UOUM（User Offloading and UAV Mobility co-optimization）。该算法在异构多智能体深度强化学习框架下，为用户和UAV两类异构智能体设计专属网络架构；引入差分奖励机制量化智能体的边际贡献，以解决多智能体的信用分配问题；同时，将人工势能场（APF）作为可微物理约束融入智能体学习框架，以确保UAV在复杂环境中的安全避障。仿真实验结果表明，与3种基准方法（仅用户卸载优化（OUO）、仅UAV轨迹优化（OUT）以及仅采用全局奖励机制的标准异构多智能体强化学习（H-MARL））相比，UOUM在不同用户数、UAV数及障碍物密度场景下均展现出优势，UOUM的最终收敛奖励比H-MARL平均提升了28.6%，且在时延控制、能耗优化和安全避障方面均展现出强大的环境适应性。

关键词: 移动边缘计算, 多智能体强化学习, 无人机避障, 任务卸载, 三维轨迹优化

CLC Number:

TN929.5

Guanliang CHEN, Yi LIU, Yi YU. Heterogeneous multi-agent reinforcement learning enabled co-optimization of UAV 3D obstacle avoidance and edge computing[J]. Journal of Computer Applications, 2026, 46(9): 2898-2909.

陈冠良, 刘义, 余意. 异构多智能体强化学习驱动的无人机三维避障与边缘计算协同优化[J]. 《计算机应用》唯一官方网站, 2026, 46(9): 2898-2909.

Figures/Tables 10

Fig. 1 Edge computing task offloading model based on multi-UAV collaborative obstacle avoidance

Fig. 2 Execution framework of UOUM algorithm

Tab. 1 Experimental parameters

参数	数值
服务区域边界	［-500，-500］m，［500，500］m
无人机数	2，3，4
附加损耗 $η L o S, η N L o S$	1.5 dB，10 dB
噪声功率 $σ 2$	10 dB
光速c	$3 × 108 m / s$
发射功率 $P n m$	$400 m W$
带宽 $B$	$30 M H z$
计算密度 $C n t$	$100,200 C P U c y c l e s / b i t$
计算功率参数 $κ n$	$1 × 10 - 28 J / H z 3 / s$
计算数据大小 $D n t$	$[2,4] M B$
旋翼桨盘面积 $A$	0.503 $m 2$
空气密度 $ρ$	1.225 $k g / m 3$
旋翼桨尖速度 $U t i p$	62 $m / s$
静态不可跨越障碍安全距离阈值 $ρ N S 0$	40 m
静态可跨越障碍水平安全距离阈值 $d S 0$	10 m
UAV间安全距离阈值 $ρ 0$	30 m

Tab. 1 Experimental parameters

参数	数值
服务区域边界	［-500，-500］m，［500，500］m
无人机数	2，3，4
附加损耗 $η L o S, η N L o S$	1.5 dB，10 dB
噪声功率 $σ 2$	10 dB
光速c	$3 × 108 m / s$
发射功率 $P n m$	$400 m W$
带宽 $B$	$30 M H z$
计算密度 $C n t$	$100,200 C P U c y c l e s / b i t$
计算功率参数 $κ n$	$1 × 10 - 28 J / H z 3 / s$
计算数据大小 $D n t$	$[2,4] M B$
旋翼桨盘面积 $A$	0.503 $m 2$
空气密度 $ρ$	1.225 $k g / m 3$
旋翼桨尖速度 $U t i p$	62 $m / s$
静态不可跨越障碍安全距离阈值 $ρ N S 0$	40 m
静态可跨越障碍水平安全距离阈值 $d S 0$	10 m
UAV间安全距离阈值 $ρ 0$	30 m

Fig. 3 Average reward of each agent

Fig. 4 Reward comparison among various algorithms

Fig. 5 UAV collision curves under different obstacle densities

Fig. 6 Comparison of transmission rates under different obstacle densities

Fig. 7 Comparison of system latency and system weighted energy consumption under different user counts

Fig. 8 Comparison of system latency and system weighted energy consumption under different UAV counts

Fig. 9 Flight trajectories of different numbers of UAVs under varying obstacle densities

References 35

[1]	Pan W， Liu Y， Yang C. Adaptive task offloading of rechargeable UAV edge computing network based on double decision value iteration ［J］. Computer Communications， 2022， 193： 136-145.
[2]	Arthurs P， Gillam L， Krause P， et al. A taxonomy and survey of edge cloud computing for intelligent transportation systems and connected vehicles ［J］. IEEE Transactions on Intelligent Transportation Systems， 2022， 23（7）： 6206-6221.
[3]	Chen H， Wen Y， Huang Y， et al. Edge computing enabling internet of ships： a survey on architectures， emerging applications， and challenges ［J］. IEEE Internet of Things Journal， 2025， 12（2）： 1509-1528.
[4]	Duan S， Wang D， Ren J， et al. Distributed artificial intelligence empowered by end-edge-cloud computing： a survey ［J］. IEEE Communications Surveys and Tutorials， 2023， 25（1）： 591-624.
[5]	Baruffa G， Detti A， Rugini L， et al. AI-driven ground robots： mobile edge computing and mmWave communications at work ［J］. IEEE Open Journal of the Communications Society， 2024， 5： 3104-3119.
[6]	Al-Doghman F， Moustafa N， Khalil I， et al. AI-enabled secure microservices in edge computing： opportunities and challenges ［J］. IEEE Transactions on Services Computing， 2023， 16（2）： 1485-1504.
[7]	李校林，江雨桑. 无人机辅助移动边缘计算中的任务卸载算法［J］. 计算机应用， 2023， 43（6）： 1893-1899.
	Li Xiaolin， Jiang Yusang. Task offloading algorithm for UAV-assisted mobile edge computing ［J］. Journal of Computer Applications， 2023， 43（6）： 1893-1899.
[8]	Zheng Y， Chen J. Geography-aware optimal UAV 3D placement for LOS relaying： a geometry approach ［J］. IEEE Transactions on Wireless Communications， 2024， 23（8）： 9301-9314.
[9]	Saoud B B， Nasraoui L. Coverage optimization for reliable UAV-assisted 5G/6G communication systems ［J］. IEEE Systems Journal， 2025， 19（1）： 65-74.
[10]	Zhao M， Zhang R， He Z， et al. Joint optimization of trajectory， offloading， caching， and migration for UAV-assisted MEC ［J］. IEEE Transactions on Mobile Computing， 2025， 24（3）： 1981-1998.
[11]	Liu D， Bao W， Zhu X， et al. Cooperative path optimization for multiple UAVs surveillance in uncertain environment ［J］. IEEE Internet of Things Journal， 2022， 9（13）： 10676-10692.
[12]	Chen J， Yang P， Ren S， et al. Enhancing AIoT device association with task offloading in aerial MEC networks ［J］. IEEE Internet of Things Journal， 2024， 11（1）： 174-187.
[13]	Wang L， Wang K， Pan C， et al. Multi-agent deep reinforcement learning-based trajectory planning for multi-UAV assisted mobile edge computing ［J］. IEEE Transactions on Cognitive Communications and Networking， 2021， 7（1）： 73-84.
[14]	赵晓焱，韩威，张俊娜，等. 基于异步深度强化学习的车联网协作卸载策略［J］. 计算机应用， 2024， 44（5）： 1501-1510.
	Zhao Xiaoyan， Han Wei， Zhang Junna， et al. Collaborative offloading strategy in internet of vehicles based on asynchronous deep reinforcement learning ［J］. Journal of Computer Applications， 2024， 44（5）： 1501-1510.
[15]	Chen Y， Ai B， Niu Y， et al. Energy-constrained computation offloading in space-air-ground integrated networks using distributionally robust optimization ［J］. IEEE Transactions on Vehicular Technology， 2021， 70（11）： 12113-12125.
[16]	Wang Y， Zhang C， Ge T， et al. Computation offloading via multi-agent deep reinforcement learning in aerial hierarchical edge computing systems ［J］. IEEE Transactions on Network Science and Engineering， 2024， 11（6）： 5253-5266.
[17]	Sun G， He L， Sun Z， et al. Joint task offloading and resource allocation in aerial-terrestrial UAV networks with edge and fog computing for post-disaster rescue ［J］. IEEE Transactions on Mobile Computing， 2024， 23（9）： 8582-8600.
[18]	Ye F， Hao W， Zhang J， et al. Energy-oriented offloading decision and multidimensional resource allocation for UAV-assisted edge computing systems ［C］// VTC 2024 - Spring. Piscataway： IEEE， 2024： 1-6.
[19]	Guo H， Li J， Liu J， et al. A survey on space-air-ground-sea integrated network security in 6G ［J］. IEEE Communications Surveys and Tutorials， 2022， 24（1）： 53-87.
[20]	Ye W， Zhao L， Zhou J， et al. Energy-efficient flight scheduling and trajectory optimization in UAV-aided edge computing networks［J］. IEEE Transactions on Network Science and Engineering， 2024， 11（5）： 4591-4602.
[21]	Han S， Liu X， Zhou M， et al. Joint association， deployment and flight trajectory optimization for multi-UAV-enabled large-scale mobile edge computing ［J］. IEEE Transactions on Mobile Computing， 2024， 23（12）： 13207-13221.
[22]	Ejaz M， Gui J， Asim M， et al. RL-planner： reinforcement learning-enabled efficient path planning in multi-UAV MEC systems ［J］. IEEE Transactions on Network and Service Management， 2024， 21（3）： 3317-3329.
[23]	Yan M， Chan C A， Gygax A F， et al. Efficient generation of optimal UAV trajectories with uncertain obstacle avoidance in MEC networks ［J］. IEEE Internet of Things Journal， 2024， 11（23）： 38380-38392.
[24]	Ouahouah S， Bagaa M， Prados-Garzon J， et al. Deep-reinforcement-learning-based collision avoidance in UAV environment ［J］. IEEE Internet of Things Journal， 2022， 9（6）： 4015-4030.
[25]	Wei Q， Zhou Z， Chen X. DRL-based energy-efficient trajectory planning， computation offloading， and charging scheduling in UAV-MEC network ［C］// ICCC 2022. Piscataway： IEEE， 2022： 1056-1061.
[26]	Yan J， Zhao X， Li Z. Deep-reinforcement-learning-based computation offloading in UAV-assisted vehicular edge computing networks ［J］. IEEE Internet of Things Journal， 2024， 11（11）： 19882-19897.
[27]	Du J， Kong Z， Sun A， et al. MADDPG-based joint service placement and task offloading in MEC empowered air-ground integrated networks ［J］. IEEE Internet of Things Journal， 2024， 11（6）： 10600-10615.
[28]	Li X， Du X， Zhao N， et al. Computing over the sky： joint UAV trajectory and task offloading scheme based on optimization-embedding multi-agent deep reinforcement learning ［J］. IEEE Transactions on Communications， 2024， 72（3）： 1355-1369.
[29]	Qin P， Fu Y， Xie Y， et al. Multi-agent learning-based optimal task offloading and UAV trajectory planning for AGIN-power IoT［J］. IEEE Transactions on Communications， 2023， 71（7）： 4005-4017.
[30]	Zeng Y， Xu J， Zhang R. Energy minimization for wireless communication with rotary-wing UAV ［J］. IEEE Transactions on Wireless Communications， 2019， 18（4）： 2329-2345.
[31]	牟文心，时宏伟. 基于改进TD3算法的无人机轨迹规划［J］. 计算机系统应用， 2024， 33（12）：197-209.
	Mou Wenxin， Shi Hongwei. UAV trajectory planning based on improved TD3 algorithm ［J］. Computer Systems and Applications， 2024， 33（12）： 197-209.
[32]	Mo W， Li J. Obstacle avoidance of multiple UAVs based on reinforcement learning ［C］// ICARCV 2024. Piscataway： IEEE， 2024： 61-66.
[33]	Li H， Chen X， Mao K， et al. Air-to-ground path loss prediction using ray tracing and measurement data jointly driven DNN ［J］. Computer Communications， 2022， 196： 268-276.
[34]	Wang S， Song X， Song T， et al. Fairness-aware computation offloading with trajectory optimization and phase-shift design in RIS-assisted multi-UAV MEC network ［J］. IEEE Internet of Things Journal， 2024， 11（11）： 20547-20561.
[35]	Gao A， Zhu Z， Zhang J， et al. Matching combined heterogeneous multi-agent reinforcement learning for resource allocation in NOMA-V2X networks ［J］. IEEE Transactions on Vehicular Technology， 2024， 73（10）： 15109-15124.

Heterogeneous multi-agent reinforcement learning enabled co-optimization of UAV 3D obstacle avoidance and edge computing

异构多智能体强化学习驱动的无人机三维避障与边缘计算协同优化

RichHTML

PDF

Knowledge

Abstract

Cite this article

share this article

Figures/Tables 10

References 35

Related Articles 15

Recommended Articles

Metrics

[1]	Juan CHEN, Yujie CHEN, Zongling WU, Di TIAN, Jie ZHONG. User-centric satellite edge computing architecture for task offloading optimization [J]. Journal of Computer Applications, 2026, 46(8): 2524-2532.
[2]	Tianyu XUE, Aiping LI, Liguo DUAN. Vehicular edge computing scheme with task offloading and resource optimization [J]. Journal of Computer Applications, 2025, 45(6): 1766-1775.
[3]	Yufei XIANG, Zhengwei NI. Edge federation dynamic analysis for hierarchical federated learning based on evolutionary game [J]. Journal of Computer Applications, 2025, 45(4): 1077-1085.
[4]	Lin WEI, Shihao ZHANG, Mengyang HE. Workflow task optimization and energy-efficient offloading method for computing power network [J]. Journal of Computer Applications, 2025, 45(12): 3916-3924.
[5]	Jian SUN, Wei ZHANG, Baoquan MA, Zhuiwei WU, Xiaohuan YANG, Tao WU. MATOS： UAV swarm assisted moving-aware adaptive-parallel computing task offloading system [J]. Journal of Computer Applications, 2025, 45(10): 3259-3269.
[6]	Junna ZHANG, Xinxin WANG, Tianze LI, Xiaoyan ZHAO, Peiyan YUAN. Task offloading method based on dynamic service cache assistance [J]. Journal of Computer Applications, 2024, 44(5): 1493-1500.
[7]	Xiaoyan ZHAO, Wei HAN, Junna ZHANG, Peiyan YUAN. Collaborative offloading strategy in internet of vehicles based on asynchronous deep reinforcement learning [J]. Journal of Computer Applications, 2024, 44(5): 1501-1510.
[8]	Xiaolin LI, Yusang JIANG. Task offloading algorithm for UAV-assisted mobile edge computing [J]. Journal of Computer Applications, 2023, 43(6): 1893-1899.
[9]	Xiaohui HUANG, Kaiming YANG, Jiahao LING. Order dispatching by multi-agent reinforcement learning based on shared attention [J]. Journal of Computer Applications, 2023, 43(5): 1620-1624.
[10]	Yanpei LIU, Ningning CHEN, Yunjing ZHU, Liping WANG. Review of mobile edge caching optimization technologies for 5G/Beyond 5G [J]. Journal of Computer Applications, 2022, 42(8): 2487-2500.
[11]	Jieqin WANG, Shihyang LIN, Shiming PENG, Shuo JIA, Miaohui YANG. Hierarchical resource allocation mechanism of cooperative mobile edge computing [J]. Journal of Computer Applications, 2022, 42(8): 2501-2510.
[12]	Shiquan DENG, Xuguo YE. Multi-objective task offloading algorithm based on deep Q-network [J]. Journal of Computer Applications, 2022, 42(6): 1668-1674.
[13]	Jingling YUAN, Huihua MAO, Nana WANG, Yao XIANG. Dynamic service deployment strategy in resource constrained mobile edge computing [J]. Journal of Computer Applications, 2022, 42(6): 1662-1667.
[14]	Yu LI, Xiping HE, Lianggui TANG. Multi-user computation offloading and resource optimization policy based on device-to-device communication [J]. Journal of Computer Applications, 2022, 42(5): 1538-1546.
[15]	Xuling ZENG, Taoshen LI, Jian GONG, Lijun DU. Secure offloading optimization of wireless powered mobile edge computing system [J]. Journal of Computer Applications, 2022, 42(4): 1216-1224.