Journal of Computer Applications ›› 2024, Vol. 44 ›› Issue (5): 1501-1510.DOI: 10.11772/j.issn.1001-9081.2023050788

Special Issue: 第十九届中国机器学习会议(CCML 2023)

• The 19th China Conference on Machine Learning (CCML 2023) • Previous Articles     Next Articles

Collaborative offloading strategy in internet of vehicles based on asynchronous deep reinforcement learning

Xiaoyan ZHAO1,2, Wei HAN1, Junna ZHANG1,2(), Peiyan YUAN1   

  1. 1.College of Computer and Information Engineering,Henan Normal University,Xinxiang Henan 453007,China
    2.Henan Engineering Lab of Intelligence Business & Internet of Things (Henan Normal University),Xinxiang Henan 453007,China
  • Received:2023-06-20 Revised:2023-07-14 Accepted:2023-07-24 Online:2023-08-03 Published:2024-05-10
  • Contact: Junna ZHANG
  • About author:ZHAO Xiaoyan, born in 1981, Ph. D., associate professor. Her research interests include edge computing, D2D communication.
    HAN Wei, born in 1995, M. S. candidate. His research interests include edge computing.
    YUAN Peiyan,born in 1978, Ph. D., professor. His research interests include edge computing, crowd sensing.
  • Supported by:
    National Natural Science Foundation of China(62072159);Science and Technology Research Project of Henan Province(222102210011)

基于异步深度强化学习的车联网协作卸载策略

赵晓焱1,2, 韩威1, 张俊娜1,2(), 袁培燕1   

  1. 1.河南师范大学 计算机与信息工程学院, 河南 新乡 453007
    2.智慧商务与物联网技术河南省工程实验室(河南师范大学), 河南 新乡 453007
  • 通讯作者: 张俊娜
  • 作者简介:赵晓焱(1981—),女,河南许昌人,副教授,博士,CCF会员,主要研究方向:边缘计算、D2D通信
    韩威(1995—),男,河南商丘人,硕士研究生,CCF会员,主要研究方向:边缘计算
    袁培燕(1978—),男,河南邓州人,教授,博士,CCF会员,主要研究方向:边缘计算、群智感知。
    第一联系人:张俊娜(1979—),女,河南扶沟人,副教授,博士,CCF会员,主要研究方向:边缘计算、服务计算
  • 基金资助:
    国家自然科学基金资助项目(62072159);河南省科技攻关项目(222102210011)

Abstract:

With the rapid development of Internet of Vehicles (IoV), smart connected vehicles generate a large number of latency-sensitive and computation-intensive tasks, and limited vehicle computing resources and traditional cloud service modes cannot meet the needs of in-vehicle users. Mobile Edge Computing (MEC) provides an effective paradigm for solving task offloading of massive data. However, when considering multi-task and multi-user scenarios, the complexity of task offloading scenarios in IoV is high due to the real-time and dynamic changes in vehicle locations, task types and vehicle density, and the offloading process is prone to problems such as unbalanced edge resource allocation, excessive communication cost overhead and slow algorithm convergence. To solve the above problems, cooperative task offloading strategy of multiple edge servers in multi-task and multi-user mobile scenarios in IoV was focused on. First, a three-layer heterogeneous network model for multi-edge collaborative processing was proposed, and dynamic collaborative clusters were introduced for the changing environment in IoV to transform the offloading problem into a joint optimization problem of delay and energy consumption. Then, the problem was divided into two subproblems of offloading decision and resource allocation, where the resource allocation problem was further split into resource allocation for edge servers and transmission bandwidth, and the two subproblems were solved based on convex optimization theory. In order to find the optimal offloading decision set, a Multi-edge Collaborative Deep Deterministic Policy Gradient (MC-DDPG) algorithm that can handle continuous problems in collaborative clusters was proposed, based on which an Asynchronous MC-DDPG (AMC-DDPG) algorithm was designed. The training parameters in collaborative clusters were asynchronously uploaded to the cloud for global update, and then the updated results were returned to each collaborative cluster to improve the convergence speed. Simulation results show that the AMC-DDPG algorithm improves the convergence speed by at least 30% over the DDPG algorithm and achieves better results in terms of reward and total cost.

Key words: Internet of Vehicles (IoV), Mobile Edge Computing (MEC), task offloading, collaboration, Deep Reinforcement Learning (DRL)

摘要:

随着车联网(IoV)的快速发展,智能网联汽车产生了大量延迟敏感型和计算密集型任务,有限的车辆计算资源以及传统的云服务模式无法满足车载用户的需求,移动边缘计算(MEC)为解决海量数据的任务卸载提供了一种有效范式。但在考虑多任务、多用户场景时,由于车辆位置、任务种类以及车辆密度的实时性和动态变化,IoV中任务卸载场景复杂度较高,卸载过程中容易出现边缘资源分配不均衡、通信成本开销过大、算法收敛慢等问题。为解决以上问题,重点研究了IoV中多任务、多用户移动场景中的多边缘服务器协同任务卸载策略。首先,提出一种多边缘协同处理的三层异构网络模型,针对IoV中不断变化的环境,引入动态协作簇,将卸载问题转化为时延和能耗的联合优化问题;其次,将问题分为卸载决策和资源分配两个子问题,其中资源分配问题又拆分为面向边缘服务器和传输带宽的资源分配,并基于凸优化理论求解。为了寻求最优卸载决策集,提出一种能在协作簇中处理连续问题的多边缘协作深度确定性策略梯度(MC-DDPG)算法,并在此基础上设计了一种异步多边缘协作深度确定性策略梯度(AMC-DDPG)算法,通过将协作簇中的训练参数异步上传至云端进行全局更新,再将更新结果返回每个协作簇中提高收敛速度。仿真结果显示,AMC-DDPG算法较DDPG算法至少提高了30%的收敛速度,且在奖励和总成本等方面也取得了较好的效果。

关键词: 车联网, 移动边缘计算, 任务卸载, 协作, 深度强化学习

CLC Number: