Abstract:

In the training process of manipulator path planning algorithm， the training efficiency of manipulator path planning is low due to the huge action space and state space leading to sparse rewards， and it becomes challenging to evaluate the value of both states and actions given the immense number of states and actions. To address the above problems， a robotic manipulator planning algorithm based on SAC （Soft Actor-Critic） reinforcement learning was proposed. The learning efficiency was improved by incorporating the demonstrated path into the reward function so that the manipulator imitated the demonstrated path during reinforcement learning， and the SAC algorithm was used to make the training of the manipulator path planning algorithm faster and more stable. The proposed algorithm and Deep Deterministic Policy Gradient （DDPG） algorithm were used to plan 10 paths respectively， and the average distances between paths planned by the proposed algorithm and the DDPG algorithm and the reference paths were 0.8 cm and 1.9 cm respectively. The experimental results show that the path imitation mechanism can improve the training efficiency， and the proposed algorithm can better explore the environment and make the planned paths more reasonable than DDPG algorithm.

Key words: imitative learning, Reinforcement Learning (RL), Soft Actor-Critic (SAC) algorithm, path planning, reward function

摘要：

在机械臂路径规划算法的训练过程中，由于动作空间和状态空间巨大导致奖励稀疏，机械臂路径规划训练效率低，面对海量的状态数和动作数较难评估状态价值和动作价值。针对上述问题，提出一种基于SAC（Soft Actor-Critic）强化学习的机械臂路径规划算法。通过将示教路径融入奖励函数使机械臂在强化学习过程中对示教路径进行模仿以提高学习效率，并采用SAC算法使机械臂路径规划算法的训练更快、稳定性更好。基于所提算法和深度确定性策略梯度（DDPG）算法分别规划10条路径，所提算法和DDPG算法规划的路径与参考路径的平均距离分别是0.8 cm和1.9 cm。实验结果表明，路径模仿机制能提高训练效率，所提算法比DDPG算法能更好地探索环境，使得规划路径更加合理。

关键词: 模仿学习, 强化学习, SAC算法, 路径规划, 奖励函数