Abstract:

Deep reinforcement learning algorithms are difficult to learn optimal policy through interaction with environment in reward sparsity environments， so that the intrinsic reward needs to be built to guide the update of algorithms. However， there are still some problems in this way： 1） statistical inaccuracy of state classification will misjudge reward value， thereby causing the agent to learn wrong behavior； 2） due to the strong ability of the prediction network to identify state information， the state freshness generated by the intrinsic reward decreases， which affects the learning effect of the optimal policy； 3） due to the random state transition， the information of the teacher strategies is not effectively utilized， which reduces the agent’s ability to explore the environment. To solve the above problems， a reward construction mechanism combining prediction error of stochastic generative network with hash discretization statistics， namely RGNP-HCE （Randomly Generated Network Prediction and Hash Count Exploration）， was proposed， and the knowledge of multi-teacher policy was transferred to student policy through distillation. In RGNP-HCE mechanism， the fusion reward was constructed through the idea of curiosity classification. In specific， the global curiosity reward was constructed by stochastic generative network’s prediction error between multiple episodes， and the local curiosity reward was constructed by hash discretization statistics in one episode， which guaranteed the rationality of intrinsic rewards and the correctness of policy gradient updates. In addition， multi-teacher policy distillation provides students with multiple reference directions for exploration， which improved environmental exploration ability of the student policy effectively. Finally， in the test environments of Montezuma’s Revenge and Breakout， experiment of comparing the proposed mechanism with four current mainstream deep reinforcement learning algorithms was carried out， and policy distillation was performed. The results show that compared with the average performance of current high-performance deep reinforcement learning algorithms， the average performance of RGNP-HCE mechanism in both test environments is improved， and the distilled student policy is further improved in average performance， indicating that RGNP-HCE mechanism and policy distillation are effective in improving the exploration ability of agent.

Key words: reward sparsity, intrinsic reward, exploration ability, policy distillation, deep reinforcement learning

摘要：

深度强化学习算法在奖励稀疏的环境下，难以通过与环境的交互学习到最优策略，因此需要构建内在奖励指导策略进行探索更新。然而，这样仍存在一些问题：1）状态分类存在的统计失准问题会造成奖励值大小被误判，使智能体（agent）学习到错误行为；2）由于预测网络识别状态信息的能力较强，内在奖励产生状态的新鲜感下降，影响了最优策略的学习效果；3）由于随机状态转移，教师策略的信息未被有效利用，降低了智能体的环境探索能力。为了解决以上问题，提出一种融合随机生成网络预测误差与哈希离散化统计的奖励构建机制RGNP-HCE （Randomly Generated Network Prediction and Hash Count Exploration），并通过蒸馏（distillation）将多教师策略的知识迁移到学生策略中。RGNP-HCE机制采用好奇心分类思想构建融合奖励：一方面在多回合间以随机生成网络预测差构建全局好奇心奖励；另一方面在单回合内以哈希离散化统计构建局部好奇心奖励，从而保证内在奖励的合理性以及策略梯度更新的正确性。此外，将多个教师策略学习到的知识通过蒸馏迁移到学生策略中，有效提升学生策略的环境探索能力。最后，在Montezuma’s Revenge与Breakout测试环境中，把所提机制与当前主流的4个深度强化学习算法进行了对比实验，并执行了策略蒸馏。结果表明，相较于当前高性能的强化学习算法，RGNP-HCE机制在两个测试环境中的平均性能均有提升，且蒸馏后学生策略的平均性能又有进一步的提升，验证了RGNP-HCE机制与策略蒸馏方法对提升智能体的环境探索能力是有效的。

关键词: 奖励稀疏, 内在奖励, 探索能力, 策略蒸馏, 深度强化学习