Abstract:

In Reinforcement Learning （RL）， the estimation bias of value functions is a critical challenge that limits the improvement of algorithm performance. It is caused by factors such as uncertainty in environmental dynamics， function approximation errors， and data distribution shifts， and makes the policy optimization direction deviate from the true optimal solution and even leads to divergence in the training process. Therefore， a Policy Optimization method Integrating overestimated and underestimated Value Functions （POIVF） was proposed. Meanwhile， a dynamic weight adjustment method was designed， through which the weight of the overestimated value function was adjusted dynamically by real-time approximation of the underestimation degree of the current underestimated value function. Additionally， a method for selecting underestimated value functions dynamically was presented. To verify the effectiveness of POIVF， the method was combined with the TD3 （Twin Delayed Deep Deterministic policy gradient）， TADD （Triplet Average Deep Deterministic policy gradient algorithm）， and QMD3 （Quasi-Median Delayed Deep Deterministic policy gradient） algorithms， respectively， and was tested and evaluated on six control tasks in OpenAI Gym. Experimental results show that POIVF achieves performance improvements in most tasks.

Key words: complementary value network, dynamic weight adjustment, policy optimization, estimation bias, underestimated value network selection

摘要：

在强化学习（RL）中，价值函数估计偏差是制约算法性能提升的关键难题，源于环境动态不确定性、函数近似误差及数据分布偏移等因素，导致策略优化方向偏离真实最优解，甚至引发训练过程发散。为系统性缓解该问题，提出一种结合高低估价值函数的策略优化方法（POIVF）。同时，设计一种动态调节权重的方法，通过实时近似判断当前低估价值函数的低估程度动态调节高估价值函数权重，并提出一种能够动态选择低估价值函数的方法。为验证POIVF的有效性，分别将它与TD3（Twin Delayed Deep Deterministic policy gradient）、TADD（Triplet Average Deep Deterministic policy gradient algorithm）、QMD3（Quasi-Median Delayed Deep Deterministic policy gradient）算法相结合，并在OpenAI Gym的6个控制任务上进行测试评估。实验结果表明，POIVF在大部分任务上取得了性能提升。

关键词: 互补价值网络, 动态权重调节, 策略优化, 估计偏差, 低估价值网络选择