Abstract: In reinforcement learning, the estimation bias of value functions is a critical challenge that restricts the improvement of algorithm performance. It stems from factors such as environmental dynamic uncertainty, function approximation errors, and data distribution shifts, causing the strategy optimization direction to deviate from the true optimal solution and even leading to divergence in the training process. To systematically alleviate this problem, a strategy optimization method combining overestimated value functions and underestimated value functions was proposed. Meanwhile, a dynamic weight adjustment method was designed, through which the weight of the overestimated value function was dynamically adjusted by real-time approximation of the underestimation degree of the current underestimated value function. Additionally, a method for dynamically selecting underestimated value functions was presented. To verify the effectiveness of the proposed strategy optimization method, it was respectively combined with the TD3 (Twin Delayed Deep Deterministic policy gradient), TADD (Triplet Average Deep Deterministic policy gradient algorithm), and QMD3 (Quasi-Median Delayed Deep Deterministic policy gradient) algorithms, and tested and evaluated on six control tasks in OpenAI Gym. The experimental results showed that the proposed method achieved performance improvements in most tasks.

Key words: complementary value network, dynamic weight adjustment, policy optimization, estimation bias, underestimated value network selection

摘要： 在强化学习中，价值函数估计偏差是制约算法性能提升的关键难题，源于环境动态不确定性、函数近似误差及数据分布偏移等因素，导致策略优化方向偏离真实最优解，甚至引发训练过程发散。为系统性缓解该问题，提出一种结合高估价值函数与低估价值函数的策略优化方法。同时，设计了一种动态调节的权重调节方法，该方法通过实时近似判断当前低估价值函数的低估程度动态调节高估价值函数权重；并提出一种能够动态选择低估价值函数的方法。为验证该策略优化方法的有效性，分别将该方法与TD3（Twin Delayed Deep Deterministic policy gradient）、TADD（Triplet Average Deep Deterministic policy gradient algorithm）、QMD3（Quasi-Median Delayed Deep Deterministic policy gradient）算法相结合，并在OpenAI Gym的6个控制任务上进行测试评估。实验结果表明，所提方法在大部分任务上取得了性能提升。

关键词: 互补价值网络, 动态权重调节, 策略优化, 估计偏差, 低估价值网络选择