关键词: 互补价值网络, 动态权重调节, 策略优化, 估计偏差, 低估价值网络选择

Abstract:

In Reinforcement Learning （RL）， the estimation bias of value functions is a critical challenge that limits the improvement of algorithm performance. It is caused by factors such as uncertainty in environmental dynamics， function approximation errors， and data distribution shifts， and makes the policy optimization direction deviate from the true optimal solution and even leads to divergence in the training process. Therefore， a Policy Optimization method Integrating overestimated and underestimated Value Functions （POIVF） was proposed. Meanwhile， a dynamic weight adjustment method was designed， through which the weight of the overestimated value function was adjusted dynamically by real-time approximation of the underestimation degree of the current underestimated value function. Additionally， a method for selecting underestimated value functions dynamically was presented. To verify the effectiveness of POIVF， the method was combined with the TD3 （Twin Delayed Deep Deterministic policy gradient）， TADD （Triplet Average Deep Deterministic policy gradient algorithm）， and QMD3 （Quasi-Median Delayed Deep Deterministic policy gradient） algorithms， respectively， and was tested and evaluated on six control tasks in OpenAI Gym. Experimental results show that POIVF achieves performance improvements in most tasks.

Key words: complementary value network, dynamic weight adjustment, policy optimization, estimation bias, underestimated value network selection