关键词: 深度强化学习, 交通信号控制, 时序预测, 蓄水池抽样算法, 长短期记忆

Abstract:

In order to cope with traffic congestion， efficient traffic signal control algorithms have been designed， which can improve the traffic efficiency of vehicles in the existing transportation network significantly. Although deep reinforcement learning algorithms have shown excellent performance in single intersection traffic signal control problems， their application in multi-intersection environments still faces major challenge — the non-stationarity problem caused by the spatiotemporal partial observability generated by Multi-Agent Reinforcement Learning （MARL） algorithm， resulting in that the deep reinforcement learning algorithms cannot guarantee stable convergence. To this end， a multi-intersection traffic signal control algorithm based on overall state prediction and fair experience replay — IS-DQN was proposed. For one thing， to avoid the problem of non-stationarity caused by spatial observability in algorithm， the state space of IS-DQN was expanded by predicting the overall state of multiple intersections based on historical traffic flow information from different lanes. For another， in order to cope with the time partial observability brought by traditional experience replay strategies， a reservoir sampling algorithm was adopted to ensure the fairness of experience replay pool， so as to avoid non-stationary problems in it. Experimental results on three different traffic pressure simulations in complex multi-intersection environments show that under different traffic pressure conditions， especially in low and medium traffic flow conditions， IS-DQN algorithm has lower average vehicle driving time， better convergence performance and convergence stability compared to independent deep reinforcement learning algorithms.

Key words: deep reinforcement learning, traffic signal control, time series prediction, reservoir sampling algorithm, Long Short-Term Memory (LSTM)