Multi-Agent Q-Learning Aided Backpressure Routing Algorithm for Delay Reduction

In queueing networks, it is well known that the throughput-optimal backpressure routing algorithm results in poor delay performance for light and moderate traffic loads. To improve delay performance, state-of-the-art backpressure routing algorithm (called BPmin [1]) exploits queue length information to direct packets to less congested routes to their destinations. However, BPmin algorithm estimates route congestion based on unrealistic assumption that every node in the network knows real-time global queue length information of all other nodes. In this paper, we propose multi-agent Q-learning aided backpressure routing algorithm, where each node estimates route congestion using only local information of neighboring nodes. Our algorithm not only outperforms state-of-the-art BPmin algorithm in delay performance but also retains the following appealing features: distributed implementation, low computation complexity and throughput-optimality. Simulation results show our algorithm reduces average packet delay by 95% for light traffic loads and by 41% for moderate traffic loads when compared to state-of-the-art BPmin algorithm.