Pith. sign in

REVIEW 3 major objections 6 minor 68 references

STMARL: A Spatio-Temporal Multi-Agent Reinforcement Learning Approach for Cooperative Traffic Light Control

T0 review · 3 major / 6 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Coordinating traffic lights through a directed adjacency graph with graph attention and recurrent memory lowers average travel time below isolated and static controllers, with the largest reported gain of 20.6% on a real-traffic…

desk verdict Solid incremental architecture for coordinated traffic signal control, but the headline empirical claim is unverifiable as reported—no error bars, no run counts, and a hand-picked Δt per dataset. read the letter →

arxiv 1908.10577 v3 pith:M3U2A7SB submitted 2019-08-28 cs.MA cs.AIcs.LG

Yanan Wang , Tong Xu , Xin Niu , Chang Tan , Enhong Chen , Hui Xiong This is my paper · ORCID
classification cs.MAcs.AIcs.LG
keywords trafficlightcontrolmulti-agentreinforcementlearninggraphneuralnetworkattentionrecurrentspatio-temporaldependencydeepQ-learningcooperative
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that traffic light control should be treated as a network coordination problem, not a set of isolated decisions, and that the network structure should be encoded explicitly as a directed graph between intersections. It proposes STMARL, in which each traffic light is a reinforcement-learning agent, an LSTM (a recurrent network that compresses recent history into a hidden state) summarizes past observations, and graph-attention message passing lets each light weigh the state of its neighbors before choosing whether to hold or switch its phase. The central empirical claim is that this combination yields lower average travel times than fixed-time, max-pressure, and several reinforcement-learning baselines on synthetic grids and simulated real-world intersections, with the largest reported margin being 20.6% over the best baseline on the Hefei dataset. If correct, the practical consequence is a controller that learns which neighboring signals matter and can coordinate green waves without a central optimizer.

What carries the argument

The load-bearing object is the directional traffic light adjacency graph $G=(V,E)$, whose nodes are traffic-light control points plus non-control endpoints and whose directed edges are roads labelled by lane count. The workhorse is the graph block: edge encoders project lane-level queue length, vehicle count, and speed into a shared space; a Long Short-Term Memory unit (a recurrent network that compresses a sequence of past observations into a hidden state) gives each node a temporal summary; attention-based node updates, repeated for a fixed number of relation-reasoning steps, pass messages along the directed edges; and a shared output MLP with a residual connection produces per-agent Q-values. This graph does double duty: it makes the spatial coordination structure explicit and it gives the temporal history a place to live.

What would settle it

Re-run the same controllers on an independently calibrated traffic simulator, or in a small field pilot on adjacent intersections, using the same Hefei phase configurations and demand traces; if STMARL's average travel time no longer beats the best baseline by a wide margin, or loses, the central claim is refuted.

Watch

Extended reading notes

Core claim

The paper's central claim is that the spatio-temporal dependency among traffic lights can be captured by a directional adjacency graph plus memory, and that this representation is what makes coordinated control work. In STMARL, each edge of the graph carries lane-level queue length, vehicle count, and average speed; each node carries the current signal phase; a Long Short-Term Memory unit folds the previous $\Delta t$ observations into each node; and iterative attention-based node updates propagate influence along the road direction. The authors report that this architecture outperforms all tested baselines on every dataset, with the largest improvement on the real-traffic Hefei network, where average travel time drops to 63.86 seconds from the best baseline's 80.45 seconds. They also report that the two design choices contribute unevenly: ablations show the spatial graph attention adds more than the temporal memory in most settings, but combining both converges faster and performs best.

Load-bearing premise

All effectiveness results come from a traffic simulator whose driver behavior, phase-change rules, and road abstraction stand in for real intersections, so the reported gains are only as trustworthy as that simulator's fidelity.

Editorial extensions

If this is right

  • If the reported results are correct, coordinating signals through a directed graph plus memory reduces average travel time compared with fixed-time, max-pressure, and isolated or rule-based reinforcement-learning controllers in the tested settings.
  • The ablation results imply that, in most tested networks, spatial neighbor information is the larger source of improvement, so adding graph structure should be the first step in upgrading an independent DQN controller.
  • The learned attention weights track incoming vehicle counts, which gives traffic engineers a mechanism-level explanation of which neighbor relations the controller prioritizes.
  • The claimed time complexity, which scales with the temporal interval $\Delta t$ and not with the number of intersections, implies the method can be applied to larger road networks without a per-intersection parameter explosion.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • My inference: the same directed-graph-plus-memory controller could be applied to virtual traffic lights, since the paper notes that its graph formulation works whenever each virtual leader is treated as a control node.
  • My inference: the attention weights could serve as a diagnostic for traffic engineers, letting them compare learned coordination priorities against measured turning movements to find intersections where cooperation is saturated.
  • My inference: a stronger generalization test would train on one week of the Hefei records and evaluate on a later week without retraining, since day-to-day demand shifts would reveal whether the learned coordination transfers or merely memorizes one traffic pattern.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. This paper proposes STMARL, a multi-agent reinforcement learning framework for coordinating traffic lights. The method constructs a directed intersection-level traffic-light adjacency graph, encodes edge-level lane observations with per-edge-type encoders, uses an LSTM to incorporate historical traffic states, and performs multi-step graph attention message passing to model spatial dependence, followed by independent DQN heads with shared parameters for decentralized control. The authors evaluate STMARL in the CityFlow simulator on two synthetic grid networks and two real-world trajectory datasets (Hangzhou public data and seven days of collected Hefei camera data), comparing average travel time against fixed-time control, max-pressure, Max-Plus, Neighbor RL, GCN-lane, GCN-inter, Colight, and three ablations. They report that STMARL significantly outperforms all baselines on all datasets, with the largest margin being 20.6% over the best baseline on DHefei, and they analyze learned attention weights and the emergence of green waves as qualitative evidence of coordination.

Significance. If the empirical claims hold, the contribution is meaningful: the framework combines a directional intersection-level adjacency graph, recurrent temporal encoding, and attention-based relational reasoning in a scalable decentralized MARL setting, and the qualitative green-wave analysis is a valuable addition beyond aggregate metrics. The paper is clearly structured, the POMDP formulation is standard, and the ablations separating spatial and temporal components are a strength. The quantitative evidence, however, is the main gate: the central "significantly outperforms all baselines" claim currently rests on tables with no run-to-run variance, unspecified t-test pairing, and hyperparameters that appear to be selected after seeing test results. The paper's significance would be substantially strengthened by seed-level statistics, a clear validation protocol, and a more detailed baseline tuning description.

major comments (3)
  1. [§5.2.1, Table 6] The central claim that "our proposed STMARL method significantly outperforms all the baseline methods in all datasets" is not supported by the evidence reported in Table 6. The table gives one average travel time per method and dataset, with no standard deviations, confidence intervals, number of independent runs, or seeds; the only note is that the DHefei result is averaged over seven days, and the seven daily values are not shown. The footnote reporting a paired t-test at p<0.01 does not state the pairing unit or sample size. This matters most for Bidirect6×6, where STMARL beats the best baseline by about 1.9 s (roughly 1%), and for DHefei, where the headline 20.6% margin is a single summary number. If the test was paired over individual vehicles, the large sample size would make even a 1% gap statistically significant without establishing a practically reliable improvement. Please report run-to-run variance, the number of seeds, and a clearly specified pairing unit (e.g., days for DHefei and episodes for the synthetic datasets) so the significance claim can be verified.
  2. [§5.1.5, §5.2.3, §5.2.4] The evaluation protocol appears to select hyperparameters after observing test performance. Section 5.1.5 states that the temporal dependency interval Δt was searched over {3,5,10,15,20} and that the activation function and MLP depth were also searched, while Section 5.2.3 reports that "STMARL achieves the best performance when Δt=10,5,3,20" for the four datasets and Section 5.2.4 selects the hidden size h=64 because it gives the best test travel times. Since Table 6 then reports the best configuration per dataset, the comparison is susceptible to selection bias on the test set, and the claim that the method reliably outperforms baselines across datasets is not established. Please add a validation-based hyperparameter selection procedure, or report results for a fixed configuration and present the sensitivity analysis as exploratory.
  3. [§5.1.6 and Table 6] The relative comparison is under-specified. For all learning-based baselines, the only protocol detail given is that they were trained for 100 episodes and tested with ε=0; there is no description of hyperparameter tuning, learning schedules, or number of runs for Max-Plus, Neighbor RL, GCN-lane, GCN-inter, or Colight. Since the paper's contribution is an empirical superiority claim, the baselines need to be shown to be comparably tuned, or the reported margins could reflect baseline under-tuning. Please provide baseline configuration details and, where possible, use the baselines' own reported or released settings for a fair comparison.
minor comments (6)
  1. [§5.1.2 and Table 6 footnote] Clarify the DHefei averaging: the text says the comparison uses the traffic flow during the most peak hour in one day, while the table footnote says the result is averaged over seven days. Please state whether the reported number is the mean over the seven daily peak-hour evaluations and provide the per-day values.
  2. [Abstract and §5.1.3] The abstract's phrase "experimental results on both synthetic and real-world data" should be qualified as "real-world data simulated in CityFlow" to avoid implying field deployment, since Section 5.1.3 makes clear that all effectiveness results come from the simulator.
  3. [Table 5 and §5.1.6] Table 5 appears inconsistent with the ablation descriptions: STMARL-T is described as not learning temporal dependency while incorporating spatial structure, but the table row for STMARL-T is printed with the same × entries as STMARL-ST. The checkmarks for the adjacency-graph, attention, and temporal-dependency columns should be reconciled with the text.
  4. [§4.2] The observation in Section 4.2 is written as "{{q_l,n_l,w_l}_{l=1}^{l_i}, phaseID_i}", but Table 1 and Section 3 define the third lane feature as speed_l, not w_l. Please define w_l or replace it with speed_l for consistency.
  5. [§4.7] The statement that the time complexity is "irrelevant to the number of intersections" relies on the stated concurrency assumption; in total serial work, the complexity grows with |V| and |E|. Please state explicitly that the bound is per-agent or parallel time.
  6. [Various] There are several typos and formatting issues, including "Sensitiveness" in Section 5.2.3, "hidden layer seize h" in Section 5.2.4, and inconsistent spacing in words such as "Traffic" in the abstract.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation: STMARL's contribution is an architecture plus external-benchmark experiments, with no fitted parameter renamed as a prediction and no load-bearing self-citation.

full rationale

The paper does not claim a mathematical derivation from first principles; its central claim is an empirical comparison against external methods. The graph construction, attention-based node update, LSTM temporal module, and DQN loss are defined independently of the evaluation metric. The average travel time reported in Table 6 is produced by the CityFlow simulator and compared with published baselines; no component is fitted to the reported travel-time outcome. Hyperparameters such as Delta t and hidden size are searched and reported, which is normal model selection rather than a fitted input called prediction. The self-citations in references [2], [6], and [7] appear only as general context for congestion and data mining; they are not used to justify the architecture or the empirical results. The skeptical concern about missing run counts or error bars concerns statistical evidence quality, not circularity. Therefore no self-definitional, fitted-input, self-citation-loaded, uniqueness-imported, ansatz-smuggled, or renaming step can be exhibited.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The empirical claim depends on the simulator being a valid proxy, on the directed graph capturing coordination structure, on the LSTM history being sufficient for partial observability, and on independent DQN converging in a non-stationary multi-agent setting. Hyperparameters such as Δt, hidden size, reasoning steps, and action duration were selected by hand or per dataset. No new physical entities are introduced.

free parameters (5)
  • Temporal dependency interval Δt = 10, 5, 3, 20 per dataset
    Chosen per dataset from {3,5,10,15,20} to produce the best travel time in Figure 6(a); the headline results use these selected values.
  • Hidden layer size h = 64
    Selected because it gives the best performance on all datasets in Figure 9; affects all model layers.
  • Relation reasoning step d = 2
    Set to 2 so each agent aggregates first- and second-order neighbor information; no sensitivity analysis provided.
  • Exploration epsilon schedule = 1 to 0.05 linear decay over first 10 episodes
    Hand-chosen standard epsilon-greedy schedule; affects exploration-exploitation tradeoff during training.
  • Action duration = 10 seconds
    Set to avoid frequent phase switches; influences the effective control frequency and thus travel time.
assumptions (4)
  • domain assumption The CityFlow simulator faithfully models traffic dynamics well enough to rank signal control policies.
    All quantitative results in Section 5 are produced inside CityFlow (Section 5.1.3); if the simulator is inaccurate, the central comparisons lose validity.
  • domain assumption The constructed directional traffic light adjacency graph captures the interaction structure relevant for coordination.
    Section 3 defines the graph with control and non-control nodes; the graph is the skeleton for all spatial reasoning, and missing edges would mean missing coordination channels.
  • domain assumption The LSTM hidden state provides a sufficient summary of history for the partially observable traffic state.
    Section 4.4 uses the standard recurrent approach to POMDPs; sufficiency is assumed, not proven, and the temporal interval is tuned rather than derived.
  • domain assumption Independent deep Q-learning with shared parameters converges to a stable cooperative policy in this setting.
    Section 4.2 adopts independent DQN and cites practical success in [45]; convergence is not guaranteed for this non-stationary multi-agent setting.

how reviews work

0 comments
Cite this review

Pith. "Pith review of STMARL: A Spatio-Temporal Multi-Agent Reinforcement Learning Approach for Cooperative Traffic Light Control." pith.science (2026). https://pith.science/paper/M3U2A7SB

@misc{pith2026190810577,
  author       = {Pith},
  title        = {Pith review of: STMARL: A Spatio-Temporal Multi-Agent Reinforcement Learning Approach for Cooperative Traffic Light Control},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/M3U2A7SB}},
  note         = {Machine review of arXiv:1908.10577}
}
read the original abstract

The development of intelligent traffic light control systems is essential for smart transportation management. While some efforts have been made to optimize the use of individual traffic lights in an isolated way, related studies have largely ignored the fact that the use of multi-intersection traffic lights is spatially influenced and there is a temporal dependency of historical traffic status for current traffic light control. To that end, in this paper, we propose a novel SpatioTemporal Multi-Agent Reinforcement Learning (STMARL) framework for effectively capturing the spatio-temporal dependency of multiple related traffic lights and control these traffic lights in a coordinating way. Specifically, we first construct the traffic light adjacency graph based on the spatial structure among traffic lights. Then, historical traffic records will be integrated with current traffic status via Recurrent Neural Network structure. Moreover, based on the temporally-dependent traffic information, we design a Graph Neural Network based model to represent relationships among multiple traffic lights, and the decision for each traffic light will be made in a distributed way by the deep Q-learning method. Finally, the experimental results on both synthetic and real-world data have demonstrated the effectiveness of our STMARL framework, which also provides an insightful understanding of the influence mechanism among multi-intersection traffic lights.

Figures

Figures reproduced from arXiv: 1908.10577 by the authors.

Figure 1
Figure 1. Example of the spatial structure among multiple traffic lights along [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. An illustration of the traffic light adjacency graph structure in the [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. One sample of fixed phase order in a cycle used in the real world with the meaning of each traffic light phase. nation among multi-intersection traffic light control. 3 PROBLEM STATEMENT In this section, we will first introduce the construction of the directional traffic light adjacency graph, and then formally define our problem. We first attempt to describe the road network structure in the graph perspective. In a… view at source ↗
Figures from the paper (7 more)
Figure 4
Figure 4. Figure 4: Overall Q-network for Spatio-Temporal Multi-Agent Reinforcement Learning. [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]
Figure 4
Figure 4. Figure 4: 4.5 Output Layer Finally, the distributed decision for each traffic light agent is now available with the learned representations for traffic lights. Based on the above learned spatial-temporal depen￾dency representation, at each time t, the decision is made for each t…
Figure 5
Figure 5. Figure 5: Training curve of our model variants across different datasets. Curves are smoothed with a moving average of five points. [PITH_FULL_IMAGE:figures/full_fig_p010_5.png]
Figure 6
Figure 6. Figure 6: Influence of Temporal Dependency Interval [PITH_FULL_IMAGE:figures/full_fig_p011_6.png]
Figure 7
Figure 7. Figure 7: Average arriving vehicles ((a), (b)) and learned attention weights ((c), (d)) by STMARL model on the four incoming edges (e.g., South-North [PITH_FULL_IMAGE:figures/full_fig_p011_7.png]
Figure 8
Figure 8. Figure 8: The emerging green waves show in different time periods on November 6th, 2018, Hefei, in which the waves pointed by the green arrows [PITH_FULL_IMAGE:figures/full_fig_p012_8.png]
Figure 9
Figure 9. Figure 9: Influence of hidden layer size h. 5.2.5 Fairness of STMARL Method In this section, we discuss the fairness of STMARL method among vehicles. As shown in [PITH_FULL_IMAGE:figures/full_fig_p012_9.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

68 extracted references · 62 canonical work pages

  1. [1]

    Traffic’s mind-boggling economic toll,

    CityLab, “Traffic’s mind-boggling economic toll,” 2018. [Online]. Available: https://www.citylab.com/transportation/ 2018/02/traffics-mind-boggling-economic-toll/552488/

    work page 2018

  2. [2]

    Exploring the so- cial learning of taxi drivers in latent vehicle-to-vehicle networks,

    T. Xu, H. Zhu, H. Xiong, H. Zhong, and E. Chen, “Exploring the so- cial learning of taxi drivers in latent vehicle-to-vehicle networks,” IEEE Transactions on Mobile Computing, 2019

    work page 2019

  3. [3]

    Bidirectionally coupled network and road traffic simulation for improved IVC analysis,

    C. Sommer, R. German, and F. Dressler, “Bidirectionally coupled network and road traffic simulation for improved IVC analysis,” IEEE Transactions on mobile computing, vol. 10, pp. 3–15, 2010

    work page 2010

  4. [4]

    Block simplex signal recovery: Methods, trade-offs, and an application to routing,

    C. Wu, A. Pozdnukhov, and A. M. Bayen, “Block simplex signal recovery: Methods, trade-offs, and an application to routing,” IEEE Transactions on Intelligent Transportation Systems , vol. 21, pp. 1547–1559, 2019

    work page 2019

  5. [5]

    Re- grets in routing networks: Measuring the impact of routing apps in traffic,

    T. Cabannes, M. Sangiovanni, A. Keimer, and A. M. Bayen, “Re- grets in routing networks: Measuring the impact of routing apps in traffic,” ACM Transactions on Spatial Algorithms and Systems (TSAS), vol. 5, pp. 1–19, 2019

    work page 2019

  6. [6]

    Competitive analysis for points of interest,

    S. Li, J. Zhou, T. Xu, H. Liu, X. Lu, and H. Xiong, “Competitive analysis for points of interest,” in Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, 2020, pp. 1265–1274

    work page 2020

  7. [7]

    Semi-supervised hierarchical recurrent graph neural network for city-wide parking availability prediction,

    W. Zhang, H. Liu, Y. Liu, J. Zhou, and H. Xiong, “Semi-supervised hierarchical recurrent graph neural network for city-wide parking availability prediction,” in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 34, no. 01, 2020, pp. 1186–1193

    work page 2020

  8. [8]

    Settings for fixed-cycle traffic signals,

    A. J. Miller, “Settings for fixed-cycle traffic signals,” Journal of the Operational Research Society, vol. 14, pp. 373–386, 1963

    work page 1963

Show all 68 references
  1. [9]

    Traffic network micro- simulation model and control algorithm based on approximate dynamic programming,

    B. Yin, M. Dridi, and A. El Moudni, “Traffic network micro- simulation model and control algorithm based on approximate dynamic programming,” IET Intelligent Transport Systems, vol. 10, pp. 186–196, 2016

    2016

  2. [10]

    Self-organizing traffic lights: A realistic simulation,

    S.-B. Cools, C. Gershenson, and B. D ´Hooghe, “Self-organizing traffic lights: A realistic simulation,” in Advances in applied self- organizing systems. Springer, 2013, pp. 45–55

    2013

  3. [11]

    Intellilight: A reinforcement learning approach for intelligent traffic light control,

    H. Wei, G. Zheng, H. Yao, and Z. Li, “Intellilight: A reinforcement learning approach for intelligent traffic light control,” in Proceed- ings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. ACM, 2018, pp. 2496–2505

    2018

  4. [12]

    An experimental review of reinforcement learning algorithms for adaptive traffic signal control,

    P . Mannion, J. Duggan, and E. Howley, “An experimental review of reinforcement learning algorithms for adaptive traffic signal control,” in Autonomic Road Transport Support Systems . Springer, 2016, pp. 47–66

    2016

  5. [13]

    Em- bed to control: A locally linear latent dynamics model for control from raw images,

    M. Watter, J. Springenberg, J. Boedecker, and M. Riedmiller, “Em- bed to control: A locally linear latent dynamics model for control from raw images,” in Advances in neural information processing systems, 2015, pp. 2746–2754

    2015

  6. [14]

    A comprehensive survey of multiagent reinforcement learning,

    L. Busoniu, R. Babuska, and B. De Schutter, “A comprehensive survey of multiagent reinforcement learning,” IEEE Transactions on Systems, Man, And Cybernetics-Part C: Applications and Reviews, 38 (2), 2008, 2008

    2008

  7. [15]

    Multiagent reinforcement learning for urban traffic control using coordination graphs,

    L. Kuyer, S. Whiteson, B. Bakker, and N. Vlassis, “Multiagent reinforcement learning for urban traffic control using coordination graphs,” in Joint European Conference on Machine Learning and Knowledge Discovery in Databases. Springer, 2008, pp. 656–671

    2008

  8. [16]

    Graph attention networks,

    P . Velickovic, G. Cucurull, A. Casanova, A. Romero, P . Li `o, and Y. Bengio, “Graph attention networks,” in 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Conference Track Proceedings . OpenReview.net, 2018...

    2018

  9. [17]

    Multiagent reinforcement learning for integrated network of adaptive traffic signal controllers (MARLIN-ATSC): methodology and large-scale application on downtown toronto,

    S. El-Tantawy, B. Abdulhai, and H. Abdelgawad, “Multiagent reinforcement learning for integrated network of adaptive traffic signal controllers (MARLIN-ATSC): methodology and large-scale application on downtown toronto,” IEEE Transactions on Intelligent Transportation Systems, ...

    2013

  10. [18]

    Adaptive multi-objective reinforce- ment learning with hybrid exploration for traffic signal control based on cooperative multi-agent framework,

    M. A. Khamis and W. Gomaa, “Adaptive multi-objective reinforce- ment learning with hybrid exploration for traffic signal control based on cooperative multi-agent framework,” Engineering Appli- cations of Artificial Intelligence, vol. 29, pp. 134–151, 2014

    2014

  11. [19]

    Reinforcement learning for true adaptive traffic signal control,

    B. Abdulhai, R. Pringle, and G. J. Karakoulas, “Reinforcement learning for true adaptive traffic signal control,” Journal of Trans- portation Engineering, vol. 129, pp. 278–285, 2003

    2003

  12. [20]

    An agent-based learning towards decentralized and coordinated traffic signal control,

    S. El-Tantawy and B. Abdulhai, “An agent-based learning towards decentralized and coordinated traffic signal control,” in 13th Inter- national IEEE Conference on Intelligent Transportation Systems. IEEE, 2010, pp. 665–670

    2010

  13. [21]

    Reinforcement learning of traffic light controllers adapting to traffic congestion

    M. Steingrover, R. Schouten, S. Peelen, E. Nijhuis, B. Bakker et al., “Reinforcement learning of traffic light controllers adapting to traffic congestion.” in BNAIC. Citeseer, 2005, pp. 216–223. 14

    2005

  14. [22]

    Reinforcement learning- based multi-agent system for network traffic signal control,

    I. Arel, C. Liu, T. Urbanik, and A. Kohls, “Reinforcement learning- based multi-agent system for network traffic signal control,” IET Intelligent Transport Systems, vol. 4, pp. 128–135, 2010

    2010

  15. [23]

    Time critic policy gradi- ent methods for traffic signal control in complex and congested scenarios,

    S. G. Rizzo, G. Vantini, and S. Chawla, “Time critic policy gradi- ent methods for traffic signal control in complex and congested scenarios,” in Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining . ACM, 2019, pp. 1654–1664

    2019

  16. [24]

    Coordinated deep reinforce- ment learners for traffic light control,

    E. Van der Pol and F. A. Oliehoek, “Coordinated deep reinforce- ment learners for traffic light control,” in NIPS’16 Workshop on Learning, Inference and Control of Multi-Agent Systems , Dec. 2016

    2016

  17. [25]

    Collaborative multiagent reinforcement learning by payoff propagation,

    J. R. Kok and N. Vlassis, “Collaborative multiagent reinforcement learning by payoff propagation,” Journal of Machine Learning Re- search, vol. 7, pp. 1789–1828, 2006

    2006

  18. [26]

    Traffic light control by multiagent reinforcement learning systems,

    B. Bakker, S. Whiteson, L. Kester, and F. C. Groen, “Traffic light control by multiagent reinforcement learning systems,” in Interac- tive Collaborative Information Systems. Springer, 2010, pp. 475–510

    2010

  19. [27]

    Multi-agent reinforcement learning for traffic light control,

    M. Wiering, “Multi-agent reinforcement learning for traffic light control,” in Machine Learning: Proceedings of the Seventeenth Interna- tional Conference (ICML’2000), 2000, pp. 1151–1158

    2000

  20. [28]

    Multi-agent deep re- inforcement learning for large-scale traffic signal control,

    T. Chu, J. Wang, L. Codec `a, and Z. Li, “Multi-agent deep re- inforcement learning for large-scale traffic signal control,” IEEE Transactions on Intelligent Transportation Systems, vol. 21, pp. 1086– 1095, 2019

    2019

  21. [29]

    Presslight: Learning max pressure control to coordinate traf- fic signals in arterial network,

    H. Wei, C. Chen, G. Zheng, K. Wu, V . Gayah, K. Xu, and Z. Li, “Presslight: Learning max pressure control to coordinate traf- fic signals in arterial network,” in Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, 2019, pp. 1290–1298

    2019

  22. [30]

    The max-pressure controller for arbitrary networks of signalized intersections,

    P . Varaiya, “The max-pressure controller for arbitrary networks of signalized intersections,” in Advances in Dynamic Network Modeling in Complex Transportation Systems. Springer, 2013, pp. 27–66

    2013

  23. [31]

    Traffic signal control based on reinforcement learning with graph convolutional neural nets,

    T. Nishi, K. Otaki, K. Hayakawa, and T. Yoshimura, “Traffic signal control based on reinforcement learning with graph convolutional neural nets,” in 2018 21st International Conference on Intelligent Transportation Systems (ITSC). IEEE, 2018, pp. 877–883

    2018

  24. [32]

    Colight: Learning network-level coop- eration for traffic signal control,

    H. Wei, N. Xu, H. Zhang, G. Zheng, X. Zang, C. Chen, W. Zhang, Y. Zhu, K. Xu, and Z. Li, “Colight: Learning network-level coop- eration for traffic signal control,” in Proceedings of the 28th ACM International Conference on Information and Knowledge Management , 2019, pp. 1913–1922

    2019

  25. [33]

    A distributed virtual traf- fic light algorithm exploiting short range V2V communications,

    A. Bazzi, A. Zanella, and B. M. Masini, “A distributed virtual traf- fic light algorithm exploiting short range V2V communications,” Ad Hoc Networks, vol. 49, pp. 42–57, 2016

    2016

  26. [34]

    On the impact of virtual traffic lights on carbon emissions mitigation,

    M. Ferreira and P . M. d’Orey, “On the impact of virtual traffic lights on carbon emissions mitigation,” IEEE Transactions on Intelligent Transportation Systems, vol. 13, pp. 284–295, 2011

    2011

  27. [35]

    Cooperative multi- agent control using deep reinforcement learning,

    J. K. Gupta, M. Egorov, and M. Kochenderfer, “Cooperative multi- agent control using deep reinforcement learning,” in International Conference on Autonomous Agents and Multiagent Systems. Springer, 2017, pp. 66–83

    2017

  28. [36]

    Cooperative multi-agent learning: The state of the art,

    L. Panait and S. Luke, “Cooperative multi-agent learning: The state of the art,” Autonomous agents and multi-agent systems, vol. 11, pp. 387–434, 2005

    2005

  29. [37]

    Learning to communicate with deep multi-agent reinforcement learning,

    J. Foerster, I. A. Assael, N. de Freitas, and S. Whiteson, “Learning to communicate with deep multi-agent reinforcement learning,” in Advances in Neural Information Processing Systems , 2016, pp. 2137– 2145

    2016

  30. [38]

    Learning multiagent communi- cation with backpropagation,

    S. Sukhbaatar, R. Fergus et al. , “Learning multiagent communi- cation with backpropagation,” in Advances in Neural Information Processing Systems, 2016, pp. 2244–2252

    2016

  31. [39]

    Soft information for localization-of-things,

    A. Conti, S. Mazuelas, S. Bartoletti, W. C. Lindsey, and M. Z. Win, “Soft information for localization-of-things,” Proceedings of the IEEE, vol. 107, pp. 2240–2264, 2019

    2019

  32. [40]

    Network operation strategies for efficient localization and navigation,

    M. Z. Win, W. Dai, Y. Shen, G. Chrisikos, and H. V . Poor, “Network operation strategies for efficient localization and navigation,” Pro- ceedings of the IEEE, vol. 106, pp. 1224–1254, 2018

    2018

  33. [41]

    Decen- tralized gaussian filters for cooperative self-localization and multi- target tracking,

    P . Sharma, A.-A. Saucan, D. J. Bucci, and P . K. Varshney, “Decen- tralized gaussian filters for cooperative self-localization and multi- target tracking,” IEEE Transactions on Signal Processing, vol. 67, pp. 5896–5911, 2019

    2019

  34. [42]

    Experiments with cooperative control of underwater robots,

    M. Dunbabin, P . Corke, I. Vasilescu, and D. Rus, “Experiments with cooperative control of underwater robots,” The International Journal of Robotics Research, vol. 28, pp. 815–833, 2009

    2009

  35. [43]

    A multiple-goal reinforcement learning method for complex vehicle overtaking maneuvers,

    D. C. K. Ngai and N. H. C. Yung, “A multiple-goal reinforcement learning method for complex vehicle overtaking maneuvers,” IEEE Transactions on Intelligent Transportation Systems , vol. 12, pp. 509–522, 2011

    2011

  36. [44]

    Playing atari with deep reinforce- ment learning,

    V . Mnih, K. Kavukcuoglu, D. Silver, A. Graves, I. Antonoglou, D. Wierstra, and M. Riedmiller, “Playing atari with deep reinforce- ment learning,” in NIPS Deep Learning Workshop, 2013

    2013

  37. [45]

    Multiagent cooperation and competition with deep reinforcement learning,

    A. Tampuu, T. Matiisen, D. Kodelja, I. Kuzovkin, K. Korjus, J. Aru, J. Aru, and R. Vicente, “Multiagent cooperation and competition with deep reinforcement learning,” PloS one, vol. 12, p. e0172395, 2017

    2017

  38. [46]

    Empiri- cally evaluating multiagent learning algorithms,

    E. Zawadzki, A. Lipson, and K. Leyton-Brown, “Empiri- cally evaluating multiagent learning algorithms,” arXiv preprint arXiv:1401.8074, 2014

    2014 arXiv

  39. [47]

    Multi-agent actor-critic for mixed cooperative-competitive envi- ronments,

    R. Lowe, Y. Wu, A. Tamar, J. Harb, O. P . Abbeel, and I. Mordatch, “Multi-agent actor-critic for mixed cooperative-competitive envi- ronments,” in Advances in Neural Information Processing Systems , 2017, pp. 6379–6390

    2017

  40. [48]

    Graph convolutional reinforcement learning,

    J. Jiang, C. Dun, T. Huang, and Z. Lu, “Graph convolutional reinforcement learning,” in 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020 . OpenReview.net, 2020. [Online]. Available: https://openreview.net/forum?id=HkxdQkSYDB

    2020

  41. [49]

    The graph neural network model,

    F. Scarselli, M. Gori, A. C. Tsoi, M. Hagenbuchner, and G. Mon- fardini, “The graph neural network model,” IEEE Transactions on Neural Networks, vol. 20, pp. 61–80, 2009

    2009

  42. [50]

    Relational inductive biases, deep learning, and graph networks,

    P . W. Battaglia, J. B. Hamrick, V . Bapst, A. Sanchez-Gonzalez, V . Zambaldi, M. Malinowski, A. Tacchetti, D. Raposo, A. Santoro, R. Faulkner et al., “Relational inductive biases, deep learning, and graph networks,” arXiv preprint arXiv:1806.01261, 2018

    2018 arXiv

  43. [51]

    Interaction networks for learning about objects, relations and physics,

    P . Battaglia, R. Pascanu, M. Lai, D. J. Rezende et al., “Interaction networks for learning about objects, relations and physics,” in Advances in neural information processing systems , 2016, pp. 4502– 4510

    2016

  44. [52]

    Gated graph sequence neural networks,

    Y. Li, D. Tarlow, M. Brockschmidt, and R. S. Zemel, “Gated graph sequence neural networks,” in 4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings , 2016. [Online]. Available: http://arxiv.org/...

    2016 arXiv

  45. [53]

    Abductive learning: towards bridging machine learning and logical reasoning,

    Z. Zhi-Hua, “Abductive learning: towards bridging machine learning and logical reasoning,” Science China (Information Sci- ences), no. 7, p. 21, 2019

    2019

  46. [54]

    Neural message passing for quantum chemistry,

    J. Gilmer, S. S. Schoenholz, P . F. Riley, O. Vinyals, and G. E. Dahl, “Neural message passing for quantum chemistry,” in Proceedings of the 34th International Conference on Machine Learning, ICML, 2017, pp. 1263–1272

    2017

  47. [55]

    Nervenet: Learning structured policy with graph neural networks,

    T. Wang, R. Liao, J. Ba, and S. Fidler, “Nervenet: Learning structured policy with graph neural networks,” in 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Conference Track Proceedings . OpenReview.net, 2018...

    2018

  48. [56]

    Deep reinforcement learning with relational inductive biases,

    V . F. Zambaldi, D. Raposo, A. Santoro, V . Bapst, Y. Li, I. Babuschkin, K. Tuyls, D. P . Reichert, T. P . Lillicrap, E. Lockhart, M. Shanahan, V . Langston, R. Pascanu, M. Botvinick, O. Vinyals, and P . W. Battaglia, “Deep reinforcement learning with relational inductive bias...

    2019

  49. [57]

    Simula- tion and optimization of traffic in a city,

    M. Wiering, J. Vreeken, J. Van Veenen, and A. Koopman, “Simula- tion and optimization of traffic in a city,” inIEEE Intelligent Vehicles Symposium, 2004. IEEE, 2004, pp. 453–458

    2004

  50. [58]

    Long short-term memory,

    S. Hochreiter and J. Schmidhuber, “Long short-term memory,” Neural computation, vol. 9, pp. 1735–1780, 1997

    1997

  51. [59]

    Multi-agent reinforcement learning: Independent vs. co- operative agents,

    M. Tan, “Multi-agent reinforcement learning: Independent vs. co- operative agents,” in Proceedings of the tenth international conference on machine learning, 1993, pp. 330–337

    1993

  52. [60]

    Empirical evaluation of rectified activations in convolutional network,

    B. Xu, N. Wang, T. Chen, and M. Li, “Empirical evaluation of rectified activations in convolutional network,” arXiv preprint arXiv:1505.00853, 2015

    2015 arXiv

  53. [61]

    Attention is all you need,

    A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” in Advances in Neural Information Processing Systems , 2017, pp. 5998– 6008

    2017

  54. [62]

    Fast and accurate deep network learning by exponential linear units (elus),

    D. Clevert, T. Unterthiner, and S. Hochreiter, “Fast and accurate deep network learning by exponential linear units (elus),” in 4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings , 15 Y. Bengi...

    2016 arXiv

  55. [63]

    Deep recurrent q-learning for partially observable mdps,

    M. J. Hausknecht and P . Stone, “Deep recurrent q-learning for partially observable mdps,” in 2015 AAAI Fall Symposia, Arlington, Virginia, USA, November 12-14, 2015. AAAI Press, 2015, pp. 29–37

    2015

  56. [64]

    Memory- based control with recurrent neural networks,

    N. Heess, J. J. Hunt, T. P . Lillicrap, and D. Silver, “Memory- based control with recurrent neural networks,” in NIPS Deep Reinforcement Learning Workshop, 2015

    2015

  57. [65]

    Human-level control through deep reinforcement learning,

    V . Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. Riedmiller, A. K. Fidjeland, G. Ostrovski et al., “Human-level control through deep reinforcement learning,” Nature, vol. 518, p. 529, 2015

    2015

  58. [66]

    Cityflow: A multi-agent reinforcement learning environment for large scale city traffic scenario,

    H. Zhang, S. Feng, C. Liu, Y. Ding, Y. Zhu, Z. Zhou, W. Zhang, Y. Yu, H. Jin, and Z. Li, “Cityflow: A multi-agent reinforcement learning environment for large scale city traffic scenario,” in The World Wide Web Conference. ACM, 2019, pp. 3620–3624

    2019

  59. [67]

    Delving deep into rectifiers: Surpassing human-level performance on imagenet classification,

    K. He, X. Zhang, S. Ren, and J. Sun, “Delving deep into rectifiers: Surpassing human-level performance on imagenet classification,” in Proceedings of the IEEE international conference on computer vision , 2015, pp. 1026–1034

    2015

  60. [68]

    Adam: A method for stochastic optimization,

    D. P . Kingma and J. Ba, “Adam: A method for stochastic optimization,” in 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings , 2015. [Online]. Available: http://arxiv.org/abs/1412.6980

    2015 arXiv

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.