Large-scale traffic signal control using machine learning: some traffic flow considerations

Hao Zhou; Jorge A. Laval

REVIEW 4 major objections 5 minor 41 references

Large-scale traffic signal control using machine learning: some traffic flow considerations

T0 review · 4 major / 5 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Traffic-signal DRL fails to learn above 75% network occupancy because signal choice stops affecting throughput.

desk verdict A clean empirical demonstration that DRL signal control fails under congested training data, with a heuristic explanation that is more speculative than the data. read the letter →

arxiv 1908.02673 v1 pith:4GSHIAPL submitted 2019-08-07 cs.AI cs.SYeess.SY

Jorge A. Laval , Hao Zhou This is my paper · ORCID

classification cs.AIcs.SYeess.SY

keywords trafficsignalcontroldeepreinforcementlearningmacroscopicfundamentaldiagramcellularautomatonrule184congestednetworkpropertypolicygradientsuperviseddensity

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks why deep reinforcement learning (DRL) methods for traffic signal control have struggled in large networks, and answers with a traffic-flow property rather than an algorithmic weakness. Using a boundary-free grid on a torus and a cellular-automaton traffic model, it shows that a policy trained only under free-flow conditions is near-optimal at every density, while training under congestion degrades the policy and training above about 75% occupancy produces policies that fail everywhere. The cause is a 'congested network property': under heavy congestion, even a deliberately bad signal policy produces more throughput than the downstream links can absorb, so total network flow becomes independent of signal choice and the learning gradient carries no useful information. A supervised policy trained on just two extreme examples outperforms the standard longest-queue-first benchmark, and random search already finds competitive policies. The practical conclusion is that DRL for signal control should train on free-flow data only and discard congested observations.

What carries the argument

The load-bearing object is the congested network property, formalized as a nonzero lower bound on the MFD in congestion. For a 'bad' policy that always serves the shortest queue, the number of vehicles on each pair of approaches is modeled as independent binomial random variables with parameters proportional to the approach length and the prevailing density; the intersection outflow is then a scaled version of the minimum of these two variables. The percentile curves of this minimum give the MFD lower bound, which rises with density and meets the upper bound (from the method of cuts) near 75% occupancy. This collision is the mechanism: it makes network throughput policy-independent in congestion, which strips the reward signal that a policy-gradient trainer needs.

What would settle it

Use the same torus-grid simulator but replace the Bernoulli initialization with platoon or bursty arrivals: if a deliberately bad policy's throughput at 75-85% occupancy drops below the LQF benchmark, or if DRL trained under that congestion learns a policy that beats LQF, then the congested network property is refuted. A simpler observational check is to measure the MFD lower bound directly with probe vehicles on a real grid and see whether it rises toward the upper bound above 75% occupancy.

Watch

Extended reading notes

Core claim

The paper's central claim is that the effectiveness of a learning-based signal controller is governed by the network's macroscopic fundamental diagram (MFD)—the plot of average flow versus average density—and specifically by the gap between its upper and lower bounds. In free flow the lower bound is zero and the upper bound leaves room for policy choice, so gradient information is meaningful. As density rises, the lower bound rises too: because queues are nearly always present, whichever approach receives green can discharge at high rate, and the intersection throughput exceeds downstream capacity. The MFD's upper and lower bounds collide around 75% density, and beyond that point signal policy has no measurable effect on network throughput. The paper establishes this by deriving the lower bound as the scaled percentile of the minimum of two binomial variables, one for each approach pair, under a Bernoulli occupancy assumption, and by using the method of cuts for the upper bound. It then shows empirically that DRL training under such conditions cannot recover sensible policies, even when initialized from a good supervised policy, while supervised learning from two extreme states succeeds.

Load-bearing premise

The lower-bound derivation assumes that, at any time, each cell is occupied independently with probability equal to the average density, so the two approach queues are independent binomial counts; if real arrivals are correlated or queues interact across intersections, the nonzero lower bound—and with it the claim that congestion leaves nothing for DRL to learn—can fail.

Editorial extensions

If this is right

Current DRL signal-control methods should filter out congested training samples; doing so should improve their performance at all traffic densities.
A free-flow-only training regime yields controllers that remain near-optimal in congestion, so congestion data is not needed for good control.
Supervised learning with only two extreme-state examples beats the longest-queue-first greedy benchmark, suggesting that expert-designed extreme states can bootstrap control.
Random search, not just deep learning, can find competitive signal policies in these networks, so the policy landscape is smooth enough for simple search.
REINFORCE-TD provides a practical way to apply policy-gradient training directly to the continuing (infinite-horizon) signal-control problem without episodic resets.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

If the congested network property transfers to real street networks, average network occupancy becomes a simple predictor for when adaptive signal control stops mattering; controllers could monitor the MFD bound gap and switch to fixed or offset-based timing beyond a threshold.
The paper's binomial-min lower bound assumes spatially independent arrivals; a testable extension is to run the same experiments with correlated platoon arrivals, which should either widen the learning window or invalidate the lower bound.
The supervised two-example result suggests a curriculum: train a policy from extreme states in free flow, freeze it, and only fine-tune on data from densities below the critical value, avoiding the degenerate gradient region.
The conjecture that optimal green time is proportional to approach length could be turned into a direct design rule and validated by comparing the MFD of proportional-timing controllers against learned policies on heterogeneous grids.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The paper is worth a look for two results: the density threshold around 0.75 where policies become throughput-independent, and the striking finding that supervised learning with just two extreme examples outperforms longest-queue-first. The empirical core is honest and fairly careful for an idealized CA grid on a torus. The authors also make a reasonable extension of REINFORCE to continuing problems, though they note other methods behave similarly.

What is genuinely new: nobody seems to have trained DRL policies under controlled density levels and shown that training occupancy is the dominant factor. The deterioration above 0.75 is directly observed, and the two-example supervised result is a useful practical insight, even if its mechanism is unexplained.

Soft spots, in proportion. First, "optimal" means "outperforms LQF." That is a weak benchmark, and the paper says so, but it is still worth flagging. Second, confidence intervals are shown for the MFDs but not for the learning curves, so some of the claims about convergence rest on visual inspection. Third, the explanatory "congested network property" is a conjecture. The Bernoulli/binomial derivation in the Discussion is explicitly heuristic, and the stress-test note is right: rule 184 creates strong correlations at high density, and the derivation assumes outgoing approaches do not block traffic, which is exactly what fails under spillback. So the lower bound shape is unverified, and the practical advice to discard all congested data from all DRL methods goes beyond the evidence. The authors are candid that this is a conjecture, but the abstract and outlook lean on it more than the derivation supports.

The empirical failure of DRL under congested training stands on its own, independent of the mechanism. I would not accept the general "nothing to learn" conclusion without a verified bound, and the recommendation to discard congested data should be limited to the conditions tested here. The paper also provides no code or data, which is a real limitation for reproducibility.

Who is this for? Researchers in traffic signal control using RL, and people studying why policy-gradient methods fail in congested, continuing environments. It deserves a serious referee: the density threshold result could affect a large body of work, and the empirical setup is clear enough to reproduce or refute. I would send it to review, but ask the authors either to verify the lower bound mechanism or to label it more explicitly as a plausible explanation, and to release the simulator code or at least detailed experiment parameters.

For your reading group: maybe. It is not deep theory, but it is a clean counterexample to the common habit of training DRL on whatever traffic data is available.

Referee Report

4 major / 5 minor

Summary. The paper studies steady-state signal control in a homogeneous toroidal grid network using Cellular Automaton rule 184. It compares supervised learning, random search, and a proposed continuing-task policy-gradient algorithm (REINFORCE-TD) against the longest-queue-first (LQF) benchmark. The main empirical findings are that a supervised policy trained on only two extreme examples outperforms LQF, random search can find competitive policies, and the occupancy level used during training is decisive: DRL policies trained at occupancies above about 75% perform very poorly and appear unable to learn. The paper proposes a 'congested network property'—a nonzero lower bound on intersection throughput that becomes policy-independent under congestion—as the explanation, and recommends that DRL practitioners discard congested training data. The theoretical explanation and the broad practical advice are the main points in need of scrutiny.

Significance. If the central mechanism were established, the paper would make an important contribution by identifying training occupancy as a decisive factor for DRL traffic-signal control on large networks. The manuscript has clear strengths: the simulation protocol is described in enough detail to be reproduced, the REINFORCE-TD algorithm is explicitly specified, and MFD comparisons are reported with confidence intervals. The supervised-learning result with two examples is striking and well supported within the paper's setting. However, the significance of the paper currently rests on two unsupported extrapolations: the heuristic derivation of the congested network property, which is not verified against the actual CA dynamics, and the generalization of the empirical failure to all DRL methods. The paper offers a falsifiable prediction about the shape of MFD lower bounds, but that prediction is not yet validated; the derivation also does not establish a true lower bound for all policies.

major comments (4)

[§5 (Discussion), 'congested network property' derivation and Fig. 10] The nonzero lower bound that drives the 'congested network property' is derived under an unverified stochastic assumption: cell occupancies are modeled as Bernoulli(k) and the queue lengths on NS and EW approaches as independent binomials. CA rule 184 (Eq. 5) is conservative and deterministic, and at k>0.75 stationary configurations contain strongly correlated platoons and holes, so the independence assumption is least plausible in the regime where it is used. The derivation also assumes 'provided outgoing approaches do not block traffic,' which fails under congestion because downstream spillback limits discharge. Finally, the percentile-of-the-min-of-two-binomials curve is not a lower bound for all policies; it is a quantile of one heuristic policy's throughput. Without verification against the actual CA dynamics or a rigorous bound, the 'nothing to learn' mechanism is unsupported.
[Abstract and §4.3.1 ('Constant demand')] The abstract claims that training under free-flow conditions yields DRL policies that are 'optimal for all traffic conditions', but §4.3.1 describes the best free-flow-trained policies as 'albeit only competitive', i.e. only overlapping LQF. Under the paper's own terminology in §4, 'optimal' means strictly outperforming LQF. These statements need to be reconciled, with the abstract's claim either weakened or supported by the relevant MFD comparisons.
[§4.3, Figs. 6-8] The central empirical claim that DRL cannot learn for training occupancies above about 75% is supported by single learning trajectories for π(s1) and π(s2); no confidence intervals or multiple-seed summaries are given for these curves, unlike the MFD evaluations. Since this claim motivates the practical recommendation to discard congested data, repeated-seed results and a quantitative definition of 'failure to learn' should be provided.
[§5, 'In the meantime' paragraph] The recommendation to discard all congested training data for current DRL methods goes beyond the evidence. The experiments use one policy class (an MLP with REINFORCE-TD), one network topology (torus with one lane per direction and constant demand), and one benchmark. The statement that 'all the DRL methods proposed in the literature to date are unable to learn' is a conjecture, not a demonstrated result; the advice should be scoped to the tested setting.

minor comments (5)

[§3, Eq. (5)] The Boolean update formula is printed as 'ci := ci−1∨ ci−1∧ ci∨ ci∧ ci+1', which appears to omit operators or parentheses; please correct and verify that it matches rule 184.
[§5, Eq. (9)] The quantities λ, ρ, and δ are not defined in the manuscript; since the consistency check w≈2/3 and λ≈1 depends on equation (17b) of [17], please reproduce the needed definitions or equation.
[§5] There is a stray phrase 'busy working so' after Eq. (9) and an ungrammatical phrase 'which is deterministic in which explains' in the paragraph about extreme congestion; the manuscript needs proofreading.
[§5 / Fig. 10] The right panel of Fig. 10 appears to be unused and unlabeled in the text; either make it part of the argument or remove it.
[§4.3.1 and §5] Several sentences state that results were verified for other cell lengths, green times, network sizes, and training algorithms, but no supporting data or appendix is provided; please document these checks or soften the statements.

Circularity Check

0 steps flagged · score 2.0 of 10

No significant circularity: the DRL-failure finding is directly observed, and the congested-network explanation is an independent heuristic bound rather than a constructed prediction.

full rationale

The paper's central claim—that DRL policies trained under free-flow are nearly optimal for all densities, while training under congestion degrades learning—is an empirical result from CA rule 184 simulations. It is not derived by fitting a parameter to a subset of data and then renaming that fit a prediction. The formal 'congested network property' lower bound is obtained from a stated probabilistic model: independent Bernoulli cell occupancies, with the flow under a smallest-queue-first policy being a function of the minimum of two independent binomials. This bound is offered as a heuristic explanation for the measured MFD behavior, not as the source of that behavior. The w≈2/3 and λ≈1 check is a post-hoc consistency read from the simulated MFDs; no equation is constructed to force that value, and it is not presented as an independent prediction. The self-citations ([33] on symmetry and the parameter-free kinematic wave model, [17] on MFD approximations) are background mathematical tools; they are not the load-bearing evidence for the DRL failure, which is observed directly in the paper's own simulator. A weakness of the lower-bound argument is that its 'provided outgoing approaches do not block traffic' assumption is questionable exactly in the congested regime, and Bernoulli independence is not the steady-state distribution of CA rule 184 at high density; however, this is a validity or correctness concern, not a circularity. No step identified reduces to its own input by definition or imports the conclusion through a self-citation chain.

Assumptions & free parameters 5 free parameters · 6 assumptions · 0 invented entities

The paper introduces no new physical entities; the 'congested network property' is an explanatory label, not a postulated entity. The main assumptions are the parameter-free CA model, the homogeneous torus with shared policy, and the Bernoulli process used in the lower bound derivation.

free parameters (5)

learning rate alpha = 0.2
Chosen via grid search, not derived.
learning rate beta = 0.05
Chosen via grid search, not derived.
segment length n = 5 cells
Fixed arbitrarily to 5 cells between signals; results claimed robust to this choice.
minimum green time g = 3 time steps
Fixed to 3 time steps; results claimed robust to this choice.
critical density threshold = 0.75
Observed empirically as the point where MFD bounds collapse; used to define 'extreme congestion'.

assumptions (6)

domain assumption The kinematic wave model with a triangular fundamental diagram is parameter-free due to symmetry.
Invoked in Problem Setup to justify using CA rule 184; relies on Laval and Chilukuri (2016).
domain assumption CA rule 184 implements the kinematic wave model.
Used as the simulation model; assumed exact.
domain assumption The network is homogeneous and boundary-free, so a single shared policy is optimal for all intersections.
Used to train one agent and share parameters across all intersections.
domain assumption Vehicle routing with equal probabilities for left, right, straight leads to uniform density distribution.
Used to justify the torus setup.
ad hoc to paper Bernoulli process assumption for vehicle distribution in lower bound derivation.
Introduced in Discussion to formalize the 'congested network property'; not derived from the CA model.
ad hoc to paper The lower bound derived from the minimum of two binomials approximates the MFD lower bound.
Used to explain the property; not rigorously justified.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Large-scale traffic signal control using machine learning: some traffic flow considerations." pith.science (2026). https://pith.science/paper/4GSHIAPL

@misc{pith2026190802673,
  author       = {Pith},
  title        = {Pith review of: Large-scale traffic signal control using machine learning: some traffic flow considerations},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/4GSHIAPL}},
  note         = {Machine review of arXiv:1908.02673}
}

read the original abstract

This paper uses supervised learning, random search and deep reinforcement learning (DRL) methods to control large signalized intersection networks. The traffic model is Cellular Automaton rule 184, which has been shown to be a parameter-free representation of traffic flow, and is the most efficient implementation of the Kinematic Wave model with triangular fundamental diagram. We are interested in the steady-state performance of the system, both spatially and temporally: we consider a homogeneous grid network inscribed on a torus, which makes the network boundary-free, and drivers choose random routes. As a benchmark we use the longest-queue-first (LQF) greedy algorithm. We find that: (i) a policy trained with supervised learning with only two examples outperforms LQF, (ii) random search is able to generate near-optimal policies, (iii) the prevailing average network occupancy during training is the major determinant of the effectiveness of DRL policies. When trained under free-flow conditions one obtains DRL policies that are optimal for all traffic conditions, but this performance deteriorates as the occupancy during training increases. For occupancies > 75% during training, DRL policies perform very poorly for all traffic conditions, which means that DRL methods cannot learn under highly congested conditions. We conjecture that DRL's inability to learn under congestion might be explained by a property of urban networks found here, whereby even a very bad policy produces an intersection throughput higher than downstream capacity. This means that the actual throughput tends to be independent of the policy. Our findings imply that it is advisable for current DRL methods in the literature to discard any congested data when training, and that doing this will improve their performance under all traffic conditions.

Figures

Figures reproduced from arXiv: 1908.02673 by the authors.

**Figure 4.** Because our network is spatially homogeneous and without boundaries, there is no reason why policies should be different across agents, and therefore we will train a single agent and share its parameters with all other agents. After training, we evaluate the performance of the policy by observing the resulting MFD. The training algorithm REINFORCE-TD In this paper we propose the training algorithm REINFORCE-TD, whic… view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

41 extracted references · 38 canonical work pages

[1]

Arel, I., C. Liu, T. Urbanik, and A. Kohls, Reinforcement learning-based multi-agent system for network trafﬁc signal control. IET Intelligent Transport Systems, V ol. 4, No. 2, 2010, pp. 128–135

work page 2010
[2]

Kavukcuoglu, D

Mnih, V ., K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. Riedmiller, A. K. Fidjeland, G. Ostrovski, et al., Human-level control through deep rein- forcement learning. Nature, V ol. 518, No. 7540, 2015, p. 529

work page 2015
[3]

Schrittwieser, K

Silver, D., J. Schrittwieser, K. Simonyan, I. Antonoglou, A. Huang, A. Guez, T. Hubert, L. Baker, M. Lai, A. Bolton, et al., Mastering the game of go without human knowledge. Nature, V ol. 550, No. 7676, 2017, p. 354

work page 2017
[4]

Model-free Deep Reinforcement Learning for Urban Autonomous Driving

Chen, J., B. Yuan, and M. Tomizuka, Model-free Deep Reinforcement Learning for Urban Autonomous Driving. arXiv preprint arXiv:1904.09503, 2019

work page Pith review arXiv 1904
[5]

Lv, and F.-Y

Li, L., Y . Lv, and F.-Y . Wang, Trafﬁc signal timing via deep reinforcement learning. IEEE/CAA Journal of Automatica Sinica, V ol. 3, No. 3, 2016, pp. 247–254

work page 2016
[6]

Genders, W. and S. Razavi, Using a deep reinforcement learning agent for trafﬁc signal con- trol. arXiv preprint arXiv:1611.01142, 2016

arXiv 2016
[7]

Chu, T. and J. Wang, Trafﬁc signal control with macroscopic fundamental diagrams. In 2015 American Control Conference (ACC), IEEE, 2015, pp. 4380–4385

work page 2015
[8]

Chu, T., J. Wang, L. Codecà, and Z. Li, Multi-Agent Deep Reinforcement Learning for Large- Scale Trafﬁc Signal Control. IEEE Transactions on Intelligent Transportation Systems, 2019

work page 2019

Show all 41 references

[9]

Tan, T., F. Bao, Y . Deng, A. Jin, Q. Dai, and J. Wang, Cooperative deep reinforcement learn- ing for large-scale trafﬁc grid signal control. IEEE transactions on cybernetics, 2019

2019
[10]

Ge, H., Y . Song, C. Wu, J. Ren, and G. Tan, Cooperative Deep Q-Learning With Q-Value Transfer for Multi-Intersection Signal Control. IEEE Access, V ol. 7, 2019, pp. 40797–40809

2019
[11]

Technical Report NU-CCS-88-3, Northeastern University, 1988

Willianms, R., Toward a theory of reinforcement-learning connectionist systems. Technical Report NU-CCS-88-3, Northeastern University, 1988

1988
[12]

F., Urban gridlock: Macroscopic modeling and mitigation approaches

Daganzo, C. F., Urban gridlock: Macroscopic modeling and mitigation approaches. Trans- portation Research Part B: Methodological, V ol. 41, No. 1, 2007, pp. 49–62

2007
[13]

Geroliminis, N. and C. F. Daganzo, Existence of urban-scale macroscopic fundamental diagrams: Some experimental ﬁndings. Transportation Research Part B: Methodological , V ol. 42, No. 9, 2008, pp. 759–770

2008
[14]

Lighthill, M. J. and G. B. Whitham, On kinematic waves II. A theory of trafﬁc ﬂow on long crowded roads. Proceedings of the Royal Society of London. Series A. Mathematical and Physical Sciences, V ol. 229, No. 1178, 1955, pp. 317–345

1955
[15]

I., Shock waves on the highway

Richards, P. I., Shock waves on the highway. Operations research, V ol. 4, No. 1, 1956, pp. 42–51

1956
[16]

Daganzo, C. F. and N. Geroliminis, An analytical approximation for the macroscopic funda- mental diagram of urban trafﬁc. Transportation Research Part B: Methodological, V ol. 42, No. 9, 2008, pp. 771–781

2008
[17]

Laval, J. A. and F. Castrillón, Stochastic approximations for the macroscopic fundamental diagram of urban networks. Transportation Research Procedia, V ol. 7, 2015, pp. 615–630. Laval, and Zhou 17

2015
[18]

Journal of mathematics and mechanics , 1957, pp

Bellman, R., A Markovian decision process. Journal of mathematics and mechanics , 1957, pp. 679–684

1957
[19]

P., Dynamic Programming: Determinist

Bertsekas, D. P., Dynamic Programming: Determinist. and Stochast. Models . Prentice-Hall, 1987

1987
[20]

A., Dynamic programming and markov processes., 1960

Howard, R. A., Dynamic programming and markov processes., 1960

1960
[21]

Puterman, M., Markovian decision problems, 1994

1994
[22]

Overseas Development Institute London, 1999

Sutton, R., The policy process: an overview. Overseas Development Institute London, 1999

1999
[23]

Qu, and J

Chu, T., S. Qu, and J. Wang, Large-scale trafﬁc grid signal control with regional reinforce- ment learning. In 2016 American Control Conference (ACC), IEEE, 2016, pp. 815–820

2016
[24]

Lillicrap, T. P., J. J. Hunt, A. Pritzel, N. Heess, T. Erez, Y . Tassa, D. Silver, and D. Wier- stra, Continuous control with deep reinforcement learning.arXiv preprint arXiv:1509.02971, 2015

2015 arXiv
[25]

Khamis, M. A. and W. Gomaa, Adaptive multi-objective reinforcement learning with hybrid exploration for trafﬁc signal control based on cooperative multi-agent framework. Engineer- ing Applications of Artiﬁcial Intelligence, V ol. 29, 2014, pp. 134–151

2014
[26]

Wei, H., N. Xu, H. Zhang, G. Zheng, X. Zang, C. Chen, W. Zhang, Y . Zhu, K. Xu, and Z. Li, CoLight: Learning Network-level Cooperation for Trafﬁc Signal Control. arXiv preprint arXiv:1905.05717, 2019

1905 arXiv
[27]

Abdel-Aty, Q

Gong, Y ., M. Abdel-Aty, Q. Cai, and M. S. Rahman,A Decentralized Network Level Adaptive Signal Control Algorithm By Deep Reinforcement Learning, 2019

2019
[28]

Parvate, C

Kheterpal, N., K. Parvate, C. Wu, A. Kreidieh, E. Vinitsky, and A. Bayen, Flow: Deep reinforcement learning for control in sumo. SUMO, 2018, pp. 134–151

2018
[29]

Kreidieh, L

Vinitsky, E., A. Kreidieh, L. Le Flem, N. Kheterpal, K. Jang, F. Wu, R. Liaw, E. Liang, and A. M. Bayen, Benchmarks for reinforcement learning in mixed-autonomy trafﬁc. In Confer- ence on Robot Learning, 2018, pp. 399–409

2018
[30]

Xu, M., J. Wu, L. Huang, R. Zhou, T. Wang, and D. Hu, Network-wide trafﬁc signal control based on the discovery of critical nodes and deep reinforcement learning. Journal of Intelli- gent Transportation Systems, 2018, pp. 1–10

2018
[31]

Camponogara, E. and W. Kraus, Distributed learning agents in urban trafﬁc control. In Por- tuguese Conference on Artiﬁcial Intelligence, Springer, 2003, pp. 324–335

2003
[32]

Dai, Y ., J. Hu, D. Zhao, and F. Zhu, Neural network based online trafﬁc signal controller de- sign with reinforcement training. In 2011 14th International IEEE Conference on Intelligent Transportation Systems (ITSC), IEEE, 2011, pp. 1045–1050

2011
[33]

Laval, J. A. and B. R. Chilukuri, Symmetries in the kinematic wave model and a parameter- free representation of trafﬁc ﬂow. Transportation Research Part B: Methodological, V ol. 89, 2016, pp. 168 – 177

2016
[34]

Nature, V ol

Wolfram, S., Cellular automata as models of complexity. Nature, V ol. 311, No. 5985, 1984, p. 419

1984
[35]

Kurková, V ., Kolmogorov’s theorem and multilayer neural networks.Neural networks, V ol. 5, No. 3, 1992, pp. 501–506

1992
[36]

Sutton, R. S. and A. G. Barto, Reinforcement learning: An introduction. MIT press, 2018

2018
[37]

S., Learning to predict by the methods of temporal differences

Sutton, R. S., Learning to predict by the methods of temporal differences. Machine learning, V ol. 3, No. 1, 1988, pp. 9–44

1988
[38]

Guy, and B

Mania, H., A. Guy, and B. Recht, Simple random search provides a competitive approach to reinforcement learning. arXiv preprint arXiv:1803.07055, 2018. Laval, and Zhou 18

2018 arXiv
[39]

de Oliveira, D., A. L. Bazzan, B. C. da Silva, E. W. Basso, L. Nunes, R. Rossetti, E. de Oliveira, R. da Silva, and L. Lamb, Reinforcement Learning based Control of Traf- ﬁc Lights in Non-stationary Environments: A Case Study in a Microscopic Simulator. In EUMAS, 2006

2006
[40]

P., D.-Y

Choi, S. P., D.-Y . Yeung, and N. L. Zhang, Hidden-mode markov decision processes for nonstationary sequential decision making. In Sequence Learning, Springer, 2000, pp. 264– 287

2000
[41]

Da Silva, B. C., E. W. Basso, A. L. Bazzan, and P. M. Engel, Dealing with non-stationary environments using context detection. In Proceedings of the 23rd international conference on Machine learning, ACM, 2006, pp. 217–224

2006

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.

[1] [1]

Arel, I., C. Liu, T. Urbanik, and A. Kohls, Reinforcement learning-based multi-agent system for network trafﬁc signal control. IET Intelligent Transport Systems, V ol. 4, No. 2, 2010, pp. 128–135

work page 2010

[2] [2]

Kavukcuoglu, D

Mnih, V ., K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. Riedmiller, A. K. Fidjeland, G. Ostrovski, et al., Human-level control through deep rein- forcement learning. Nature, V ol. 518, No. 7540, 2015, p. 529

work page 2015

[3] [3]

Schrittwieser, K

Silver, D., J. Schrittwieser, K. Simonyan, I. Antonoglou, A. Huang, A. Guez, T. Hubert, L. Baker, M. Lai, A. Bolton, et al., Mastering the game of go without human knowledge. Nature, V ol. 550, No. 7676, 2017, p. 354

work page 2017

[4] [4]

Model-free Deep Reinforcement Learning for Urban Autonomous Driving

Chen, J., B. Yuan, and M. Tomizuka, Model-free Deep Reinforcement Learning for Urban Autonomous Driving. arXiv preprint arXiv:1904.09503, 2019

work page Pith review arXiv 1904

[5] [5]

Lv, and F.-Y

Li, L., Y . Lv, and F.-Y . Wang, Trafﬁc signal timing via deep reinforcement learning. IEEE/CAA Journal of Automatica Sinica, V ol. 3, No. 3, 2016, pp. 247–254

work page 2016

[6] [6]

Genders, W. and S. Razavi, Using a deep reinforcement learning agent for trafﬁc signal con- trol. arXiv preprint arXiv:1611.01142, 2016

arXiv 2016

[7] [7]

Chu, T. and J. Wang, Trafﬁc signal control with macroscopic fundamental diagrams. In 2015 American Control Conference (ACC), IEEE, 2015, pp. 4380–4385

work page 2015

[8] [8]

Chu, T., J. Wang, L. Codecà, and Z. Li, Multi-Agent Deep Reinforcement Learning for Large- Scale Trafﬁc Signal Control. IEEE Transactions on Intelligent Transportation Systems, 2019

work page 2019