Pith. sign in

REVIEW 4 major objections 5 minor 75 references

An Open-Source Framework for Adaptive Traffic Signal Control

T0 review · 4 major / 5 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read An open-source framework for adaptive traffic signal control reports that the Max-pressure algorithm, once hyperparameters are optimized, outperforms deep Q-network and deep deterministic policy gradient controllers on a two-intersection…

desk verdict A genuinely useful open-source framework for adaptive signal control, wrapped around a comparative experiment whose 'best controller' claim is undermined by in-sample hyperparameter tuning. read the letter →

arxiv 1909.00395 v1 pith:LO2UB3Z5 submitted 2019-09-01 eess.SY cs.AIcs.LGcs.SY

Wade Genders , Saiedeh Razavi This is my paper · ORCID
classification eess.SYcs.AIcs.LGcs.SY
keywords adaptivetrafficsignalcontrolMax-pressuredeepQ-networkdeterministicpolicygradienthyperparameteroptimizationSUMOself-organizinglightstraveltime
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper contributes an open-source framework that implements five adaptive traffic signal controllers (Webster's, Max-pressure, Self-Organizing Traffic Lights, deep Q-network, and deep deterministic policy gradient) for the SUMO traffic microsimulator, and demonstrates the framework by tuning and comparing the controllers on a two-intersection network. The central empirical claim is that, once every controller's hyperparameters are optimized, the Max-pressure heuristic achieves the lowest travel time, the lowest queue and delay, and the lowest variance of all tested methods, including the deep reinforcement learning controllers. The paper also claims that controllers with more hyperparameters, especially the learning-based ones, show much larger performance variance across hyperparameter settings, and that a poor hyperparameter choice can lead to erroneous conclusions. The broader contribution is a reproducible starting point intended to let researchers generate adaptive signal control results faster and with less effort.

What carries the argument

The framework combines the SUMO traffic microsimulator with a distributed acting, centralized learning architecture for the reinforcement learning controllers, and grid-search hyperparameter optimization for all controllers. The load-bearing identity is the Max-pressure rule, which for a candidate green phase $p$ computes the pressure as the sum of queue lengths on incoming lanes minus the sum on outgoing lanes, $\mathrm{Pressure}(p)=\sum_{l\in L_{p,\mathrm{inc}}}|V_l|-\sum_{l\in L_{p,\mathrm{out}}}|V_l|$, and switches to the phase with the largest pressure after a minimum green time. The other controllers include Webster's fixed-cycle timing computed from recent flow, SOTL's vehicle-time integral threshold with platoon protection, and deep neural network policies for DQN and DDPG. The comparison is carried out by optimizing each controller's hyperparameters with a grid search and then estimating performance from 32 seeded simulations.

What would settle it

Run the same framework with the optimized hyperparameters on a different network, such as a city grid or a corridor with oversaturated demand, using multiple random seeds; if a learning-based controller or Webster's produces a lower mean and median travel time than Max-pressure, the paper's central comparative claim fails.

Watch

Extended reading notes

Core claim

On the authors' terms, the paper establishes that in a simulated two-intersection network under a three-hour dynamic demand scenario, the Max-pressure controller is the best of the five adaptive controllers: it has the lowest mean and median travel time and the lowest standard deviation across 32 randomized simulations. DQN and DDPG perform about as well as Uniform and Webster's on average, but DQN exhibits significant outliers. The paper also establishes that hyperparameter choice dramatically changes controller performance, with learning-based controllers showing high variance, and that the DQN controller performs poorly at low demand but competitively at peak demand, a pattern the authors attribute to possible overfitting to high-reward periods.

Load-bearing premise

The ranking of controllers rests on a single synthetic two-intersection network with one three-hour demand scenario; if that network is not representative of other networks or real demand patterns, the comparative conclusions may not generalize.

Editorial extensions

If this is right

  • Researchers using the framework can obtain reproducible baseline comparisons without reimplementing controllers from scratch.
  • A fair comparison between traffic signal controllers requires optimizing each controller's hyperparameters rather than relying on default settings, because poor choices can drastically alter rankings.
  • The Max-pressure controller's low variance and strong average performance on the tested network make it a compelling candidate for deployment on small networks similar to the experiment.
  • Learning-based controllers, at least in this setup, do not yet outperform simple heuristics, so future work on function approximators or training algorithms is needed to close the gap.
  • The DQN's pattern of poor low-demand performance and strong peak-demand performance suggests a concrete direction for studying overfitting in reward-driven signal control.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the Max-pressure advantage holds across more networks, cities could adopt simpler, interpretable controllers rather than deep reinforcement learning, which would ease certification, maintenance, and explanation to the public.
  • Because pressure-based control has theoretical throughput guarantees in certain network models, combining this framework's benchmark methodology with that theory could sharpen predictions about which network structures favour Max-pressure.
  • A testable extension is to retrain the DQN with a reward that does not normalize by the peak-demand magnitude, to see whether the observed low-demand underperformance disappears.
  • The open-source nature of the framework allows other researchers to add new controllers to the same benchmark, turning the comparison into a living leaderboard rather than a one-off result.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper presents an open-source framework for adaptive traffic signal control in the SUMO microsimulator, with implementations of Uniform, Webster's, Max-pressure, SOTL, DQN, and DDPG controllers. The framework is demonstrated on a two-intersection network under a simulated three-hour dynamic demand scenario: Section IV-A performs a grid search over each controller's hyperparameters using eight random seeds, and Section IV-B compares the selected 'optimal' configurations over 32 additional simulations using travel time, queue, and delay measures. The authors report that Max-pressure achieves the lowest mean, median, and standard deviation of travel time, while learning-based controllers show higher hyperparameter sensitivity and larger variance. The manuscript emphasizes the framework's value as a reusable, parallelized starting point for adaptive signal control research.

Significance. The framework itself is a useful community resource: the code is open-source, integrates with SUMO, and provides a distributed actor-learner architecture that makes the included RL controllers tractable on modest hardware. The authors are explicit (Section V) that their results are not intended to establish the superiority of Max-pressure in all environments, which moderates the scope of the empirical claim. If the comparison is made fair and reproducible, the finding that a hyperparameter-sparse heuristic can outperform heavily-tuned deep RL controllers on a small network is a plausible and useful counterpoint to the prevailing emphasis on learning-based methods. The reported multi-seed boxplots and confidence intervals support internal comparisons, although formal significance testing is absent.

major comments (4)
  1. [Section IV-A and IV-B] The same two-intersection network and the same three-hour dynamic demand scenario are used for both hyperparameter selection (Section IV-A) and the final performance evaluation (Section IV-B). The controllers are therefore compared in-sample: the hyperparameter configurations are selected on the exact scenario used to rank the controllers, which asymmetrically favors low-variance methods such as Max-pressure over the more hyperparameter-rich DQN and DDPG. This undermines the Section IV-B claim that Max-pressure achieves the best performance, because the ranking may reflect the tuning protocol rather than the algorithms. A held-out evaluation on a different demand profile or network, or an explicit cross-validation scheme, is needed before the ranking is attributed to the controllers.
  2. [Section IV-A] The grid-search details are not reported: the manuscript does not state the number of hyperparameter configurations per controller, the ranges of the grid, or the criterion used to select the 'best' configuration (the Figure 3 caption mentions ordering by sum of mean and standard deviation, but the formal selection rule is not defined). For DQN and DDPG, the training hyperparameters (learning rate, batch size, target update interval, exploration schedule, action repeat, replay buffer size, and network size) are omitted entirely, although these are listed as free parameters in the framework. Without these details, a reader cannot determine whether the learning controllers were tuned as thoroughly as the non-learning controllers, which is load-bearing for the comparative conclusion.
  3. [Section IV-B] The performance differences among controllers are described qualitatively (e.g., 'approximately equal performance' for DQN, DDPG, Uniform, and Webster's) with boxplots and confidence intervals, but no significance tests or effect-size measures are reported. The claim that Max-pressure has the lowest mean, median, and standard deviation would be strengthened by a statistical test of paired or unpaired differences across the 32 seeds, and the 'approximately equal' grouping should be supported by a test showing that the observed differences are not statistically significant.
  4. [Section V] The conclusion states that 'the Max-pressure controller was found to achieve the best performance' and that 'learning-based controllers can be further developed to offer improved performance.' While the authors disclaim universal generalization, the conclusion presents the in-sample comparison as an established finding. The wording should be tempered to reflect that the result is specific to the tested network and demand scenario, and that the comparison was not performed on a held-out scenario.
minor comments (5)
  1. [Figure 3] The caption acknowledges that differing axis scales make direct visual comparison biased. Consider using a common normalized scale or subplots with identical axis limits so that hyperparameter sensitivity is visually comparable across controllers.
  2. [Section IV-A] The manuscript does not state how many hyperparameter configurations were tested for each controller, nor does it list the grid ranges. Adding a table of grid ranges and the number of configurations would improve reproducibility and would help readers assess whether the search was exhaustive enough.
  3. [Section III-A.3, Eq. (1)] The Max-pressure pressure definition uses the difference of total vehicle counts on incoming and outgoing lanes, whereas the original max-pressure controller uses turning-movement-specific pressures. The authors should state that this is a simplified version and justify its adequacy for the tested network.
  4. [Section II-C] The contribution statement says the framework 'was designed to scale to develop adaptive controllers for any SUMO network,' but the experiments only cover a two-intersection network. The claim would be better phrased as a design goal rather than an achieved property.
  5. [Figure 6] SOTL is omitted from the intersection-level queue and delay plots to improve readability. The text explains this, but it would be clearer to state that SOTL values are outside the ordinate range in the caption as well.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity found: the paper's comparative claims are empirical simulation results, not derivations that reduce to fitted inputs or self-citation chains.

full rationale

The paper's central contribution is an open-source SUMO-based framework and an empirical comparison of five adaptive traffic signal controllers. The claim that Max-pressure achieves the lowest mean, median, and standard deviation of travel time is an experimental measurement on a specific two-intersection network under a specific demand scenario, not a quantity determined by construction from the controllers' definitions. Hyperparameter optimization in Section IV-A is a grid-search procedure that selects configurations, and Section IV-B then evaluates those configurations; this raises an in-sample evaluation concern, but it is not circularity under the stated criteria because the reported performance measures are not analytically forced by the hyperparameter selection process. The DQN and DDPG controllers are related to the authors' prior work (refs 32, 38, 52), but those citations are background literature and implementation precursors, not load-bearing uniqueness theorems or ansatz smuggled in by citation; the algorithms themselves are standard deep reinforcement learning methods described in the paper and appendix. No fitted parameter is renamed as a prediction, and no equation is equivalent to its own input by definition. The authors also explicitly disclaim generalization beyond the tested environments in Section V, which further limits the reach of the claim. Accordingly, no specific circular step can be quoted, and the appropriate finding is no significant circularity.

Assumptions & free parameters 6 free parameters · 4 assumptions · 0 invented entities

The central empirical comparison depends on the hyperparameters chosen for each controller (not fully reported), the assumption that SUMO is a valid proxy, and the representativeness of the single test network. No new physical or conceptual entities are introduced.

free parameters (6)
  • Uniform green time u
    Hyperparameter for the Uniform baseline controller, chosen by grid search in Section IV-A.
  • Webster's parameters (W, cmin, cmax, s)
    Hyperparameters for the adaptive Webster's controller; s is the saturation flow rate assumption.
  • Max-pressure minimum green gmin
    Minimum green time for the Max-pressure controller, chosen by grid search.
  • SOTL parameters (gmin, theta, omega, mu)
    Thresholds and timing parameters for Self-Organizing Traffic Lights, chosen by grid search.
  • DQN hyperparameters (arepeat, learning rate, replay size, target interval, network size, exploration)
    Deep Q-network training and control hyperparameters; many are not listed in the text but are part of the grid search.
  • DDPG hyperparameters (gmin, gmax, learning rates, replay size, tau, network size)
    DDPG training and control hyperparameters, chosen by grid search.
assumptions (4)
  • domain assumption SUMO microsimulator faithfully models traffic signal control dynamics.
    All conclusions are drawn from SUMO simulations; the framework's value depends on the simulator's fidelity.
  • domain assumption Webster's cycle formula applies to the simulated intersection and traffic conditions.
    The Webster's controller uses the standard cycle length formula, assuming the network satisfies its underlying assumptions.
  • domain assumption The reward function (negative delay) correlates with travel time and overall performance.
    RL agents are trained to minimize delay, while the comparison uses travel time; it is assumed optimizing delay is a good proxy for improving travel time.
  • ad hoc to paper The two-intersection network and synthetic demand are representative enough for comparative conclusions.
    The paper draws general conclusions about controller performance from a single small network, which is a load-bearing assumption for the empirical ranking.

how reviews work

0 comments
Cite this review

Pith. "Pith review of An Open-Source Framework for Adaptive Traffic Signal Control." pith.science (2026). https://pith.science/paper/LO2UB3Z5

@misc{pith2026190900395,
  author       = {Pith},
  title        = {Pith review of: An Open-Source Framework for Adaptive Traffic Signal Control},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/LO2UB3Z5}},
  note         = {Machine review of arXiv:1909.00395}
}
read the original abstract

Sub-optimal control policies in transportation systems negatively impact mobility, the environment and human health. Developing optimal transportation control systems at the appropriate scale can be difficult as cities' transportation systems can be large, complex and stochastic. Intersection traffic signal controllers are an important element of modern transportation infrastructure where sub-optimal control policies can incur high costs to many users. Many adaptive traffic signal controllers have been proposed by the community but research is lacking regarding their relative performance difference - which adaptive traffic signal controller is best remains an open question. This research contributes a framework for developing and evaluating different adaptive traffic signal controller models in simulation - both learning and non-learning - and demonstrates its capabilities. The framework is used to first, investigate the performance variance of the modelled adaptive traffic signal controllers with respect to their hyperparameters and second, analyze the performance differences between controllers with optimal hyperparameters. The proposed framework contains implementations of some of the most popular adaptive traffic signal controllers from the literature; Webster's, Max-pressure and Self-Organizing Traffic Lights, along with deep Q-network and deep deterministic policy gradient reinforcement learning controllers. This framework will aid researchers by accelerating their work from a common starting point, allowing them to generate results faster with less effort. All framework source code is available at https://github.com/docwza/sumolights.

Figures

Figures reproduced from arXiv: 1909.00395 by the authors.

Figure 1
Figure 1. Adaptive traffic signal control DDPG and DQN neural network agents (left) and distributed acting, centralized learning architecture (right) composed [PITH_FULL_IMAGE:figures/full_fig_p005_1.png] view at source ↗
Figure 2
Figure 2. Two intersection SUMO network used for hyperparameter ex [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Individual hyperparameter results for each traffic signal controller. Travel time is used as a measure of effectiveness and is estimated for each [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Comparison of all traffic signal controller hyperparameter travel time performance. Note both vertical and horizontal axis limits have been clipped at [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: Boxplots depicting the distribution of travel times for each traffic signal controller. The solid white line represents the median, solid coloured box the [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 6
Figure 6. Figure 6: Comparison of traffic signal controller individual intersection queue and delay measure of effectiveness in units of vehicles [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

75 extracted references · 62 canonical work pages

  1. [1]

    The influence of intersections on fuel consumption in urban arterial road traffic: a single vehicle test in harbin, china,

    L. Wu, Y . Ci, J. Chu, and H. Zhang, “The influence of intersections on fuel consumption in urban arterial road traffic: a single vehicle test in harbin, china,” PloS one, vol. 10, no. 9, 2015

    work page 2015

  2. [2]

    The impact of individual anthropogenic emissions sectors on the global burden of human mortality due to ambient air pollution,

    R. A. Silva, Z. Adelman, M. M. Fry, and J. J. West, “The impact of individual anthropogenic emissions sectors on the global burden of human mortality due to ambient air pollution,” Environmental health perspectives, vol. 124, no. 11, p. 1776, 2016

    work page 2016

  3. [3]

    Ambient air pollution: A global assessment of exposure and burden of disease,

    World Health Organization et al. , “Ambient air pollution: A global assessment of exposure and burden of disease,” 2016

    work page 2016

  4. [4]

    INRIX global traffic scorecard,

    G. Cookson, “INRIX global traffic scorecard,” INRIX, Tech. Rep., 2018

    work page 2018

  5. [5]

    Recent devel- opment and applications of SUMO - Simulation of Urban MObility,

    D. Krajzewicz, J. Erdmann, M. Behrisch, and L. Bieker, “Recent devel- opment and applications of SUMO - Simulation of Urban MObility,” International Journal On Advances in Systems and Measurements , vol. 5, no. 3&4, pp. 128–138, December 2012

    work page 2012

  6. [6]

    Genetic reinforcement learning for cooper- ative traffic signal control,

    S. Mikami and Y . Kakazu, “Genetic reinforcement learning for cooper- ative traffic signal control,” in Evolutionary Computation, 1994. IEEE World Congress on Computational Intelligence., Proceedings of the First IEEE Conference on . IEEE, 1994, pp. 223–228

    work page 1994

  7. [7]

    Real-time optimiza- tion for adaptive traffic signal control using genetic algorithms,

    J. Lee, B. Abdulhai, A. Shalaby, and E.-H. Chung, “Real-time optimiza- tion for adaptive traffic signal control using genetic algorithms,” Journal of Intelligent Transportation Systems , vol. 9, no. 3, pp. 111–122, 2005

    work page 2005

  8. [8]

    Organic control of traffic lights,

    H. Prothmann, F. Rochner, S. Tomforde, J. Branke, C. M ¨uller-Schloer, and H. Schmeck, “Organic control of traffic lights,” in International Conference on Autonomic and Trusted Computing . Springer, 2008, pp. 219–233

    work page 2008

Show all 75 references
  1. [9]

    Time optimization for traffic signal control using genetic algorithm,

    L. Singh, S. Tripathi, and H. Arora, “Time optimization for traffic signal control using genetic algorithm,” International Journal of Recent Trends in Engineering, vol. 2, no. 2, p. 4, 2009

    2009

  2. [10]

    Evolving adaptive traffic signal controllers for a real scenario using genetic programming with an epigenetic mechanism,

    E. Ricalde and W. Banzhaf, “Evolving adaptive traffic signal controllers for a real scenario using genetic programming with an epigenetic mechanism,” in 2017 16th IEEE International Conference on Machine Learning and Applications (ICMLA) . IEEE, 2017, pp. 897–902

    2017

  3. [11]

    Signal multiobjective optimization for urban traffic network,

    X. Li and J.-Q. Sun, “Signal multiobjective optimization for urban traffic network,” IEEE Transactions on Intelligent Transportation Systems , vol. 19, no. 11, pp. 3529–3537, 2018

    2018

  4. [12]

    Distributed traffic signal control for maximum network throughput,

    T. Wongpiromsarn, T. Uthaicharoenpong, Y . Wang, E. Frazzoli, and D. Wang, “Distributed traffic signal control for maximum network throughput,” in Intelligent Transportation Systems (ITSC), 2012 15th International IEEE Conference on . IEEE, 2012, pp. 588–595

    2012

  5. [13]

    The max-pressure controller for arbitrary networks of signalized intersections,

    P. Varaiya, “The max-pressure controller for arbitrary networks of signalized intersections,” in Advances in Dynamic Network Modeling in Complex Transportation Systems . Springer, 2013, pp. 27–66

    2013

  6. [14]

    Capacity-aware backpressure traffic signal control,

    J. Gregoire, X. Qian, E. Frazzoli, A. De La Fortelle, and T. Wong- piromsarn, “Capacity-aware backpressure traffic signal control,” IEEE Transactions on Control of Network Systems , vol. 2, no. 2, pp. 164– 173, 2015

    2015

  7. [15]

    Multi-agent immune networks to control interrupted flow at signalized intersections,

    S. Darmoul, S. Elkosantini, A. Louati, and L. B. Said, “Multi-agent immune networks to control interrupted flow at signalized intersections,” Transportation Research Part C: Emerging Technologies , vol. 82, pp. 290–313, 2017

    2017

  8. [16]

    An artificial immune network to control interrupted flow at a signalized intersection,

    A. Louati, S. Darmoul, S. Elkosantini, and L. ben Said, “An artificial immune network to control interrupted flow at a signalized intersection,” Information Sciences, vol. 433, pp. 70–95, 2018

    2018

  9. [17]

    Self-organizing traffic lights,

    C. Gershenson, “Self-organizing traffic lights,” arXiv preprint nlin/0411066, 2004. JOURNAL OF TRANSACTIONS ON INTELLIGENT TRANSPORTATION SYSTEMS, VOL. X, NO. X, AUGUST 2019 10

    2004 arXiv

  10. [18]

    Self-organizing traffic lights: A realistic simulation,

    S.-B. Cools, C. Gershenson, and B. DHooghe, “Self-organizing traffic lights: A realistic simulation,” in Advances in applied self-organizing systems. Springer, 2013, pp. 45–55

    2013

  11. [19]

    Self-organization in traffic lights: Evolution of signal control with advances in sensors and com- munications,

    S. Goel, S. F. Bush, and C. Gershenson, “Self-organization in traffic lights: Evolution of signal control with advances in sensors and com- munications,” arXiv preprint arXiv:1708.07188 , 2017

    2017 arXiv

  12. [20]

    A demand-responsive strategy for traffic signal control,

    N. Gartner, “A demand-responsive strategy for traffic signal control,” Transportation Research Record, vol. 906, pp. 75–81, 1983

    1983

  13. [21]

    Scats, sydney co-ordinated adaptive traffic system: A traffic responsive method of controlling urban traffic,

    P. Lowrie, “Scats, sydney co-ordinated adaptive traffic system: A traffic responsive method of controlling urban traffic,” 1990

    1990

  14. [22]

    A real-time traffic signal control system: architecture, algorithms, and analysis,

    P. Mirchandani and L. Head, “A real-time traffic signal control system: architecture, algorithms, and analysis,” Transportation Research Part C: Emerging Technologies, vol. 9, no. 6, pp. 415–432, 2001

    2001

  15. [23]

    Acs-lite algorithmic architecture: applying adaptive control system technology to closed-loop traffic signal control systems,

    F. Luyanda, D. Gettman, L. Head, S. Shelby, D. Bullock, and P. Mirchan- dani, “Acs-lite algorithmic architecture: applying adaptive control system technology to closed-loop traffic signal control systems,” Transportation Research Record: Journal of the Transportation Research B...

    2003

  16. [24]

    Traffic light control using sarsa with three state representations,

    T. L. Thorpe and C. W. Anderson, “Traffic light control using sarsa with three state representations,” Citeseer, Tech. Rep., 1996

    1996

  17. [25]

    Reinforcement learning in neurofuzzy traffic signal con- trol,

    E. Bingham, “Reinforcement learning in neurofuzzy traffic signal con- trol,” European Journal of Operational Research , vol. 131, no. 2, pp. 232–241, 2001

    2001

  18. [26]

    Reinforcement learning for true adaptive traffic signal control,

    B. Abdulhai, R. Pringle, and G. J. Karakoulas, “Reinforcement learning for true adaptive traffic signal control,” Journal of Transportation Engineering, vol. 129, no. 3, pp. 278–285, 2003

    2003

  19. [27]

    Reinforcement learning with function approximation for traffic signal control,

    L. Prashanth and S. Bhatnagar, “Reinforcement learning with function approximation for traffic signal control,” IEEE Transactions on Intelli- gent Transportation Systems, vol. 12, no. 2, pp. 412–421, 2011

    2011

  20. [28]

    Multiagent rein- forcement learning for integrated network of adaptive traffic signal controllers (marlin-atsc): methodology and large-scale application on downtown toronto,

    S. El-Tantawy, B. Abdulhai, and H. Abdelgawad, “Multiagent rein- forcement learning for integrated network of adaptive traffic signal controllers (marlin-atsc): methodology and large-scale application on downtown toronto,” IEEE Transactions on Intelligent Transportation Systems...

    2013

  21. [29]

    Deeplight: Deep reinforcement learning for signalised traffic control,

    T. Rijken, “Deeplight: Deep reinforcement learning for signalised traffic control,” Ph.D. dissertation, Masters Thesis. University College London, 2015

    2015

  22. [30]

    Deep reinforcement learning for coordination in traffic light control,

    E. van der Pol, “Deep reinforcement learning for coordination in traffic light control,” Ph.D. dissertation, Masters Thesis. University of Amsterdam, 2016

    2016

  23. [31]

    Traffic signal timing via deep reinforce- ment learning,

    L. Li, Y . Lv, and F.-Y . Wang, “Traffic signal timing via deep reinforce- ment learning,” IEEE/CAA Journal of Automatica Sinica , vol. 3, no. 3, pp. 247–254, 2016

    2016

  24. [32]

    Using a deep reinforcement learning agent for traffic signal control,

    W. Genders and S. Razavi, “Using a deep reinforcement learning agent for traffic signal control,” arXiv preprint arXiv:1611.01142, 2016, https: //arxiv.org/abs/1611.01142

    2016 arXiv

  25. [33]

    Adaptive traffic signal control with actor-critic methods in a real-world traffic network with different traffic disruption events,

    M. Aslani, M. S. Mesgari, and M. Wiering, “Adaptive traffic signal control with actor-critic methods in a real-world traffic network with different traffic disruption events,” Transportation Research Part C: Emerging Technologies, vol. 85, pp. 732–752, 2017

    2017

  26. [34]

    Traffic light control using deep policy-gradient and value-function based reinforce- ment learning,

    S. S. Mousavi, M. Schukat, P. Corcoran, and E. Howley, “Traffic light control using deep policy-gradient and value-function based reinforce- ment learning,” arXiv preprint arXiv:1704.08883 , 2017

    2017 arXiv

  27. [35]

    Deep reinforcement learn- ing for traffic light control in vehicular networks,

    X. Liang, X. Du, G. Wang, and Z. Han, “Deep reinforcement learn- ing for traffic light control in vehicular networks,” arXiv preprint arXiv:1803.11115, 2018

    2018 arXiv

  28. [36]

    A deep reinforcement learning network for traffic light cycle control,

    ——, “A deep reinforcement learning network for traffic light cycle control,” IEEE Transactions on Vehicular Technology , vol. 68, no. 2, pp. 1243–1253, 2019

    2019

  29. [37]

    Deep reinforcement learning-based traffic signal control using high-resolution event-based data,

    S. Wang, X. Xie, K. Huang, J. Zeng, and Z. Cai, “Deep reinforcement learning-based traffic signal control using high-resolution event-based data,” Entropy, vol. 21, no. 8, p. 744, 2019

    2019

  30. [38]

    Asynchronous n-step q-learning adaptive traffic signal control,

    W. Genders and S. Razavi, “Asynchronous n-step q-learning adaptive traffic signal control,” Journal of Intelligent Transportation Systems , vol. 23, no. 4, pp. 319–331, 2019

    2019

  31. [39]

    Multi-agent deep reinforcement learning for large-scale traffic signal control,

    T. Chu, J. Wang, L. Codec `a, and Z. Li, “Multi-agent deep reinforcement learning for large-scale traffic signal control,” IEEE Transactions on Intelligent Transportation Systems, 2019

    2019

  32. [40]

    Stevanovic, Adaptive traffic control systems: domestic and foreign state of practice , 2010, no

    A. Stevanovic, Adaptive traffic control systems: domestic and foreign state of practice , 2010, no. Project 20-5 (Topic 40-03)

    2010

  33. [41]

    Design of reinforce- ment learning parameters for seamless application of adaptive traffic signal control,

    S. El-Tantawy, B. Abdulhai, and H. Abdelgawad, “Design of reinforce- ment learning parameters for seamless application of adaptive traffic signal control,” Journal of Intelligent Transportation Systems , vol. 18, no. 3, pp. 227–245, 2014

    2014

  34. [42]

    A review on computational intelligence methods for controlling traffic signal timing,

    S. Araghi, A. Khosravi, and D. Creighton, “A review on computational intelligence methods for controlling traffic signal timing,”Expert systems with applications, vol. 42, no. 3, pp. 1538–1550, 2015

    2015

  35. [43]

    An experimental review of reinforcement learning algorithms for adaptive traffic signal control,

    P. Mannion, J. Duggan, and E. Howley, “An experimental review of reinforcement learning algorithms for adaptive traffic signal control,” in Autonomic Road Transport Support Systems . Springer, 2016, pp. 47–66

    2016

  36. [44]

    A survey on reinforcement learning models and algorithms for traffic signal control,

    K.-L. A. Yau, J. Qadir, H. L. Khoo, M. H. Ling, and P. Komisarczuk, “A survey on reinforcement learning models and algorithms for traffic signal control,” ACM Computing Surveys (CSUR) , vol. 50, no. 3, p. 34, 2017

    2017

  37. [45]

    Multiagent rein- forcement learning for urban traffic control using coordination graphs,

    L. Kuyer, S. Whiteson, B. Bakker, and N. Vlassis, “Multiagent rein- forcement learning for urban traffic control using coordination graphs,” in Joint European Conference on Machine Learning and Knowledge Discovery in Databases . Springer, 2008, pp. 656–671

    2008

  38. [46]

    Traffic signal control using reinforce- ment learning and the max-plus algorithm as a coordinating strategy,

    J. C. Medina and R. F. Benekohal, “Traffic signal control using reinforce- ment learning and the max-plus algorithm as a coordinating strategy,” in Intelligent Transportation Systems (ITSC), 2012 15th International IEEE Conference on . IEEE, 2012, pp. 596–601

    2012

  39. [47]

    Holonic multi-agent system for traffic signals control,

    M. Abdoos, N. Mozayani, and A. L. Bazzan, “Holonic multi-agent system for traffic signals control,” Engineering Applications of Artificial Intelligence, vol. 26, no. 5, pp. 1575–1587, 2013

    2013

  40. [48]

    Adaptive multi-objective reinforcement learning with hybrid exploration for traffic signal control based on co- operative multi-agent framework,

    M. A. Khamis and W. Gomaa, “Adaptive multi-objective reinforcement learning with hybrid exploration for traffic signal control based on co- operative multi-agent framework,”Engineering Applications of Artificial Intelligence, vol. 29, pp. 134–151, 2014

    2014

  41. [49]

    Large-scale traffic grid signal control with regional reinforcement learning,

    T. Chu, S. Qu, and J. Wang, “Large-scale traffic grid signal control with regional reinforcement learning,” in American Control Conference (ACC), 2016. IEEE, 2016, pp. 815–820

    2016

  42. [50]

    Deep deterministic policy gradient for urban traffic light control,

    N. Casas, “Deep deterministic policy gradient for urban traffic light control,” arXiv preprint arXiv:1703.09035 , 2017

    2017 arXiv

  43. [51]

    Distributed cooperative rein- forcement learning-based traffic signal control that integrates v2x net- works dynamic clustering,

    W. Liu, G. Qin, Y . He, and F. Jiang, “Distributed cooperative rein- forcement learning-based traffic signal control that integrates v2x net- works dynamic clustering,” IEEE Transactions on Vehicular Technology, vol. 66, no. 10, pp. 8667–8681, 2017

    2017

  44. [52]

    Deep reinforcement learning adaptive traffic signal con- trol,

    W. Genders, “Deep reinforcement learning adaptive traffic signal con- trol,” Ph.D. dissertation, McMaster University, 2018

    2018

  45. [53]

    Traffic signal settings, road research technical paper no. 39,

    F. Webster, “Traffic signal settings, road research technical paper no. 39,” Road Research Laboratory , 1958

    1958

  46. [54]

    Taylor expansion of the accumulated rounding error,

    S. Linnainmaa, “Taylor expansion of the accumulated rounding error,” BIT Numerical Mathematics , vol. 16, no. 2, pp. 146–160, 1976

    1976

  47. [55]

    Learning represen- tations by back-propagating errors,

    D. E. Rumelhart, G. E. Hinton, and R. J. Williams, “Learning represen- tations by back-propagating errors,” nature, vol. 323, no. 6088, p. 533, 1986

    1986

  48. [56]

    Asynchronous methods for deep rein- forcement learning,

    V . Mnih, A. P. Badia, M. Mirza, A. Graves, T. Lillicrap, T. Harley, D. Silver, and K. Kavukcuoglu, “Asynchronous methods for deep rein- forcement learning,” in International Conference on Machine Learning , 2016, pp. 1928–1937

    2016

  49. [57]

    Distributed prioritized experience replay,

    D. Horgan, J. Quan, D. Budden, G. Barth-Maron, M. Hessel, H. Van Hasselt, and D. Silver, “Distributed prioritized experience replay,” arXiv preprint arXiv:1803.00933 , 2018

    2018 arXiv

  50. [58]

    Impala: Scalable dis- tributed deep-rl with importance weighted actor-learner architectures,

    L. Espeholt, H. Soyer, R. Munos, K. Simonyan, V . Mnih, T. Ward, Y . Doron, V . Firoiu, T. Harley, I. Dunninget al., “Impala: Scalable dis- tributed deep-rl with importance weighted actor-learner architectures,” arXiv preprint arXiv:1802.01561 , 2018

    2018 arXiv

  51. [59]

    A distributional perspec- tive on reinforcement learning,

    M. G. Bellemare, W. Dabney, and R. Munos, “A distributional perspec- tive on reinforcement learning,” in Proceedings of the 34th International Conference on Machine Learning-Volume 70 . JMLR. org, 2017, pp. 449–458

    2017

  52. [60]

    Distribu- tional reinforcement learning with quantile regression,

    W. Dabney, M. Rowland, M. G. Bellemare, and R. Munos, “Distribu- tional reinforcement learning with quantile regression,” in Thirty-Second AAAI Conference on Artificial Intelligence , 2018

    2018

  53. [61]

    Implicit quan- tile networks for distributional reinforcement learning,

    W. Dabney, G. Ostrovski, D. Silver, and R. Munos, “Implicit quan- tile networks for distributional reinforcement learning,” arXiv preprint arXiv:1806.06923, 2018

    2018 arXiv

  54. [62]

    Between mdps and semi-mdps: A framework for temporal abstraction in reinforcement learning,

    R. S. Sutton, D. Precup, and S. Singh, “Between mdps and semi-mdps: A framework for temporal abstraction in reinforcement learning,” Artificial intelligence, vol. 112, no. 1-2, pp. 181–211, 1999

    1999

  55. [63]

    The option-critic architecture,

    P.-L. Bacon, J. Harb, and D. Precup, “The option-critic architecture,” in Thirty-First AAAI Conference on Artificial Intelligence , 2017

    2017

  56. [64]

    Learning to repeat: Fine grained action repetition for deep reinforcement learning,

    S. Sharma, A. S. Lakshminarayanan, and B. Ravindran, “Learning to repeat: Fine grained action repetition for deep reinforcement learning,” arXiv preprint arXiv:1702.06054 , 2017

    2017 arXiv

  57. [65]

    Human-level control through deep reinforcement learning,

    V . Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. Riedmiller, A. K. Fidjeland, G. Ostrovski et al. , “Human-level control through deep reinforcement learning,” Nature, vol. 518, no. 7540, pp. 529–533, 2015

    2015

  58. [66]

    Q-learning,

    C. J. Watkins and P. Dayan, “Q-learning,” Machine learning, vol. 8, no. 3-4, pp. 279–292, 1992. JOURNAL OF TRANSACTIONS ON INTELLIGENT TRANSPORTATION SYSTEMS, VOL. X, NO. X, AUGUST 2019 11

    1992

  59. [67]

    Learning to predict by the methods of temporal differ- ences,

    R. S. Sutton, “Learning to predict by the methods of temporal differ- ences,” Machine learning, vol. 3, no. 1, pp. 9–44, 1988

    1988

  60. [68]

    R. S. Sutton and A. G. Barto, Reinforcement learning: An introduction . MIT press Cambridge, 1998, vol. 1, no. 1

    1998

  61. [69]

    Continuous control with deep reinforcement learning,

    T. P. Lillicrap, J. J. Hunt, A. Pritzel, N. Heess, T. Erez, Y . Tassa, D. Silver, and D. Wierstra, “Continuous control with deep reinforcement learning,” arXiv preprint arXiv:1509.02971 , 2015

    2015 arXiv

  62. [70]

    Self-improving reactive agents based on reinforcement learn- ing, planning and teaching,

    L.-J. Lin, “Self-improving reactive agents based on reinforcement learn- ing, planning and teaching,” Machine learning, vol. 8, no. 3-4, pp. 293– 321, 1992

    1992

  63. [71]

    TensorFlow: Large-scale machine learning on heterogeneous systems,

    M. Abadi, A. Agarwal, P. Barham, E. Brevdo, Z. Chen, C. Citro, G. S. Corrado, A. Davis, J. Dean, M. Devin, S. Ghemawat, I. Goodfellow, A. Harp, G. Irving, M. Isard, Y . Jia, R. Jozefowicz, L. Kaiser, M. Kudlur, J. Levenberg, D. Man ´e, R. Monga, S. Moore, D. Murray, C. Olah, M...

    2015

  64. [72]

    SciPy: Open source scientific tools for Python,

    E. Jones, T. Oliphant, P. Peterson et al., “SciPy: Open source scientific tools for Python,” 2001, ”http://www.scipy.org/”

    2001

  65. [73]

    Tabor, 2019

    P. Tabor, 2019. [Online]. Available: https://github.com/philtabor/ Youtube-Code-Repository/blob/master/ReinforcementLearning/ PolicyGradient/DDPG/pendulum/tensorflow/ddpg orig tf.py

    2019

  66. [74]

    Delving deep into rectifiers: Surpassing human-level performance on imagenet classification,

    K. He, X. Zhang, S. Ren, and J. Sun, “Delving deep into rectifiers: Surpassing human-level performance on imagenet classification,” in Proceedings of the IEEE international conference on computer vision , 2015, pp. 1026–1034

    2015

  67. [75]

    Adam: A method for stochastic optimization,

    D. Kingma and J. Ba, “Adam: A method for stochastic optimization,” arXiv preprint arXiv:1412.6980 , 2014. Wade Genders earned a Software B.Eng. & Society in 2013, Civil M.A.Sc. in 2014 and Civil Ph.D in 2018 from McMaster University. His research interests include traffic signa...

    2014 arXiv

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.