Pith. sign in

REVIEW 4 major objections 8 minor 25 references

NS-Gym: Open-Source Simulation Environments and Benchmarks for Non-Stationary Markov Decision Processes

T0 review · 4 major / 8 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper presents NS-Gym, the first simulation toolkit built explicitly for non-stationary Markov decision processes, along with the first comparative benchmark of six solution algorithms on standardized non-stationary environments.

desk verdict NS-Gym is a genuinely useful toolkit, but the benchmark section needs a revision before the 'first standardized benchmark' claim holds. read the letter →

arxiv 2501.09646 v1 pith:7ZC4ZZ3E submitted 2025-01-16 cs.AI

Nathaniel S. Keplinger , Baiting Luo , Iliyas Bektas , Yunuo Zhang , Kyle Hollins Wray , Aron Laszka , Abhishek Dubey , Ayan Mukhopadhyay This is my paper · ORCID
classification cs.AI
keywords non-stationaryMDPsimulationtoolkitGymnasiumbenchmarkreinforcementlearningsemi-Markovdecisionprocessadaptivedecision-makingMCTS
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper's goal is to fix the lack of standardized simulation environments for non-stationary Markov decision processes (NS-MDPs), where the rules, transitions, or rewards drift over time. It introduces NS-Gym, a Python toolkit built on the Gymnasium framework that turns ordinary stationary environments into non-stationary ones by letting users declare which exposed parameters change, when they change (schedulers), and how (update functions). To show the toolkit works, the paper also runs the first comparative benchmark of six prior algorithms — MCTS, DDQN, AlphaZero, PA-MCTS, ADA-MCTS, and RATS — across four environments with single or continuous changes, with or without notification of the change. If the toolkit is adopted, decision-making research gains a common yardstick: the same changing environments, the same interfaces, and baseline numbers that future methods must be compared against.

What carries the argument

The load-bearing mechanism is the NS-Gym wrapper: it sits between the agent and an underlying Gymnasium environment and holds a dictionary of tunable parameters, each mapped to a scheduler (continuous, stepwise, random, periodic) and an update function (e.g., increment or random walk). The wrapper checks the scheduler at every decision epoch, applies the update, computes the magnitude of change (Wasserstein distance for distributions), and packages the result into a custom observation with env_change and delta_change fields, while an optional get_planning_env() returns a snapshot of the base environment at the appropriate notification level.

What would settle it

Define an NS-MDP where, after some decision epoch, a new action becomes available (the action set changes size) and try to express it with NS-Gym's parameter-update wrapper; the wrapper has no slot for structural changes, so it would fail to simulate the problem, contradicting the claim of a general-purpose NS-MDP toolkit.

Watch

Extended reading notes

Core claim

The central discovery is that a broad class of non-stationary decision problems can be captured by a thin wrapper around standard Gymnasium environments: treat the environment's exposed parameters (gravity, pole mass, transition probabilities) as a semi-Markov process that evolves independently of the agent, and let a scheduler plus update function drive that evolution. The wrapper returns custom observations containing the state, a change flag, the magnitude of change, and relative time, and can optionally hand the agent a stationary snapshot of the environment for planning at one of three notification levels. On the benchmark side, the paper reports that detailed notification generally improves most methods, that model-based online search methods (MCTS, PA-MCTS, AlphaZero with notifications) handle continuous change well, and that risk-averse tree searches (ADA-MCTS, RATS) fare better in stochastic grid worlds like FrozenLake.

Load-bearing premise

The toolkit only models non-stationarity as changes to observable parameters that an existing stationary Gymnasium environment already exposes, so shifts that alter the state space, action space, or the very structure of the dynamics cannot be represented.

Editorial extensions

If this is right

  • Researchers can now report results on shared NS-MDP environments with a common interface, making head-to-head algorithm comparison meaningful and reproducible.
  • New decision-making algorithms can plug into NS-Gym with minimal code by wrapping an existing Gymnasium environment and declaring parameter mappings.
  • The three notification levels give a principled way to test how much information an algorithm needs: a boolean flag, exact parameter magnitudes, or a full planning model.
  • The benchmark baseline suggests that notification information matters more for online search methods than for pure RL like DDQN, and that risk-averse strategies pay off in stochastic non-stationary domains.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The wrapper architecture implicitly bounds the toolkit: any NS-MDP whose non-stationarity is not a parameter change of a base environment (e.g., changing state or action spaces) cannot be represented; a natural extension would be a wrapper that can also modify observation and action spaces.
  • The paper's separation of change detection (notification) from model update suggests a taxonomy where detection is assumed external; an extension could benchmark agents that must detect changes themselves from raw observations, which NS-Gym currently does not simulate.
  • The benchmark metric is mean cumulative episode reward; extending the harness to regret or dynamic regret would make results more informative for the bandit-flavored prior work the paper surveys.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 8 minor

Summary. The paper introduces NS-Gym, an open-source Python library built on Gymnasium for constructing non-stationary Markov decision processes (NS-MDPs). The design wraps existing Gymnasium environments and lets users specify which environment parameters change, when they change (via schedulers), and how they change (via update functions), with three notification levels. The paper also presents a taxonomy of prior work, a set of environment instances (CartPole, Mountain Car, Acrobot, Pendulum, FrozenLake, CliffWalking, Bridge), and a benchmark comparing MCTS, AlphaZero, DDQN, PA-MCTS, ADA-MCTS, and RATS under single-change and continuous-change protocols with and without notification. The central claim is that NS-Gym is the first standardized, reproducible NS-MDP toolkit and benchmark suite.

Significance. NS-Gym addresses a real gap: prior NS-MDP papers use ad hoc, mutually incompatible environments, making cross-paper comparison difficult. The library is open-source, integrates directly with the Gymnasium API, and cleanly separates the parameter-evolution process from the agent, with configurable schedulers and update functions. The benchmark breadth—six algorithms, four base environments, two change regimes, and three notification levels—is a useful starting point if the numbers are made reproducible. I credit the authors for releasing the code and for providing a tutorial-style pipeline (Section 3.6) that lowers the adoption barrier. There is no circularity concern: benchmarking the authors' own algorithms is standard practice. However, the benchmark results as currently reported cannot yet serve as a standardized reference because of missing run counts, incomplete hyperparameters, asymmetric information across algorithms, and corrupted table entries.

major comments (4)
  1. [§4.1, §3.3, Tables 4 and 7] The 'with notification' condition is not the same intervention for all algorithms. Model-based methods (MCTS, AlphaZero, PA-MCTS) receive the current environment model through get_planning_env() at the configured notification level (§3.3), whereas DDQN is given only a fixed wall-clock budget ('time 0.4' in Table 7) to collect data and perform gradient updates (§4.1, item 3). The 'without notification' condition is also asymmetric: model-based planners are handed a stationary snapshot of the environment while DDQN learns only from raw interactions. Consequently, column-wise differences in Tables 3 and 4 confound the notification modality with the amount of privileged model information and with training budget; they do not measure a common 'notification' treatment. The authors should specify exactly what information each algorithm receives in each condition and, if the comparison is meant to reflect each algorithm's intended interface, should say so explicitly and avoid the implication that the columns differ only in notification level.
  2. [§4.2, Tables 3, 4, and 11] The benchmark tables report only mean ± standard error, with no statement of the number of independent runs, the number of random seeds, or the number of evaluation episodes per condition. Without these counts, the bold 'best' entries cannot be judged against the reported error bars, and the results cannot be independently reproduced. Several table entries are additionally corrupted or unreadable: Table 3's CliffWalking 0.6 row reads '-1216.72±63.686.97±8.2' (two numbers fused), Table 11 contains '2 740.84±43.23' and '0.60.31' with missing separators, and Table 4 uses '+-', '+/', and '+/-' for the same quantity. These data-integrity problems are central to the paper's benchmark contribution and must be fixed, not merely reformatted.
  3. [§3.2, Abstract, and Contribution 1] The architecture is restricted to non-stationarity that can be expressed as modifications to parameters that an existing stationary Gymnasium environment exposes ('each wrapper introduces non-stationarity by modifying some parameters that the base environment exposes'). Structural changes to the state or action space, changes in observation modality, and non-parametric changes in the transition kernel that cannot be captured by a scalar or distributional parameter of the base environment cannot be represented. This is a real limitation of the toolkit's current design, and it is not stated in the abstract or in Contribution 1, which claim a toolkit for NS-MDPs without qualification. Please state this limitation prominently and adjust the scope claims accordingly.
  4. [Appendix C, Tables 6–8, and §4.1] The algorithm configuration is incomplete for reproducibility. For DDQN and for the PA-MCTS value/policy network, Tables 7 and 8 list only network sizes and, for DDQN, a notification-time budget; missing are the learning rate, optimizer, batch size, replay-buffer size, target-network update period, exploration schedule, and number of training episodes. For AlphaZero, Table 6 does not state how long the stationary policy network was trained, the number of self-play games, or the training data distribution. Since all of these choices materially change the benchmark numbers, they need to be reported in the paper or supplied as a versioned configuration in the released repository.
minor comments (8)
  1. [Section 1] The sentence 'We make the following contributions. We make the following contributions:' is duplicated before the contribution list.
  2. [Section 2, Figure 1 caption] 'textitsojourn' is a LaTeX artifact; the intended text is 'sojourn time'.
  3. [Section 4.1, item 1] 'UTC' should be 'UCT' (Upper Confidence bounds for Trees).
  4. [Table 4 caption] The caption says 'WO and WON' but the rows are labeled 'WN' and 'WON'; it also says 'for with agent' instead of 'for an agent'.
  5. [Section 3.3, paragraph 2] In the second problem setting, 'the user is aware of the exact environmental change' should refer to the agent, not the programmer; the footnote defines 'user' as the programmer.
  6. [Abstract and Contribution 5] The 'first' claims are not substantiated by a systematic comparison with existing non-stationary benchmark or wrapper efforts; please provide evidence for the priority claim or soften it.
  7. [Tables 3, 4, and 11] Notation for uncertainty is inconsistent across tables ('±', '+-', '+/-'); unify it, and ensure that all decimal separators and spaces are unambiguous.
  8. [Appendix A.2.3 and A.2.5] Appendix A.2.3 says 'jail cell' where 'goal cell' is presumably meant, and Appendix A.2.5 has 'is moves' instead of 'it moves'.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: NS-Gym is a software and benchmark contribution whose claims rest on an implementation and on measured evaluations, not on a derivation that reduces to its own inputs.

full rationale

This paper makes no formal derivation that could be circular. Its central claims are (1) the presentation of a simulation toolkit and (2) benchmark results comparing six algorithms on non-stationary MDP instances. The toolkit design is described operationally: wrappers introduce non-stationarity by modifying parameters exposed by base Gymnasium environments, schedulers determine when changes occur, and update functions determine how parameters change. The benchmark section reports measured episodic rewards from executing the algorithms in those environments. No parameter is fitted to a subset of data and then presented as a prediction; no quantity is defined in terms of the quantity it is supposed to establish. The authors do benchmark algorithms from their own prior work (PA-MCTS, ADA-MCTS), but the benchmark numbers are produced by running the algorithms in this paper rather than imported by citation, and the central toolkit claim does not rest on a self-citation. The review of prior work and the 'first toolkit' novelty assertion are historical and narrative claims rather than derivations; even if contested, a novelty dispute is not a circularity. No uniqueness theorem, ansatz, or renaming of a known result is invoked to make the argument load-bearing. Accordingly, the analysis finds no significant circularity.

Assumptions & free parameters 4 free parameters · 4 assumptions · 2 invented entities

The paper's central contribution is software, not a theory. It rests on domain assumptions that NS-MDPs can be captured by observable parameter evolution on top of existing Gymnasium environments, and that the four design questions are an adequate taxonomy. No new physical entities are introduced; the runtime monitor and model updater are software components with implementations.

free parameters (4)
  • Environment change magnitudes = varied: pole mass 1.0/1.5, transition probabilities 0.4/0.6/0.8, decrements 0.1/0.2/0.02
    Hand-chosen test conditions for the benchmark; they do not enter the toolkit's design, but the reported algorithm rankings could change under other values.
  • PAMCTS alpha interpolation weights = 0.25, 0.5, 0.75
    Three values hand-picked to vary how much the stale policy is trusted; results depend on this choice.
  • DDQN notification update time = 0.4 seconds
    Hand-chosen compute budget given to DDQN in the 'with notification' setting; no principled basis, and it affects cross-algorithm comparisons.
  • Episode truncation lengths = 2500 (CartPole), 100 (FrozenLake), 200 (CliffWalking, Bridge)
    Hand-set limits that cap the maximum reward and influence comparisons.
assumptions (4)
  • domain assumption The dynamics of an NS-MDP are fully determined by observable parameters θ that the agent or wrapper can read and update.
    Section 2 and Figure 1 assume θ is observable and explicitly specified; if non-stationarity involves hidden regime changes that cannot be exposed as parameters, the wrapper design cannot capture them.
  • domain assumption Non-stationarity can be layered onto existing Gymnasium environments by changing exposed scalar parameters.
    Section 3.2 states 'each wrapper introduces non-stationarity by modifying some parameters that the base environment exposes.' Structural changes to the state/action space or unobservable dynamics are not handled.
  • domain assumption The four questions (what changes, how, detect, know) are sufficient to categorize the space of NS-MDPs.
    Section 2 uses these questions to organize prior work and to define NS-Gym problem types; the paper does not prove coverage.
  • standard math Standard MDP definitions and the semi-Markov formalism from Campo et al. [1991] apply.
    Section 2 builds on standard MDP definitions and the semi-Markov chain formulation.
invented entities (2)
  • Runtime monitor independent evidence
    purpose: Conceptual component that tracks θ and detects changes, delivering notifications to the agent.
    Implemented in NS-Gym as boolean/detailed notification flags in the observation type (Table 2); it is a software abstraction, not a physical entity.
  • Model updater independent evidence
    purpose: Conceptual component that updates the agent's transition model when θ changes, enabling planning against a current snapshot.
    Implemented via get_planning_env() and model snapshot functionality (Section 3.3); again a software abstraction.

how reviews work

0 comments
Cite this review

Pith. "Pith review of NS-Gym: Open-Source Simulation Environments and Benchmarks for Non-Stationary Markov Decision Processes." pith.science (2026). https://pith.science/paper/7ZC4ZZ3E

@misc{pith2026250109646,
  author       = {Pith},
  title        = {Pith review of: NS-Gym: Open-Source Simulation Environments and Benchmarks for Non-Stationary Markov Decision Processes},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/7ZC4ZZ3E}},
  note         = {Machine review of arXiv:2501.09646}
}
read the original abstract

In many real-world applications, agents must make sequential decisions in environments where conditions are subject to change due to various exogenous factors. These non-stationary environments pose significant challenges to traditional decision-making models, which typically assume stationary dynamics. Non-stationary Markov decision processes (NS-MDPs) offer a framework to model and solve decision problems under such changing conditions. However, the lack of standardized benchmarks and simulation tools has hindered systematic evaluation and advance in this field. We present NS-Gym, the first simulation toolkit designed explicitly for NS-MDPs, integrated within the popular Gymnasium framework. In NS-Gym, we segregate the evolution of the environmental parameters that characterize non-stationarity from the agent's decision-making module, allowing for modular and flexible adaptations to dynamic environments. We review prior work in this domain and present a toolkit encapsulating key problem characteristics and types in NS-MDPs. This toolkit is the first effort to develop a set of standardized interfaces and benchmark problems to enable consistent and reproducible evaluation of algorithms under non-stationary conditions. We also benchmark six algorithmic approaches from prior work on NS-MDPs using NS-Gym. Our vision is that NS-Gym will enable researchers to assess the adaptability and robustness of their decision-making algorithms to non-stationary conditions.

Figures

Figures reproduced from arXiv: 2501.09646 by the authors.

Figure 1
Figure 1. An overall framework for non-stationary Markov decision processes. At time [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. A sequence diagram of the agent-environment interaction in NS-Gym. Steps 4–9 in the diagram show how [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. The Gymnasium CartPole environment. A.2 Mountain Car The MountainCar environment (see [PITH_FULL_IMAGE:figures/full_fig_p012_3.png] view at source ↗
Figures from the paper (14 more)
Figure 4
Figure 4. Figure 4: The Gymnasium MountainCar environment. A.2.1 Acrobot [PITH_FULL_IMAGE:figures/full_fig_p013_4.png]
Figure 5
Figure 5. Figure 5: The Gymnasium Acrobot environment. The Acrobot environment is a double pendulum (see [PITH_FULL_IMAGE:figures/full_fig_p013_5.png]
Figure 6
Figure 6. Figure 6: The Gymnasium Pendulum environment. A.2.3 FrozenLake The FrozenLake environment ( [PITH_FULL_IMAGE:figures/full_fig_p014_6.png]
Figure 7
Figure 7. Figure 7: The Gymnasium FrozenLake environment. A.2.4 CliffWalker The CliffWalking environment ( [PITH_FULL_IMAGE:figures/full_fig_p014_7.png]
Figure 8
Figure 8. Figure 8: The Gymnasium CliffWalking environment. reach a goal cell, the agent must cross a “bridge” surrounded by terminal cells. The secondary goal cell is farther from the starting location but less risky because fewer holes surround it. Unlike the CliffWalking environment, w…
Figure 9
Figure 9. Figure 9: The Bridge environment. The start cell is in red, the two goals are in green, and the terminal "holes" are in [PITH_FULL_IMAGE:figures/full_fig_p015_9.png]
Figure 10
Figure 10. Figure 10: Distribution of rewards for the CliffWalking experiments with a single change. [PITH_FULL_IMAGE:figures/full_fig_p020_10.png]
Figure 11
Figure 11. Figure 11: Average success rate (i.e., the agent finds the goal state) for each agent in the single change experiments. [PITH_FULL_IMAGE:figures/full_fig_p020_11.png]
Figure 12
Figure 12. Figure 12: Distribution of episode rewards for each agent tested on non-stationary CartPole environment with and [PITH_FULL_IMAGE:figures/full_fig_p021_12.png]
Figure 13
Figure 13. Figure 13: Mean episode reward and standard error for each agent in a non-stationary FrozenLake environment with a [PITH_FULL_IMAGE:figures/full_fig_p021_13.png]
Figure 14
Figure 14. Figure 14: Distribution of episode reward for each agent under the continuous change experiment conditions. [PITH_FULL_IMAGE:figures/full_fig_p022_14.png]
Figure 15
Figure 15. Figure 15: Mean reward and standard error for agents in the non-stationary Bridge environment under the continuous [PITH_FULL_IMAGE:figures/full_fig_p022_15.png]
Figure 16
Figure 16. Figure 16: Mean reward and standard error for agents in the non-stationary FrozenLake environment under continuous [PITH_FULL_IMAGE:figures/full_fig_p023_16.png]
Figure 17
Figure 17. Figure 17: Distribution of episode rewards for agents in the continuous non-stationary CartPole environment. [PITH_FULL_IMAGE:figures/full_fig_p023_17.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

25 extracted references · 13 canonical work pages

  1. [1]

    On state estimation in switching environments

    Guy Ackerson and K Fu. On state estimation in switching environments. IEEE transactions on automatic control, 15 0 (1): 0 10--17, 1970

    work page 1970

  2. [2]

    Near-optimal regret bounds for reinforcement learning

    Peter Auer, Thomas Jaksch, and Ronald Ortner. Near-optimal regret bounds for reinforcement learning. Advances in neural information processing systems, 21, 2008

    2008

  3. [3]

    Stochastic multi-armed-bandit problem with non-stationary rewards

    Omar Besbes, Yonatan Gur, and Assaf Zeevi. Stochastic multi-armed-bandit problem with non-stationary rewards. Advances in neural information processing systems, 27, 2014

    2014

  4. [4]

    State estimation for systems with sojourn-time-dependent markov model switching

    L Campo, P Mookerjee, and Y Bar-Shalom. State estimation for systems with sojourn-time-dependent markov model switching. IEEE Transactions on Automatic Control, 36 0 (2): 0 238--243, 1991

    work page 1991

  5. [5]

    Non stationary bandits with periodic variation

    Titas Chakraborty and Parth Shettiwar. Non stationary bandits with periodic variation. In Proceedings of the 23rd International Conference on Autonomous Agents and Multiagent Systems, pages 2177--2179, 2024

    work page 2024

  6. [6]

    Towards safe policy improvement for non-stationary mdps

    Yash Chandak, Scott Jordan, Georgios Theocharous, Martha White, and Philip S Thomas. Towards safe policy improvement for non-stationary mdps. Advances in Neural Information Processing Systems, 33: 0 9156--9168, 2020 a

    work page 2020

  7. [7]

    Optimizing for the future in non-stationary mdps

    Yash Chandak, Georgios Theocharous, Shiv Shankar, Sridhar Mahadevan, Martha White, and Philip S Thomas. Optimizing for the future in non-stationary mdps. Thirty-seventh International Conference on Machine Learning (ICML), 2020 b

    work page 2020

  8. [8]

    Anomaly detection: A survey

    Varun Chandola, Arindam Banerjee, and Vipin Kumar. Anomaly detection: A survey. ACM computing surveys (CSUR), 41 0 (3): 0 1--58, 2009

    2009

Show all 25 references
  1. [9]

    Reinforcement learning for non-stationary markov decision processes: The blessing of (more) optimism

    Wang Chi Cheung, David Simchi-Levi, and Ruihao Zhu. Reinforcement learning for non-stationary markov decision processes: The blessing of (more) optimism. In International Conference on Machine Learning, pages 1843--1854. PMLR, 2020

    2020

  2. [10]

    On upper-confidence bound policies for switching bandit problems

    Aur \'e lien Garivier and Eric Moulines. On upper-confidence bound policies for switching bandit problems. In International Conference on Algorithmic Learning Theory, pages 174--188. Springer, 2011

    2011

  3. [11]

    Markov decision processes with their applications, volume 14

    Qiying Hu and Wuyi Yue. Markov decision processes with their applications, volume 14. Springer Science & Business Media, 2007

    2007

  4. [12]

    Deep reinforcement learning for autonomous driving: A survey

    B Ravi Kiran, Ibrahim Sobh, Victor Talpaert, Patrick Mannion, Ahmad A Al Sallab, Senthil Yogamani, and Patrick P \'e rez. Deep reinforcement learning for autonomous driving: A survey. IEEE Transactions on Intelligent Transportation Systems, 23 0 (6): 0 4909--4926, 2021

    2021

  5. [13]

    Algorithms for decision making

    Mykel J Kochenderfer, Tim A Wheeler, and Kyle H Wray. Algorithms for decision making. MIT press, 2022

    2022

  6. [14]

    Bandit Based Monte - Carlo Planning

    Levente Kocsis and Csaba Szepesvári. Bandit Based Monte - Carlo Planning . In Johannes Fürnkranz, Tobias Scheffer, and Myra Spiliopoulou, editors, Machine Learning : ECML 2006 , pages 282--293, Berlin, Heidelberg, 2006. Springer Berlin Heidelberg. ISBN 978-3-540-46056-5

    2006

  7. [15]

    Non-stationary markov decision processes, a worst-case approach using model-based reinforcement learning

    Erwan Lecarpentier and Emmanuel Rachelson. Non-stationary markov decision processes, a worst-case approach using model-based reinforcement learning. Advances in neural information processing systems, 32, 2019

    2019

  8. [16]

    Learning to delegate for large-scale vehicle routing

    Sirui Li, Zhongxia Yan, and Cathy Wu. Learning to delegate for large-scale vehicle routing. Advances in Neural Information Processing Systems, 34: 0 26198--26211, 2021

    2021

  9. [17]

    Act as you learn: Adaptive decision-making in non-stationary markov decision processes

    Baiting Luo, Yunuo Zhang, Abhishek Dubey, and Ayan Mukhopadhyay. Act as you learn: Adaptive decision-making in non-stationary markov decision processes. arXiv preprint arXiv:2401.01841, 2024

    2024 arXiv

  10. [18]

    A review of incident prediction, resource allocation, and dispatch models for emergency management

    Ayan Mukhopadhyay, Geoffrey Pettet, Sayyed Mohsen Vazirizade, Di Lu, Alejandro Jaimes, Said El Said, Hiba Baroud, Yevgeniy Vorobeychik, Mykel Kochenderfer, and Abhishek Dubey. A review of incident prediction, resource allocation, and dispatch models for emergency management. A...

    2022

  11. [19]

    Trading financial indices with reinforcement learning agents

    Parag C Pendharkar and Patrick Cusatis. Trading financial indices with reinforcement learning agents. Expert Systems with Applications, 103: 0 1--13, 2018

    2018

  12. [20]

    Decision making in non-stationary environments with policy-augmented search

    Ava Pettet, Yunuo Zhang, Baiting Luo, Kyle Wray, Hendrik Baier, Aron Laszka, Abhishek Dubey, and Ayan Mukhopadhyay. Decision making in non-stationary environments with policy-augmented search. arXiv preprint arXiv:2401.03197, 2024

    2024 arXiv

  13. [21]

    Markov decision processes: discrete stochastic dynamic programming

    Martin L Puterman. Markov decision processes: discrete stochastic dynamic programming. John Wiley & Sons, 2014

    2014

  14. [22]

    Mastering Chess and Shogi by Self - Play with a General Reinforcement Learning Algorithm , December 2017

    David Silver, Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Matthew Lai, Arthur Guez, Marc Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, Timothy Lillicrap, Karen Simonyan, and Demis Hassabis. Mastering Chess and Shogi by Self - Play with a General Reinfor...

    2017 arXiv

  15. [23]

    Terry, Ariel Kwiatkowski, John U

    Mark Towers, Jordan K. Terry, Ariel Kwiatkowski, John U. Balis, Gianluca de Cola, Tristan Deleu, Manuel Goulão, Andreas Kallinteris, Arjun KG, Markus Krimmel, Rodrigo Perez-Vicente, Andrea Pierré, Sander Schulhoff, Jun Jet Tai, Andrew Tan Jin Shen, and Omar G. Younis. Gymnasiu...

    2023

  16. [24]

    Deep reinforcement learning with double q-learning

    Hado van Hasselt, Arthur Guez, and David Silver. Deep reinforcement learning with double q-learning. CoRR, abs/1509.06461, 2015. URL http://arxiv.org/abs/1509.06461

    2015 arXiv

  17. [25]

    Reinforcement learning in healthcare: A survey

    Chao Yu, Jiming Liu, Shamim Nemati, and Guosheng Yin. Reinforcement learning in healthcare: A survey. ACM Computing Surveys (CSUR), 55 0 (1): 0 1--36, 2021

    2021

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.