{"id":"01fc2295-990d-4592-b50e-ed76d96f1d1a","arxiv_id":"2501.09646","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"NS-Gym provides the first standardized Gymnasium-based toolkit and benchmark suite for non-stationary Markov decision processes.","lead":"This paper introduces NS-Gym, a set of Python simulation environments built on top of the Gymnasium library, designed to let researchers test decision-making algorithms under changing conditions. It also reports benchmark results for six algorithms, giving the non-stationary reinforcement learning community a common testbed.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The benchmark half of the central claim is under-supported: the six-algorithm comparison gives algorithms different information and omits seed counts, so the reported rankings cannot yet be treated as a standardized, reproducible benchmark.","rationale":"The reader's selected weakest assumption was the wrapper-only parameterization of NS-MDPs. That is a real scope limitation, but the paper's toolkit still covers a meaningful class of non-stationary problems, and the authors are reasonably explicit about their design. The more load-bearing issue for the central claim is the reliability of the benchmark comparison itself: different algorithms receive different forms and amounts of information, seed counts are absent, and at least two table entries appear corrupted. These problems directly affect the contribution that the paper is 'the first set of benchmark results comparing six algorithms.' They do not invalidate the toolkit, but they make the benchmark claims non-reproducible as written. Consequently, I agree with the reader's CONDITIONAL verdict and with the suggested revision points, though I would prioritize the experimental protocol and metadata over the representational limitation in the revision.","tokens_in":16537,"tokens_out":4713,"duration_ms":54845,"concrete_test":"Re-run one full benchmark table, e.g., single-change CliffWalking without notification, with a stated seed count (say 50) and with all model-based algorithms restricted to the same stationary snapshot while logging exactly which algorithms receive the true post-change parameters; then re-derive the 'notification helps' conclusion from the replotted distributions. If the algorithm rankings shift when the information sets are equalized, the benchmark is protocol-dependent rather than a measure of algorithm quality.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim includes 'the first set of benchmark results that compares six algorithmic approaches for solving NS-MDPs.' For this claim to hold, the comparison must be a fair, reproducible comparison under the same NS-MDP instances. In the current protocol, algorithms receive unequal information: in the with-notification setting, MCTS, AlphaZero, and PAMCTS obtain the current planning model, while DDQN receives a time budget for gradient updates; in the without-notification setting, DDQN must learn from raw interactions while model-based methods still receive a stationary snapshot of the environment. Additionally, the paper does not state the number of seeds used for any result, and several table entries appear to be formatting errors (e.g., '686.97±8.2' without a minus sign in Table 3 and '2 740.84' in Table 11). These issues mean the reported numbers cannot be independently reproduced or used to rank algorithms reliably. The NS-Gym library itself may still be a useful contribution, but the benchmark portion of the central claim is not yet supported as a standardized comparison.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces NS-Gym, an open-source Python library built on Gymnasium for constructing non-stationary Markov decision processes (NS-MDPs). The design wraps existing Gymnasium environments and lets users specify which environment parameters change, when they change (via schedulers), and how they change (via update functions), with three notification levels. The paper also presents a taxonomy of prior work, a set of environment instances (CartPole, Mountain Car, Acrobot, Pendulum, FrozenLake, CliffWalking, Bridge), and a benchmark comparing MCTS, AlphaZero, DDQN, PA-MCTS, ADA-MCTS, and RATS under single-change and continuous-change protocols with and without notification. The central claim is that NS-Gym is the first standardized, reproducible NS-MDP toolkit and benchmark suite.","tokens_in":16696,"tokens_out":11741,"duration_ms":120001,"significance":"NS-Gym addresses a real gap: prior NS-MDP papers use ad hoc, mutually incompatible environments, making cross-paper comparison difficult. The library is open-source, integrates directly with the Gymnasium API, and cleanly separates the parameter-evolution process from the agent, with configurable schedulers and update functions. The benchmark breadth—six algorithms, four base environments, two change regimes, and three notification levels—is a useful starting point if the numbers are made reproducible. I credit the authors for releasing the code and for providing a tutorial-style pipeline (Section 3.6) that lowers the adoption barrier. There is no circularity concern: benchmarking the authors' own algorithms is standard practice. However, the benchmark results as currently reported cannot yet serve as a standardized reference because of missing run counts, incomplete hyperparameters, asymmetric information across algorithms, and corrupted table entries.","major_comments":[{"comment":"The 'with notification' condition is not the same intervention for all algorithms. Model-based methods (MCTS, AlphaZero, PA-MCTS) receive the current environment model through get_planning_env() at the configured notification level (§3.3), whereas DDQN is given only a fixed wall-clock budget ('time 0.4' in Table 7) to collect data and perform gradient updates (§4.1, item 3). The 'without notification' condition is also asymmetric: model-based planners are handed a stationary snapshot of the environment while DDQN learns only from raw interactions. Consequently, column-wise differences in Tables 3 and 4 confound the notification modality with the amount of privileged model information and with training budget; they do not measure a common 'notification' treatment. The authors should specify exactly what information each algorithm receives in each condition and, if the comparison is meant to reflect each algorithm's intended interface, should say so explicitly and avoid the implication that the columns differ only in notification level.","section":"§4.1, §3.3, Tables 4 and 7"},{"comment":"The benchmark tables report only mean ± standard error, with no statement of the number of independent runs, the number of random seeds, or the number of evaluation episodes per condition. Without these counts, the bold 'best' entries cannot be judged against the reported error bars, and the results cannot be independently reproduced. Several table entries are additionally corrupted or unreadable: Table 3's CliffWalking 0.6 row reads '-1216.72±63.686.97±8.2' (two numbers fused), Table 11 contains '2 740.84±43.23' and '0.60.31' with missing separators, and Table 4 uses '+-', '+/', and '+/-' for the same quantity. These data-integrity problems are central to the paper's benchmark contribution and must be fixed, not merely reformatted.","section":"§4.2, Tables 3, 4, and 11"},{"comment":"The architecture is restricted to non-stationarity that can be expressed as modifications to parameters that an existing stationary Gymnasium environment exposes ('each wrapper introduces non-stationarity by modifying some parameters that the base environment exposes'). Structural changes to the state or action space, changes in observation modality, and non-parametric changes in the transition kernel that cannot be captured by a scalar or distributional parameter of the base environment cannot be represented. This is a real limitation of the toolkit's current design, and it is not stated in the abstract or in Contribution 1, which claim a toolkit for NS-MDPs without qualification. Please state this limitation prominently and adjust the scope claims accordingly.","section":"§3.2, Abstract, and Contribution 1"},{"comment":"The algorithm configuration is incomplete for reproducibility. For DDQN and for the PA-MCTS value/policy network, Tables 7 and 8 list only network sizes and, for DDQN, a notification-time budget; missing are the learning rate, optimizer, batch size, replay-buffer size, target-network update period, exploration schedule, and number of training episodes. For AlphaZero, Table 6 does not state how long the stationary policy network was trained, the number of self-play games, or the training data distribution. Since all of these choices materially change the benchmark numbers, they need to be reported in the paper or supplied as a versioned configuration in the released repository.","section":"Appendix C, Tables 6–8, and §4.1"}],"minor_comments":[{"comment":"The sentence 'We make the following contributions. We make the following contributions:' is duplicated before the contribution list.","section":"Section 1"},{"comment":"'textitsojourn' is a LaTeX artifact; the intended text is 'sojourn time'.","section":"Section 2, Figure 1 caption"},{"comment":"'UTC' should be 'UCT' (Upper Confidence bounds for Trees).","section":"Section 4.1, item 1"},{"comment":"The caption says 'WO and WON' but the rows are labeled 'WN' and 'WON'; it also says 'for with agent' instead of 'for an agent'.","section":"Table 4 caption"},{"comment":"In the second problem setting, 'the user is aware of the exact environmental change' should refer to the agent, not the programmer; the footnote defines 'user' as the programmer.","section":"Section 3.3, paragraph 2"},{"comment":"The 'first' claims are not substantiated by a systematic comparison with existing non-stationary benchmark or wrapper efforts; please provide evidence for the priority claim or soften it.","section":"Abstract and Contribution 5"},{"comment":"Notation for uncertainty is inconsistent across tables ('±', '+-', '+/-'); unify it, and ensure that all decimal separators and spaces are unambiguous.","section":"Tables 3, 4, and 11"},{"comment":"Appendix A.2.3 says 'jail cell' where 'goal cell' is presumably meant, and Appendix A.2.5 has 'is moves' instead of 'it moves'.","section":"Appendix A.2.3 and A.2.5"}],"recommendation":"major_revision","confidential_remarks":"The toolkit is a genuinely useful contribution and the topic is timely. The benchmark section, however, is not yet at the standard of a citable reference: it needs seed counts, complete hyperparameters, explicit per-algorithm information sets, and repaired tables. I would not require a new algorithmic contribution, but the empirical claims must be made reproducible before acceptance. I also suggest asking the authors to moderate the 'first' claims unless they can document a thorough search of prior toolkits and benchmark suites."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a real contribution—the NS-Gym codebase and wrapper design fill a genuine gap—but the benchmark results as reported are not yet a trustworthy standardized comparison. The toolkit deserves attention; the numbers need a revision.\n\nWhat's new: a Gymnasium-compatible library that lets you inject non-stationarity into existing environments via schedulers and parameter update functions, with three notification levels for the agent. That modular separation—scheduler, update function, notification—is a sensible abstraction, and it's genuinely the first time someone has packaged this in an open-source, usable form. The code is on GitHub, there's a tutorial, and the paper walks through a clean example. For a field that has been running on ad-hoc custom environments, this is useful infrastructure.\n\nThe benchmark section is where I part ways. The claim of 'the first set of benchmark results that compares six algorithmic approaches' is too strong given the current protocol. MCTS/AlphaZero/PA-MCTS get a planning model at their notification level; DDQN gets a time budget for gradient updates. That is not a level playing field. In the no-notification case, model-based agents still receive a stationary snapshot of the base environment, which leaks information that a no-notification agent shouldn't have. No seed counts are reported anywhere—just means and standard errors. There are also clear formatting errors in the tables (e.g., the CliffWalking AlphaZero entry in Table 3 reads '686.97±8.2' with a missing minus sign, and '2 740.84' appears in the appendix table). These are fixable, but they undermine the reproducibility the paper claims.\n\nThe wrapper-only parameterization is a real limitation: it can't represent structural changes to state/action spaces or unobservable regime shifts. The authors are upfront that this is what the wrapper does, so it's not a sleight of hand, but it does mean the toolkit covers a subset of NS-MDPs, not all of them. That should be stated more prominently.\n\nI'd send this to review. A serious referee can push for seed counts, a fair notification protocol, and a clearer scope statement. The toolkit itself is worth having in the literature.","headline":"NS-Gym is a genuinely useful toolkit, but the benchmark section needs a revision before the 'first standardized benchmark' claim holds.","tokens_in":17278,"tokens_out":2198,"would_cite":true,"duration_ms":23326,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper presents NS-Gym, the first simulation toolkit built explicitly for non-stationary Markov decision processes, along with the first comparative benchmark of six solution algorithms on standardized non-stationary environments.","keywords":["non-stationary MDP","simulation toolkit","Gymnasium","benchmark","reinforcement learning","semi-Markov decision process","adaptive decision-making","MCTS"],"falsifier":"Define an NS-MDP where, after some decision epoch, a new action becomes available (the action set changes size) and try to express it with NS-Gym's parameter-update wrapper; the wrapper has no slot for structural changes, so it would fail to simulate the problem, contradicting the claim of a general-purpose NS-MDP toolkit.","tokens_in":16305,"feed_emoji":"🎮","tokens_out":4276,"duration_ms":37472,"temperature":0.7,"pith_summary":"The paper's goal is to fix the lack of standardized simulation environments for non-stationary Markov decision processes (NS-MDPs), where the rules, transitions, or rewards drift over time. It introduces NS-Gym, a Python toolkit built on the Gymnasium framework that turns ordinary stationary environments into non-stationary ones by letting users declare which exposed parameters change, when they change (schedulers), and how (update functions). To show the toolkit works, the paper also runs the first comparative benchmark of six prior algorithms — MCTS, DDQN, AlphaZero, PA-MCTS, ADA-MCTS, and RATS — across four environments with single or continuous changes, with or without notification of the change. If the toolkit is adopted, decision-making research gains a common yardstick: the same changing environments, the same interfaces, and baseline numbers that future methods must be compared against.","feed_headline":"NS-Gym: first standardized testbed for non-stationary MDPs","feed_subtitle":"Wraps Gymnasium environments with schedulable parameter changes and benchmarks six algorithms across four domains.","key_machinery":"The load-bearing mechanism is the NS-Gym wrapper: it sits between the agent and an underlying Gymnasium environment and holds a dictionary of tunable parameters, each mapped to a scheduler (continuous, stepwise, random, periodic) and an update function (e.g., increment or random walk). The wrapper checks the scheduler at every decision epoch, applies the update, computes the magnitude of change (Wasserstein distance for distributions), and packages the result into a custom observation with env_change and delta_change fields, while an optional get_planning_env() returns a snapshot of the base environment at the appropriate notification level.","core_discovery":"The central discovery is that a broad class of non-stationary decision problems can be captured by a thin wrapper around standard Gymnasium environments: treat the environment's exposed parameters (gravity, pole mass, transition probabilities) as a semi-Markov process that evolves independently of the agent, and let a scheduler plus update function drive that evolution. The wrapper returns custom observations containing the state, a change flag, the magnitude of change, and relative time, and can optionally hand the agent a stationary snapshot of the environment for planning at one of three notification levels. On the benchmark side, the paper reports that detailed notification generally improves most methods, that model-based online search methods (MCTS, PA-MCTS, AlphaZero with notifications) handle continuous change well, and that risk-averse tree searches (ADA-MCTS, RATS) fare better in stochastic grid worlds like FrozenLake.","pith_inferences":["The wrapper architecture implicitly bounds the toolkit: any NS-MDP whose non-stationarity is not a parameter change of a base environment (e.g., changing state or action spaces) cannot be represented; a natural extension would be a wrapper that can also modify observation and action spaces.","The paper's separation of change detection (notification) from model update suggests a taxonomy where detection is assumed external; an extension could benchmark agents that must detect changes themselves from raw observations, which NS-Gym currently does not simulate.","The benchmark metric is mean cumulative episode reward; extending the harness to regret or dynamic regret would make results more informative for the bandit-flavored prior work the paper surveys."],"forward_implications":["Researchers can now report results on shared NS-MDP environments with a common interface, making head-to-head algorithm comparison meaningful and reproducible.","New decision-making algorithms can plug into NS-Gym with minimal code by wrapping an existing Gymnasium environment and declaring parameter mappings.","The three notification levels give a principled way to test how much information an algorithm needs: a boolean flag, exact parameter magnitudes, or a full planning model.","The benchmark baseline suggests that notification information matters more for online search methods than for pure RL like DDQN, and that risk-averse strategies pay off in stochastic non-stationary domains."],"supporting_citations":[{"why":"Supplies the base Gymnasium framework and environments that NS-Gym wraps.","marker":"[Towers et al., 2023]"},{"why":"Supplies the semi-Markov formulation of parameter evolution that NS-Gym's scheduler/update-function design follows.","marker":"[Campo et al., 1991]"},{"why":"Contributes the Bridge environment and the RATS worst-case search algorithm benchmarked here.","marker":"[Lecarpentier and Rachelson, 2019]"},{"why":"Contributes the policy-augmented MCTS (PA-MCTS) algorithm and the single-change setting with notification.","marker":"[Pettet et al., 2024]"},{"why":"Contributes ADA-MCTS and the setting where the agent is notified of a change but must learn its magnitude.","marker":"[Luo et al., 2024]"},{"why":"Motivates the no-notification bandit-style category in the paper's taxonomy.","marker":"[Garivier and Moulines, 2011]"},{"why":"Motivates the forecast-based objective and episode-level changes in the taxonomy.","marker":"[Chandak et al., 2020b]"}],"fun_headline_variants":["NS-Gym: open-source testbed for non-stationary MDPs","Test your RL agents on shifting dynamics with NS-Gym","Standard benchmarks for changing environments from NS-Gym","Six RL methods benchmarked on non-stationary MDPs","NS-Gym: modular sims for non-stationary decision making"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The toolkit only models non-stationarity as changes to observable parameters that an existing stationary Gymnasium environment already exposes, so shifts that alter the state space, action space, or the very structure of the dynamics cannot be represented.","fun_headline_variants_meta":{"raw":{"variants":["NS-Gym: open-source testbed for non-stationary MDPs","Test your RL agents on shifting dynamics with NS-Gym","Standard benchmarks for changing environments from NS-Gym","Six RL methods benchmarked on non-stationary MDPs","NS-Gym: modular sims for non-stationary decision making"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001142,"raw_usage":{"total_tokens":4743,"prompt_tokens":954,"completion_tokens":3789,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":570,"completion_tokens_details":{"reasoning_tokens":3700}},"tokens_in":570,"tokens_out":3789,"duration_ms":27707,"temperature":1.0,"reasoning_tokens":3700,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T19:47:54.647886+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Define an NS-MDP where, after some decision epoch, a new action becomes available (the action set changes size) and try to express it with NS-Gym's parameter-update wrapper; the wrapper has no slot for structural changes, so it would fail to simulate the problem, contradicting the claim of a general-purpose NS-MDP toolkit.","supporting_citations":[{"cited_title":"State estimation for systems with sojourn-time-dependent markov model switching","cited_arxiv_id":null,"evidence_quote":"Supplies the semi-Markov formulation of parameter evolution that NS-Gym's scheduler/update-function design follows."},{"cited_title":"Non-stationary markov decision processes, a worst-case approach using model-based reinforcement learning","cited_arxiv_id":null,"evidence_quote":"Contributes the Bridge environment and the RATS worst-case search algorithm benchmarked here."},{"cited_title":"On upper-confidence bound policies for switching bandit problems","cited_arxiv_id":null,"evidence_quote":"Motivates the no-notification bandit-style category in the paper's taxonomy."}],"review_version":1}