Pith. sign in

REVIEW 3 major objections 7 minor 39 references

Gym4ReaL: A Suite for Benchmarking Real-World Reinforcement Learning

T0 review · 3 major / 7 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Gym4ReaL claims that a six-environment suite can reproduce the structural difficulties of real-world reinforcement learning, and benchmark results show standard RL methods at least matching rule-based baselines in five of the six tasks.

desk verdict A useful, thoroughly documented RL benchmark suite whose central 'real-world challenge' claims are asserted rather than experimentally verified. read the letter →

arxiv 2507.00257 v1 pith:HUQA63QU submitted 2025-06-30 cs.LG cs.AI

Davide Salaorni , Vincenzo De Paola , Samuele Delpero , Giovanni Dispoto , Paolo Bonetti , Alessio Russo , Giuseppe Calcagno , Francesco Trovò
show 4 more authors
Matteo Papini Alberto Maria Metelli Marco Mussi Marcello Restelli
This is my paper · ORCID
classification cs.LGcs.AI
keywords reinforcementlearningbenchmarksuitereal-worldenvironmentsnon-stationaritypartialobservabilitycontinuouscontrolrule-basedbaselineswaterdistributionsystems
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Gym4ReaL is a benchmarking suite of six realistic reinforcement-learning environments: a dam controller, an elevator dispatcher, a residential microgrid, a robotic feeder, a foreign-exchange trading desk, and a water distribution network. The paper aims to show that these environments capture the obstacles that make real-world RL hard—non-stationarity, partial observability, large or continuous state and action spaces, and expensive or risky exploration—inside a standardized environment interface. Across the six tasks, standard off-the-shelf algorithms (PPO, DQN, Q-Learning, SARSA) generally match or beat rule-based expert policies; the main exceptions are that neither deep RL agent beats passive trading baselines, and the microgrid results show high variance. The authors offer the suite as a methodologically agnostic testbed for developing and comparing RL algorithms aimed at real-world deployment.

What carries the argument

The load-bearing object is the environment suite itself: six environments, each implemented behind the same standard RL interface, whose realism comes from physical simulators or historical datasets rather than from idealized dynamics. Within each environment, the machinery that makes the setting hard is the combination of partially hidden or delayed observations, exogenous uncontrollable signals, and rewards with several contrastive terms that penalize constraint violations—for example, DamEnv clips actions against piecewise rating curves, MicrogridEnv subtracts battery degradation from trading profit, and WDSEnv couples demand satisfaction to an overflow-risk penalty. This design is what turns each task from a clean mathematical problem into a test bed for non-stationarity, partial observability, constrained exploration, and risk-sensitive decision making.

What would settle it

Take the best-performing policy on any Gym4ReaL environment and evaluate it under a distribution shift the suite does not include—for example, retraining MicrogridEnv or TradingEnv on profiles generated by a different stochastic process, or running the WDSEnv policy on a different network topology—and check whether its advantage over rule-based baselines collapses.

Watch

Extended reading notes

Core claim

The central discovery is that a curated collection of six domain environments can reproduce the structural difficulties of real-world RL while remaining solvable by existing methods. Each environment embeds a real-world failure mode: DamEnv pairs a continuous release action with hydraulic rating curves and daily demand; ElevatorEnv turns a classic dispatching problem into a stochastic, restricted-capacity scheduling task; MicrogridEnv combines energy forecasts, market prices, and battery degradation; RoboFeederEnv operates from camera images at the trajectory-planning level; TradingEnv uses high-frequency market data with transaction costs; and WDSEnv couples pump scheduling to a step-wise hydraulic solver. Benchmarking with PPO, DQN, Q-Learning, and SARSA shows these standard algorithms outperforming rule-based baselines in five of the six settings, with trading as the notable exception and microgrid control as the high-variance case. The paper reads this as evidence that standard RL remains competitive under realistic conditions, and that the suite can act as a common benchmark for methods aimed at practical deployment.

Load-bearing premise

The load-bearing premise is that simulated, historically grounded versions of these problems are a reliable proxy for the real-world difficulties the suite claims to model, so a policy that exploits the fixed simulator would not necessarily transfer to living systems.

Editorial extensions

If this is right

  • Off-the-shelf RL algorithms remain viable for realistic control and scheduling tasks, so a new method that cannot match PPO, DQN, Q-Learning, or SARSA on these environments is unlikely to be ready for deployment.
  • The suite provides a single place to evaluate methods intended for non-stationary, partially observable, and constrained-exploration settings, where the relevant algorithms and paradigms are explicitly mapped to environments.
  • The supported RL paradigms beyond the benchmarked ones—imitation learning in DamEnv and WDSEnv, hierarchical RL in RoboFeederEnv, risk-averse trading, frequency adaptation, and multi-objective RL—extend the suite's usefulness beyond the reported baselines.
  • The negative and high-variance results (TradingEnv RL not beating passive strategies, high PPO variance on MicrogridEnv) identify concrete gaps where new algorithms are needed.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: because the historical data are used as a fixed test set, the suite measures non-stationarity within a dataset but not the distribution shift caused by a deployed policy changing the world; wrapping each environment in a resampling or adversarial data layer would test transfer directly.
  • Editorial inference: the same standard interface makes cross-environment transfer experiments natural, so one could train on one demand regime and evaluate on another to turn Gym4ReaL into a domain-shift stress test.
  • Editorial inference: if the trading null result holds under broader hyperparameter search, it suggests that practical RL for finance needs explicit risk-averse or transaction-cost-aware objectives rather than generic deep RL.
  • Editorial inference: comparing Gym4ReaL scores with actual deployment performance on the corresponding physical systems would quantify how much realism is still missing; large gaps would pinpoint where the benchmarks overstate readiness.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 7 minor

Summary. Gym4ReaL is introduced as a Gymnasium-compatible benchmarking suite of six environments: dam release control, elevator dispatching, residential microgrid management, robotic feeding (with separate picking and planning tasks), Forex trading, and municipal water distribution. The paper specifies the observation, action, and reward structure of each environment, documents the underlying datasets and simulators, and reports baseline experiments in which standard RL algorithms (PPO, DQN, Q-Learning, SARSA) are compared with rule-based policies. The central empirical claim is that the RL methods are competitive with or superior to the rule-based baselines on most tasks, with TradingEnv as the acknowledged exception. The appendix contains environment-specific parameters, dataset provenance and licenses, hyperparameter tables, and learning curves.

Significance. Should the suite live up to its description, Gym4ReaL is a potentially useful community resource: it is multi-domain, configurable, open-sourced, and documented with tutorials; the authors provide dataset provenance, licenses, hyperparameters, seeds, and learning curves, which makes the reported comparisons reproducible in practice. The A.4 limitation statement is honest and appropriately scopes the suite to benchmarking rather than deployment guarantees. The baseline results also provide falsifiable reference points, such as PPO's overflow avoidance on DamEnv and DQN's low-variance performance on WDSEnv. The principal contribution is breadth and standardization rather than algorithmic novelty; however, the advertised challenge properties (partial observability, non-stationarity) are not yet empirically verified, and the benchmarking evidence rests on small numbers of episodes and seeds.

major comments (3)
  1. [Table 1; §2.3, §2.5; App. D.3, E.5] The central claim that the suite exposes algorithms to real-world challenges such as partial observability and non-stationarity rests on design assertions in Table 1 and in Appendix D, not on empirical verification. For MicrogridEnv, the observation (Eq. 6) contains previous-step estimates bPD,t and bPG,t rather than the true demand and generation, which is the stated basis for the 'Partially Observable' label; there is no ablation comparing an agent trained on these estimates with an agent given oracle access to the true values. For TradingEnv, non-stationarity is inferred from the train (2019-2020), validation (2021), and test (2022) split (App. E.5), but no same-distribution control (training and testing on the same year) is reported to show that the performance gap is due to distribution shift rather than to overfitting or noise. Appendix A.4 appropriately limits deployment claims, but it does not address whether the simulated challenges themselves bind. I request such diagnostics (oracle ablations and same-distribution controls), or a moderation of the Table 1 claims.
  2. [§2.1-§2.6; Tables 9, 11, 14] The experimental support for the claim that standard RL algorithms 'confirm their competitiveness' is statistically thin. The test sets consist of 13 episodes for DamEnv, 30 for ElevatorEnv, 28 for MicrogridEnv, 20 for WDSEnv, 50 episodes over 5 seeds for RoboFeeder, and 6 seeds for TradingEnv, and DamEnv, MicrogridEnv, and WDSEnv are each reported with a single seed (Tables 9, 11, 14). Statements such as 'larger average return with small variability' (§2.1) and 'consistently outperform' (§2.6) need multi-seed evidence with per-seed variability; the large PPO variance reported for MicrogridEnv (§2.3) suggests that single-seed conclusions may be fragile. The paper's own remark in App. E.5 that untuned standard hyperparameters were used and that manual selection can easily overfit further tempers these comparisons.
  3. [App. D.6, Eq. (20)] Equation (20) defines the demand-satisfaction ratio as a fraction whose numerator and denominator are written with the identical symbol (d_{j,t} on both sides), so as printed the expression is identically 1 and cannot represent the DSR term described in the text. Distinct notation for actual supplied demand and expected demand (e.g., d_{j,t} versus d-bar_{j,t}) must be restored; otherwise the WDSEnv reward specification is vacuous as a formal definition.
minor comments (7)
  1. [App. D.6] The paragraph describing the simulator contains a duplicated passage (the text about tanks and junctions appears twice, separated by a colon) and should be edited to remove the repetition.
  2. [App. E.5] The sentence 'we considered the base setting, as referred in the Appendix E.5' points to the very section it is in; the intended reference is presumably Appendix C.5 or D.5.
  3. [App. E.5; Table 13] The text states that training was performed for approximately 7M steps, which is hard to reconcile with the Table 13 entries '# Episodes 30' and '# Steps 708'; please clarify whether these denote total timesteps, per-update rollout lengths, or evaluation episodes.
  4. [§1, Related Work] The statement that suites such as RWRL Suite and RRLS 'do not aim to address real-world problems directly' mischaracterizes them, since both explicitly target real-world deployment challenges such as partial observability, non-stationarity, and robustness; the contrast should be drawn on breadth and on grounding in concrete application domains rather than on intent.
  5. [Figure 2] The boxplot in Figure 2b is labeled 'Return (s)', yet the plotted values are negative cumulative waiting times; the sign convention should be clarified or the axis relabeled.
  6. [App. B; App. D.4.1] There are typos in environment names: 'TradinvEnv' should be 'TradingEnv', and 'Gym4Real' should be 'Gym4ReaL'.
  7. [§3, Discussion] The concluding claim that the suite captures 'limited data availability' is not reflected in the environments as instantiated, since most draw on years of historical data or unlimited synthetic sampling; either identify which environment exhibits data scarcity or remove this item from the list.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: the benchmark results are empirical evaluations of RL on externally defined environments, and self-citations are not load-bearing.

full rationale

Gym4ReaL does not derive its headline results from fitted parameters or from self-citations. Each environment has dynamics and reward functions defined externally: Lake Como historical data for DamEnv, Poisson arrivals for ElevatorEnv, a BESS digital twin and market/demand datasets for MicrogridEnv, MuJoCo for RoboFeederEnv, EUR/USD tick data for TradingEnv, and EPANET for WDSEnv. RL agents are then trained on these fixed environments and compared with rule-based policies on held-out episodes or years, e.g., TradingEnv trains on 2019-2020, validates on 2021, and tests on 2022, while MicrogridEnv trains on 2015-2019 and tests on 2020. The reported competitiveness is an empirical outcome rather than a construction; the TradingEnv results even show RL failing to consistently beat buy-and-hold baselines. The only quasi-circular element is that Table 1 labels such as 'Partially Observable' and 'Non-Stationary' are design assertions rather than verified properties, with no oracle-privilege ablations or same-distribution controls to show that these challenges actually bind. That is a benchmark-validity gap, and Appendix A.4 appropriately disclaims deployment transfer, but neither constitutes a reduction of a result to its own inputs. Self-citations ([4], [15], [20]) point to simulators, datasets, and prior software used as building blocks; they do not supply the paper's central empirical claim about RL competitiveness, so they are not load-bearing circularity.

Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

The paper's contribution is infrastructural, so there are no newly invented physical entities. The free parameters are reward weights and algorithm hyperparameters, which are standard for RL benchmarks. The main axioms are the fidelity of the simulations and the validity of historical data as a test distribution.

free parameters (2)
  • PPO hyperparameters = varying per environment, e.g. learning rate 8e-6 for DamEnv, 5e-5 for MicrogridEnv
    These are standard training hyperparameters chosen by the authors for each environment. They are not derived from theory and could affect the baseline comparisons, but they are not the central claim of the paper.
  • reward weights = lambda1, lambda2 for DamEnv; lambda for MicrogridEnv; lambda for RoboFeeder
    The balance between reward terms is chosen by hand and affects what behavior is optimal. The paper does not report the specific values in the main text.
assumptions (3)
  • domain assumption The simulated environments faithfully represent the real-world systems they claim to model.
    This is the central premise of the paper. DamEnv models Lake Como, WDSEnv uses EPANET, RoboFeeder uses MuJoCo, and TradingEnv uses historical data. The paper acknowledges in Appendix A.4 that performance in these simulations does not guarantee real-world performance.
  • domain assumption Historical data used in MicrogridEnv and TradingEnv is representative of future conditions and of test conditions.
    TradingEnv tests on 2022 after training on 2019-2020, and MicrogridEnv trains on four years and tests on 2020. The evaluation therefore assumes that the test distribution is reflected in the historical data.
  • domain assumption Standard RL implementations (SB3, skrl) provide fair baselines for RL performance.
    The paper uses off-the-shelf implementations of PPO, DQN, Q-Learning, and SARSA, and does not tune them extensively. The conclusions about RL competitiveness are limited to these default configurations.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Gym4ReaL: A Suite for Benchmarking Real-World Reinforcement Learning." pith.science (2026). https://pith.science/paper/HUQA63QU

@misc{pith2026250700257,
  author       = {Pith},
  title        = {Pith review of: Gym4ReaL: A Suite for Benchmarking Real-World Reinforcement Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/HUQA63QU}},
  note         = {Machine review of arXiv:2507.00257}
}
read the original abstract

In recent years, \emph{Reinforcement Learning} (RL) has made remarkable progress, achieving superhuman performance in a wide range of simulated environments. As research moves toward deploying RL in real-world applications, the field faces a new set of challenges inherent to real-world settings, such as large state-action spaces, non-stationarity, and partial observability. Despite their importance, these challenges are often underexplored in current benchmarks, which tend to focus on idealized, fully observable, and stationary environments, often neglecting to incorporate real-world complexities explicitly. In this paper, we introduce \texttt{Gym4ReaL}, a comprehensive suite of realistic environments designed to support the development and evaluation of RL algorithms that can operate in real-world scenarios. The suite includes a diverse set of tasks that expose algorithms to a variety of practical challenges. Our experimental results show that, in these settings, standard RL algorithms confirm their competitiveness against rule-based benchmarks, motivating the development of new methods to fully exploit the potential of RL to tackle the complexities of real-world tasks.

Figures

Figures reproduced from arXiv: 2507.00257 by the authors.

Figure 1
Figure 1. Test performances with confidence intervals on [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Performance of baselines in terms of mean cumulative reward (a) and average return (b) on [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Performance of baselines in terms of mean cumulative reward (a) and average return (b) on [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (14 more)
Figure 4
Figure 4. Figure 4: Heatmap of the success rate of picking tasks across the entire workspace with baseline [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: Performances of PPO and DQN against baselines B&H and S&H on Test (a) Daily [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 6
Figure 6. Figure 6: Performance of baselines in terms of mean cumulative resilience (a) and average return (b) [PITH_FULL_IMAGE:figures/full_fig_p009_6.png]
Figure 7
Figure 7. Figure 7: Rendering of MuJoCo simulator from RoboFeederEnv [PITH_FULL_IMAGE:figures/full_fig_p022_7.png]
Figure 9
Figure 9. Figure 9: Learning curves of PPO on DamEnv [PITH_FULL_IMAGE:figures/full_fig_p028_9.png]
Figure 10
Figure 10. Figure 10: Learning curves of Q-Learning and SARSA on [PITH_FULL_IMAGE:figures/full_fig_p029_10.png]
Figure 11
Figure 11. Figure 11: Learning curves of PPO on MicrogridEnv. E.4 RoboFeederEnv This section gives further details on the training of RL agents on RoboFeeder for reproducibility pur￾poses [PITH_FULL_IMAGE:figures/full_fig_p030_11.png]
Figure 12
Figure 12. Figure 12: RoboFeeder-picking-v0, PPO training curve of mean episode length and reward over 5 seeds. E.4.2 RoboFeeder-planning Figures 13a and 13b present the results of training the PPO algorithm over 10k iterations. The environment is set up with three objects positioned so th…
Figure 13
Figure 13. Figure 13: RoboFeeder-planning, training curve of mean episode reward and episode length over 5 seeds. and 15. We used the Stable-Baselines3 [18] implementations of the algorithms, specifically the SBX (Stable-Baselines3 in JAX) implementation of PPO and the standard SB3 impleme…
Figure 14
Figure 14. Figure 14: Training Curves PPO of the 6 seeds considered for [PITH_FULL_IMAGE:figures/full_fig_p033_14.png]
Figure 15
Figure 15. Figure 15: Training Curves DQN of the 6 seeds considered for [PITH_FULL_IMAGE:figures/full_fig_p033_15.png]
Figure 16
Figure 16. Figure 16: Performances of PPO and DQN w.r.t. common baselines (i.e., passive strategies) on [PITH_FULL_IMAGE:figures/full_fig_p034_16.png]
Figure 17
Figure 17. Figure 17: Policy learned by PPO on 3 different seeds in the [PITH_FULL_IMAGE:figures/full_fig_p034_17.png]
Figure 18
Figure 18. Figure 18: Learning curves of DQN on WDSEnv. 35 [PITH_FULL_IMAGE:figures/full_fig_p035_18.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

39 extracted references · 26 canonical work pages

  1. [1]

    Almgren and N

    R. Almgren and N. A. Chriss. Optimal execution of portfolio trans-actions. 2000. URL https://api.semanticscholar.org/CorpusID:15502295

    work page 2000

  2. [2]

    Cominola, M

    A. Cominola, M. Giuliani, A. Castelleti, A. Abdallah, and D. E. Rosenberg. Developing a stochastic simulation model for the generation of residential water end-use demand time series. 2016

    work page 2016

  3. [3]

    Crites and A

    R. Crites and A. Barto. Improving elevator performance using reinforcement learning. In D. Touretzky, M. Mozer, and M. Hasselmo, editors,Advances in Neural Information Processing Systems, volume 8. MIT Press, 1995. URL https://proceedings.neurips.cc/paper_ files/paper/1995/file/390e982518a50e280d8e2b535462ec1f-Paper.pdf

    work page 1995

  4. [4]

    De Paola, G

    V . De Paola, G. Calcagno, A. M. Metelli, and M. Restelli. The power of hybrid learning in industrial robotics: Efficient grasping strategies with supervised-driven reinforcement learning. In 2024 International Joint Conference on Neural Networks (IJCNN), pages 1–9, 2024. doi: 10.1109/IJCNN60899.2024.10650627

    arXiv 2024

  5. [5]

    Dulac-Arnold, N

    G. Dulac-Arnold, N. Levine, D. J. Mankowitz, J. Li, C. Paduraru, S. Gowal, and T. Hester. An empirical investigation of the challenges of real-world reinforcement learning. arXiv preprint arXiv:2003.11881, 2020

    arXiv 2003

  6. [6]

    Fioriti, L

    D. Fioriti, L. Pellegrino, G. Lutzemberger, E. Micolano, and D. Poli. Optimal sizing of residential battery systems with multi-year dynamics and a novel rainflow-based model of storage degradation: An extensive italian case study. Electric Power Systems Research, 203,

    work page

  7. [7]

    Historical data mgp, 2015-2020

    Gestore dei Mercati Energetici S.p.A. Historical data mgp, 2015-2020. Data retrieved from GME: https://www.mercatoelettrico.org/it/download/DatiStorici.aspx

    work page 2015

  8. [8]

    Heinsbroek

    A. Heinsbroek. Epynet. https://github.com/Vitens/epynet, 2016

    work page 2016

Show all 39 references
  1. [9]

    K. A. Klise, R. Murray, and T. Haxton. An overview of the water network tool for resilience (WNTR), 2018

    2018

  2. [10]

    Kober, J

    J. Kober, J. A. Bagnell, and J. Peters. Reinforcement learning in robotics: A survey. The International Journal of Robotics Research, 32(11):1238–1274, 2013

    2013

  3. [11]

    Q. Li, Z. Peng, L. Feng, Q. Zhang, Z. Xue, and B. Zhou. Metadrive: Composing diverse driving scenarios for generalizable reinforcement learning. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022

    2022

  4. [12]

    X.-Y . Liu, H. Yang, J. Gao, and C. D. Wang. FinRL: Deep reinforcement learning framework to automate trading in quantitative finance. ACM International Conference on AI in Finance (ICAIF), 2021

    2021

  5. [13]

    V . Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. Riedmiller, A. K. Fidjeland, G. Ostrovski, S. Petersen, C. Beattie, A. Sadik, I. Antonoglou, H. King, D. Kumaran, D. Wierstra, S. Legg, and D. Hassabis. Human-level control through deep ...

    2015 doi

  6. [14]

    Morad, R

    S. Morad, R. Kortvelesy, M. Bettini, S. Liwicki, and A. Prorok. POPGym: Benchmarking par- tially observable reinforcement learning. In The Eleventh International Conference on Learning Representations, 2023. URL https://openreview.net/forum?id=chDrutUTs0K

    2023

  7. [15]

    Murillo, R

    A. Murillo, R. Taormina, N. O. Tippenhauer, D. Salaorni, R. van Dijk, L. Jonker, S. V os, M. Weyns, and S. Galelli. High-fidelity cyber and physical simulation of water distribution systems. i: Models and data. Journal of Water Resources Planning and Management , 149 (5):04023...

    2023 doi

  8. [16]

    A. Naug, A. Guillen, R. Luna, V . Gundecha, C. Bash, S. Ghorbanpour, S. Mousavi, A. R. Babu, D. Markovikj, L. D. Kashyap, D. Rengarajan, and S. Sarkar. Sustaindc: Bench- marking for sustainable data center control. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J....

  9. [17]

    Pfenninger and I

    S. Pfenninger and I. Staffell. Long-term patterns of european pv output using 30 years of validated hourly reanalysis and satellite data. Energy, 114:1251–1265, 2016. ISSN 0360-5442. doi: https://doi.org/10.1016/j.energy.2016.08.060

    2016 doi

  10. [18]

    Raffin, A

    A. Raffin, A. Hill, A. Gleave, A. Kanervisto, M. Ernestus, and N. Dormann. Stable-baselines3: Reliable reinforcement learning implementations. Journal of Machine Learning Research, 22 (268):1–8, 2021. URL http://jmlr.org/papers/v22/20-1364.html

    2021

  11. [19]

    L. A. Rossman. EPANET 2: Users Manual. U.S. Environmental Protection Agency, Cincinnati, OH, 2000. https://www.epa.gov/water-research/epanet

    2000

  12. [20]

    Salaorni

    D. Salaorni. Ernesto-dt, 2023. https://github.com/Daveonwave/ErNESTO-DT

    2023

  13. [21]

    Sandler, A

    M. Sandler, A. G. Howard, M. Zhu, A. Zhmoginov, and L. Chen. Inverted residuals and linear bottlenecks: Mobile networks for classification, detection and segmentation. CoRR, abs/1801.04381, 2018. URL http://arxiv.org/abs/1801.04381

    2018 arXiv

  14. [22]

    Schulman, F

    J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal policy optimization algorithms. CoRR, abs/1707.06347, 2017. URL http://arxiv.org/abs/1707.06347

    2017 arXiv

  15. [23]

    Serrano-Muñoz, D

    A. Serrano-Muñoz, D. Chrysostomou, S. Bøgh, and N. Arana-Arexolaleiba. skrl: Modular and flexible library for reinforcement learning. Journal of Machine Learning Research, 24(254): 1–9, 2023. URL http://jmlr.org/papers/v24/23-0112.html

    2023

  16. [25]

    Silver, J

    D. Silver, J. Schrittwieser, K. Simonyan, I. Antonoglou, A. Huang, A. Guez, T. Hubert, L. Baker, M. Lai, A. Bolton, Y . Chen, T. Lillicrap, F. Hui, L. Sifre, G. van den Driessche, T. Graepel, and D. Hassabis. Mastering the game of go without human knowledge. Nature, 550(7676):...

    2017 doi

  17. [26]

    Staffell, S

    I. Staffell, S. Pfenninger, and N. Johnson. A global model of hourly space heating and cooling de- mand at multiple spatial scales. Nature Energy, 8, 09 2023. doi: 10.1038/s41560-023-01341-5

    2023 doi

  18. [27]

    R. S. Sutton and A. G. Barto. Reinforcement Learning: An Introduction. The MIT Press, second edition, 2018

    2018

  19. [28]

    Todorov, T

    E. Todorov, T. Erez, and Y . Tassa. Mujoco: A physics engine for model-based control. In2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pages 5026–5033. IEEE,

  20. [29]

    Towers, A

    M. Towers, A. Kwiatkowski, J. Terry, J. U. Balis, G. De Cola, T. Deleu, M. Goulão, A. Kallinteris, M. Krimmel, A. KG, et al. Gymnasium: A standard interface for reinforcement learning environments. arXiv preprint arXiv:2407.17032, 2024

    2024 arXiv

  21. [30]

    C. J. Watkins and P. Dayan. Q-learning. Machine learning, 8:279–292, 1992

    1992

  22. [31]

    C. Yeh, V . Li, R. Datta, Y . Yue, and A. Wierman. SustainGym: A benchmark suite of re- inforcement learning for sustainability applications. In NeurIPS 2022 Workshop on Tack- ling Climate Change with Machine Learning , New Orleans, LA, USA, 12 2022. URL https://www.climatecha...

    2022

  23. [32]

    X. Yuan, L. Bu¸ soniu, and R. Babuška. Reinforcement learning for elevator control. IFAC Proceedings Volumes, 41(2):2212–2217, 2008. ISSN 1474-6670. doi: https://doi.org/10. 3182/20080706-5-KR-1001.00373. URL https://www.sciencedirect.com/science/ article/pii/S1474667016392783...

    2008

  24. [33]

    Z. Yuan, A. W. Hall, S. Zhou, L. Brunke, M. Greeff, J. Panerati, and A. P. Schoellig. Safe- control-gym: A unified benchmark suite for safe learning-based control and reinforcement learning in robotics. IEEE Robotics and Automation Letters, 7(4):11142–11149, 2022. doi: 10.1109...

    2022

  25. [34]

    Zouitine, D

    A. Zouitine, D. Bertoin, P. Clavier, M. Geist, and E. Rachelson. Rrls : Robust reinforcement learning suite, 2024. URL https://arxiv.org/abs/2406.08406. 12 A Metadata A.1 Hosting and Maintenance Gym4ReaL is distributed as a Python package and is publicly available on both GitH...

    2024 arXiv

  26. [39]

    In the first part of the table, we report the key features of each environment with respect to the standard RL taxonomy

    Characteristics. In the first part of the table, we report the key features of each environment with respect to the standard RL taxonomy. These Characteristics are inherent to the design and implementation of the environments provided by Gym4ReaL

  27. [40]

    This second part of the table describes the RL subfields that can be associated with each environment

    RL Paradigms. This second part of the table describes the RL subfields that can be associated with each environment. Specifically, although the benchmarking results provided in the main paper employ standard RL approaches, the environments can be adapted to test novel algorith...

    2021

  28. [2012]

    doi: 10.1109/IROS.2012.6386109

    2012

  29. [2017]

    URL http://arxiv.org/abs/1712.01815

    arXiv

  30. [2022]

    doi: https://doi.org/10.1016/j.epsr.2021.107675

    ISSN 0378-7796. doi: https://doi.org/10.1016/j.epsr.2021.107675

    2021

  31. [2024]

    URL https://proceedings.neurips.cc/paper_files/paper/2024/file/ b6676756f8a935e208f394a1ba47f0bc-Paper-Datasets_and_Benchmarks_Track. pdf

    2024

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.