Pith. sign in

REVIEW 3 major objections 6 minor 24 references

Benchmarking Partial Observability in Reinforcement Learning with a Suite of Memory-Improvable Domains

T0 review · 3 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper argues that a usable partial-observability benchmark needs broad coverage across forms of state aliasing plus a clear floor-to-ceiling performance gap, and introduces POBAX, a JAX suite whose tasks are all memory improvable and…

desk verdict A useful JAX benchmark and a sensible selection criterion, but the 'all environments memory improvable' claim is confounded by architecture mismatches in five of eleven tasks. read the letter →

arxiv 2508.00046 v1 pith:RDSNABQB submitted 2025-07-31 cs.LG cs.AI

Ruo Yu Tao , Kaicheng Guo , Cameron Allen , George Konidaris This is my paper
classification cs.LGcs.AI
keywords reinforcementlearningpartialobservabilityPOMDPbenchmarkingmemoryimprovabilityJAXrecurrentpoliciesstatealiasing
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Mitigating partial observability is necessary for general reinforcement learning agents, but most existing benchmarks test only narrow forms of state aliasing such as feature masking and Gaussian noise, where measured gains can come from hyperparameter choices, network size, or learning dynamics instead of memory. The paper argues that a useful partially observable benchmark needs two properties: coverage across the many distinct forms of partial observability, and memory improvability, meaning a clear performance gap between agents with more and less state information while everything else is held roughly equal. It introduces POBAX, a JAX-based suite of environments spanning localization and mapping, visual occlusion, object tracking, moment features, games, and more. The central claim is that every POBAX environment is memory improvable and requires hard-to-learn memory functions, giving the field a concrete signal for judging whether an algorithm actually mitigates partial observability.

What carries the argument

The mechanism that carries the argument is the memory improvability gap: the difference in return between an agent fed partial observations and an agent fed a richer state representation (or an optimal belief policy), with the learning algorithm, network size, and other hyperparameters held fixed. The formal setting is the POMDP, whose observation function $\Phi : S \to \Delta\Omega$ maps hidden states to incomplete observations, and whose memory functions $\mu(h_t)$ condense the history $h_t = (o_t, a_t, \ldots, o_0, a_0)$ into a fixed-size memory state. The paper also contributes an eight-category taxonomy of partial observability—noisy state features, visual occlusion, object uncertainty and tracking, spatial uncertainty, moment features, unknown opposition, episode nonstationarity, and needle-in-a-haystack tasks—and selects at least one representative environment per category so the suite has coverage as well as signal. The gap is the instrument: a large floor-to-ceiling distance means added memory is what would close the distance, so an improvement by a memory-based algorithm is attributable to mitigating partial observability rather than to a better function approximator.

What would settle it

Retrain the full-state ceiling agents for RockSample(11,11), RockSample(15,15), velocity-only Walker and HalfCheetah, and no-inventory Crafter with the same memoryless MLP used for the observation-only floor; if the performance gap largely collapses or reverses, those environments' memory improvability is an artifact of function approximation power rather than a signal about partial observability.

Watch

Extended reading notes

Core claim

The paper's central claim is that all POBAX environments are memory improvable and require hard-to-learn memory functions. Memory improvability is the existence of a performance gap between an observations-only agent (the floor) and an agent that receives more state information (the ceiling), all other factors roughly equal; the paper verifies this gap with 30-seed experiments across T-Maze, RockSample(11,11), RockSample(15,15), Battleship, velocity-only Walker and HalfCheetah, two DeepMind Lab MiniGrid mazes, pixel-based Ant and HalfCheetah, and no-inventory Crafter. In every environment three memory-based algorithms—recurrent PPO, $\lambda$-discrepancy with recurrent PPO, and Transformer-XL—improve on the observation-only agent and land below the full-state agent, which the authors read as evidence that the gains come from mitigating partial observability rather than from function approximation or hyperparameters. The suite is implemented entirely in JAX, and the paper supplies recommended per-environment hyperparameters so the floor and ceiling can be reproduced out of the box.

Load-bearing premise

The load-bearing premise is that the gap between the observation-only floor and the full-state ceiling measures hidden information rather than architectural power, because in several environments the ceiling agent uses a recurrent or transformer network while the floor agent uses a memoryless multilayer perceptron.

Editorial extensions

If this is right

  • Researchers can benchmark new history-based algorithms out of the box: POBAX ships recommended hyperparameters for every environment, and the floor-to-ceiling gap is the declared measure of progress.
  • An algorithm that substantially closes the memory improvability gap has positive evidence that its gains come from mitigating partial observability, because the gap is constructed to hold architecture and hyperparameters fixed.
  • The gap size becomes a diagnostic: a large gap tells a practitioner that adding an RNN, transformer, or other memory mechanism is likely to pay off, while a small gap signals that memory is not the bottleneck.
  • Because every environment runs in JAX and scales on GPU, researchers can afford the large hyperparameter sweeps the paper shows are needed for reliable floor and ceiling curves.
  • The eight-category taxonomy gives benchmark designers a coverage checklist when assembling or extending their own partial-observability suites.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural extension is to make the memory-improvability test a gate for admitting new tasks to the benchmark: compute the floor-ceiling gap under a fixed architecture budget and keep only environments where the gap is large and positive.
  • The paper suggests, though it does not test, a stronger claim: an algorithm that performs well across all POBAX categories simultaneously—pixels, sparse rewards, long horizons, object tracking—should transfer better to real deployments, which usually mix several forms of partial observability at once.
  • A stricter benchmark variant would require the floor and ceiling to share the same network family; testing whether the improvability gaps survive that constraint would separate the information gap from the function-approximation gap.
  • If the paper's reasoning is right, a memory-algorithm result published without its floor and ceiling curves cannot be attributed to memory; making all three curves a reporting norm would make benchmark comparisons decisive.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper argues that a partial-observability benchmark should satisfy two properties: coverage of diverse forms of state aliasing, and a memory-improvability signal defined as a performance gap between agents with more versus less state information when other factors are held roughly equal. It introduces POBAX, a JAX-based suite containing T-Maze, RockSample(11,11), RockSample(15,15), Battleship, Walker-V, HalfCheetah-V, two DeepMind Lab MiniGrid mazes, two visual Mujoco tasks, and No-inventory Crafter. The paper categorizes partial observability into eight forms, reports ablations on the number of parallel environments and network width, recommends per-environment hyperparameters, and evaluates recurrent PPO, λ-discrepancy PPO, and Transformer-XL with 30-seed confidence intervals. The central claim is that all POBAX environments are memory improvable and require hard-to-learn memory functions.

Significance. If the central claim holds, POBAX would be a valuable community resource: it covers several distinct forms of partial observability, is implemented entirely in JAX for GPU-scalable experimentation, ships with recommended hyperparameters, and provides extensive baselines. The paper's strengths include the open-source release, the detailed appendices, the systematic hyperparameter sweeps, the 30-seed confidence intervals, and the explicit investigation of confounding factors such as parallel-env count and network width. The timing experiments in Appendix A also support the scalability argument. However, the suite-level memory-improvability claim is only as strong as the floor/ceiling comparisons, and several of those comparisons do not satisfy the paper's own 'all other factors roughly equal' condition.

major comments (3)
  1. [§6.1, §6.2, App. C.3/C.5/C.8] The central claim that all POBAX environments are memory improvable is not established for five of the configurations. For RockSample(11,11) and RockSample(15,15), Appendix C.3 states that the perfect-memory agent was trained with an RNN rather than a memoryless MLP; for Walker-V and HalfCheetah-V, Appendix C.5 states that an RNN was used for the fully observable results; and for No-inventory Crafter, Section 6.2 states that the 'full state' agent is a transformer trained on regular Crafter observations with the inventory included. The observation-only floors in Figure 6 are memoryless MLPs. Under the Section 4 definition, memory improvability requires a gap with 'all other factors roughly equal,' and an RNN or transformer ceiling does not satisfy that condition, since the gap may reflect greater function-approximation capacity rather than additional state information. The appendix disclosures do not repair the abstract's unqualified claim. Please rerun these ceilings with a memoryless MLP, or otherwise match the architecture between floor and ceiling, or restrict the memory-improvable claim to the configurations without this confound.
  2. [§6.2, Fig. 6, App. C.4] For Battleship, the green 'STATE' ceiling in Figure 6 is not a neural agent trained with more state information; it is the mean return of an optimal belief-state policy computed programmatically. The floor is a memoryless PPO agent trained from hit/miss observations. The gap between an optimal solver and a sample-based learner can be large even in the absence of partial observability, due to sample complexity, exploration difficulty, or optimization error. This violates the Section 4 requirement that other factors be roughly equal, in a different way from the architecture mismatch. Please add a trained ceiling using the same PPO learning setup with full-state or perfect-memory observations, and report the optimal belief policy as a separate reference line rather than as the 'STATE' ceiling.
  3. [Abstract, §6, Fig. 6] The abstract and Section 6 assert that the POBAX tasks require hard-to-learn memory functions, but the paper never defines or operationalizes hardness. The only supporting evidence offered is that the three memory-based algorithms improve over the observation-only floor and underperform the ceiling. Given the architecture and ceiling mismatches described above, that evidence does not establish hardness. Please provide an operational criterion for 'hard-to-learn'—for example, required memory length, information-theoretic complexity, or the number of training steps needed to close a fixed fraction of the gap—and report it per environment.
minor comments (6)
  1. [Abstract] The phrase 'performance of a agents' should be 'performance of agents'.
  2. [§3.2] The sentence 'In Figure 3 investigate the effect of network width' is missing a subject; it should read 'In Figure 3 we investigate'.
  3. [App. C.3] The sentence 'avoid all the back ones' appears to be a typo for 'avoid all the bad rocks.'
  4. [App. C.5] The exact masked features for each masked-control environment are not specified in the appendix; the text directs readers to the code repository. Since the information content of the floor and ceiling observations is central to the paper's claims, please list the observation vectors explicitly.
  5. [App. C.7] The architecture used for the fully observable ceiling in the visual Mujoco environments is not stated. Please specify explicitly whether that ceiling uses a memoryless MLP, an RNN, or a transformer.
  6. [§6.2] For No-inventory Crafter, the curve labeled 'STATE' in Figure 6 is actually the regular Crafter observation with the inventory visible, not the underlying environment state; please relabel this curve in the figure and caption to avoid confusion.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: the memory-improvability claims are empirical benchmark measurements, not derivations from fitted inputs or self-citations.

full rationale

The paper's central claim—that all POBAX environments are memory improvable—is an empirical measurement of a performance gap between baseline and ceiling agents, not a quantity derived from fitted parameters or from a self-citation. No equation reconstructs the gap by construction; the recommended hyperparameters are chosen so that baseline and skyline agents 'both learn' (Section 3.1), not to force a target gap. The λ-discrepancy algorithm (Allen et al., 2024) is co-authored by an author of this paper, but it is evaluated alongside recurrent PPO and Transformer-XL, and the benchmark's utility does not depend on λ-discrepancy winning. The appendix disclosures that the full-state ceiling for RockSample, Masked Mujoco, and no-inventory Crafter uses an RNN or transformer while the observation-only floor uses a memoryless MLP (C.3, C.5, C.8) are genuine confounds for the 'all other factors roughly equal' criterion, but they are validity concerns, not circular reductions: the gap is not constructed to equal a fitted value, and the paper does not hide the difference. There is no self-citation chain invoked as an external uniqueness theorem, and no known result is renamed as a derivation. The honest finding is no significant circularity.

Assumptions & free parameters 6 free parameters · 4 assumptions · 0 invented entities

The central claim depends on the recommended hyperparameters, which are tuned per environment and per algorithm (free parameters), and on the domain assumption that the performance gap between floor and ceiling agents is attributable to information rather than architecture. Since the ceiling often uses a different architecture, this assumption is not fully satisfied. No new entities are introduced.

free parameters (6)
  • per-environment hidden_size = 32 for T-Maze, 256 for RockSample 11 and masked control, 512 for RockSample 15, DMLab, visual control, Crafter
    Network width is hand-set per environment; Section 3.2 shows it changes performance, and the gaps are reported at these widths.
  • per-environment num_envs = 4 default; 8 for RockSample 11; 16 for RockSample 15; 32 for Battleship; 256 for DMLab-01 and Crafter; 512 for DMLab-02
    Number of parallel environments is chosen per task; Section 3.1 shows it changes the observation/state gap.
  • algorithm learning rate and GAE lambda = swept per algorithm, e.g., lr in {2.5e-3, 2.5e-4, 2.5e-5, 2.5e-6}, lambda0 in {0.1, 0.3, 0.5, 0.7, 0.9, 0.95}
    Selected by highest AUC on the benchmark environments, so reported algorithm curves are best-case after tuning.
  • lambda-discrepancy beta and lambda1 = beta in {0.25, 0.5}, lambda1 in {0.5, 0.7, 0.95}
    Algorithm-specific weights swept for the LD agent; they affect the LD results only.
  • entropy coefficient = 0.01 default; 0.05 for Battleship; 0.2 for RockSample
    Hand-adjusted per environment to encourage exploration; changes the floor and ceiling curves.
  • transformer embed size / qkv features = 256 default, 220 or 96 in some environments
    Chosen in part to match recurrent PPO parameter counts; affects transformer performance.
assumptions (4)
  • standard math The POMDP formalization with observation function Phi is the right model for the environments
    Section 2 defines the setting; standard in the RL literature and used as background.
  • domain assumption If agents with more state information outperform agents with less, under otherwise equal conditions, then memory can improve performance
    This is the memory improvability definition in Section 4. It assumes the gap is not caused by architecture, featurization, or learning dynamics.
  • domain assumption The eight listed categories of partial observability adequately span the forms relevant to RL benchmarks
    Section 5 says the list is not exhaustive and excludes unknown opposition and needle-in-a-haystack, so the coverage claim depends on this selection being representative.
  • domain assumption The failure of tested memory algorithms to fully close the floor-ceiling gap indicates the memory function is hard to learn
    Section 6.2 interprets the partially closed gap as evidence of hard-to-learn memory functions; no independent measure of difficulty is provided.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Benchmarking Partial Observability in Reinforcement Learning with a Suite of Memory-Improvable Domains." pith.science (2026). https://pith.science/paper/RDSNABQB

@misc{pith2026250800046,
  author       = {Pith},
  title        = {Pith review of: Benchmarking Partial Observability in Reinforcement Learning with a Suite of Memory-Improvable Domains},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/RDSNABQB}},
  note         = {Machine review of arXiv:2508.00046}
}
read the original abstract

Mitigating partial observability is a necessary but challenging task for general reinforcement learning algorithms. To improve an algorithm's ability to mitigate partial observability, researchers need comprehensive benchmarks to gauge progress. Most algorithms tackling partial observability are only evaluated on benchmarks with simple forms of state aliasing, such as feature masking and Gaussian noise. Such benchmarks do not represent the many forms of partial observability seen in real domains, like visual occlusion or unknown opponent intent. We argue that a partially observable benchmark should have two key properties. The first is coverage in its forms of partial observability, to ensure an algorithm's generalizability. The second is a large gap between the performance of a agents with more or less state information, all other factors roughly equal. This gap implies that an environment is memory improvable: where performance gains in a domain are from an algorithm's ability to cope with partial observability as opposed to other factors. We introduce best-practice guidelines for empirically benchmarking reinforcement learning under partial observability, as well as the open-source library POBAX: Partially Observable Benchmarks in JAX. We characterize the types of partial observability present in various environments and select representative environments for our benchmark. These environments include localization and mapping, visual control, games, and more. Additionally, we show that these tasks are all memory improvable and require hard-to-learn memory functions, providing a concrete signal for partial observability research. This framework includes recommended hyperparameters as well as algorithm implementations for fast, out-of-the-box evaluation, as well as highly performant environments implemented in JAX for GPU-scalable experimentation.

Figures

Figures reproduced from arXiv: 2508.00046 by the authors.

Figure 1
Figure 1. Masked continuous control online undiscounted returns for observations only (gray), full [PITH_FULL_IMAGE:figures/full_fig_p005_1.png] view at source ↗
Figure 2
Figure 2. (Left) Image of the DMLab Minigrid maze environment for [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Online undiscounted returns comparing network hidden sizes 32, 64 and 256 (left to right) [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: Different levels of observability in 5 × 5 Battleship. (Left) Observations in this version of Battleship are whether or not the previous action hit. (Middle) “Perfect memory” observability, where observations include all previous position hit and missed. Grayed out gri…
Figure 5
Figure 5. Figure 5: Pixel-based environments in POBAX. (Left) Ant and HalfCheetah in visual continuous [PITH_FULL_IMAGE:figures/full_fig_p009_5.png]
Figure 6
Figure 6. Figure 6: Performance across all POBAX domains. Experiments are run over 30 seeds, with shaded [PITH_FULL_IMAGE:figures/full_fig_p011_6.png]
Figure 7
Figure 7. Figure 7: Wall clock speeds over number of parallel environments run for 10M steps. Dashed verti [PITH_FULL_IMAGE:figures/full_fig_p016_7.png]
Figure 8
Figure 8. Figure 8: (Left to right, top to down) Three DeepMind Lab MiniGrid mazes, [PITH_FULL_IMAGE:figures/full_fig_p025_8.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

24 extracted references · 15 canonical work pages

  1. [1]

    fully observable

    Unless otherwise specified, these default parameters were used in every experiment. We also note that unless otherwise stated, the “fully observable” agent was trained with a memoryless MLP. We begin with a discussion on differences observed between our results and other benchmark results in partial observability, then elucidate the full details of each e...

    work page 2023

  2. [7]

    Then we rerun the experiments over 30 seeds using best hyperparameters. Step size λ0 λ1 β Fully Observable – – – – Memoryless 2.5 × 10−3 0.1 – – RNN 2.5 × 10−3 0.7 – – Transformer-XL 2.5 × 10−5 0.1 – – Lambda Discrepancy 2.5 × 10−3 0.1 0 .95 0 .5 Table 7: Battleship Best Hyperparameters C.5 Masked Continuous Control Masked continuous control are Mujoco en...

    work page 2012

  3. [8]

    At the beginning of every episode, both agent start state and goal state are randomly initialized

    These three mazes get increasingly complex and large. At the beginning of every episode, both agent start state and goal state are randomly initialized. Maximum number of episode steps is 2000, 4000 and 6000 for each maze, from lowest ID to highest ID. Observation Space One-hot first-person images of size(2, 3, 2), where the two channels represent the wal...

    work page 2000

  4. [9]

    Jaxmarl: Multi-agent rl environments in jax

    Alexander Rutherford, Benjamin Ellis, Matteo Gallici, Jonathan Cook, Andrei Lupu, Gardar Ing- varsson, Timon Willi, Akbir Khan, Christian Schroeder de Witt, Alexandra Souly, Saptarashmi Bandyopadhyay, Mikayel Samvelyan, Minqi Jiang, Robert Tjarko Lange, Shimon Whiteson, Bruno Lacerda, Nick Hawes, Tim Rocktaschel, Chris Lu, and Jakob Nicolaus Foerster. Jax...

    arXiv

  5. [10]

    The first two elements dictate which hallway the agent is in (reward up or reward down) and is only set at the start grid

    Observation Space The agent’s observation is a binary vector with 4 elements. The first two elements dictate which hallway the agent is in (reward up or reward down) and is only set at the start grid. The next element is 1 if the agent is in the hallway. The third elemnt is 1 if the agent is in the junction. Full Observation Space The full observation spa...

    work page 2025

  6. [13]

    As maze_id increases, maze complexity and size increases as well

    Reinforcement Learning Journal 2025 Figure 8: (Left to right, top to down) Three DeepMind Lab MiniGrid mazes, maze_id = 01, 02, 03 . As maze_id increases, maze complexity and size increases as well. Hyperparameter Step size {2.5 × 10−3, 2.5 × 10−4, 2.5 × 10−5, 2.5 × 10−6} λ0 {0.1, 0.5, 0.7, 0.9, 0.95} Table 10: DeepMind Lab MiniGrid Maze hyperparameters s...

    work page 2025

  7. [15]

    Our experiments consist of two steps

    based on a batch experimen- tation library written in JAX (Lu et al., 2022). Our experiments consist of two steps. First, we perform a hyperparameter sweep over all envi- ronments, using a small number of seeds. Then, we select the best hyperparameters based on the highest area under the curve (AUC) score. After selection, we rerun the best hyperparameter...

    work page 2022

  8. [19]

    Reinforcement Learning Journal 2025 Observation Space After each step, the agent only receives a single binary signal

    The environment terminates when all positions on the grid with a ship are hit. Reinforcement Learning Journal 2025 Observation Space After each step, the agent only receives a single binary signal. A 0 indicate no ship is hit and a 1 indicate the opposite. To simplify the learning process, we concatenate the agent’s last action to the observation. Since t...

    work page 2025

Show all 24 references
  1. [24]

    C.8 No-inventory Crafter No-inventory Crafter is a more partially observable variant of Crafter (Hafner, 2021)

    After the selection, we rerun the experiments over 30 seeds using best hyperparameters. C.8 No-inventory Crafter No-inventory Crafter is a more partially observable variant of Crafter (Hafner, 2021). This environ- ments was built on top of the Craftax framework (Matthews et al...

    2021

  2. [96]

    We refer to our codebase for the best hyperparameters se- lected for the full masked mujoco hyperparameter sweep

    We list the best hyperparameters for the Walker-V and HalfCheetah-V environments in Tables 8 and 9 as they appear in our benchmark. We refer to our codebase for the best hyperparameters se- lected for the full masked mujoco hyperparameter sweep. For both of these environments,...

    2023

  3. [1964]

    Proximal policy optimization algorithms

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint, abs/1707.06347,

    arXiv

  4. [1992]

    Leveraging procedural genera- tion to benchmark reinforcement learning

    Karl Cobbe, Christopher Hesse, Jacob Hilton, and John Schulman. Leveraging procedural genera- tion to benchmark reinforcement learning. arXiv preprint arXiv:1912.01588,

    1912 arXiv

  5. [1997]

    Tutorial on training recurrent neural networks, covering bppt, rtrl, ekf and the echo state network approach

    Herbert Jaeger. Tutorial on training recurrent neural networks, covering bppt, rtrl, ekf and the echo state network approach. GMD-Forschungszentrum Informationstechnik, 2002., 5,

    2002

  6. [1998]

    Open- Spiel: A framework for reinforcement learning in games

    Marc Lanctot, Edward Lockhart, Jean-Baptiste Lespiau, Vinicius Zambaldi, Satyaki Upadhyay, Julien Pérolat, Sriram Srinivasan, Finbarr Timbers, Karl Tuyls, Shayegan Omidshafiei, Daniel Hennes, Dustin Morrill, Paul Muller, Timo Ewalds, Ryan Faulkner, János Kramár, Bart De Vylder...

    1908 arXiv

  7. [1999]

    Eduardo Pignatelli, Jarek Liesen, Robert Tjarko Lange, Chris Lu, Pablo Samuel Castro, and Laura Toni

    Morgan Kaufmann Publishers Inc. Eduardo Pignatelli, Jarek Liesen, Robert Tjarko Lange, Chris Lu, Pablo Samuel Castro, and Laura Toni. Navix: Scaling minigrid environments with jax. arXiv preprint arXiv:2407.19396,

    arXiv

  8. [2004]

    Sutton and Andrew G

    Reinforcement Learning Journal 2025 Richard S. Sutton and Andrew G. Barto. Reinforcement Learning: An Introduction . A Bradford Book,

    2025

  9. [2005]

    Mujoco: A physics engine for model-based control

    Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems , pp. 5026–5033. IEEE,

    2012

  10. [2012]

    Gymnasium: A standard interface for reinforcement learning environments

    Mark Towers, Ariel Kwiatkowski, Jordan Terry, John U Balis, Gianluca De Cola, Tristan Deleu, Manuel Goulão, Andreas Kallinteris, Markus Krimmel, Arjun KG, et al. Gymnasium: A standard interface for reinforcement learning environments. arXiv preprint arXiv:2407.17032,

    arXiv

  11. [2016]

    Benchmarking the spectrum of agent capabilities.arXiv preprint arXiv:2109.06780,

    Danijar Hafner. Benchmarking the spectrum of agent capabilities.arXiv preprint arXiv:2109.06780,

    arXiv

  12. [2018]

    Maxime Chevalier-Boisvert, Bolun Dai, Mark Towers, Rodrigo de Lazcano, Lucas Willems, Salem Lahlou, Suman Pal, Pablo Samuel Castro, and Jordan Terry

    URL http: //github.com/jax-ml/jax. Maxime Chevalier-Boisvert, Bolun Dai, Mark Towers, Rodrigo de Lazcano, Lucas Willems, Salem Lahlou, Suman Pal, Pablo Samuel Castro, and Jordan Terry. Minigrid & miniworld: Modular & customizable reinforcement learning environments for goal-or...

    arXiv

  13. [2020]

    Deep recurrent q-learning for partially observable MDPs

    Matthew Hausknecht and Peter Stone. Deep recurrent q-learning for partially observable MDPs. In Proceedings of the 2015 American Association for Artificial Intelligence,

    2015

  14. [2021]

    Reinforcement Learning Journal 2025 Ian Goodfellow, Yoshua Bengio, and Aaron Courville

    URL http: //github.com/google/brax. Reinforcement Learning Journal 2025 Ian Goodfellow, Yoshua Bengio, and Aaron Courville. Deep Learning. MIT Press,

    2025

  15. [2022]

    Transformer-XL is a memory-augmented algorithm that extends from the conventional architecture of transformers by incorporating segment-level recurrence

    and we implement the λ-discrepancy algorithm following the implementation of (Allen et al., 2024). Transformer-XL is a memory-augmented algorithm that extends from the conventional architecture of transformers by incorporating segment-level recurrence. Our algorithm followed t...

    2024

  16. [2024]

    The dashed blue line represents the time it takes per environment for 10M steps when running 2000 environments on GPU

    im- plementation (orange). The dashed blue line represents the time it takes per environment for 10M steps when running 2000 environments on GPU. The scaling curve for the GPU accelerated runs are desirable for large-scale experimentation. In this setting, each run may run wit...

    2000

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.