Pith. sign in

REVIEW 5 major objections 5 minor 1 cited by

RLBenchNet: The Right Network for the Right Reinforcement Learning Task

T0 review · 5 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A controlled PPO benchmark of seven network architectures across nine environments claims that the right choice is task-dependent: MLPs for Markovian control, recurrent networks for moderate partial observability, and only long-context…

desk verdict Useful benchmark with real efficiency numbers, but the MemoryS11 'Mamba-2 wins' claim is confounded, not significant, and inconsistent with the paper's own text. read the letter →

arxiv 2505.15040 v1 pith:B4FGTPU3 submitted 2025-05-21 cs.LG

Ivan Smirnov , Shangding Gu This is my paper · ORCID
classification cs.LG
keywords ReinforcementLearningNeuralArchitectureBenchmarkPPOPartialObservabilityMemoryMambaTransformer-XLState-SpaceModels
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Reinforcement-learning agents usually pick their neural network by habit rather than by task. This paper tests six families of networks—MLP, LSTM, GRU, Transformer-XL, Gated Transformer-XL, and the state-space models Mamba and Mamba-2—under a fixed Proximal Policy Optimization (PPO) algorithm, with network sizes matched so that capacity is not the confound, across nine environments spanning continuous control, discrete games, partially observable control, and memory-intensive grid tasks. Its central claim is that the best architecture depends on the task's memory horizon: MLPs win or tie in fully observable Markovian control, recurrent networks hold up in moderately partial settings, and only the long-context models Transformer-XL, Gated Transformer-XL, and Mamba-2 solve the hardest long-horizon memory task, with Mamba-2 doing so at roughly 8× lower GPU memory than Transformer-XL. It also claims large efficiency gains for Mamba, which trains 4.5× faster than LSTM and 3.9× faster than GRU at comparable returns. If these results generalize, they give practitioners a concrete rule: start with an MLP, move to Mamba-2 when temporal dependencies appear, and reserve transformers for extreme memory demands.

What carries the argument

The carrying object is a controlled benchmark suite built on a single-file reference PPO implementation, with the policy network replaced by each candidate architecture and layer widths adjusted to keep parameter counts roughly equal across models. The protocol fixes the RL algorithm, training schedule, hardware, and evaluation procedure, and varies only the network, producing paired measurements of final return, learning-curve shape, training throughput, inference latency, and GPU memory use. The decisive mechanism is this pairing of task performance with resource cost, because it is what lets the authors convert raw scores into a performance-per-byte and performance-per-second ranking, and it is what supports the claim that Mamba-2 is not merely good at memory tasks but good at them cheaply.

What would settle it

Run the same nine environments with per-architecture hyperparameter optimization at a fixed total budget; if a recurrent model such as LSTM or GRU then matches or beats Mamba-2 on the Memory-S11 task, the paper's headline ranking is not robust. Alternatively, re-running the Mamba variants with hidden-state resets at episode boundaries would test whether state leakage explains their weak episodic results.

Watch

Extended reading notes

Core claim

The central discovery is a task–architecture compatibility map for PPO agents. Across the nine environments, the authors find that added architectural complexity pays off only when the task's memory horizon demands it. Plain and frame-stacked MLPs match or beat more complex models in smooth continuous control and a reactive deterministic game; LSTM and GRU lead in moderately partial tasks such as masked CartPole and Breakout, where short temporal structure matters; Transformer-XL leads in a more complex partially observable control task; and in the most demanding long-horizon memory task, Memory-S11, only Transformer-XL, Gated Transformer-XL, and Mamba-2 learn meaningfully. Mamba-2 reaches near-optimal reward while using about 8× less GPU memory than Transformer-XL, making it the paper's recommended default for sequence-heavy RL. A secondary result is that the original Mamba is fast but inconsistent—strong in one memory task and unstable in others—which the authors attribute partly to missing hidden-state resets at episode boundaries.

Load-bearing premise

The rankings rest on the assumption that the chosen hyperparameters—mostly defaults, with a reduced learning rate for Mamba and task-specific settings for transformers—do not systematically favor one architecture. A different tuning budget per architecture could change which network wins each task.

Editorial extensions

If this is right

  • In Markovian control tasks, starting with an MLP is the recommended default: it matches or exceeds recurrent and attention-based models at a fraction of the training cost.
  • For tasks with short- or medium-term partial observability, LSTM and GRU remain strong choices, and Mamba-2 offers comparable returns while training about 4.5× faster than LSTM.
  • For long-horizon memory tasks, the practical shortlist narrows to Transformer-XL, Gated Transformer-XL, and Mamba-2, with Mamba-2 providing the best performance-to-memory ratio.
  • Gating in Gated Transformer-XL produces smoother, lower-variance learning curves than Transformer-XL but not a large final-performance advantage, so gating alone does not decide the choice.
  • The paper's results challenge the default assumption that larger or more expressive architectures are inherently better for RL; simpler models are preferable unless the environment demonstrably requires long-range memory.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The reported rankings are tied to the paper's tuning budget: hyperparameters are mostly defaults from the reference PPO implementation, with a reduced learning rate for Mamba and some hand-set transformer settings, so a version of this benchmark with equal per-architecture hyperparameter optimization could reorder the winners.
  • The Mamba implementation does not reset its hidden state across episode boundaries, and the paper flags this as a possible source of state leakage; fixing the reset could improve Mamba's results in episodic tasks and close some of the gap to Mamba-2.
  • Because all experiments use on-policy PPO, the architecture ranking may not carry over to off-policy methods such as SAC or TD3; testing the top architectures under those algorithms would reveal whether the map is algorithm-independent.
  • If the 8× memory advantage of Mamba-2 holds as model and batch sizes grow, state-space networks become the practical route to long-memory RL on memory-limited hardware, where transformer footprints are prohibitive.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. This paper benchmarks PPO agents with seven neural network architectures (MLP with one or four stacked observations, LSTM, GRU, Transformer-XL, GTrXL, Mamba, Mamba-2) across nine environments spanning continuous control, Atari, partially observable classic control, and MiniGrid memory tasks. The study controls parameter counts approximately, uses CleanRL defaults with a reduced learning rate for Mamba models, and measures return, throughput, latency, and GPU memory. The central claims are: (1) MLPs are strong in Markovian continuous control; (2) LSTM/GRU are robust in moderate partial observability; (3) Mamba variants are roughly 4–5× more efficient than LSTM/GRU; and (4) only Transformer-XL, GTrXL, and Mamba-2 solve the hardest long-memory task (MiniGrid Memory-S11), with Mamba-2 best overall and 8× more memory-efficient than Transformer-XL. The paper concludes with practitioner guidelines and suggestions for future work.

Significance. If the main comparative claims were fully supported, the paper would be a useful addition to the emerging literature on architecture selection in RL, especially for practitioners weighing state-space models against recurrent and attention-based memory. The work ships single-file CleanRL-based implementations and makes code available, and the efficiency measurements (throughput, latency, memory) are concrete and potentially reusable. The qualitative pattern that sequence-oriented architectures matter mainly in memory-demanding tasks, while MLPs remain competitive in Markovian domains, is consistent with prior work and is a plausible and valuable empirical message. However, the benchmark's scientific value is currently gated by unresolved confounds in the Mamba implementation and by inconsistencies between the text and the reported tables, so the contribution is real but provisional.

major comments (5)
  1. [Section 3.1, 3.3, 4.8 and Table 3] The headline claim that Mamba-2 is the best architecture on MiniGrid Memory-S11 is confounded. Section 3.3 states that the learning rate was reduced for Mamba-based models relative to all other architectures, and Section 4.8 admits the Mamba implementation has no episode-boundary hidden-state reset, 'potentially affecting performance in episodic tasks'. The Memory-S11 comparison in Table 3 (0.96±0.01 for Mamba-2 vs 0.93±0.11 for GTrXL and 0.88±0.18 for TrXL, 8 seeds) rests on overlapping intervals, and no significance test is reported. Because the abstract and Section 4.6 base their strongest claims on this comparison, the paper needs either a reset-equipped Mamba baseline, an equal-learning-rate control, or a statistical analysis demonstrating the differences are reliable.
  2. [Abstract, Section 4.4, and Section 4.6] There is a direct internal contradiction about which architectures solve Memory-S11. The abstract says only Transformer-XL, GTrXL, and Mamba-2 'successfully solve' the task; Section 4.4 says the same; but Section 4.6 says 'in long-horizon tasks only Mamba-2 and Transformer-XL can effectively solve tasks requiring extensive memory'. The paper needs a consistent criterion for 'solving' and a single, justified statement of which architectures meet it; otherwise the central finding is not unambiguously defined.
  3. [Section 4.1 and Table 3] The text claims in Finding 1 that 'Mamba has the worst performance across all environments', but Table 3 shows Mamba outperforming several architectures in multiple environments: on Memory-S11 it is tied with LSTM and PPO-4 at 0.49, on DoorKey it is best (0.85 vs 0.84 for GTrXL and 0.73 for TrXL), and on Pong it scores 20.82, comparable to the top models. This undermines the credibility of the prose summaries as a faithful representation of the data and needs correction.
  4. [Section 3.1 and Table 4] The claim of 'approximately equal parameter counts' is not met for several environments. For example, in Pong-v5 the parameter counts range from 2271.7k (GRU) to 2805.3k (Mamba-2), a ratio of 1.23, and in CartPole the GTrXL count of 264.5k is 1.17× the Mamba-2 count of 226.0k. The table also omits PPO-4 parameter counts entirely (Table 4 has no PPO-4 column despite the main text saying PPO-4 is used throughout). Authors should report PPO-4 parameters and either tighten the capacity matching or state that capacity differences of this magnitude are acceptable.
  5. [Section 4.5, Table 1, and Table 3] The abstract and Section 4.5 state that Mamba is '4.5x faster than LSTM and 3.9x faster than GRU' while maintaining 'comparable performance', but the measured throughputs in Table 1 (2734 SPS for Mamba vs 604 for LSTM and 701 for GRU) actually contradict the reported table values. The first row of Table 1 gives MiniGrid-MemoryS11 as 802/924 for LSTM/GRU, and the claimed averages cannot be derived from the listed entries; the LSTM/GRU row values shown (604/701) appear inconsistent with the detailed per-environment columns. This numerical inconsistency affects a headline efficiency claim and must be corrected.
minor comments (5)
  1. [Section 3.3] The phrase 'reduced following recommendations from recent literature [19]' is not sufficient; the exact reasoning for choosing 1.5e-4 and whether the same reduced rate was considered fair for all architectures should be stated, since Table 10 shows this rate is only applied to Mamba and Mamba-2.
  2. [Section 4.4 and Figure 4] The caption of Figure 4 says 'only Transformer-XL and Mamba-2 achieve meaningful learning', which contradicts the bullet point and Section 4.4 saying GTrXL also achieves meaningful learning; the figure caption and text should agree.
  3. [Section 4.1] The numbers quoted in the prose (MLP ~3250 in Walker2d, LSTM ~3350 in HalfCheetah) do not match Table 3 (3379.3 for Walker2d and 3997.4 for LSTM HalfCheetah); either the text should cite the table values consistently or the figures should be referenced with approximate values.
  4. [Table 7] Table 7 shows the 'GPU Memory Reserved' for Mamba-2 in classic control as 0.512–0.541 GB, which is sharply larger than the corresponding LSTM/GRU/Mamba reserved memory (0.049–0.068 GB); this is not discussed in Section 4.5, which instead emphasizes only average allocated memory.
  5. [Section 4.2 and Figure 2] The Pong results in Figure 2 are mislabeled with a caption saying 'Mamba and MLP with frame stacking excel' and the y-axis label '20 10 0 10 20' appears truncated or misformatted; the label should be fixed to show the full range.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: this is an empirical benchmark whose findings summarize measured returns and throughput, not a derivation that reduces to its own inputs.

full rationale

The paper is an empirical architecture benchmark, not a derivation. Claims such as 'MLPs excel in fully observable continuous control tasks' and 'only Transformer-XL, Gated Transformer-XL, and Mamba-2 successfully solve the most challenging memory-intensive tasks' are summaries of measured returns and throughput reported in Tables 1-3 and Figures 1-4. There is no equation that is defined in terms of a claimed result, no fitted parameter renamed as a prediction, and no uniqueness theorem imported from the authors' own prior work to force a conclusion. The authors' self-citations ([11], [12]) are background references on safe multi-agent RL and constrained multi-objective RL and play no role in the benchmark's conclusions. The skeptical concerns raised in the manuscript itself are explicitly stated as limitations rather than smuggled assumptions: Section 3.1 discloses that 'Mamba models run ... without resetting at episode boundaries' and Section 4.8 states that this 'potentially affecting performance in episodic tasks' is 'another important technical improvement' for future work. Likewise, Section 4.8 concedes that 'a more thorough hyperparameter optimization process ... could provide a fairer comparison across architectures,' which addresses the weaker-assumption concern directly rather than hiding it. Inconsistencies between the abstract ('only Transformer-XL, Gated Transformer-XL, and Mamba-2') and Section 4.6 ('only Mamba-2 and Transformer-XL') are internal-reporting consistency issues and statistical-significance concerns, which fall under correctness risk, not circularity under the specified criteria. The practical guidelines in Section 4.7 are a restatement of observed results; restating one's own measurements as advice is not a circular derivation because the advice is not used as evidence for the measurements. No step in the paper's reasoning reduces an output to an input by construction, so the appropriate finding is no significant circularity with score 0.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The paper's claims rest on a set of hyperparameter and implementation choices (learning rate adjustments, hidden dims, memory lengths, no state reset for Mamba) rather than on fitted parameters in a derivation. These choices are reasonable but not fully controlled, so they limit the strength of the comparative conclusions.

free parameters (4)
  • Mamba/Mamba-2 learning rate = 1.5e-4 (MiniGrid, Classic, Atari), 3e-4 (MuJoCo)
    Reduced from CleanRL default 2.5e-4 following [19]; the paper calls this the only notable hyperparameter exception, but it is applied only to Mamba models.
  • Transformer-XL/GTrXL memory length = 119 (MiniGrid), 64 (MuJoCo), 32 (Classic), 64 (Atari)
    Set based on prior work or task complexity rather than a grid search; directly controls how much history the model can use.
  • Hidden dimensions per architecture/domain = see Table 10 (e.g., 512/90/256/512 for PPO-4; 380/70/284/450 for Mamba)
    Adjusted to approximately match parameter counts (Table 4), but the match is approximate (up to ~17% deviation in MuJoCo), leaving residual capacity differences.
  • Entropy coefficient for TrXL/GTrXL = 0.01 across all domains
    Higher than the 0/1e-4 used for other architectures; different exploration incentives are a confound.
assumptions (5)
  • domain assumption PPO is representative of RL algorithms for architecture benchmarking
    Only PPO is tested; architecture conclusions may not transfer to off-policy methods (SAC, TD3), which the paper lists as future work.
  • domain assumption Approximately equal parameter counts make capacity differences negligible
    Table 4 shows deviations up to 17% (e.g., MuJoCo TrXL 37.8k vs Mamba-2 44.4k), so capacity is matched only loosely.
  • domain assumption The nine environments are representative of continuous control, discrete decisions, and memory demands
    No results on other benchmark families (DM Control, ProcGen, POPGym); scope is limited to these environments.
  • domain assumption CleanRL defaults are an appropriate common baseline for all architectures
    Defaults are tuned for MLP PPO; applying them to recurrent/transformer models could disadvantage those architectures.
  • ad hoc to paper Mamba without episode-boundary hidden-state reset is a fair evaluation of the architecture
    Section 3.1 states Mamba runs 'without resetting at episode boundaries'; Section 4.8 acknowledges this 'may affect performance in episodic tasks'. The main results do not correct for this.

how reviews work

0 comments
Cite this review

Pith. "Pith review of RLBenchNet: The Right Network for the Right Reinforcement Learning Task." pith.science (2026). https://pith.science/paper/B4FGTPU3

@misc{pith2026250515040,
  author       = {Pith},
  title        = {Pith review of: RLBenchNet: The Right Network for the Right Reinforcement Learning Task},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/B4FGTPU3}},
  note         = {Machine review of arXiv:2505.15040}
}
read the original abstract

Reinforcement learning (RL) has seen significant advancements through the application of various neural network architectures. In this study, we systematically investigate the performance of several neural networks in RL tasks, including Long Short-Term Memory (LSTM), Multi-Layer Perceptron (MLP), Mamba/Mamba-2, Transformer-XL, Gated Transformer-XL, and Gated Recurrent Unit (GRU). Through comprehensive evaluation across continuous control, discrete decision-making, and memory-based environments, we identify architecture-specific strengths and limitations. Our results reveal that: (1) MLPs excel in fully observable continuous control tasks, providing an optimal balance of performance and efficiency; (2) recurrent architectures like LSTM and GRU offer robust performance in partially observable environments with moderate memory requirements; (3) Mamba models achieve a 4.5x higher throughput compared to LSTM and a 3.9x increase over GRU, all while maintaining comparable performance; and (4) only Transformer-XL, Gated Transformer-XL, and Mamba-2 successfully solve the most challenging memory-intensive tasks, with Mamba-2 requiring 8x less memory than Transformer-XL. These findings provide insights for researchers and practitioners, enabling more informed architecture selection based on specific task characteristics and computational constraints. Code is available at: https://github.com/SafeRL-Lab/RLBenchNet

Figures

Figures reproduced from arXiv: 2505.15040 by the authors.

Figure 1
Figure 1. Average returns for MuJoCo tasks. MLP and LSTM demonstrate competitive or superior [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Average returns across random seeds for Atari environments. Mamba and MLP with frame [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Average returns across seeds for masked classic control tasks. Recurrent architectures and [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Average returns across random seeds for MiniGrid environments. In DoorKey-8x8, original [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Reward Structure Shapes the Interaction Between Episodic Exploration and Neural Memory in Reinforcement Learning

    cs.LG 2026-08 conditional novelty 7.0 of 10

    An identical exploration bonus amplifies, equalizes, or has no effect on memory architectures depending on whether the task requires active discovery, a single reward-supervised cue, or follows a fixed schedule.

Reference graph

Works this paper leans on

30 extracted references · 20 canonical work pages · cited by 1 Pith paper

  1. [1]

    What matters in on-policy reinforcement learning? a large-scale empirical study, 2020

    Marcin Andrychowicz, Anton Raichuk, Piotr Sta´nczyk, Manu Orsini, Sertan Girgin, Raphael Marinier, Léonard Hussenot, Matthieu Geist, Olivier Pietquin, Marcin Michalski, Sylvain Gelly, and Olivier Bachem. What matters in on-policy reinforcement learning? a large-scale empirical study, 2020

    work page 2020

  2. [2]

    M. G. Bellemare, Y . Naddaf, J. Veness, and M. Bowling. The arcade learning environment: An evaluation platform for general agents.Journal of Artificial Intelligence Research, 47:253–279, June 2013

    2013

  3. [3]

    Aydar Bulatov, Yuri Kuratov, Yermek Kapushev, and Mikhail S. Burtsev. Scaling transformer to 1m tokens and beyond with rmt, 2024

    work page 2024

  4. [4]

    Kovalev, and Aleksandr I

    Egor Cherepanov, Alexey Staroverov, Dmitry Yudin, Alexey K. Kovalev, and Aleksandr I. Panov. Recurrent action transformer with memory, 2024

    work page 2024

  5. [5]

    Minigrid & miniworld: Modular & customizable reinforcement learning environments for goal-oriented tasks, 2023

    Maxime Chevalier-Boisvert, Bolun Dai, Mark Towers, Rodrigo de Lazcano, Lucas Willems, Salem Lahlou, Suman Pal, Pablo Samuel Castro, and Jordan Terry. Minigrid & miniworld: Modular & customizable reinforcement learning environments for goal-oriented tasks, 2023

    work page 2023

  6. [6]

    Empirical evaluation of gated recurrent neural networks on sequence modeling, 2014

    Junyoung Chung, Caglar Gulcehre, KyungHyun Cho, and Yoshua Bengio. Empirical evaluation of gated recurrent neural networks on sequence modeling, 2014

    2014

  7. [7]

    Le, and Ruslan Salakhutdinov

    Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoc V . Le, and Ruslan Salakhutdinov. Transformer-xl: Attentive language models beyond a fixed-length context, 2019

    2019

  8. [8]

    Transformers are ssms: Generalized models and efficient algorithms through structured state space duality, 2024

    Tri Dao and Albert Gu. Transformers are ssms: Generalized models and efficient algorithms through structured state space duality, 2024. 9

    work page 2024

Show all 30 references
  1. [9]

    Addressing function approximation error in actor-critic methods

    Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error in actor-critic methods. InProceedings of the 35th International Conference on Machine Learning (ICML), pages 1587–1596. PMLR, 2018

    2018

  2. [10]

    Mamba: Linear-time sequence modeling with selective state spaces, 2024

    Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces, 2024

    2024

  3. [11]

    Safe multi-agent reinforcement learning for multi-robot control.Artificial Intelligence, 319:103905, 2023

    Shangding Gu, Jakub Grudzien Kuba, Yuanpei Chen, Yali Du, Long Yang, Alois Knoll, and Yaodong Yang. Safe multi-agent reinforcement learning for multi-robot control.Artificial Intelligence, 319:103905, 2023

    2023

  4. [12]

    Safe and balanced: A framework for constrained multi-objective reinforcement learning.IEEE Transactions on Pattern Analysis and Machine Intelligence, 2025

    Shangding Gu, Bilgehan Sel, Yuhao Ding, Lu Wang, Qingwei Lin, Alois Knoll, and Ming Jin. Safe and balanced: A framework for constrained multi-objective reinforcement learning.IEEE Transactions on Pattern Analysis and Machine Intelligence, 2025

    2025

  5. [13]

    Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor, 2018

    Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor, 2018

    2018

  6. [14]

    Long short-term memory.Neural computation, 9(8):1735–1780, 1997

    Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory.Neural computation, 9(8):1735–1780, 1997

    1997

  7. [15]

    The 37 implementation details of proximal policy optimization

    Shengyi Huang, Rousslan Fernand Julien Dossa, Antonin Raffin, Anssi Kanervisto, and Weixun Wang. The 37 implementation details of proximal policy optimization. InICLR Blog Track,

  8. [16]

    Shengyi Huang, Rousslan Fernand Julien Dossa, Chang Ye, Jeff Braga, Dipam Chakraborty, Kinal Mehta, and João G.M. Araújo. Cleanrl: High-quality single-file implementations of deep reinforcement learning algorithms.Journal of Machine Learning Research, 23(274):1–18, 2022

    2022

  9. [17]

    Decision mamba: Reinforcement learning via hybrid selective sequence modeling, 2024

    Sili Huang, Jifeng Hu, Zhejian Yang, Liwei Yang, Tao Luo, Hechang Chen, Lichao Sun, and Bo Yang. Decision mamba: Reinforcement learning via hybrid selective sequence modeling, 2024

    2024

  10. [18]

    Du, and Huazhe Xu

    Chenhao Lu, Ruizhe Shi, Yuyao Liu, Kaizhe Hu, Simon S. Du, and Huazhe Xu. Rethinking transformers in solving pomdps, 2024

    2024

  11. [19]

    Efficient recurrent off-policy RL requires a context-encoder-specific learning rate

    Fan-Ming Luo, Zuolin Tu, Zefang Huang, and Yang Yu. Efficient recurrent off-policy RL requires a context-encoder-specific learning rate. InThe Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024

    2024

  12. [20]

    Decision mamba: A multi-grained state space model with self-evolution regularization for offline rl, 2025

    Qi Lv, Xiang Deng, Gongwei Chen, Michael Yu Wang, and Liqiang Nie. Decision mamba: A multi-grained state space model with self-evolution regularization for offline rl, 2025

    2025

  13. [21]

    Popgym: Benchmarking partially observable reinforcement learning, 2023

    Steven Morad, Ryan Kortvelesy, Matteo Bettini, Stephan Liwicki, and Amanda Prorok. Popgym: Benchmarking partially observable reinforcement learning, 2023

    2023

  14. [22]

    When do transformers shine in rl? decoupling memory from credit assignment, 2023

    Tianwei Ni, Michel Ma, Benjamin Eysenbach, and Pierre-Luc Bacon. When do transformers shine in rl? decoupling memory from credit assignment, 2023

    2023

  15. [23]

    Decision mamba: Reinforcement learning via sequence modeling with selective state spaces, 2024

    Toshihiro Ota. Decision mamba: Reinforcement learning via sequence modeling with selective state spaces, 2024

    2024

  16. [24]

    Francis Song, Jack W

    Emilio Parisotto, H. Francis Song, Jack W. Rae, Razvan Pascanu, Caglar Gulcehre, Siddhant M. Jayakumar, Max Jaderberg, Raphael Lopez Kaufman, Aidan Clark, Seb Noury, Matthew M. Botvinick, Nicolas Heess, and Raia Hadsell. Stabilizing transformers for reinforcement learning, 2019

    2019

  17. [25]

    Transformerxl as episodic memory in proximal policy optimization.Github Repository, 2023

    Marco Pleines, Matthias Pallasch, Frank Zimmer, and Mike Preuss. Transformerxl as episodic memory in proximal policy optimization.Github Repository, 2023

    2023

  18. [26]

    Memory gym: Towards endless tasks to benchmark memory capabilities of agents, 2024

    Marco Pleines, Matthias Pallasch, Frank Zimmer, and Mike Preuss. Memory gym: Towards endless tasks to benchmark memory capabilities of agents, 2024. 10

    2024

  19. [27]

    Proximal policy optimization algorithms, 2017

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms, 2017

    2017

  20. [28]

    Mas- tering the game of go with deep neural networks and tree search.nature, 529(7587):484–489, 2016

    David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den Driess- che, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, et al. Mas- tering the game of go with deep neural networks and tree search.nature, 529(7587):484–489, 2016

    2016

  21. [29]

    Mujoco: A physics engine for model-based control

    Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In2012 IEEE/RSJ international conference on intelligent robots and systems, pages 5026–5033. IEEE, 2012. 11 A Performance Metrics. Table 1: Final Steps Per Second (SPS) for Various Ar...

    2012

  22. [2022]

    https://iclr-blog-track.github.io/2022/03/25/ppo-implementation-details/

    2022

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.