Pith. sign in

REVIEW 4 major objections 6 minor 61 references

Bi-directional Recurrence Improves Transformer in Partially Observable Markov Decision Processes

T0 review · 4 major / 6 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper argues that replacing a transformer's feed-forward network with a single bidirectional GRU layer produces a more sample-efficient and 25% smaller Q-network for partially observable RL, with success-rate improvements of 87–482%…

desk verdict A plausible architectural tweak with an unreproducible headline number and an unaddressed causality question; worth a serious referee after the authors fix the evaluation. read the letter →

arxiv 2505.11153 v1 pith:SPXRYCLC submitted 2025-05-16 cs.LG

Ashok Arora , Neetesh Kumar This is my paper · ORCID
classification cs.LG
keywords partiallyobservableMarkovdecisionprocessesreinforcementlearningtransformerbidirectionalgatedrecurrentunitsampleefficiencyQ-networkgridverseneuralnetworks
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Partially observable environments force an RL agent to remember and recombine past observations, and transformers deliver that memory at a high parameter cost. This paper claims that the transformer's multi-layer feed-forward block can be replaced by a single layer of bidirectional gated recurrent units, producing the Deep BiGRUformer Q-Network (DBGFQN). Across 23 POMDP benchmarks, the network is reported to outperform transformer-, attention-, and recurrence-based baselines by 87.39% to 482.04% on average, while cutting total parameters by about 25%. If true, this would make memory-equipped RL agents substantially more parameter-efficient and improve their sample efficiency in partially observable settings.

What carries the argument

The load-bearing mechanism is the BiGRUformer encoder block: self-attention over a context of K observations, followed by layer normalization, then a single bidirectional GRU that reads the sequence forward and backward, followed by residual addition and a second layer normalization, with a final linear map to Q-values. The BiGRU's two hidden-state updates — reset gate, update gate, and candidate state in both directions — let the network combine forward context (past observations) with backward context (later observations in the context window) before Q-values are computed. It works because the backward pass acts as a learned attention-like summarizer over the whole window, while the forward pass preserves causal order; together they refine the self-attention representation before the value head.

What would settle it

Re-run the 23-environment benchmark with a strictly causal variant of DBGFQN — the same BiGRU block but with the backward pass masked so it never sees observations after the decision timestep — and check whether the 87.39% average gain over DTQN survives; if it collapses, the reported margins depend on future leakage rather than on genuine memory.

Watch

Extended reading notes

Core claim

The central claim is that feed-forward expansion inside a transformer Q-network is not necessary for partial observability: a single BiGRU layer placed after multi-head self-attention captures the temporal dependencies that POMDP agents need, and does so better than the feed-forward blocks it replaces. The paper further claims that this design beats DTQN by 87.39%, DGFQN by 96.14%, and DRQN by 482.04% on average over 23 environments, and that bidirectional recurrence matters most in environments with dense structure and strong conditional dependencies, such as Gridverse Memory Four Rooms and Keydoor. The authors therefore conclude that POMDPs divide into classes for which different architectural choices are appropriate, and that large feed-forward expansions can be abandoned in favor of a compact recurrent layer.

Load-bearing premise

The load-bearing premise is that the agent's action is chosen from Q-values computed only at the final timestep of the BiGRU-processed context; if intermediate outputs are used, the backward GRU leaks future observations into the decision and the reported gains would not reflect a valid POMDP policy.

Editorial extensions

If this is right

  • POMDP agents can get transformer-level memory at roughly 25% fewer parameters, making memory-equipped policies more practical for edge devices and real-time control.
  • The standard 4x embedding-dimension feed-forward expansion is not essential; a single recurrent layer can replace it, which may simplify transformer design in RL.
  • Performance differences across the 23 environments indicate that POMDPs should be classified by structural density and conditional dependencies when choosing between recurrent and non-recurrent architectures.
  • Artificially adding structure (hallucinated rooms) to a memory-light grid improves base-model performance, suggesting that environment structure itself drives the benefit of recurrence.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the Q-values for action selection are read only from the final timestep of the BiGRU-processed context, the backward pass is safely causal; this is the natural deployment rule, but the paper should state it explicitly because intermediate-timestep outputs would leak future observations.
  • The claimed margins of 87.39% and 482.04% depend on an unspecified aggregation rule across 23 environments; recomputing them as medians or per-environment geometric means would show whether the gains are consistent or concentrated in a few tasks.
  • The hallucination experiment suggests a practical diagnostic: injecting fake structure into a new POMDP could predict whether a recurrent agent will beat a feed-forward one before training the full model.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The manuscript proposes DBGFQN, a deep Q-network that replaces the feed-forward network in a transformer encoder with a single bidirectional GRU layer, and evaluates it on 23 POMDP-style environments against DRQN, DTQN, and several uni- and bi-directional recurrent variants. The authors claim that this architecture swap improves success rates by margins ranging from 87.39% to 482.04% on average across the 23 environments and reduces the parameter count by about 25% compared with traditional transformers. The paper also reports ablations addressing five research questions about the necessity of feed-forward expansion, the choice of recurrent layer, environment-dependent behavior, and a hallucinated-structure experiment.

Significance. The question of whether a recurrent layer can replace the feed-forward network in transformer-based POMDP agents is timely and practically relevant, and the paper contains useful infrastructure: a 23-environment benchmark, forward-pass pseudocode, and a hyperparameter table. If the causal-inference ambiguity were resolved and the reported margins were reproducible from the tables, the architecture would be a meaningful contribution. As presented, however, the central quantitative claims are not substantiated: the bidirectional GRU as described may leak future observations into the Q-values used for action selection, and the headline improvement figures are not reproducible from Table 1. The manuscript therefore needs substantial clarification and re-analysis before its claims can be assessed.

major comments (4)
  1. [§3, Appendix Eqs. (12), (14), (16), (18), Algorithm 1] The bidirectional GRU is defined in the Appendix with a backward pass whose hidden state at time t depends on h_{t+1}, and Algorithm 1 applies self.biGRU(x) to the entire context window x in a single call. The text and Figure 1 show Q-values at every timestep (Eq. (23), Q_t), so if the action at time t is selected using Q_t computed from the full context window, the backward pass leaks information from observations O_{t+1..K}. The paper never states that actions are selected only from the final token of the BiGRU output or that a causal mask is applied. This is a load-bearing ambiguity: without a causal inference procedure, the reported gains may be an artifact of lookahead rather than improved memory. The authors must specify the exact inference-time protocol and empirically verify that no future information is used.
  2. [§5.3 and Table 1] The headline claim of an 87.39% average improvement over DTQN is not reproducible from Table 1. Averaging the success rates over the 23 environments gives approximately 0.466 for DBGFQN versus 0.365 for DTQN, a relative improvement of roughly 28%, not 87.39%. The paper does not state whether the reported margin is a mean of per-environment ratios, a ratio of means, a percentage-point difference, or some other aggregation. The authors should report the aggregation formula, per-environment margins, standard errors, and either the code or raw results needed to verify the 87.39% and 482.04% figures.
  3. [§1 and Table 3] The claim that DBGFQN reduces the overall parameter count by 25% compared with traditional transformers is not supported by Table 3, which lists parameter counts for DTQN1-DTQN3, DRFQN, DLFQN, DGFQN, DBRFQN, and DBLFQN, but no DBGFQN column. Moreover, the DGFQN counts in Table 3 (e.g., 97,285 for Hallway) are comparable to DTQN3 (97,029), not 25% smaller. The authors should add the DBGFQN parameter count and state exactly which baseline and configuration yields the claimed 25% reduction.
  4. [§4 and §5.3] The paper claims improved sample efficiency, but Table 1 reports only a final mean success rate and the figures are not described in enough detail to show learning curves or fixed-budget comparisons. A single scalar success rate cannot demonstrate that DBGFQN reaches a given performance level with fewer interactions. The authors should present learning curves or interaction-budget comparisons, ideally with variance across seeds, to support the sample-efficiency claim.
minor comments (6)
  1. [Figure captions] The captions for Figures 5-7 contain a typo ('Comparitive') and the environment ranges are inconsistent: Figure 7 is titled 'environments 7-23' while the preceding figures cover 1-8 and 9-16; please clarify the intended ranges.
  2. [Appendix Eq. (7)] The softmax notation in Eq. (7) is unclear: the summation index and upper limit appear garbled (P over O_{t+1} o=1), and the notation should be rewritten as a standard softmax over the context positions.
  3. [§5.2 and §2] There are grammatical errors, including 'the feed-forward layers was replaced' in §5.2 and 'have also gaining popularity' in §2; these should be corrected.
  4. [Table 3] Table 3 repeats identical parameter counts for different Gridverse sizes (e.g., GV 5x5, GV 7x7, GV 9x9), which is confusing; clarify whether the counts are environment-specific or the same architecture weights.
  5. [§5.3 and §5.4] The paper introduces RQ4 but then moves to RQ5 without an explicit subsection that answers RQ4; the discussion in §5.3 should be explicitly connected to RQ4.
  6. [§5.4] The hallucination experiment is described only qualitatively; please report the actual numbers behind Figure 8 and clarify whether the same artificial-room layout and training budget were used for all models compared.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation: the paper's claims are empirical architecture comparisons against external baselines, with no fitted input renamed as prediction and no load-bearing self-citation chain.

full rationale

The central claims are empirical comparisons against external baselines (DTQN, DRQN, DGFQN) and an architectural ablation. Walking the derivation chain, the model equations (Eqs. 1-24) simply define a transformer encoder whose FFN is replaced by a BiGRU; no prediction is obtained by fitting a parameter to the benchmark numbers and then renaming the fit as a result. The reported margins (87.39% over DTQN, 482.04% over DRQN) are claimed from Table 1, and while I did not verify the aggregation rule from the table, that is a reproducibility/statistics concern, not circularity. There are no load-bearing self-citations: references [23], [24], and [55] are external prior works, and the paper does not invoke a uniqueness theorem or an author-imported ansatz. The hallucination experiment (RQ5, Section 5.4) is post hoc but is an empirical manipulation, not a conclusion that reduces to its premises by definition. The paper's own limitation statement, 'the analysis of causality remains empirical and may not generalize to all POMDP settings,' further confirms that the architecture-level claims are not presented as derived from a closed-form equivalence. Therefore no circular step can be exhibited with a quoted reduction, and the correct finding is no significant circularity.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

The central claim rests on standard architectural hyperparameters plus unstated evaluation assumptions. No new physical or mathematical entities are introduced. The main unexamined premise is the causal use of the BiGRU at inference, which directly affects whether the reported gains are valid.

free parameters (3)
  • Embedding dimension D = 64 or 128
    Set to 128 for most environments and 64 for Hallway, HeavenHell, and CarFlag to match DTQN; an architecture hyperparameter, not fitted to the target result.
  • BiGRU hidden dimension H = 32 or 64
    Chosen by hand; affects parameter count and capacity, but not fitted to the success-rate claim.
  • Context length K = 50
    Standard choice from DTQN; not fitted to the target result.
assumptions (3)
  • domain assumption The 23 POMDP environments are representative of partial observability, and the running average success rate is a valid performance metric.
    The paper generalizes from gridworld benchmarks to POMDPs broadly, but does not justify representativeness beyond selecting existing domains.
  • domain assumption The DTQN and DRQN baselines are correctly implemented and tuned for fair comparison.
    The results depend on fair baseline comparisons, but no code is provided to verify implementation quality or hyperparameter matching.
  • domain assumption The bidirectional GRU does not use future observations for action selection at inference.
    The paper does not describe the inference procedure, so this assumption is unstated and structurally important for causal validity.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Bi-directional Recurrence Improves Transformer in Partially Observable Markov Decision Processes." pith.science (2026). https://pith.science/paper/SPXRYCLC

@misc{pith2026250511153,
  author       = {Pith},
  title        = {Pith review of: Bi-directional Recurrence Improves Transformer in Partially Observable Markov Decision Processes},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/SPXRYCLC}},
  note         = {Machine review of arXiv:2505.11153}
}
read the original abstract

In real-world reinforcement learning (RL) scenarios, agents often encounter partial observability, where incomplete or noisy information obscures the true state of the environment. Partially Observable Markov Decision Processes (POMDPs) are commonly used to model these environments, but effective performance requires memory mechanisms to utilise past observations. While recurrence networks have traditionally addressed this need, transformer-based models have recently shown improved sample efficiency in RL tasks. However, their application to POMDPs remains underdeveloped, and their real-world deployment is constrained due to the high parameter count. This work introduces a novel bi-recurrent model architecture that improves sample efficiency and reduces model parameter count in POMDP scenarios. The architecture replaces the multiple feed forward layers with a single layer of bi-directional recurrence unit to better capture and utilize sequential dependencies and contextual information. This approach improves the model's ability to handle partial observability and increases sample efficiency, enabling effective learning from comparatively fewer interactions. To evaluate the performance of the proposed model architecture, experiments were conducted on a total of 23 POMDP environments. The proposed model architecture outperforms existing transformer-based, attention-based, and recurrence-based methods by a margin ranging from 87.39% to 482.04% on average across the 23 POMDP environments.

Figures

Figures reproduced from arXiv: 2505.11153 by the authors.

Figure 1
Figure 1. Architecture of the proposed DBGFQN model. [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. The effect of varying the number of FFN layers in the Transformer model. [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. The effect of uni-directional recurrent layers in the Transformer model. [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: The effect of bi-directional recurrent layers in the Transformer model. [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 5
Figure 5. Figure 5: Comparitive analysis on POMDP environments 1-8. [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: Comparitive analysis on POMDP environments 9-16. [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]
Figure 7
Figure 7. Figure 7: Comparitive analysis on POMDP environments 7-23. [PITH_FULL_IMAGE:figures/full_fig_p008_7.png]
Figure 8
Figure 8. Figure 8: Effect of hallucination on the base model. [PITH_FULL_IMAGE:figures/full_fig_p008_8.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

61 extracted references · 27 canonical work pages

  1. [1]

    Reinforcement learning in game industry—review, prospects and challenges,

    K. Souchleris, G. K. Sidiropoulos, and G. A. Papakostas, “Reinforcement learning in game industry—review, prospects and challenges,”Applied Sciences, vol. 13, no. 4, p. 2443, 2023

    work page 2023

  2. [2]

    Artificial intelligence, machine learning and deep learning in advanced robotics, a review,

    M. Soori, B. Arezoo, and R. Dastres, “Artificial intelligence, machine learning and deep learning in advanced robotics, a review,”Cognitive Robotics, vol. 3, pp. 54–70, 2023

    2023

  3. [3]

    Playing atari with deep reinforcement learning,

    V . Mnih, K. Kavukcuoglu, D. Silver, A. Graves, I. Antonoglou, D. Wierstra, and M. Riedmiller, “Playing atari with deep reinforcement learning,” 2013. [Online]. Available: https://arxiv.org/abs/1312.5602

    arXiv 2013

  4. [4]

    Weakly coupled deep q-networks,

    I. El Shar and D. Jiang, “Weakly coupled deep q-networks,”Advances in Neural Information Processing Systems, vol. 36, 2024

    work page 2024

  5. [5]

    Tuning apex dqn: A reinforcement learning based deep q-network algorithm,

    D. Ruhela and A. Ruhela, “Tuning apex dqn: A reinforcement learning based deep q-network algorithm,” inPractice and Experience in Advanced Research Computing 2024: Human Powered Computing, 2024, pp. 1–5

    work page 2024

  6. [6]

    Safe reinforcement learning via shielding under partial observability,

    S. Carr, N. Jansen, S. Junges, and U. Topcu, “Safe reinforcement learning via shielding under partial observability,” inProceedings of the AAAI Conference on Artificial Intelligence, vol. 37, no. 12, 2023, pp. 14 748–14 756

    work page 2023

  7. [7]

    Integrated task and motion planning for safe legged navigation in partially observable environments,

    A. Shamsah, Z. Gu, J. Warnke, S. Hutchinson, and Y . Zhao, “Integrated task and motion planning for safe legged navigation in partially observable environments,”IEEE Transactions on Robotics, 2023

    work page 2023

  8. [8]

    Deep reinforcement learning for multiagent systems: A review of challenges, solutions, and applications,

    T. T. Nguyen, N. D. Nguyen, and S. Nahavandi, “Deep reinforcement learning for multiagent systems: A review of challenges, solutions, and applications,”IEEE transactions on cybernetics, vol. 50, no. 9, pp. 3826–3839, 2020

    2020

Show all 61 references
  1. [9]

    A definition of continual reinforcement learning,

    D. Abel, A. Barreto, B. Van Roy, D. Precup, H. P. van Hasselt, and S. Singh, “A definition of continual reinforcement learning,”Advances in Neural Information Processing Systems, vol. 36, 2024

    2024

  2. [10]

    Deep reinforcement learning unleashing the power of ai in decision-making,

    J. Shuford, “Deep reinforcement learning unleashing the power of ai in decision-making,” Journal of Artificial Intelligence General science (JAIGS) ISSN: 3006-4023, vol. 1, no. 1, 2024

    2024

  3. [11]

    Exploration in deep reinforcement learning: A survey,

    P. Ladosz, L. Weng, M. Kim, and H. Oh, “Exploration in deep reinforcement learning: A survey,” Information Fusion, vol. 85, pp. 1–22, 2022

    2022

  4. [12]

    Memory gym: Partially observable challenges to memory-based agents,

    M. Pleines, M. Pallasch, F. Zimmer, and M. Preuss, “Memory gym: Partially observable challenges to memory-based agents,” inThe eleventh international conference on learning representations, 2023

    2023

  5. [14]

    Deep rein- forcement learning: A survey,

    X. Wang, S. Wang, X. Liang, D. Zhao, J. Huang, X. Xu, B. Dai, and Q. Miao, “Deep rein- forcement learning: A survey,”IEEE Transactions on Neural Networks and Learning Systems, vol. 35, no. 4, pp. 5064–5078, 2022

    2022

  6. [15]

    Partially observable markov decision processes (pomdps) and robotics,

    H. Kurniawati, “Partially observable markov decision processes (pomdps) and robotics,”arXiv preprint arXiv:2107.07599, 2021

    2021 arXiv

  7. [16]

    Recurrent neural networks,

    L. R. Medsker, L. Jainet al., “Recurrent neural networks,”Design and Applications, vol. 5, no. 64-67, p. 2, 2001

    2001

  8. [17]

    Long short-term memory,

    S. Hochreiter and J. Schmidhuber, “Long short-term memory,”Neural computation, vol. 9, no. 8, pp. 1735–1780, 1997

    1997

  9. [18]

    Gate-variants of gated recurrent unit (gru) neural networks,

    R. Dey and F. M. Salem, “Gate-variants of gated recurrent unit (gru) neural networks,” in2017 IEEE 60th international midwest symposium on circuits and systems (MWSCAS). IEEE, 2017, pp. 1597–1600. 10

    2017

  10. [19]

    Recurrent prediction model for partially observable mdps,

    S. Xie, Z. Zhang, H. Yu, and X. Luo, “Recurrent prediction model for partially observable mdps,”Information Sciences, vol. 620, pp. 125–141, 2023

    2023

  11. [20]

    Visualizing transformers for nlp: a brief survey,

    A. M. Bra¸ soveanu and R. Andonie, “Visualizing transformers for nlp: a brief survey,” in2020 24th International Conference Information Visualisation (IV). IEEE, 2020, pp. 270–279

    2020

  12. [21]

    Transformers in vision: A survey,

    S. Khan, M. Naseer, M. Hayat, S. W. Zamir, F. S. Khan, and M. Shah, “Transformers in vision: A survey,”ACM computing surveys (CSUR), vol. 54, no. 10s, pp. 1–41, 2022

    2022

  13. [22]

    Windows deep transformer q-networks: an extended variance reduction architecture for partially observable reinforcement learning,

    Z. Wang, B. Wang, H. Dou, and Z. Liu, “Windows deep transformer q-networks: an extended variance reduction architecture for partially observable reinforcement learning,” 2024

    2024

  14. [23]

    Deep transformer q-networks for partially observable reinforcement learning,

    K. Esslinger, R. Platt, and C. Amato, “Deep transformer q-networks for partially observable reinforcement learning,”arXiv preprint arXiv:2206.01078, 2022

    2022 arXiv

  15. [24]

    Deep recurrent q-learning for partially observable mdps,

    M. Hausknecht and P. Stone, “Deep recurrent q-learning for partially observable mdps,” 2017. [Online]. Available: https://arxiv.org/abs/1507.06527

    2017 arXiv

  16. [25]

    On improving deep reinforcement learning for pomdps,

    P. Zhu, X. Li, P. Poupart, and G. Miao, “On improving deep reinforcement learning for pomdps,”

  17. [26]

    Learning to communicate to solve riddles with deep distributed recurrent q-networks,

    J. N. Foerster, Y . M. Assael, N. de Freitas, and S. Whiteson, “Learning to communicate to solve riddles with deep distributed recurrent q-networks,” 2016. [Online]. Available: https://arxiv.org/abs/1602.02672

    2016 arXiv

  18. [27]

    Deep reinforcement learning with bidirectional recurrent neural networks for dynamic spectrum access,

    P. Chen, S. Guo, and Y . Gao, “Deep reinforcement learning with bidirectional recurrent neural networks for dynamic spectrum access,” in2021 IEEE 94th Vehicular Technology Conference (VTC2021-Fall), 2021, pp. 1–5

    2021

  19. [28]

    On transforming reinforcement learning with transformers: The development trajectory,

    S. Hu, L. Shen, Y . Zhang, Y . Chen, and D. Tao, “On transforming reinforcement learning with transformers: The development trajectory,”IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024

    2024

  20. [29]

    Transformer in reinforcement learning for decision-making: A survey,

    W. Yuan, J. Chen, S. Chen, D. Feng, Z. Hu, P. Li, and W. Zhao, “Transformer in reinforcement learning for decision-making: A survey,”Frontiers of Information Technology & Electronic Engineering, vol. 25, no. 6, pp. 763–790, 2024

    2024

  21. [30]

    Attention is all you need,

    A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin, “Attention is all you need,” 2023. [Online]. Available: https: //arxiv.org/abs/1706.03762

    2023 arXiv

  22. [31]

    Decision transformer: Reinforcement learning via sequence modeling,

    L. Chen, K. Lu, A. Rajeswaran, K. Lee, A. Grover, M. Laskin, P. Abbeel, A. Srinivas, and I. Mordatch, “Decision transformer: Reinforcement learning via sequence modeling,” 2021. [Online]. Available: https://arxiv.org/abs/2106.01345

    2021 arXiv

  23. [32]

    Deep attention recurrent q-network,

    I. Sorokin, A. Seleznev, M. Pavlov, A. Fedorov, and A. Ignateva, “Deep attention recurrent q-network,” 2015. [Online]. Available: https://arxiv.org/abs/1512.01693

    2015 arXiv

  24. [33]

    Towards interpretable reinforcement learning using attention augmented agents,

    A. Mott, D. Zoran, M. Chrzanowski, D. Wierstra, and D. J. Rezende, “Towards interpretable reinforcement learning using attention augmented agents,” 2019. [Online]. Available: https://arxiv.org/abs/1906.02500

    2019 arXiv

  25. [34]

    Efficient transformers in reinforcement learning using actor-learner distillation,

    E. Parisotto and R. Salakhutdinov, “Efficient transformers in reinforcement learning using actor-learner distillation,” 2021. [Online]. Available: https://arxiv.org/abs/2104.01655

    2021 arXiv

  26. [35]

    Gated linear attention transformers with hardware-efficient training,

    S. Yang, B. Wang, Y . Shen, R. Panda, and Y . Kim, “Gated linear attention transformers with hardware-efficient training,” 2024. [Online]. Available: https://arxiv.org/abs/2312.06635

    2024 arXiv

  27. [36]

    Offline reinforcement learning as one big sequence modeling problem,

    M. Janner, Q. Li, and S. Levine, “Offline reinforcement learning as one big sequence modeling problem,” inAdvances in Neural Information Processing Systems, 2021

    2021

  28. [37]

    Online decision transformer,

    Q. Zheng, A. Zhang, and A. Grover, “Online decision transformer,” 2022. [Online]. Available: https://arxiv.org/abs/2202.05607

    2022 arXiv

  29. [38]

    Structured state space models for in-context reinforcement learning,

    C. Lu, Y . Schroecker, A. Gu, E. Parisotto, J. Foerster, S. Singh, and F. Behbahani, “Structured state space models for in-context reinforcement learning,” 2023. [Online]. Available: https://arxiv.org/abs/2303.03982

    2023 arXiv

  30. [39]

    Mastering memory tasks with world models,

    M. R. Samsami, A. Zholus, J. Rajendran, and S. Chandar, “Mastering memory tasks with world models,” 2024. [Online]. Available: https://arxiv.org/abs/2403.04253

    2024 arXiv

  31. [40]

    Mastering atari with discrete world models,

    D. Hafner, T. Lillicrap, M. Norouzi, and J. Ba, “Mastering atari with discrete world models,”

  32. [41]

    Transformers are rnns: Fast autoregressive transformers with linear attention,

    A. Katharopoulos, A. Vyas, N. Pappas, and F. Fleuret, “Transformers are rnns: Fast autoregressive transformers with linear attention,” 2020. [Online]. Available: https://arxiv.org/abs/2006.16236

    2020 arXiv

  33. [42]

    Rwkv: Reinventing rnns for the transformer era,

    B. Peng, E. Alcaide, Q. Anthony, A. Albalak, S. Arcadinho, S. Biderman, H. Cao, X. Cheng, M. Chung, M. Grella, K. K. GV , X. He, H. Hou, J. Lin, P. Kazienko, J. Kocon, J. Kong, B. Koptyra, H. Lau, K. S. I. Mantri, F. Mom, A. Saito, G. Song, X. Tang, B. Wang, J. S. Wind, S. Woz...

    2023 arXiv

  34. [43]

    Resurrecting recurrent neural networks for long sequences,

    A. Orvieto, S. L. Smith, A. Gu, A. Fernando, C. Gulcehre, R. Pascanu, and S. De, “Resurrecting recurrent neural networks for long sequences,” 2023. [Online]. Available: https://arxiv.org/abs/2303.06349

    2023 arXiv

  35. [44]

    Efficiently modeling long sequences with structured state spaces,

    A. Gu, K. Goel, and C. Ré, “Efficiently modeling long sequences with structured state spaces,”

  36. [45]

    Simplified state space layers for sequence modeling,

    J. T. H. Smith, A. Warrington, and S. W. Linderman, “Simplified state space layers for sequence modeling,” 2023. [Online]. Available: https://arxiv.org/abs/2208.04933

    2023 arXiv

  37. [46]

    Reinforcement learning upside down: Don’t predict rewards – just map them to actions,

    J. Schmidhuber, “Reinforcement learning upside down: Don’t predict rewards – just map them to actions,” 2020. [Online]. Available: https://arxiv.org/abs/1912.02875

    2020 arXiv

  38. [47]

    Available: https://arxiv.org/abs/2111.00396

    [Online]. Available: https://arxiv.org/abs/2111.00396

    arXiv

  39. [48]

    Longformer: The long-document transformer,

    I. Beltagy, M. E. Peters, and A. Cohan, “Longformer: The long-document transformer,” 2020. [Online]. Available: https://arxiv.org/abs/2004.05150

    2020 arXiv

  40. [49]

    Transformer-XL: Attentive language models beyond a fixed-length context,

    Z. Dai, Z. Yang, Y . Yang, J. Carbonell, Q. Le, and R. Salakhutdinov, “Transformer-XL: Attentive language models beyond a fixed-length context,” inProceedings of the 57th Annual Meeting of the Association for Computational Linguistics, A. Korhonen, D. Traum, and L. Màrquez, Ed...

    2019

  41. [50]

    Stabilizing transformers for reinforcement learning,

    E. Parisotto, H. F. Song, J. W. Rae, R. Pascanu, C. Gulcehre, S. M. Jayakumar, M. Jaderberg, R. L. Kaufman, A. Clark, S. Noury, M. M. Botvinick, N. Heess, and R. Hadsell, “Stabilizing transformers for reinforcement learning,” 2019. [Online]. Available: https://arxiv.org/abs/1910.06764

    2019 arXiv

  42. [51]

    Recurrent memory transformer,

    A. Bulatov, Y . Kuratov, and M. S. Burtsev, “Recurrent memory transformer,” 2022. [Online]. Available: https://arxiv.org/abs/2207.06881

    2022 arXiv

  43. [52]

    Reformer: The efficient transformer,

    N. Kitaev, Łukasz Kaiser, and A. Levskaya, “Reformer: The efficient transformer,” 2020. [Online]. Available: https://arxiv.org/abs/2001.04451

    2020 arXiv

  44. [53]

    Linear transformers are secretly fast weight programmers,

    I. Schlag, K. Irie, and J. Schmidhuber, “Linear transformers are secretly fast weight programmers,” inProceedings of the 38th International Conference on Machine Learning, ser. Proceedings of Machine Learning Research, M. Meila and T. Zhang, Eds., vol. 139. PMLR, 18–24 Jul 202...

    2021

  45. [54]

    Investigating the role of feed-forward networks in transformers using parallel attention and feed-forward net design,

    S. Sonkar and R. G. Baraniuk, “Investigating the role of feed-forward networks in transformers using parallel attention and feed-forward net design,” 2023. [Online]. Available: https://arxiv.org/abs/2305.13297

    2023 arXiv

  46. [55]

    Rethinking transformers in solving pomdps,

    C. Lu, R. Shi, Y . Liu, K. Hu, S. S. Du, and H. Xu, “Rethinking transformers in solving pomdps,” inProceedings of the 41st International Conference on Machine Learning, ser. ICML’24. JMLR.org, 2024

    2024

  47. [56]

    Rethinking attention with performers,

    K. Choromanski, V . Likhosherstov, D. Dohan, X. Song, A. Gane, T. Sarlos, P. Hawkins, J. Davis, A. Mohiuddin, L. Kaiser, D. Belanger, L. Colwell, and A. Weller, “Rethinking attention with performers,” 2022. [Online]. Available: https://arxiv.org/abs/2009.14794

    2022 arXiv

  48. [57]

    Pomdp robot domains,

    H. Nguyen, “Pomdp robot domains,” https://github.com/hai-h-nguyen/pomdp-domains, 2021

    2021

  49. [58]

    Learning policies for partially observable environments: Scaling up,

    M. L. Littman, A. R. Cassandra, and L. P. Kaelbling, “Learning policies for partially observable environments: Scaling up,” inMachine Learning Proceedings 1995. Elsevier, 1995, pp. 362–370. 12

    1995

  50. [59]

    gym-gridverse: Gridworld domains for fully and partially observable reinforcement learning,

    A. Baisero and S. Katt, “gym-gridverse: Gridworld domains for fully and partially observable reinforcement learning,” 2021

    2021

  51. [62]

    Solving large pomdps using real time dynamic programming,

    H. Geffner and B. Bonet, “Solving large pomdps using real time dynamic programming,” in Working Notes Fall AAAI Symposium on POMDPs, vol. 218, 1998. A Appendix / supplemental material Model Architecture Initially, the observations from the environment for timesteps 1 to t, den...

    1998

  52. [2018]

    Available: https://arxiv.org/abs/1704.07978

    [Online]. Available: https://arxiv.org/abs/1704.07978

    arXiv

  53. [2022]

    Available: https://arxiv.org/abs/2010.02193 11

    [Online]. Available: https://arxiv.org/abs/2010.02193 11

    2010 arXiv

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.