Pith. sign in

REVIEW 4 major objections 6 minor 24 references

Decision Transformer vs. Decision Mamba: Analysing the Complexity of Sequential Decision Making in Atari Games

T0 review · 4 major / 6 minor · reviewed 2026-08-12 · deepseek-v4-flash

Pith's one-line read Across 12 Atari games, the gap between Decision Mamba and Decision Transformer is driven mainly by two cheap game statistics: the number of actions and the compressibility of game frames.

desk verdict Useful 12-game DT/DM comparison, but the claim that action/visual complexity are primary determinants is not supported by a 12-game random forest without significance tests. read the letter →

arxiv 2412.00725 v1 pith:SVUDZPM7 submitted 2024-12-01 cs.LG cs.AI

classification cs.LGcs.AI
keywords DecisionTransformerMambaofflinereinforcementlearningAtarigamesvisualcomplexityactionspacecompressionratiorandomforestregression
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that the performance gap between the Decision Transformer (DT) and Decision Mamba (DM) in Atari games is driven mainly by two game properties: the size of the action space and the visual complexity of the game, measured most usefully by compression ratio. It argues that DM tends to win in visually simple, low-action environments, while DT tends to win in visually complex, high-action games. If this is right, a practitioner could choose between the two architectures based on two cheap numbers computed from the environment, rather than training both models. The paper supports the claim with random forest feature importance, SHAP values, Pearson correlations, and action-fusion experiments that manipulate the action space directly.

What carries the argument

The argument is carried by a specific target variable and a small feature set. The target is the performance difference $S_{\text{DT}} - S_{\text{DM}}$ computed from normalized scores, where 0 is a random agent and 100 is human-level play. The explanatory features are six game characteristics: number of actions, average trajectory length, average steps to the first non-zero reward, image entropy, compression ratio (uncompressed over compressed frame size, computed losslessly with zlib), and SIFT feature count. The inference machinery is a random forest regressor with 1000 trees and six-fold cross-validation, whose feature importances are reported alongside SHAP values, plus Pearson correlations and two action-space simplification interventions (simple action fusion and frequency-based action fusion) that reduce the action space while preserving all original actions during evaluation.

What would settle it

A decisive check would be to decouple the two factors by modifying games: take a visually simple game such as Breakout and add visual texture without changing its 4-action set, or take Hero and keep its 18 actions while simplifying the graphics. If the DT–DM gap tracks compression ratio when action count is held fixed, the visual-complexity claim is supported; if it tracks only action count, the paper's hierarchy is wrong. A cheaper statistical falsifier is to resample the 12 games thousands of times and check whether action count remains the top random-forest feature; if the top feature changes frequently, the reported importance ranking is unstable.

Watch

Extended reading notes

Core claim

The central discovery claimed is that, across 12 Atari games trained offline on the DQN-replay dataset, the normalized-score gap between Decision Transformer and Decision Mamba is not random: it tracks the number of actions and the compression ratio of game frames. DM generally wins on visually simple, low-action games (Breakout, Pong, Seaquest, RoadRunner, SpaceInvaders), while DT generally wins on visually complex, high-action games (Hero, KungFuMaster). The paper supports this with feature-importance and SHAP analyses that rank action count first and compression ratio second, and with action-fusion experiments in which reducing the action space of Hero and KungFuMaster shrinks DT's advantage over DM. The conclusion is that both action-space complexity and visual complexity, particularly as captured by compression ratio, are the primary determinants of the DT–DM performance gap.

Load-bearing premise

The ranking of factors is read out of a random forest fitted to only 12 games, with six-fold cross-validation and post-hoc outlier removal, so the feature-importance ordering may be a property of a few games rather than of Atari games in general.

Editorial extensions

If this is right

  • In Atari-like offline reinforcement learning, Decision Mamba is the safer choice in visually simple games with small action sets, while Decision Transformer is preferable in games with many actions and complex visuals.
  • Action space complexity plays a causal role: simplifying the action space of Hero and KungFuMaster via action fusion drops DT's normalized score from about 30 to roughly 16–19, erasing most of DT's advantage over DM.
  • Compression ratio can serve as a low-cost proxy for the visual complexity that matters to the DT–DM gap, since it ranks second in importance in both random forest and SHAP analyses.
  • Architecture evaluation in offline RL should report game-characteristic metrics alongside normalized scores, because the win/loss pattern is not intrinsic to the architecture alone but depends on the environment.
  • If the two-metric rule holds, model selection between DT and DM can be guided by a quick pre-training scan of action count and frame compressibility for each new Atari environment.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A testable screening rule that the paper does not itself propose is: before training on a new Atari-like environment, compute the action count and zlib compression ratio of its frames; if both are low, expect DM to win, otherwise expect DT to win.
  • Because action count and image entropy correlate strongly (r = 0.68) in the paper's 12-game table, the regression may not fully separate the two factors; an experiment that manipulates visual complexity while holding action count fixed would sharpen the causal claim.
  • The action-fusion results suggest that DT's advantage is not purely architectural but comes from exploiting a larger action vocabulary; extending the comparison to continuous-control tasks with discrete action abstractions would test whether the pattern generalizes beyond Atari.
  • A direct architectural probe, such as examining DT's attention patterns and DM's selection vectors on visually simple versus visually complex frames, could reveal the mechanism behind the claimed complexity-dependent gap.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper compares Decision Transformer (DT) and Decision Mamba (DM) on 12 Atari games, using normalized scores from offline DQN-replay data. It introduces three visual-complexity metrics (image entropy, compression ratio, feature count) and analyzes the performance gap with random forest regression, SHAP values, and Pearson correlations. The authors conclude that action-space complexity and visual complexity (especially compression ratio) are the primary determinants of the DT–DM gap, and they test two action-fusion strategies on Hero and KungFuMaster to probe the action-space hypothesis. The stated contributions are an empirical comparison, identification of influential game characteristics, and an action-space simplification method.

Significance. If the central claim were well supported, it would offer a practical rule for architecture selection in offline RL: measure the number of actions and frame compressibility, then choose DT or DM accordingly. The paper does provide a reproducible experimental setup, a novel set of visual-complexity metrics, and open code, which are useful building blocks. However, the quantitative evidence for the claim rests on a random forest fit to 12 games, with no significance tests, no reported R-squared, and post-hoc outlier removal that changes headline numbers. The result is therefore best read as a hypothesis-generation exercise, not a validated explanation of DT–DM differences.

major comments (4)
  1. [§4.2.1, Figure 3] The central claim that action-space complexity and visual complexity are the primary determinants of the DT–DM gap is not supported by the statistical evidence. The random forest is fitted to n=12 games with 6 features, and the reported 6-fold CV RMSE is 17.57 ± 12.35, which is larger than the magnitude of most observed performance differences in Table 5 (all but three |y| ≤ 10.05). The feature importances are thus likely driven by the three extreme games (Hero, KungFuMaster, Breakout) rather than a generalizable relationship. I recommend reporting leave-one-out predictions, permutation-based importance, and a confidence interval or significance test for the importances, or substantially tempering the causal wording in the abstract and conclusion.
  2. [Table 3 vs. Appendix E] The outlier-removal protocol is not pre-registered and changes the headline results. For KungFuMaster at context length 10, the DT normalized score changes from 29.41 (Table 3, after outlier removal) to 19.65 (Appendix E, raw), and for Qbert at context length 30 it changes from 4.94 to 9.06. Because the paper's narrative about which model wins and by how much depends on these numbers, the analysis should present both raw and trimmed results and justify the trimming criteria. Without that, the performance gaps used in the regression and correlation analyses are not stable.
  3. [§4.2.2, Figure 5] The feature set is highly collinear: Num_Actions correlates 0.68 with Image_Entropy and −0.53 with Compression_Ratio, while Image_Entropy correlates −0.70 with Compression_Ratio. With only 12 data points, random forest feature importances and SHAP values cannot reliably disentangle these correlated contributions, so the ranking in Figures 3 and 4 is not robust. I recommend reporting variance inflation factors, partial correlations, or a sensitivity analysis with orthogonalized features before drawing conclusions about which metric is 'primary.'
  4. [§4.3.2, Tables 7–8] The action-fusion experiments are limited to two games and do not isolate action-space complexity from other changes. When actions are fused, the training distribution changes and the model's effective policy class changes, so the observed performance drop in DT could be due to altered data statistics rather than to the reduced action count. The paper's conclusion that 'action space complexity alone cannot fully account' for the gaps is therefore not directly tested. A control condition, such as random action relabeling that preserves the action count, or additional games, would be needed to support that inference.
minor comments (6)
  1. [§1, last paragraph] The text refers to 'the rest of the dissertation'; in a journal paper this should be 'paper' or 'manuscript.'
  2. [§3.3.2, Expected Return Setup] The choice to use 5× the maximum return for all games is a free parameter that may differentially affect DT and DM. Please report a sensitivity analysis for at least one game, or justify why a single multiplier is fair across the 12 games.
  3. [Algorithm 1, line 11] The variable n_test is used in the MSE formula but not defined; specify that it is the size of the held-out fold.
  4. [§4.3.2, first paragraph] There is a typo: 'Random Walk scores and Human Players soirees' should be 'scores.' Also, the sentence about bias is vague; specify what bias is introduced by action fusion.
  5. [Figure 1 caption] The caption says 'σ is SiLU function' but does not mention the ⊗ symbol; expand the caption for readability.
  6. [Table 5] For Pong, the DM score has a very large standard deviation (64.31 ± 53.24), which suggests instability; the paper does not discuss this outlier in the context of the correlation results.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the DT–DM comparison is an empirical regression/intervention study whose claims do not reduce to their inputs.

full rationale

The paper does not derive a prediction from a fitted parameter or import a load-bearing conclusion via self-citation. The central claim (abstract, Section 4.2.1, Section 6) is supported by directly measured performance gaps (Table 5), random-forest feature importances and SHAP values computed on the 12-game dataset, Pearson correlations, and an action-fusion intervention on Hero and KungFuMaster. None of these steps defines the target variable in terms of the explanatory features or vice versa: y = DT_norm - DM_norm is an independent measurement from the online ALE evaluation, while the game metrics (number of actions, image entropy, compression ratio, feature count, trajectory length, steps to first reward) are computed separately from the DQN-replay dataset. The action-fusion experiment is a genuine manipulation of the action space and therefore provides non-circular evidence for the role of action-space complexity. The main weaknesses - small sample size (n=12), unstable feature importances, a cross-validated RMSE (17.57 +/- 12.35) larger than most observed |y|, high inter-feature correlations, and the use of causal language such as 'primary determining factors' - are validity and generalization concerns, not circularity under the stated criteria. There is no load-bearing self-citation: the cited prior works (Decision Transformer, Decision Mamba, Mamba, ALE, etc.) are external sources, and the paper does not invoke any uniqueness theorem or ansatz from the authors' own prior work to force its conclusions.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The central conclusion depends on assumptions about the representativeness of the 1% DQN-replay sample, the validity of the hand-chosen expected-return multiplier, the adequacy of the visual-complexity metrics, and the statistical reliability of a 6-feature regression built from 12 games. The first three axioms are domain assumptions; the fourth is a statistical assumption introduced specifically for this analysis.

free parameters (4)
  • Expected return multiplier = 5
    All games use 5 times the max return from the DQN-replay dataset as the evaluation target (Section 3.3.2). This hand-chosen value can shift the DT-DM performance balance and is not ablated.
  • Random forest tree count = 1000
    Selected without reported tuning; feature importances can change with tree count, especially on 12 samples.
  • Cross-validation folds = 6
    Chosen as a round number for n=12; each test fold has only 2 games, making RMSE and importance estimates unstable.
  • Context length for extended experiments = 10
    Chosen for computational efficiency rather than performance; Section 3.3 states this, and it affects all 12-game comparisons.
assumptions (5)
  • domain assumption The 1% sample of DQN-replay data is representative of each game's state and action distribution.
    Section 3.1.1; the metrics and training are computed on this sample, and the paper does not test sensitivity to the sample.
  • domain assumption 5 times max return is an appropriate expected-return initialization for all games.
    Section 3.3.2; the paper calls this a naive assumption and does not vary it.
  • domain assumption Compression ratio, image entropy, and SIFT feature count are valid measures of task-relevant visual complexity.
    Section 4.2; these metrics are computed on dataset frames but never validated against human perception or game mechanics.
  • ad hoc to paper Random forest feature importances generalize from 12 games to other Atari-like environments.
    Section 4.2.1; with n=12 and six features, this is a strong and unsupported statistical assumption.
  • domain assumption Action fusion preserves the game dynamics that matter for comparing DT and DM.
    Section 4.3.2; the paper acknowledges the method 'may introduce slight bias' while still using it to draw conclusions about action space complexity.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Decision Transformer vs. Decision Mamba: Analysing the Complexity of Sequential Decision Making in Atari Games." pith.science (2026). https://pith.science/paper/SVUDZPM7

@misc{pith2026241200725,
  author       = {Pith},
  title        = {Pith review of: Decision Transformer vs. Decision Mamba: Analysing the Complexity of Sequential Decision Making in Atari Games},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/SVUDZPM7}},
  note         = {Machine review of arXiv:2412.00725}
}
read the original abstract

This work analyses the disparity in performance between Decision Transformer (DT) and Decision Mamba (DM) in sequence modelling reinforcement learning tasks for different Atari games. The study first observed that DM generally outperformed DT in the games Breakout and Qbert, while DT performed better in more complicated games, such as Hero and Kung Fu Master. To understand these differences, we expanded the number of games to 12 and performed a comprehensive analysis of game characteristics, including action space complexity, visual complexity, average trajectory length, and average steps to the first non-zero reward. In order to further analyse the key factors that impact the disparity in performance between DT and DM, we employ various approaches, including quantifying visual complexity, random forest regression, correlation analysis, and action space simplification strategies. The results indicate that the performance gap between DT and DM is affected by the complex interaction of multiple factors, with the complexity of the action space and visual complexity (particularly evaluated by compression ratio) being the primary determining factors. DM performs well in environments with simple action and visual elements, while DT shows an advantage in games with higher action and visual complexity. Our findings contribute to a deeper understanding of how the game characteristics affect the performance difference in sequential modelling reinforcement learning, potentially guiding the development of future model design and applications for diverse and complex environments.

Figures

Figures reproduced from arXiv: 2412.00725 by the authors.

Figure 1
Figure 1. Mamba block, σ is SiLU function, and ⊗ stands for elementwise multiplication. As illustrated in [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. The Architecture of Decision Transformer (Left) and Decision Mamba (Right). [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. Feature importance of each game metrics. Higher scores indicate a greater [PITH_FULL_IMAGE:figures/full_fig_p015_3.png] view at source ↗
Figures from the paper (8 more)
Figure 4
Figure 4. Figure 4: SHAP value feature importance. Higher values suggest a greater influence. [PITH_FULL_IMAGE:figures/full_fig_p015_4.png]
Figure 5
Figure 5. Figure 5: Correlation Matrix of Performance Difference and the Game Metrics. The [PITH_FULL_IMAGE:figures/full_fig_p016_5.png]
Figure 6
Figure 6. Figure 6: Simple Action Fusion: Fuse Move Actions and Fire Actions [PITH_FULL_IMAGE:figures/full_fig_p018_6.png]
Figure 7
Figure 7. Figure 7: Action Distribution of the Last 1% of the Dataset (Game: Hero) [PITH_FULL_IMAGE:figures/full_fig_p019_7.png]
Figure 8
Figure 8. Figure 8: Breakout [PITH_FULL_IMAGE:figures/full_fig_p024_8.png]
Figure 9
Figure 9. Figure 9: Qbert 24 [PITH_FULL_IMAGE:figures/full_fig_p024_9.png]
Figure 10
Figure 10. Figure 10: Hero [PITH_FULL_IMAGE:figures/full_fig_p025_10.png]
Figure 11
Figure 11. Figure 11: Kung Fu Master C Max Return and Expected Return Games Max Return Expected Return (5*max) Breakout 104 520 Qbert 640 3200 Hero 190 950 KungFuMaster 284 1420 Pong 21 105 Seaquest 314 1570 Alien 213 1065 BankHeist 136 680 BattleZone 32 160 RoadRunner 270 1350 FishingDerb…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

24 extracted references · 17 canonical work pages

  1. [1]

    Decision transformer: Reinforcement learning via sequence modeling,

    L. Chen, K. Lu, A. Rajeswaran, K. Lee, A. Grover, M. Laskin, P. Abbeel, A. Srinivas, and I. Mordatch, “Decision transformer: Reinforcement learning via sequence modeling,” in Advances in Neural Information Processing Systems (NeurIPS) , vol. 34. Curran Associates, Inc., 2021, pp. 15 084–15 097. [Online]. Available: https://proceedings.neurips.cc/paper/ 20...

  2. [2]

    Decision mamba: Reinforcement learning via sequence modeling with selective state spaces,

    T. Ota, “Decision mamba: Reinforcement learning via sequence modeling with selective state spaces,” arXiv preprint arXiv:2403.19925 , 2024

  3. [3]

    Mamba: Linear-time sequence modeling with selective state spaces,

    A. Gu and T. Dao, “Mamba: Linear-time sequence modeling with selective state spaces,” arXiv preprint arXiv:2312.00752 , 2023

  4. [4]

    Attention is all you need,

    A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. u. Kaiser, and I. Polosukhin, “Attention is all you need,” in Advances in Neural Information Processing Systems (NeurIPS) , vol. 30. Curran Associates, Inc., 2017. [Online]. Available: https://papers.nips.cc/paper files/paper/2017/hash/ 3f5ee243547dee91fbd053c1c4a845aa-Abstract.html

  5. [5]

    BERT: pre-training of deep bidirectional transformers for language understanding,

    J. Devlin, M. Chang, K. Lee, and K. Toutanova, “BERT: pre-training of deep bidirectional transformers for language understanding,” in Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short ...

  6. [6]

    Efficiently modeling long sequences with structured state spaces,

    A. Gu, K. Goel, and C. R´ e, “Efficiently modeling long sequences with structured state spaces,” in The International Conference on Learning Representations (ICLR) , 2022

  7. [7]

    Hippo: Recurrent memory with optimal polynomial projections,

    A. Gu, T. Dao, S. Ermon, A. Rudra, and C. R´ e, “Hippo: Recurrent memory with optimal polynomial projections,” in Advances in Neural Information Processing Systems , H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin, Eds., vol. 33. Curran Associates, Inc., 2020, pp. 1474–1487. [Online]. Available: https://proceedings.neurips.cc/ paper files/pap...

  8. [8]

    Long short-term memory,

    S. Hochreiter and J. Schmidhuber, “Long short-term memory,” Neural Comput., vol. 9, no. 8, p. 1735–1780, nov 1997. [Online]. Available: https://doi.org/10.1162/neco.1997.9.8.1735

Show all 24 references
  1. [9]

    R. S. Sutton and A. G. Barto, Reinforcement Learning: An Introduction , 2nd ed. The MIT Press, 2018. [Online]. Available: http://incompleteideas.net/book/the-book-2nd.html

  2. [10]

    Decision convformer: Local filtering in metaformer is sufficient for decision making,

    J. Kim, S. Lee, W. Kim, and Y. Sung, “Decision convformer: Local filtering in metaformer is sufficient for decision making,” in International Conference on Learning Representations , 2024

  3. [11]

    Decision s4: Efficient sequence-based rl via state spaces layers,

    S. Bar-David, I. Zimerman, E. Nachmani, and L. Wolf, “Decision s4: Efficient sequence-based rl via state spaces layers,” 2023

  4. [12]

    An optimistic perspective on offline reinforce- ment learning,

    R. Agarwal, D. Schuurmans, and M. Norouzi, “An optimistic perspective on offline reinforce- ment learning,” in Proceedings of the 37th International Conference on Machine Learning , ser. ICML’20. JMLR.org, 2020

  5. [13]

    The arcade learning environment: An evaluation platform for general agents,

    M. G. Bellemare, Y. Naddaf, J. Veness, and M. Bowling, “The arcade learning environment: An evaluation platform for general agents,” Journal of Artificial Intelligence Research , vol. 47, pp. 253–279, jun 2013

  6. [14]

    Mastering atari with discrete world models,

    D. Hafner, T. P. Lillicrap, M. Norouzi, and J. Ba, “Mastering atari with discrete world models,” in 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021 . OpenReview.net, 2021. [Online]. Available: https://openreview.net/fo...

  7. [15]

    Mastering atari games with limited data,

    W. Ye, S. Liu, T. Kurutach, P. Abbeel, and Y. Gao, “Mastering atari games with limited data,” in Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, virtual , 2021, pp. 25 476...

  8. [16]

    Image entropy equalization: A novel preprocessing technique for image recognition tasks,

    T. Hayashi, D. Cimr, H. Fujita, and R. Cimler, “Image entropy equalization: A novel preprocessing technique for image recognition tasks,” Information Sciences , vol. 647, p. 119539, 2023. [Online]. Available: https://www.sciencedirect.com/science/article/pii/ S0020025523011246

  9. [17]

    A mathematical theory of communication,

    C. E. Shannon, “A mathematical theory of communication,” The Bell System Technical Journal, vol. 27, no. 3, pp. 379–423, 1948

  10. [18]

    Salomon and G

    D. Salomon and G. Motta, Handbook of Data Compression , 5th ed. Springer Publishing Company, Incorporated, 2009. 22

  11. [19]

    zlib: A massively spiffy yet delicately unobtrusive compression library,

    J.-l. Gailly and M. Adler, “zlib: A massively spiffy yet delicately unobtrusive compression library,” 2004. [Online]. Available: https://github.com/madler/zlib

  12. [20]

    Object recognition from local scale-invariant features,

    D. G. Lowe, “Object recognition from local scale-invariant features,” in Proceedings of the seventh IEEE international conference on computer vision , vol. 2. Ieee, 1999, pp. 1150–1157

  13. [21]

    Random forests,

    L. Breiman, “Random forests,” Machine Learning, vol. 45, no. 1, p. 5–32, oct 2001. [Online]. Available: https://doi.org/10.1023/A:1010933404324

  14. [22]

    A unified approach to interpreting model predictions,

    S. M. Lundberg and S.-I. Lee, “A unified approach to interpreting model predictions,” in Proceedings of the 31st International Conference on Neural Information Processing Systems , ser. NIPS’17. Red Hook, NY, USA: Curran Associates Inc., 2017, p. 4768–4777

  15. [23]

    Chapter 4 - understanding your data,

    J. J. Berman, “Chapter 4 - understanding your data,” in Data Simplification , J. J. Berman, Ed. Morgan Kaufmann, 2016, pp. 135–187. [Online]. Available: https://www.sciencedirect.com/science/article/pii/B9780128037812000047

  16. [24]

    Correlation coefficients: appropriate use and interpretation,

    P. Schober, C. Boer, and L. A. Schwarte, “Correlation coefficients: appropriate use and interpretation,” Anesthesia & analgesia , vol. 126, no. 5, pp. 1763–1768, 2018. 23 Appendices A Hyperparameters Hyperparameter V alue Number of layers 6 Embedding dimension 128 Batch size 2...

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.