REVIEW 4 major objections 6 minor 24 references
Decision Transformer vs. Decision Mamba: Analysing the Complexity of Sequential Decision Making in Atari Games
T0 review · 4 major / 6 minor · reviewed 2026-08-12 · deepseek-v4-flash
Pith's one-line read Across 12 Atari games, the gap between Decision Mamba and Decision Transformer is driven mainly by two cheap game statistics: the number of actions and the compressibility of game frames.
desk verdict Useful 12-game DT/DM comparison, but the claim that action/visual complexity are primary determinants is not supported by a 12-game random forest without significance tests. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The argument is carried by a specific target variable and a small feature set. The target is the performance difference $S_{\text{DT}} - S_{\text{DM}}$ computed from normalized scores, where 0 is a random agent and 100 is human-level play. The explanatory features are six game characteristics: number of actions, average trajectory length, average steps to the first non-zero reward, image entropy, compression ratio (uncompressed over compressed frame size, computed losslessly with zlib), and SIFT feature count. The inference machinery is a random forest regressor with 1000 trees and six-fold cross-validation, whose feature importances are reported alongside SHAP values, plus Pearson correlations and two action-space simplification interventions (simple action fusion and frequency-based action fusion) that reduce the action space while preserving all original actions during evaluation.
What would settle it
A decisive check would be to decouple the two factors by modifying games: take a visually simple game such as Breakout and add visual texture without changing its 4-action set, or take Hero and keep its 18 actions while simplifying the graphics. If the DT–DM gap tracks compression ratio when action count is held fixed, the visual-complexity claim is supported; if it tracks only action count, the paper's hierarchy is wrong. A cheaper statistical falsifier is to resample the 12 games thousands of times and check whether action count remains the top random-forest feature; if the top feature changes frequently, the reported importance ranking is unstable.
Extended reading notes
Core claim
The central discovery claimed is that, across 12 Atari games trained offline on the DQN-replay dataset, the normalized-score gap between Decision Transformer and Decision Mamba is not random: it tracks the number of actions and the compression ratio of game frames. DM generally wins on visually simple, low-action games (Breakout, Pong, Seaquest, RoadRunner, SpaceInvaders), while DT generally wins on visually complex, high-action games (Hero, KungFuMaster). The paper supports this with feature-importance and SHAP analyses that rank action count first and compression ratio second, and with action-fusion experiments in which reducing the action space of Hero and KungFuMaster shrinks DT's advantage over DM. The conclusion is that both action-space complexity and visual complexity, particularly as captured by compression ratio, are the primary determinants of the DT–DM performance gap.
Load-bearing premise
The ranking of factors is read out of a random forest fitted to only 12 games, with six-fold cross-validation and post-hoc outlier removal, so the feature-importance ordering may be a property of a few games rather than of Atari games in general.
Editorial extensions
If this is right
- In Atari-like offline reinforcement learning, Decision Mamba is the safer choice in visually simple games with small action sets, while Decision Transformer is preferable in games with many actions and complex visuals.
- Action space complexity plays a causal role: simplifying the action space of Hero and KungFuMaster via action fusion drops DT's normalized score from about 30 to roughly 16–19, erasing most of DT's advantage over DM.
- Compression ratio can serve as a low-cost proxy for the visual complexity that matters to the DT–DM gap, since it ranks second in importance in both random forest and SHAP analyses.
- Architecture evaluation in offline RL should report game-characteristic metrics alongside normalized scores, because the win/loss pattern is not intrinsic to the architecture alone but depends on the environment.
- If the two-metric rule holds, model selection between DT and DM can be guided by a quick pre-training scan of action count and frame compressibility for each new Atari environment.
Reading between the lines
- A testable screening rule that the paper does not itself propose is: before training on a new Atari-like environment, compute the action count and zlib compression ratio of its frames; if both are low, expect DM to win, otherwise expect DT to win.
- Because action count and image entropy correlate strongly (r = 0.68) in the paper's 12-game table, the regression may not fully separate the two factors; an experiment that manipulates visual complexity while holding action count fixed would sharpen the causal claim.
- The action-fusion results suggest that DT's advantage is not purely architectural but comes from exploiting a larger action vocabulary; extending the comparison to continuous-control tasks with discrete action abstractions would test whether the pattern generalizes beyond Atari.
- A direct architectural probe, such as examining DT's attention patterns and DM's selection vectors on visually simple versus visually complex frames, could reveal the mechanism behind the claimed complexity-dependent gap.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper compares Decision Transformer (DT) and Decision Mamba (DM) on 12 Atari games, using normalized scores from offline DQN-replay data. It introduces three visual-complexity metrics (image entropy, compression ratio, feature count) and analyzes the performance gap with random forest regression, SHAP values, and Pearson correlations. The authors conclude that action-space complexity and visual complexity (especially compression ratio) are the primary determinants of the DT–DM gap, and they test two action-fusion strategies on Hero and KungFuMaster to probe the action-space hypothesis. The stated contributions are an empirical comparison, identification of influential game characteristics, and an action-space simplification method.
Significance. If the central claim were well supported, it would offer a practical rule for architecture selection in offline RL: measure the number of actions and frame compressibility, then choose DT or DM accordingly. The paper does provide a reproducible experimental setup, a novel set of visual-complexity metrics, and open code, which are useful building blocks. However, the quantitative evidence for the claim rests on a random forest fit to 12 games, with no significance tests, no reported R-squared, and post-hoc outlier removal that changes headline numbers. The result is therefore best read as a hypothesis-generation exercise, not a validated explanation of DT–DM differences.
major comments (4)
- [§4.2.1, Figure 3] The central claim that action-space complexity and visual complexity are the primary determinants of the DT–DM gap is not supported by the statistical evidence. The random forest is fitted to n=12 games with 6 features, and the reported 6-fold CV RMSE is 17.57 ± 12.35, which is larger than the magnitude of most observed performance differences in Table 5 (all but three |y| ≤ 10.05). The feature importances are thus likely driven by the three extreme games (Hero, KungFuMaster, Breakout) rather than a generalizable relationship. I recommend reporting leave-one-out predictions, permutation-based importance, and a confidence interval or significance test for the importances, or substantially tempering the causal wording in the abstract and conclusion.
- [Table 3 vs. Appendix E] The outlier-removal protocol is not pre-registered and changes the headline results. For KungFuMaster at context length 10, the DT normalized score changes from 29.41 (Table 3, after outlier removal) to 19.65 (Appendix E, raw), and for Qbert at context length 30 it changes from 4.94 to 9.06. Because the paper's narrative about which model wins and by how much depends on these numbers, the analysis should present both raw and trimmed results and justify the trimming criteria. Without that, the performance gaps used in the regression and correlation analyses are not stable.
- [§4.2.2, Figure 5] The feature set is highly collinear: Num_Actions correlates 0.68 with Image_Entropy and −0.53 with Compression_Ratio, while Image_Entropy correlates −0.70 with Compression_Ratio. With only 12 data points, random forest feature importances and SHAP values cannot reliably disentangle these correlated contributions, so the ranking in Figures 3 and 4 is not robust. I recommend reporting variance inflation factors, partial correlations, or a sensitivity analysis with orthogonalized features before drawing conclusions about which metric is 'primary.'
- [§4.3.2, Tables 7–8] The action-fusion experiments are limited to two games and do not isolate action-space complexity from other changes. When actions are fused, the training distribution changes and the model's effective policy class changes, so the observed performance drop in DT could be due to altered data statistics rather than to the reduced action count. The paper's conclusion that 'action space complexity alone cannot fully account' for the gaps is therefore not directly tested. A control condition, such as random action relabeling that preserves the action count, or additional games, would be needed to support that inference.
minor comments (6)
- [§1, last paragraph] The text refers to 'the rest of the dissertation'; in a journal paper this should be 'paper' or 'manuscript.'
- [§3.3.2, Expected Return Setup] The choice to use 5× the maximum return for all games is a free parameter that may differentially affect DT and DM. Please report a sensitivity analysis for at least one game, or justify why a single multiplier is fair across the 12 games.
- [Algorithm 1, line 11] The variable n_test is used in the MSE formula but not defined; specify that it is the size of the held-out fold.
- [§4.3.2, first paragraph] There is a typo: 'Random Walk scores and Human Players soirees' should be 'scores.' Also, the sentence about bias is vague; specify what bias is introduced by action fusion.
- [Figure 1 caption] The caption says 'σ is SiLU function' but does not mention the ⊗ symbol; expand the caption for readability.
- [Table 5] For Pong, the DM score has a very large standard deviation (64.31 ± 53.24), which suggests instability; the paper does not discuss this outlier in the context of the correlation results.
Circularity Check
No circularity: the DT–DM comparison is an empirical regression/intervention study whose claims do not reduce to their inputs.
full rationale
The paper does not derive a prediction from a fitted parameter or import a load-bearing conclusion via self-citation. The central claim (abstract, Section 4.2.1, Section 6) is supported by directly measured performance gaps (Table 5), random-forest feature importances and SHAP values computed on the 12-game dataset, Pearson correlations, and an action-fusion intervention on Hero and KungFuMaster. None of these steps defines the target variable in terms of the explanatory features or vice versa: y = DT_norm - DM_norm is an independent measurement from the online ALE evaluation, while the game metrics (number of actions, image entropy, compression ratio, feature count, trajectory length, steps to first reward) are computed separately from the DQN-replay dataset. The action-fusion experiment is a genuine manipulation of the action space and therefore provides non-circular evidence for the role of action-space complexity. The main weaknesses - small sample size (n=12), unstable feature importances, a cross-validated RMSE (17.57 +/- 12.35) larger than most observed |y|, high inter-feature correlations, and the use of causal language such as 'primary determining factors' - are validity and generalization concerns, not circularity under the stated criteria. There is no load-bearing self-citation: the cited prior works (Decision Transformer, Decision Mamba, Mamba, ALE, etc.) are external sources, and the paper does not invoke any uniqueness theorem or ansatz from the authors' own prior work to force its conclusions.
Assumptions & free parameters
free parameters (4)
- Expected return multiplier =
5
- Random forest tree count =
1000
- Cross-validation folds =
6
- Context length for extended experiments =
10
assumptions (5)
- domain assumption The 1% sample of DQN-replay data is representative of each game's state and action distribution.
- domain assumption 5 times max return is an appropriate expected-return initialization for all games.
- domain assumption Compression ratio, image entropy, and SIFT feature count are valid measures of task-relevant visual complexity.
- ad hoc to paper Random forest feature importances generalize from 12 games to other Atari-like environments.
- domain assumption Action fusion preserves the game dynamics that matter for comparing DT and DM.
Cite this review
Pith. "Pith review of Decision Transformer vs. Decision Mamba: Analysing the Complexity of Sequential Decision Making in Atari Games." pith.science (2026). https://pith.science/paper/SVUDZPM7
@misc{pith2026241200725,
author = {Pith},
title = {Pith review of: Decision Transformer vs. Decision Mamba: Analysing the Complexity of Sequential Decision Making in Atari Games},
year = {2026},
howpublished = {\url{https://pith.science/paper/SVUDZPM7}},
note = {Machine review of arXiv:2412.00725}
}
read the original abstract
This work analyses the disparity in performance between Decision Transformer (DT) and Decision Mamba (DM) in sequence modelling reinforcement learning tasks for different Atari games. The study first observed that DM generally outperformed DT in the games Breakout and Qbert, while DT performed better in more complicated games, such as Hero and Kung Fu Master. To understand these differences, we expanded the number of games to 12 and performed a comprehensive analysis of game characteristics, including action space complexity, visual complexity, average trajectory length, and average steps to the first non-zero reward. In order to further analyse the key factors that impact the disparity in performance between DT and DM, we employ various approaches, including quantifying visual complexity, random forest regression, correlation analysis, and action space simplification strategies. The results indicate that the performance gap between DT and DM is affected by the complex interaction of multiple factors, with the complexity of the action space and visual complexity (particularly evaluated by compression ratio) being the primary determining factors. DM performs well in environments with simple action and visual elements, while DT shows an advantage in games with higher action and visual complexity. Our findings contribute to a deeper understanding of how the game characteristics affect the performance difference in sequential modelling reinforcement learning, potentially guiding the development of future model design and applications for diverse and complex environments.
Figures
Figures from the paper (8 more)
Reference graph
Works this paper leans on
-
[1]
Decision transformer: Reinforcement learning via sequence modeling,
L. Chen, K. Lu, A. Rajeswaran, K. Lee, A. Grover, M. Laskin, P. Abbeel, A. Srinivas, and I. Mordatch, “Decision transformer: Reinforcement learning via sequence modeling,” in Advances in Neural Information Processing Systems (NeurIPS) , vol. 34. Curran Associates, Inc., 2021, pp. 15 084–15 097. [Online]. Available: https://proceedings.neurips.cc/paper/ 20...
work page 2021
-
[2]
Decision mamba: Reinforcement learning via sequence modeling with selective state spaces,
T. Ota, “Decision mamba: Reinforcement learning via sequence modeling with selective state spaces,” arXiv preprint arXiv:2403.19925 , 2024
arXiv 2024
-
[3]
Mamba: Linear-time sequence modeling with selective state spaces,
A. Gu and T. Dao, “Mamba: Linear-time sequence modeling with selective state spaces,” arXiv preprint arXiv:2312.00752 , 2023
arXiv 2023
-
[4]
A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. u. Kaiser, and I. Polosukhin, “Attention is all you need,” in Advances in Neural Information Processing Systems (NeurIPS) , vol. 30. Curran Associates, Inc., 2017. [Online]. Available: https://papers.nips.cc/paper files/paper/2017/hash/ 3f5ee243547dee91fbd053c1c4a845aa-Abstract.html
work page 2017
-
[5]
BERT: pre-training of deep bidirectional transformers for language understanding,
J. Devlin, M. Chang, K. Lee, and K. Toutanova, “BERT: pre-training of deep bidirectional transformers for language understanding,” in Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short ...
2019
-
[6]
Efficiently modeling long sequences with structured state spaces,
A. Gu, K. Goel, and C. R´ e, “Efficiently modeling long sequences with structured state spaces,” in The International Conference on Learning Representations (ICLR) , 2022
work page 2022
-
[7]
Hippo: Recurrent memory with optimal polynomial projections,
A. Gu, T. Dao, S. Ermon, A. Rudra, and C. R´ e, “Hippo: Recurrent memory with optimal polynomial projections,” in Advances in Neural Information Processing Systems , H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin, Eds., vol. 33. Curran Associates, Inc., 2020, pp. 1474–1487. [Online]. Available: https://proceedings.neurips.cc/ paper files/pap...
work page 2020
-
[8]
S. Hochreiter and J. Schmidhuber, “Long short-term memory,” Neural Comput., vol. 9, no. 8, p. 1735–1780, nov 1997. [Online]. Available: https://doi.org/10.1162/neco.1997.9.8.1735
Show all 24 references
-
[9]
R. S. Sutton and A. G. Barto, Reinforcement Learning: An Introduction , 2nd ed. The MIT Press, 2018. [Online]. Available: http://incompleteideas.net/book/the-book-2nd.html
2018
-
[10]
Decision convformer: Local filtering in metaformer is sufficient for decision making,
J. Kim, S. Lee, W. Kim, and Y. Sung, “Decision convformer: Local filtering in metaformer is sufficient for decision making,” in International Conference on Learning Representations , 2024
2024
-
[11]
Decision s4: Efficient sequence-based rl via state spaces layers,
S. Bar-David, I. Zimerman, E. Nachmani, and L. Wolf, “Decision s4: Efficient sequence-based rl via state spaces layers,” 2023
2023
-
[12]
An optimistic perspective on offline reinforce- ment learning,
R. Agarwal, D. Schuurmans, and M. Norouzi, “An optimistic perspective on offline reinforce- ment learning,” in Proceedings of the 37th International Conference on Machine Learning , ser. ICML’20. JMLR.org, 2020
2020
-
[13]
The arcade learning environment: An evaluation platform for general agents,
M. G. Bellemare, Y. Naddaf, J. Veness, and M. Bowling, “The arcade learning environment: An evaluation platform for general agents,” Journal of Artificial Intelligence Research , vol. 47, pp. 253–279, jun 2013
2013
-
[14]
Mastering atari with discrete world models,
D. Hafner, T. P. Lillicrap, M. Norouzi, and J. Ba, “Mastering atari with discrete world models,” in 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021 . OpenReview.net, 2021. [Online]. Available: https://openreview.net/fo...
2021
-
[15]
Mastering atari games with limited data,
W. Ye, S. Liu, T. Kurutach, P. Abbeel, and Y. Gao, “Mastering atari games with limited data,” in Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, virtual , 2021, pp. 25 476...
2021
-
[16]
Image entropy equalization: A novel preprocessing technique for image recognition tasks,
T. Hayashi, D. Cimr, H. Fujita, and R. Cimler, “Image entropy equalization: A novel preprocessing technique for image recognition tasks,” Information Sciences , vol. 647, p. 119539, 2023. [Online]. Available: https://www.sciencedirect.com/science/article/pii/ S0020025523011246
2023
-
[17]
A mathematical theory of communication,
C. E. Shannon, “A mathematical theory of communication,” The Bell System Technical Journal, vol. 27, no. 3, pp. 379–423, 1948
1948
-
[18]
Salomon and G
D. Salomon and G. Motta, Handbook of Data Compression , 5th ed. Springer Publishing Company, Incorporated, 2009. 22
2009
-
[19]
zlib: A massively spiffy yet delicately unobtrusive compression library,
J.-l. Gailly and M. Adler, “zlib: A massively spiffy yet delicately unobtrusive compression library,” 2004. [Online]. Available: https://github.com/madler/zlib
2004
-
[20]
Object recognition from local scale-invariant features,
D. G. Lowe, “Object recognition from local scale-invariant features,” in Proceedings of the seventh IEEE international conference on computer vision , vol. 2. Ieee, 1999, pp. 1150–1157
1999
-
[21]
Random forests,
L. Breiman, “Random forests,” Machine Learning, vol. 45, no. 1, p. 5–32, oct 2001. [Online]. Available: https://doi.org/10.1023/A:1010933404324
2001 doi
-
[22]
A unified approach to interpreting model predictions,
S. M. Lundberg and S.-I. Lee, “A unified approach to interpreting model predictions,” in Proceedings of the 31st International Conference on Neural Information Processing Systems , ser. NIPS’17. Red Hook, NY, USA: Curran Associates Inc., 2017, p. 4768–4777
2017
-
[23]
Chapter 4 - understanding your data,
J. J. Berman, “Chapter 4 - understanding your data,” in Data Simplification , J. J. Berman, Ed. Morgan Kaufmann, 2016, pp. 135–187. [Online]. Available: https://www.sciencedirect.com/science/article/pii/B9780128037812000047
2016
-
[24]
Correlation coefficients: appropriate use and interpretation,
P. Schober, C. Boer, and L. A. Schwarte, “Correlation coefficients: appropriate use and interpretation,” Anesthesia & analgesia , vol. 126, no. 5, pp. 1763–1768, 2018. 23 Appendices A Hyperparameters Hyperparameter V alue Number of layers 6 Embedding dimension 128 Batch size 2...
2018
Reviewed August 12, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.