Pith. sign in

REVIEW 3 major objections 5 minor 25 references

Should We Ever Prefer Decision Transformer for Offline Reinforcement Learning?

T0 review · 3 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Filtered Behavior Cloning, an MLP trained only on the best trajectories, matches or beats Decision Transformer in sparse-reward offline RL, leading the authors to conclude that DT is not preferable in this setting.

desk verdict A useful empirical check on Decision Transformer's sparse-reward advantage, but the Robomimic best-epoch reporting and hand-set filter threshold mean the strong conclusion outruns the evidence. read the letter →

arxiv 2507.10174 v1 pith:IRUN6QO4 submitted 2025-07-14 cs.AI cs.LG

Yumi Omori , Zixuan Dong , Keith Ross This is my paper · ORCID
classification cs.AIcs.LG
keywords offlinereinforcementlearningDecisionTransformerFilteredBehaviorCloningsparserewardsD4RLRobomimicreturn-to-goconditioning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper asks whether the Transformer-based Decision Transformer (DT) is ever the right choice for offline reinforcement learning, focusing on the sparse-reward case where a recent systematic comparison argued DT is superior. It proposes Filtered Behavior Cloning (FBC): keep only the high-performing trajectories in the offline dataset and then run ordinary behavior cloning with a small MLP. On nine sparsified D4RL locomotion datasets FBC beats DT in seven and raises the aggregate normalized score from 75.39 to 78.22; on the two sparse Robomimic manipulation tasks it beats DT on both, with aggregate success 0.89 versus 0.86. FBC also uses roughly half the parameters, less training data, and about one third of the wall-clock time. The paper therefore concludes that DT is not preferable for sparse-reward raw-state tasks, and that the burden is on showing any setting where vanilla DT wins.

What carries the argument

The central object is Filtered Behavior Cloning (FBC), an MLP that applies vanilla behavior cloning to a filtered dataset. In the sparse setting it keeps only successful trajectories; in the sparsified setting it keeps trajectories whose final return is in the top 10% of the dataset. The conceptual machinery is the observation that DT's return-to-go input in these settings is just a quality label: conditioning on it tells the policy which training trajectories are good, which is exactly the information FBC uses to discard the rest. Filtered Decision Transformer (FDT), the same filter applied to DT, serves as the ablation that isolates filtering from architecture.

What would settle it

Re-run the sparsified D4RL experiments while sweeping the FBC filter threshold (for example x=1%, 5%, 20%, 50%) and also run FBC on Robomimic without access to success labels, filtering by predicted or proxy quality instead. If DT's normalized score or success rate exceeds FBC at any reasonable threshold, or if FBC's ranking flips when labels are unavailable, the paper's central claim fails.

Watch

Extended reading notes

Core claim

In sparse-reward offline RL, the return-to-go conditioning that defines DT does little more than separate good trajectories from bad ones, and a plain MLP trained on the good trajectories alone can match or beat DT. The paper's empirical evidence is FBC outperforming DT on 7 of 9 sparsified D4RL datasets (aggregate 78.22 vs 75.39) and on both Robomimic sparse tasks (Lift 0.94 vs 0.92; PickPlaceCan 0.84 vs 0.79), while using fewer parameters, less data, and much less training time. Training DT on the same filtered data (FDT) does not recover an advantage, which the paper reads as evidence that DT's benefit in sparse settings is essentially its implicit selection by return-to-go rather than its sequence modeling. The conclusion is stated directly: for sparse reward environments, DT is not preferable.

Load-bearing premise

The result depends on the hand-chosen rule that keeping only the top 10% of trajectories (or only the labeled successes) is the right way to filter; the paper reports no sensitivity analysis, so if that threshold is wrong for a given task, FBC's edge over DT could disappear.

Editorial extensions

If this is right

  • In sparse-reward offline RL, practitioners can replace DT with filtered behavior cloning and get comparable or better scores at a fraction of the compute and data.
  • Claims that DT is preferable in sparse-reward or low-quality-data settings should be tested against a filtered behavior cloning baseline before being accepted.
  • Because FDT does not beat DT, filtering alone is not what makes transformers competitive; the MLP backbone of FBC carries the advantage in these tasks.
  • Together with the dense-reward evidence the paper cites, the sparse-reward results leave no demonstrated raw-state robotic task where vanilla DT is the preferred algorithm.
  • The question in the title remains open for dense-reward settings, but the paper narrows the candidate regime by removing sparse-reward tasks.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper fixes the filter threshold at 10% for sparsified D4RL. A natural next test is to sweep that threshold: if another threshold makes FBC clearly worse than DT, the practical conclusion would need to be qualified.
  • The Robomimic experiment uses ground-truth success labels to filter. In applications where such labels do not exist, one could estimate trajectory quality from episode return or learned classifiers and check whether FBC's edge survives.
  • The comparison is on raw-state, proprioceptive tasks. Testing FBC on image-based or stochastic environments would show whether the conclusion extends beyond the settings studied here.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper compares Decision Transformer (DT) with a proposed Filtered Behavior Cloning (FBC) baseline in two sparse-reward offline RL settings: sparsified D4RL locomotion datasets and sparse Robomimic manipulation tasks. FBC filters trajectories by success (sparse setting) or by top-10% final return (sparsified setting) and then trains a standard MLP with behavior cloning. The authors report that FBC achieves higher aggregate D4RL scores than DT, higher best success rates on both Robomimic tasks, and shorter wall-clock training time, and they conclude that DT is not preferable for sparse-reward environments.

Significance. If the empirical claims are robust, the paper provides a useful negative result for offline RL: a simple MLP with a trajectory-filtering step can match or beat a transformer-based sequence model in sparse-reward settings, with lower computational cost. The paper's FBC is refreshingly simple, the experimental setup closely follows Bhargava et al. (2024), and the authors report hyperparameters, 5-seed averages, and learning curves. However, the central comparison is not yet established: the Robomimic results are based on best-epoch selection, the D4RL '7 of 9' claim ignores overlapping standard deviations, and the 10% filter threshold is unexamined. These issues directly affect the paper's main conclusion, so the contribution is currently conditional on additional analysis.

major comments (3)
  1. [§5.2, Table 2] The table reports 'the best success rate' achieved during training, not final performance. Selecting the best evaluation epoch introduces selection bias that favors methods with higher variance, and the reported FBC-over-DT gaps (Lift 0.94 vs 0.92 with std 0.03; Can 0.84 vs 0.79 with std 0.06) are within one standard deviation of each other. Since Robomimic is the only genuinely sparse-reward (binary terminal reward) benchmark in the paper, the Section 6 conclusion that 'DT is not preferable' for sparse-reward environments is not supported until final-epoch or checkpoint-averaged success rates, with confidence intervals, are reported.
  2. [§5.1, Table 1] The claim that 'FBC beats DT in 7 of 9 datasets' is not accompanied by any significance analysis. For example, Hopper Medium-Expert differs by only 0.09 (111.46 vs 111.37) with overlapping standard deviations, and several other per-dataset differences are within one standard deviation. The aggregate total average (78.22 vs 75.39) is also close relative to the quoted variability. Please report paired comparisons or bootstrap confidence intervals across the 5 seeds and state how many of the 7 wins are statistically reliable; relatedly, the Section 6 statement that FBC 'has better performance than DT' overstates the per-dataset results, since DT is higher on two of the nine D4RL tasks.
  3. [§4, filtering rule] The FBC advantage depends on the hand-chosen top-10% threshold x=10% for sparsified D4RL, and the paper provides no sensitivity analysis. A brief sweep (e.g., x=5%, 10%, 20% or a validation-based selection) is needed to show that the conclusion is not an artifact of this threshold. For the sparse setting, the reliance on ground-truth success labels should also be discussed, since such labels may not be available in real applications.
minor comments (5)
  1. [§7 vs §2] Section 7 refers to 'Graph Decision Transformer (GDT)' citing Hu et al. (2023), while Section 2 uses the same acronym for 'Generalized Decision Transformer' citing Furuta et al. (2022); please disambiguate the two methods and acronyms.
  2. [§5 and Appendix A.3] Section 5 says evaluation is conducted every 50 epochs, but Table 4 in Appendix A.3 lists 'Every 50 epochs (Robomimic), Every 100 epochs (D4RL)'; these statements should be reconciled.
  3. [Figure 2 caption] The caption uses the task name 'PickupCan', whereas the main text and Table 3 use 'PickPlaceCan'; please make the naming consistent.
  4. [Reproducibility] No code or data release is described; providing an anonymous repository would improve reproducibility, especially since the experimental setup is claimed to follow Bhargava et al. (2024).
  5. [Table 4, RTG values] The return-to-go values for DT (e.g., 6000 for HalfCheetah, 120 for Robomimic) are listed without explanation of how they were chosen; please clarify whether they are taken directly from Bhargava et al. (2024) or tuned separately.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper's claim is an independent empirical comparison, not a derivation that reduces to its own inputs.

full rationale

The paper's central claim is that Filtered Behavior Cloning (FBC) achieves competitive or superior performance relative to Decision Transformer (DT) on sparse-reward benchmarks. This is supported by direct evaluations on external D4RL and Robomimic datasets, where all scores are measured from rollouts rather than derived from any fitted parameter. FBC is defined by a fixed filtering rule: keep successful trajectories in the sparse setting and keep the top 10% of final returns in the sparsified setting, with x = 10% stated explicitly and not tuned to the reported outcomes. The 10% threshold is a hyperparameter chosen before evaluation, not a quantity fitted to the data and then re-predicted, so it does not create a fitted-input-called-prediction loop. The Robomimic tables report the best success rate achieved during training, and no sensitivity analysis is provided for the 10% threshold; these are legitimate methodological validity concerns, but they concern robustness and statistical reporting rather than circular reasoning. The paper cites Bhargava et al. (2024) for the experimental protocol and for the claim that DT beats BC and CQL, but the FBC and FDT results are newly measured in this paper, and the conclusion that FBC is competitive with DT is not forced by any equation, definition, or self-citation chain. There is no load-bearing self-citation, no imported uniqueness theorem, and no ansatz smuggled in via citation. The comparison is self-contained against external benchmarks, so no circularity is present.

Assumptions & free parameters 1 free parameters · 4 assumptions · 0 invented entities

The central empirical claim rests on the sparsified-reward equivalence, the representativeness of two MG Robomimic tasks, and reuse of prior hyperparameters. The main free parameter of the paper's own method is the FBC filtering threshold, which is chosen without sensitivity analysis.

free parameters (1)
  • FBC top-x% return filter threshold = x = 10% (sparsified D4RL); ground-truth success in Robomimic sparse
    Set by hand in Section 4 ('In this paper, we use x = 10%.') with no sensitivity analysis; the claimed FBC advantage over DT depends on this choice and on access to success labels.
assumptions (4)
  • domain assumption Sparsifying dense rewards by zeroing intermediate rewards and moving total return to the final timestep faithfully represents true sparse-reward environments.
    Section 3 defines the sparsified setting and the paper's D4RL conclusions rest entirely on this transformation, which is inherited from Bhargava et al. (2024).
  • domain assumption The Machine-Generated (MG) Robomimic datasets are representative of sparse-reward robotic manipulation tasks with low-quality data.
    Section 5.2 and Appendix A.1 restrict evaluation to the MG variant, chosen because DT was previously reported to excel there; generalizing beyond these two tasks is assumed.
  • domain assumption Hyperparameters taken from Bhargava et al. (2024) are appropriate for all compared methods, including FBC, so no per-method tuning is needed.
    Section 5 and Appendix A.3 state all experiments follow the prior setup; if these hyperparameters are suboptimal for DT or unusually favorable to FBC, the comparison could change.
  • domain assumption Success labels are available for filtering in the Robomimic sparse setting.
    The FBC filtering rule in Section 4 retains only successful trajectories; in applications where success is not labeled, FBC cannot be applied as described.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Should We Ever Prefer Decision Transformer for Offline Reinforcement Learning?." pith.science (2026). https://pith.science/paper/IRUN6QO4

@misc{pith2026250710174,
  author       = {Pith},
  title        = {Pith review of: Should We Ever Prefer Decision Transformer for Offline Reinforcement Learning?},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/IRUN6QO4}},
  note         = {Machine review of arXiv:2507.10174}
}
read the original abstract

In recent years, extensive work has explored the application of the Transformer architecture to reinforcement learning problems. Among these, Decision Transformer (DT) has gained particular attention in the context of offline reinforcement learning due to its ability to frame return-conditioned policy learning as a sequence modeling task. Most recently, Bhargava et al. (2024) provided a systematic comparison of DT with more conventional MLP-based offline RL algorithms, including Behavior Cloning (BC) and Conservative Q-Learning (CQL), and claimed that DT exhibits superior performance in sparse-reward and low-quality data settings. In this paper, through experimentation on robotic manipulation tasks (Robomimic) and locomotion benchmarks (D4RL), we show that MLP-based Filtered Behavior Cloning (FBC) achieves competitive or superior performance compared to DT in sparse-reward environments. FBC simply filters out low-performing trajectories from the dataset and then performs ordinary behavior cloning on the filtered dataset. FBC is not only very straightforward, but it also requires less training data and is computationally more efficient. The results therefore suggest that DT is not preferable for sparse-reward environments. From prior work, arguably, DT is also not preferable for dense-reward environments. Thus, we pose the question: Is DT ever preferable?

Figures

Figures reproduced from arXiv: 2507.10174 by the authors.

Figure 1
Figure 1. Illustrations of the Lift and PickPlaceCan tasks in Robosuite (Zhu et al., 2020b). We evaluate algorithms in sparse-reward robotic manipulation tasks using the Robosuite simulator (Zhu et al., 2020a) and the Robomimic dataset suite (Mandlekar et al., 2021). Robosuite is a mod￾ular simulation environment supporting both dense and sparse reward configurations. Following the experimental setup of Bhargava et al. (2024)… view at source ↗
Figure 2
Figure 2. Results for Robomimic tasks. Performance comparison of FBC, FDT, DT. [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗
Figure 3
Figure 3. Performance of FBC, FDT, and DT on D4RL. [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

25 extracted references · 22 canonical work pages

  1. [1]

    Bhargava, S

    D. Bhargava, S. Paternain, M. R. Rilo, L. Pinto, P. Abbeel, and A. Filos. When should we prefer decision transformers for offline reinforcement learning? In The Twelfth International Conference on Learning Representations (ICLR), 2024

    work page 2024

  2. [2]

    Q-transformer: Scalable offline reinforcement learning via autoregressive q-functions

    Yevgen Chebotar, Quan Vuong, Karol Hausman, Fei Xia, Yao Lu, Alex Irpan, Aviral Kumar, Tianhe Yu, Alexander Herzog, Karl Pertsch, et al. Q-transformer: Scalable offline reinforcement learning via autoregressive q-functions. In Conference on Robot Learning, 2023

    work page 2023

  3. [3]

    L. Chen, K. Lu, A. Rajeswaran, K. Lee, A. Grover, M. Laskin, P. Abbeel, A. Srinivas, and I. Mordatch. Decision transformer: Reinforcement learning via sequence modeling. In Advances in Neural Information Processing Systems (NeurIPS), 2021

    work page 2021

  4. [4]

    RvS : What is essential for offline rl via supervised learning? In NeurIPS Offline Reinforcement Learning Workshop, 2021

    Scott Emmons, Zichuan Wang, Aditya Anand, Evan Zheran Langlois, Yikai Tian, Xuechen Li, Yilun Du, Ahmed Touati, Sudeep Choudhury, Seyed Kamyar Seyed Ghasemipour, et al. RvS : What is essential for offline rl via supervised learning? In NeurIPS Offline Reinforcement Learning Workshop, 2021

    work page 2021

  5. [5]

    D4rl: Datasets for deep data-driven reinforcement learning

    Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.07219, 2020

    arXiv 2004

  6. [6]

    Generalized decision transformer for offline hindsight information matching

    Hiroki Furuta, Yutaka Matsuo, and Shixiang Shane Gu. Generalized decision transformer for offline hindsight information matching. In International Conference on Learning Representations, 2022

    work page 2022

  7. [7]

    Instruction-driven history-aware policies for robotic manipulations

    Pierre-Louis Guhur, Shizhe Chen, Ricardo Garcia Pinel, Makarand Tapaswi, Ivan Laptev, and Cordelia Schmid. Instruction-driven history-aware policies for robotic manipulations. In Conference on Robot Learning, 2023

    work page 2023

  8. [8]

    Graph Decision Transformer

    Shengchao Hu, Li Shen, Ya Zhang, and Dacheng Tao. Graph decision transformer. arXiv preprint arXiv:2303.03747, 2023

    work page Pith review arXiv 2023

Show all 25 references
  1. [9]

    Q-value regularized transformer for offline reinforcement learning

    Shengchao Hu, Ziqing Fan, Chaoqin Huang, Li Shen, Ya Zhang, Yanfeng Wang, and Dacheng Tao. Q-value regularized transformer for offline reinforcement learning. In Proceedings of the 41st International Conference on Machine Learning, 2024

    2024

  2. [10]

    Janner, Q

    M. Janner, Q. Li, S. Levine, and C. Finn. Trajectory transformer. In Advances in Neural Information Processing Systems (NeurIPS), 2021

    2021

  3. [11]

    Kumar, A

    A. Kumar, A. Zhou, G. Tucker, and S. Levine. Conservative Q -learning for offline reinforcement learning. In H. Larochelle, M. Ranzato, R. Hadsell, M.-F. Balcan, and H.-T. Lin (eds.), Advances in Neural Information Processing Systems (NeurIPS), volume 33, pp.\ 1179--1191. Curr...

    2020

  4. [12]

    Pre-trained language models for interactive decision-making

    Shuang Li, Xavier Puig, Chris Paxton, Yilun Du, Clinton Wang, Linxi Fan, Tao Chen, De-An Huang, Ekin Aky \"u rek, Anima Anandkumar, et al. Pre-trained language models for interactive decision-making. Advances in Neural Information Processing Systems, 2022

    2022

  5. [13]

    A survey on transformers in reinforcement learning

    Wenzhe Li, Hao Luo, Zichuan Lin, Chongjie Zhang, Zongqing Lu, and Deheng Ye. A survey on transformers in reinforcement learning. Transactions on Machine Learning Research, 2023

    2023

  6. [14]

    What matters in learning from offline human demonstrations for robot manipulation

    Ajay Mandlekar, Danfei Xu, Roberto Mart \' n-Mart \' n Wang, Li Fei-Fei, Silvio Savarese, Yuke Zhu, et al. What matters in learning from offline human demonstrations for robot manipulation. Robotics: Science and Systems (RSS), 2021

    2021

  7. [15]

    Stabilizing transformers for reinforcement learning

    Emilio Parisotto, Francis Song, Jack Rae, Razvan Pascanu, Caglar Gulcehre, Siddhant Jayakumar, Max Jaderberg, Raphael Lopez Kaufman, Aidan Clark, Seb Noury, et al. Stabilizing transformers for reinforcement learning. In International conference on machine learning, 2020

    2020

  8. [16]

    You can’t count on luck: Why decision transformers and rvs fail in stochastic environments

    Keiran Paster, Sheila McIlraith, and Jimmy Ba. You can’t count on luck: Why decision transformers and rvs fail in stochastic environments. Advances in neural information processing systems, 2022

    2022

  9. [17]

    N. M. M. Shafiullah, Z. An, Q. Luo, M. Janner, T. Jaakkola, B. Boots, and S. Levine. Behavior transformers: Cloning k modes with one stone. In Advances in Neural Information Processing Systems (NeurIPS), 2022

    2022

  10. [18]

    The sensory neuron as a transformer: Permutation-invariant neural networks for reinforcement learning

    Yujin Tang and David Ha. The sensory neuron as a transformer: Permutation-invariant neural networks for reinforcement learning. Advances in Neural Information Processing Systems, 2021

    2021

  11. [19]

    CORL : Research-oriented deep offline reinforcement learning library

    Denis Tarasov, Alexander Nikulin, Dmitry Akimov, Vladislav Kurenkov, and Sergey Kolesnikov. CORL : Research-oriented deep offline reinforcement learning library. Advances in Neural Information Processing Systems, 2023

    2023

  12. [20]

    Vaswani, N

    A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin. Attention is all you need. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neural Information Processing Syst...

    2017

  13. [21]

    Bootstrapped transformer for offline reinforcement learning

    Kerong Wang, Hanye Zhao, Xufang Luo, Kan Ren, Weinan Zhang, and Dongsheng Li. Bootstrapped transformer for offline reinforcement learning. Advances in Neural Information Processing Systems, 2022

    2022

  14. [22]

    Q-learning decision transformer: Leveraging dynamic programming for conditional sequence modelling in offline rl

    Taku Yamagata, Ahmed Khalil, and Raul Santos-Rodriguez. Q-learning decision transformer: Leveraging dynamic programming for conditional sequence modelling in offline rl. In International Conference on Machine Learning, 2023

    2023

  15. [23]

    Robosuite: A modular simulation framework and benchmark for robot learning

    Yuke Zhu, Ajay Mandlekar, Animesh Gao, Li Fei-Fei, Silvio Savarese, et al. Robosuite: A modular simulation framework and benchmark for robot learning. arXiv preprint arXiv:2009.12293, 2020 a

    2009 arXiv

  16. [24]

    Robosuite environments documentation

    Yuke Zhu, Josiah Wong, Ajay Mandlekar, Roberto Mart\' i n-Mart\' i n, Abhishek Joshi, Kevin Lin, Abhiram Maddukuri, Soroush Nasiriany, and Yifeng Zhu. Robosuite environments documentation. https://robosuite.ai/docs/modules/environments.html, 2020 b

    2020

  17. [25]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.