pith. sign in

arxiv: 2606.19721 · v1 · pith:YRDM2IGWnew · submitted 2026-06-18 · 💻 cs.LG · cs.AI

OnDeFog: Online Decision Transformer under Frame Dropping

Daiki Yotsufuji , Kenta Nishihara , Shoma Shimizu , Kento Uchida , Shinichi Shirakawa This is my paper

Pith reviewed 2026-06-26 18:33 UTC · model grok-4.3

classification 💻 cs.LG cs.AI
keywords reinforcement learningdecision transformerframe droppingonline learningoffline learningpolicy learning
0
0 comments X

The pith

OnDeFog integrates DeFog mechanisms into the online decision transformer to maintain performance when frames drop at high rates and when training data contains many low-reward episodes.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper addresses frame dropping in reinforcement learning, where agents miss states and rewards due to delays or sensor issues. It combines the frame-handling additions from the offline DeFog method with the online decision transformer so the agent can continue learning through direct environment interaction rather than relying solely on a fixed dataset. This targets DeFog's weakness on unseen states and the online baseline's weakness at high drop rates. A reader would care because many deployed RL systems face unreliable observation streams, and a method that works without perfect data could make policies more practical.

Core claim

OnDeFog integrates the mechanisms developed in DeFog for handling random frame dropping with the online decision transformer. Comprehensive experimental evaluation demonstrates that OnDeFog achieves superior performance compared to ODT in environments characterized by high dropping frame rate and outperforms DeFog on datasets containing a large amount of low-reward data.

What carries the argument

Integration of DeFog's frame-dropping mechanisms into the online decision transformer architecture to support live interaction under missing observations.

If this is right

  • OnDeFog produces higher returns than the base online decision transformer when frame dropping rates increase.
  • OnDeFog produces higher returns than DeFog when the offline dataset includes many low-reward trajectories.
  • Online learning can incorporate offline frame-drop mitigations to reduce reliance on complete high-quality datasets.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Similar mechanism transfers could be tested on other online transformer variants to handle partial observations.
  • The method may reduce the volume of high-reward data needed to train reliable policies in unreliable sensor settings.

Load-bearing premise

The mechanisms added in DeFog transfer directly to the online decision transformer setting without requiring substantial new adaptations or introducing instability during live interaction.

What would settle it

Running the evaluated environments at high frame drop rates and observing that OnDeFog returns are not higher than those of ODT, or that OnDeFog returns are not higher than DeFog on low-reward datasets, would falsify the performance advantage.

Figures

Figures reproduced from arXiv: 2606.19721 by Daiki Yotsufuji, Kenta Nishihara, Kento Uchida, Shinichi Shirakawa, Shoma Shimizu.

Figure 1
Figure 1. Figure 1: Overview of OnDeFog model that no frame dropping occurs in the trajectories τ obtained during the online RL stage. This two-staged training enables efficient exploration while leveraging the pre-collected dataset. Under the assumption that optimal return for the task is positive, the target return gonline during online learning is set to twice the optimal return. This scaled target value encourages the age… view at source ↗
Figure 2
Figure 2. Figure 2: IQMs of return for varying frame drop rates and their 95% confidence intervals analyzing RL experimental results aligns with recommendations from existing research [1]. 5.4 Comparative Experiments This experiment was conducted to perform a comparative analysis of the pro￾posed OnDeFog against the existing DeFog and ODT. The evaluation results for different datasets and frame drop rates are presented in [P… view at source ↗
Figure 3
Figure 3. Figure 3: IQMs of return for frame drop rates and their 95% confidence intervals Based on these findings, train-time frame dropping in the proposed OnDe￾Fog is crucial for adapting to dropping environments. Additionally, drop-span embedding is indispensable for enhancing stable learning in the presence of dropping. Meanwhile, freeze-trunk fine-tuning demonstrated limited effective￾ness compared to other components … view at source ↗
read the original abstract

In challenging real-world reinforcement learning applications, communication delays or sensor failures often cause frame dropping, in which the agent cannot receive the dropped states and associated rewards. To address the performance degradation caused by frame dropping, the Decision Transformer under Random Frame Dropping (DeFog) was developed by incorporating additional mechanisms into the decision transformer to tackle frame dropping. Although DeFog can mitigate performance degradation in frame-dropping environments, since DeFog is an offline learning method, it struggles to effectively generalize to novel states not adequately represented in the training dataset. In this study, we propose OnDeFog, which integrates the mechanisms in DeFog with the online decision transformer (ODT), an online reinforcement learning method that learns policies through direct environmental interaction. Comprehensive experimental evaluation demonstrates that our proposed OnDeFog achieves superior performance compared to ODT in environments characterized by high dropping frame rate and outperforms DeFog on datasets containing a large amount of low-reward data.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 0 minor

Summary. The paper proposes OnDeFog by integrating the frame-dropping mechanisms from the offline DeFog method into the Online Decision Transformer (ODT). It claims that this yields superior performance to ODT under high frame-drop rates and to DeFog on datasets containing large amounts of low-reward data, supported by comprehensive experimental evaluation.

Significance. If the experimental claims are substantiated with adequate controls and the integration is shown to preserve stability, the work would usefully extend decision-transformer methods to online RL under realistic sensor/communication failures. It addresses a practical gap between offline robustness techniques and online policy improvement.

major comments (2)
  1. [Abstract] Abstract: the assertion of 'comprehensive experimental evaluation' demonstrating superiority supplies no information on the baselines compared, number of independent runs, statistical tests, or the precise procedure used to simulate frame dropping. These omissions are load-bearing for the central performance claims.
  2. [Method] Method / integration description: the manuscript states that OnDeFog 'integrates the mechanisms in DeFog with the online decision transformer' but provides no evidence or description confirming that the mechanisms transferred without modification or that the online interaction loop remained stable. This assumption directly underpins the reported gains versus ODT and DeFog.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the constructive comments. We address each major point below and indicate where revisions will be made to the manuscript.

read point-by-point responses

  1. Referee: [Abstract] Abstract: the assertion of 'comprehensive experimental evaluation' demonstrating superiority supplies no information on the baselines compared, number of independent runs, statistical tests, or the precise procedure used to simulate frame dropping. These omissions are load-bearing for the central performance claims.

    Authors: We agree the abstract is concise and omits these specifics. The body of the manuscript specifies the baselines (ODT and DeFog), frame-dropping simulation (random dropping at given rates), and results over multiple independent runs. To improve clarity, we will revise the abstract to briefly note the key comparisons and frame-dropping procedure. revision: yes

  2. Referee: [Method] Method / integration description: the manuscript states that OnDeFog 'integrates the mechanisms in DeFog with the online decision transformer' but provides no evidence or description confirming that the mechanisms transferred without modification or that the online interaction loop remained stable. This assumption directly underpins the reported gains versus ODT and DeFog.

    Authors: We acknowledge that the current description is high-level. In the revised version we will expand the method section with a detailed account of how each DeFog mechanism is adapted into the ODT architecture (including any modifications), together with experimental evidence confirming stability of the online loop under frame dropping. revision: yes

Circularity Check

0 steps flagged

No derivation chain present; claims rest on experimental comparisons only.

full rationale

The paper proposes OnDeFog via integration of prior DeFog mechanisms into ODT and supports superiority via experimental results on frame-dropping environments and low-reward datasets. No equations, derivations, fitted parameters, or self-citation chains are described in the abstract or referenced structure. The central claim does not reduce to any input by construction, self-definition, or renaming; it is an empirical assertion open to external verification. This matches the default expectation of no significant circularity.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 0 invented entities

Only the abstract is available; no details on parameters, assumptions, or new entities are provided.

pith-pipeline@v0.9.1-grok · 5702 in / 948 out tokens · 20482 ms · 2026-06-26T18:33:57.830106+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

19 extracted references · 5 canonical work pages · 4 internal anchors

  1. [1]

    Advances in Neural Information Processing Systems (2021)

    Agarwal, R., Schwarzer, M., Castro, P.S., Courville, A., Bellemare, M.G.: Deep reinforcement learning at the edge of the statistical precipice. Advances in Neural Information Processing Systems (2021)

    2021

  2. [2]

    In: 1992 American Control Conference

    Balemi, S., Brunner, U.: Supervision of discrete event systems with communication delays. In: 1992 American Control Conference. pp. 2794–2798 (1992)

    1992

  3. [3]

    In: International Conference on Learning Represen- tations (2021)

    Bouteiller, Y., Ramstedt, S., Beltrame, G., Pal, C.J., Binas, J.: Reinforcement learning with random delays. In: International Conference on Learning Represen- tations (2021)

    2021

  4. [4]

    OpenAI Gym

    Brockman, G., Cheung, V., Pettersson, L., Schneider, J., Schulman, J., Tang, J., Zaremba, W.: OpenAI Gym. arXiv:1606.01540 (2016)

    work page internal anchor Pith review Pith/arXiv arXiv 2016

  5. [5]

    Chapman, D., Kaelbling, L.P.: Input generalization in delayed reinforcement learn- ing:Analgorithmandperformancecomparisons.In:InternationalJointConference on Artificial Intelligence. vol. 91, pp. 726–731 (1991)

    1991

  6. [6]

    Advances in Neural Information Processing Systems34, 15084–15097 (2021)

    Chen,L.,Lu, K.,Rajeswaran, A.,Lee,K., Grover, A.,Laskin,M., Abbeel, P.,Srini- vas, A., Mordatch, I.: Decision Transformer: Reinforcement learning via sequence modeling. Advances in Neural Information Processing Systems34, 15084–15097 (2021)

    2021

  7. [7]

    Machine Learning110(9), 2419–2468 (2021)

    Dulac-Arnold, G., Levine, N., Mankowitz, D.J., Li, J., Paduraru, C., Gowal, S., Hester,T.:Challengesofreal-worldreinforcementlearning:definitions,benchmarks and analysis. Machine Learning110(9), 2419–2468 (2021)

    2021

  8. [8]

    D4RL: Datasets for Deep Data-Driven Reinforcement Learning

    Fu, J., Kumar, A., Nachum, O., Tucker, G., Levine, S.: D4RL: Datasets for deep data-driven reinforcement learning. arXiv:2004.07219 (2020)

    work page internal anchor Pith review Pith/arXiv arXiv 2004

  9. [9]

    In: International Conference on Advanced Robotics

    Funda, J., Paul, R.: Efficient control of a robotic system for time-delayed environ- ments. In: International Conference on Advanced Robotics. pp. 219–224 (1991)

    1991

  10. [10]

    In: International Conference on Machine Learning

    Haarnoja, T., Zhou, A., Abbeel, P., Levine, S.: Soft Actor-Critic: Off-policy maxi- mum entropy deep reinforcement learning with a stochastic actor. In: International Conference on Machine Learning. pp. 1861–1870. PMLR (2018)

    2018

  11. [11]

    In: International Conference on Learning Representations (2023)

    Hu, K., Zheng, R.C., Gao, Y., Xu, H.: Decision transformer under random frame dropping. In: International Conference on Learning Representations (2023)

    2023

  12. [12]

    Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems

    Levine, S., Kumar, A., Tucker, G., Fu, J.: Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv:2005.01643 (2020)

    work page internal anchor Pith review Pith/arXiv arXiv 2005

  13. [13]

    Playing Atari with Deep Reinforcement Learning

    Mnih, V., Kavukcuoglu, K., Silver, D., Graves, A., Antonoglou, I., Wierstra, D., Riedmiller, M.A.: Playing Atari with deep reinforcement learning. arXiv:1312.5602 (2013)

    work page internal anchor Pith review Pith/arXiv arXiv 2013

  14. [14]

    In: ACM International Conference on Information & Knowledge Management

    Nath, S., Baranwal, M., Khadilkar, H.: Revisiting state augmentation methods for reinforcement learning with stochastic delays. In: ACM International Conference on Information & Knowledge Management. pp. 1346–1355 (2021)

    2021

  15. [15]

    arXiv:1912.10944 (2019)

    Shao, K., Tang, Z., Zhu, Y., Li, N., Zhao, D.: A survey of deep reinforcement learning in video games. arXiv:1912.10944 (2019)

    work page arXiv 1912

  16. [16]

    In: Deep Reinforcement Learning Workshop NeurIPS 2022 (2022)

    Siedler, P.D.: Dynamic collaborative multi-agent reinforcement learning commu- nication for autonomous drone reforestation. In: Deep Reinforcement Learning Workshop NeurIPS 2022 (2022)

    2022

  17. [17]

    Advances in Neural Information Pro- cessing Systems (2017)

    Vaswani,A.,Shazeer,N.,Parmar,N.,Uszkoreit,J.,Jones,L.,Gomez,A.N.,Kaiser, L., Polosukhin, I.: Attention is all you need. Advances in Neural Information Pro- cessing Systems (2017)

    2017

  18. [18]

    In: International Conference on Learning Representations (2020) OnDeFog: Online Decision Transformer under Frame Dropping 11

    You, Y., Li, J., Reddi, S., Hseu, J., Kumar, S., Bhojanapalli, S., Song, X., Demmel, J., Keutzer, K., Hsieh, C.J.: Large batch optimization for deep learning: Train- ing bert in 76 minutes. In: International Conference on Learning Representations (2020) OnDeFog: Online Decision Transformer under Frame Dropping 11

    2020

  19. [19]

    In: International Conference on Machine Learning

    Zheng, Q., Zhang, A., Grover, A.: Online decision transformer. In: International Conference on Machine Learning. pp. 27042–27059. PMLR (2022) 12 Y. Daiki et al. T able 1.Model architecture and hyperparameters of Decision Transformer Hyperparameter Value Number of Layers 4 Number of Attention Heads 4 Number of Embedding Dimensions 512 Training Context Leng...

    2022