Pith. sign in

REVIEW 3 major objections 5 minor 13 references

Directly Forecasting Belief for Reinforcement Learning with Delays

T0 review · 3 major / 5 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A transformer that directly forecasts delayed states avoids the exponential compounding errors of step-by-step belief prediction in reinforcement learning.

desk verdict Sensible direct-forecasting idea with strong empirical support, but the theory overclaims: the recursion bound is only exponential when L_P > 1, and Assumption 5.6 is nearly tautological. read the letter →

arxiv 2505.00546 v2 pith:CJHZPJGL submitted 2025-05-01 cs.LG

Qingyuan Wu , Yuhui Wang , Simon Sinong Zhan , Yixuan Wang , Chung-Wei Lin , Chen Lv , Qi Zhu , Jürgen Schmidhuber
show 1 more author
Chao Huang
This is my paper · ORCID
classification cs.LG
keywords reinforcementlearningwithdelaysbeliefrepresentationcompoundingerrorssequencemodelingtransformerdirectforecastingmulti-stepbootstrappingsoftactor-critic
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Reinforcement learning with delayed observations is hard because the agent must infer the current state from outdated sensory data. Existing belief-based methods do this by recursively predicting the missing states one step at a time, and each step multiplies the approximation error, so the belief degrades exponentially with the delay length. This paper argues that the recursion itself is the problem, and proposes a transformer-based method that directly forecasts all missing states at once from the delayed observation window. The claim, backed by a theoretical comparison of performance-degeneration bounds, is that direct forecasting replaces an exponentially growing error term with a delay-independent one, and the paper shows empirically on the MuJoCo benchmark that this yields better prediction accuracy and much stronger reinforcement learning performance at long delays.

What carries the argument

The central object is the directly forecasting belief transformer, a transformer that maps the delayed information state—reformulated as a sequence of tokens $\{s_{t-\Delta}, a_{t-\Delta+i}, r_{t-\Delta+i}\}_{i=0}^{\Delta-1}$—directly to the block of $\Delta$ missing states. It replaces the recursive invocation of an approximate dynamics function $\Delta$ times with a single parallel prediction, so that the per-step approximation error $\epsilon_P$ is never iterated and compounded. The theoretical argument also relies on a Lipschitz continuity assumption on the MDP dynamics and the value function to translate belief error into a performance-degeneration bound.

What would settle it

Train a direct forecaster and a recursive forecaster on the same delayed Markov decision process, and measure the belief error at increasing delays beyond the training distribution: if the direct forecast's error grows at a comparable exponential rate to the recursive one, the paper's central claim in Proposition 5.7 collapses.

Watch

Extended reading notes

Core claim

The central claim is that the belief-estimation bottleneck in delayed RL is not the difficulty of predicting future states, but the recursive way existing methods chain predictions together, and that this bottleneck can be bypassed by treating belief forecasting as a direct sequence-modeling task. The paper introduces the Directly Forecasting Belief Transformer (DFBT), which takes the augmented state of the last observable state and the intervening actions plus rewards, reformatted as a token sequence, and predicts the full block of unobserved states $s_{t-\Delta+1:t}$ in parallel. The theoretical result compares the performance difference of a recursively forecast belief, bounded by $L_V \frac{1-L_P^\Delta}{1-L_P}\epsilon_P$, with the directly forecast belief, bounded by $L_V\epsilon_{\text{direct}}$, showing the recursive error grows exponentially with delay while the direct error does not. The paper further shows that, when the direct forecast is accurate enough, its performance guarantee is strictly better than the recursive one, and integrates this direct forecast with multi-step bootstrapping in a Soft Actor-Critic agent, which both improves learning efficiency and delivers substantially higher returns than augmentation-based and belief-based baselines on MuJoCo at delays from 8 to 128 steps.

Load-bearing premise

The theoretical advantage of direct forecasting rests entirely on the assumption that the one-shot direct forecast error $\epsilon_{\text{direct}}$ has a fixed bound that does not grow with the delay length, yet the paper never derives or proves that bound.

Editorial extensions

If this is right

  • If the direct-forecast error stays bounded independently of delay, belief-based delayed RL can handle far longer delays than recursive methods, because the dominant exponential error term is removed.
  • The same direct-forecasting principle applies to any partially observable or delayed control problem where the hidden quantity is the state and the delay window is known, including action-delayed systems after the standard reduction.
  • Multi-step bootstrapping on the forecasted state sequence can shorten the effective credit-assignment horizon in delayed RL, improving sample efficiency without requiring a learned world model.
  • Because DFBT does not invoke the forward dynamics model repeatedly, its inference cost stays roughly constant with delay, offering a computational advantage over recursive belief methods at long horizons.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The theoretical comparison is conditional: the direct method wins only if its one-shot error $\epsilon_{\text{direct}}$ stays meaningfully below the recursively accumulated error. A long-horizon direct forecast may itself degrade with delay in harder environments, so the practical claim should be read as 'the error grows more slowly than the compounding recursive error', which the paper's own expe
  • The direct-forecast architecture is a strong candidate for any setting where state is hidden by latency—not just fixed delays—and where a transformer can be pre-trained on recorded trajectories, such as teleoperation logs in robotics.
  • A natural stress test of the method's reach is to evaluate it in stochastic environments with noisy dynamics, where the belief is a true distribution rather than a point estimate, and where error accumulation may resurface in the form of variance growth.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper addresses reinforcement learning with observation delays. It proposes DFBT, a transformer-based belief estimator that directly forecasts the sequence of unobserved states from the augmented delayed observation, rather than recursively applying a learned one-step dynamics model. The trained DFBT is then combined with Soft Actor-Critic to form DFBT-SAC, which also uses multi-step bootstrapping on the predicted states. The paper claims a theoretical demonstration that direct forecasting reduces compounding errors relative to recursive belief forecasting, and supports this with D4RL-based belief accuracy experiments and MuJoCo performance comparisons against augmentation-based and belief-based baselines.

Significance. If the claims were fully established, this would be a useful contribution to delayed RL: direct multi-step belief forecasting is a natural idea, and the empirical results are encouraging. DFBT-SAC consistently outperforms belief-based baselines in the reported MuJoCo tasks, especially at long delays, and the paper includes useful ablations (bootstrapping steps, belief fine-tuning, stochastic environments, inference speed) plus released code. The theoretical part, however, is currently the weakest link: the exponential-error claim in Theorem 5.5 depends on an unstated L_P > 1 condition, and the direct-method guarantee rests on an assumption about epsilon_direct whose magnitude and delay-dependence are never analyzed. The paper would be sound if the theoretical claims were reworded as conditional statements and the empirical results were presented as the main evidence.

major comments (3)
  1. [Section 5.1, Theorem 5.5] The bound in Theorem 5.5 has the form L_V (1 - L_P^Delta)/(1 - L_P) epsilon_P. For L_P < 1, this expression converges to L_V epsilon_P/(1 - L_P) as Delta grows, so it is not exponential in the delay; for L_P > 1 it does grow exponentially, but the authors never state or justify the condition L_P > 1. The repeated claim in Section 5.1 and the abstract that the error is 'exponentially increased' with delays is therefore only true under an unstated and nonstandard assumption. This is load-bearing because it motivates the entire direct-forecasting design.
  2. [Section 5.2, Assumption 5.6 and Propositions 5.7-5.8] Assumption 5.6 defines epsilon_direct := max_i W(b^(i)(.|x_t) || b_theta^(i)(.|x_t)) and then asserts W(b(.|x_t) || b_theta(.|x_t)) <= epsilon_direct. Since W(b || b_theta) is one of the terms in the maximum, this inequality is automatic; the real content is that epsilon_direct is small and delay-independent. No bound on epsilon_direct is derived from sample size, model capacity, or the training loss, and Section 6.3 explicitly concedes that the sample complexity of directly forecasting belief is not theoretically analyzed. The comparison in Proposition 5.8 is therefore conditional on exactly the quantity that the paper needs to control. The abstract and conclusion should be reworded to state this conditionality clearly.
  3. [Section 6.2.1 and Remark 5.9] Figure 2 reports L1 belief error, while Theorem 5.5 and Proposition 5.8 are stated in terms of W(p||q) and the performance differences I_recursive and I_direct. The empirical curves are not a direct validation of Proposition 5.8 as claimed in Remark 5.9. The authors should either compute the corresponding Wasserstein distances or explicitly describe Figure 2 as indirect evidence rather than validation of the theorem.
minor comments (5)
  1. [Definition 5.2] There are typos in Definition 5.2: 'distrance' should be 'distance' and 'repectively' should be 'respectively'.
  2. [Definition 5.3] The notation LQ-LC and Lpi-LC is used without being defined; please define these or explain the terminology in the text.
  3. [Section 4.1, Equation (2)] The gradient expression in Equation (2) is written without an expectation or sample-average notation; clarify that the loss is an empirical expectation over the offline dataset.
  4. [Appendix B.2 and Remark 5.9] References are inconsistent: the proof of Proposition B.2 says 'Applying Theorem 5.6' when it should refer to Assumption 5.6, and Remark 5.9 refers to 'Theorem 5.8' when the statement is Proposition 5.8.
  5. [Section 6.2.5 and Table 4] Table 4's rows 'Online', 'Offline', and 'Offline + Fine-tuning' are not defined in the caption or the surrounding text; clarify whether the results in Tables 1 and 2 use the offline-trained belief without fine-tuning, and explain how fine-tuning is integrated with Algorithm 1.

Circularity Check

1 steps flagged · score 6.0 of 10

The theoretical guarantee reduces to a self-defined bound: Assumption 5.6 defines epsilon_direct as the maximum direct-forecast belief error, making the assumption tautological, and Proposition 5.8 then assumes epsilon_direct is small instead of deriving it; empirical benchmarks remain independent.

  1. self definitional [Section 5.2, Assumption 5.6 and Proposition 5.8; cf. Section 6.3]
    "Assumption 5.6 (Directly Forecasting Belief Difference Bound). The distance between the directly forecasting belief belief bθ parameterized by θ and the ground-truth belief b is bounded, it satisfies that ∀xt ∈ X, we have W(b(·|xt)||bθ(·|xt))≤ϵ direct, where ϵdirect := maxi=1,...,∆W(b(i)(·|xt)||b(i) θ (·|xt))."

    epsilon_direct is defined as max_i W(b^(i)||b_theta^(i)); since b=b^(Delta) is one term of that max, the 'assumed' bound is a tautology that constrains nothing. Proposition 5.8 then conditions the claimed advantage on epsilon_direct <= ((1-L_P^Delta)/(1-L_P)) epsilon_P, i.e., on exactly the desired smallness of the direct error. No bound on epsilon_direct is derived; Section 6.3 concedes the sample complexity is not analyzed, and Remark 5.10 cites multi-step forecasting literature where direct error grows with horizon. Thus the reduction of compounding errors is assumed under the name epsilon_direct, not proved.

full rationale

The empirical core of the paper is not circular: DFBT is trained on D4RL data and its belief error and MuJoCo returns are compared against published baselines (DATS, D-Dreamer, D-SAC, BPQL, ADRL), with hyperparameters reported in the appendix and no fitted constant used to produce the theoretical results. Self-citations to the authors' own prior delayed-RL papers (ADRL, VDPO) occur mainly as baselines and are not load-bearing. However, the theoretical claim in the abstract—that DFBT 'greatly reduces compounding errors ... yielding stronger performance guarantees'—rests on Assumption 5.6 and Propositions 5.7–5.8. Assumption 5.6 defines epsilon_direct as the maximum direct-forecast error, so the assumed bound is a tautology; Proposition 5.8's comparison is conditional on exactly the quantity that needs to be controlled. The paper explicitly defers the sample-complexity analysis that would bound epsilon_direct (Section 6.3). Thus the central theoretical guarantee partially reduces to its own input, while the empirical demonstration remains independent evidence. This warrants a partial-circularity score of 6 rather than a higher score, because no fitted prediction is relabeled as a result and the benchmark comparisons stand on their own.

Assumptions & free parameters 1 free parameters · 4 assumptions · 0 invented entities

The theoretical comparison rests on four assumptions: two Lipschitz conditions, a per-step model error bound, and a delay-independent direct-forecast error bound. The delay-independent bound is critical and unproven; it is essentially the result being argued. No fitted constants or new entities are introduced. The experimental results provide independent support for the practical claim.

free parameters (1)
  • Bootstrapping steps N = 8 (default)
    Chosen by hand and ablated in Table 3; N=8 gives the best average performance on Walker2d-v2, but the gain shrinks at delays 64 and 128.
assumptions (4)
  • domain assumption MDP is Lipschitz continuous with constant L_P (Definition 5.2).
    Used in Theorem 5.5 to propagate belief error through one-step dynamics. This is common in model-based RL but is not guaranteed by the MuJoCo environments.
  • domain assumption Value function V^π is Lipschitz with constant L_V (Definition 5.3).
    Used to convert Wasserstein distance between beliefs into a value-function difference in the performance bounds.
  • ad hoc to paper The approximated dynamic P_θ has error bounded by ϵ_P (Assumption 5.4).
    Provides the per-step error injected into recursive forecasting; no constructive guarantee of such a bound is given.
  • ad hoc to paper The directly forecasting belief has a delay-independent error bound ϵ_direct (Assumption 5.6).
    This is the load-bearing premise: it assumes the direct forecaster's error does not grow with delay. The desired conclusion, that direct forecasting is better, is effectively embedded in this assumption, and no proof or sample-complexity bound is supplied. The Limitations section admits this.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Directly Forecasting Belief for Reinforcement Learning with Delays." pith.science (2026). https://pith.science/paper/CJHZPJGL

@misc{pith2026250500546,
  author       = {Pith},
  title        = {Pith review of: Directly Forecasting Belief for Reinforcement Learning with Delays},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/CJHZPJGL}},
  note         = {Machine review of arXiv:2505.00546}
}
read the original abstract

Reinforcement learning (RL) with delays is challenging as sensory perceptions lag behind the actual events: the RL agent needs to estimate the real state of its environment based on past observations. State-of-the-art (SOTA) methods typically employ recursive, step-by-step forecasting of states. This can cause the accumulation of compounding errors. To tackle this problem, our novel belief estimation method, named Directly Forecasting Belief Transformer (DFBT), directly forecasts states from observations without incrementally estimating intermediate states step-by-step. We theoretically demonstrate that DFBT greatly reduces compounding errors of existing recursively forecasting methods, yielding stronger performance guarantees. In experiments with D4RL offline datasets, DFBT reduces compounding errors with remarkable prediction accuracy. DFBT's capability to forecast state sequences also facilitates multi-step bootstrapping, thus greatly improving learning efficiency. On the MuJoCo benchmark, our DFBT-based method substantially outperforms SOTA baselines. Code is available at https://github.com/QingyuanWuNothing/DFBT.

Figures

Figures reproduced from arXiv: 2505.00546 by the authors.

Figure 1
Figure 1. Pipeline of DFBT-SAC. (a) Training DFBT on the trajectory dataset. (b) The agent can interact and learn with the delay-free environment recovered by the DFBT as highlighted in the dashline box. (c) Multi-step bootstrapping on the forecasted states from DFBT. Before starting the theoretical analysis, we introduce the definition of the performance degeneration of the ground￾truth belief b as follows. Definition 5.1 (P… view at source ↗
Figure 2
Figure 2. Belief errors comparison on (a) HalfCheetah-v2, (b) Hopper-v2, and (c) Walker2d-v2. sequence-modeling method estimates the states st−∆+1:t based on the augmented state xt in parallel, instead of in￾voking ∆ times iteratively, which effectively alleviates the source of compounding errors. Especially, the attention mechanism in the transformer can selectively capture the long-range relationships within the augmented s… view at source ↗
Figure 3
Figure 3. Learning Curves on MuJoCo with 128 Delays and U(1, 128) Delays. delays, DFBT-SAC can learn remarkably faster than all baselines, resulting in 173.3% (HalfCheetah-v2), 122.2% (Hopper-v2), and 263.6% (Walker2d-v2) final performance improvement better than the second-best baselines. Results Analysis and Discussion. Based on the above experimental results, we can observe a general trend in both deterministic and stochas… view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: Learning Curves on MuJoCo with Deterministic Delays. 18 [PITH_FULL_IMAGE:figures/full_fig_p018_4.png]
Figure 5
Figure 5. Figure 5: Learning Curves on MuJoCo with Stochastic Delays. 19 [PITH_FULL_IMAGE:figures/full_fig_p019_5.png]
Figure 6
Figure 6. Figure 6: Belief qualitative comparison on HalfCheetah-v2 with different delays. 21 [PITH_FULL_IMAGE:figures/full_fig_p021_6.png]
Figure 7
Figure 7. Figure 7: Belief qualitative comparison on Hopper-v2 with different delays. 22 [PITH_FULL_IMAGE:figures/full_fig_p022_7.png]
Figure 8
Figure 8. Figure 8: Belief qualitative comparison on Walker2d-v2 with different delays. 23 [PITH_FULL_IMAGE:figures/full_fig_p023_8.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

13 extracted references · 5 canonical work pages

  1. [4]

    Soft actor-critic algorithms and applications.arXiv preprint arXiv:1812.05905,

    Haarnoja, T., Zhou, A., Hartikainen, K., Tucker, G., Ha, S., Tan, J., Kumar, V ., Zhu, H., Gupta, A., Abbeel, P., et al. Soft actor-critic algorithms and applications.arXiv preprint arXiv:1812.05905,

    arXiv

  2. [5]

    Dream to control: Learning behaviors by latent imagination.arXiv preprint arXiv:1912.01603,

    Hafner, D., Lillicrap, T., Ba, J., and Norouzi, M. Dream to control: Learning behaviors by latent imagination.arXiv preprint arXiv:1912.01603,

    arXiv 1912

  3. [9]

    Playing atari with deep reinforcement learning.arXiv preprint arXiv:1312.5602,

    Mnih, V ., Kavukcuoglu, K., Silver, D., Graves, A., Antonoglou, I., Wierstra, D., and Riedmiller, M. Playing atari with deep reinforcement learning.arXiv preprint arXiv:1312.5602,

    arXiv

  4. [13]

    1−L P δ 1−L P ϵP # , then it is obvious that we have Itrue ∆ (xt) +LVϵdirect | {z } |Idirect(xt)| ≤ Itrue ∆ (xt) +LV E δ∼d∆(·)

    and CleanRL (Huang et al., 2022). The codebase for reproducing our experimental results is also provided in the Supplementary Material. We detail the hyperparameter settings of DFBT and DFBT-SAC in Table 8 and Table 9, respectively. Table 8.Hyper-parameters table of DFBT. Hyper-parameter Value Epoch 1e3 Batch Size 256 Attention Heads Num 4 Layers Num 10 H...

    work page 2022

  5. [126]

    f¨ur Informatik, 1990b

    Inst. f¨ur Informatik, 1990b. Schmidhuber, J. Reinforcement learning upside down: Don’t predict rewards–just map them to actions.arXiv preprint arXiv:1912.02875,

    arXiv 1912

  6. [2009]

    Highway Reinforcement Learning

    Wang, W., Han, D., Luo, X., and Li, D. Addressing signal delay in deep reinforcement learning. InThe Twelfth 11 Directly Forecasting Belief for Reinforcement Learning with Delays International Conference on Learning Representations, 2023a. Wang, Y ., Zhan, S., Wang, Z., Huang, C., Wang, Z., Yang, Z., and Zhu, Q. Joint differentiable optimization and verif...

    work page Pith review arXiv 2023

  7. [2017]

    Wu, Q., Zhan, S

    doi: 10.1145/3061639.3062224. Wu, Q., Zhan, S. S., Wang, Y ., Wang, Y ., Lin, C.-W., Lv, C., Zhu, Q., and Huang, C. Variational delayed policy optimization.Advances in neural information processing systems, 2024a. Wu, Q., Zhan, S. S., Wang, Y ., Wang, Y ., Lin, C.-W., Lv, C., Zhu, Q., Schmidhuber, J., and Huang, C. Boosting reinforcement learning with str...

    arXiv

  8. [2018]

    Dota 2 with large scale deep reinforcement learning.arXiv preprint arXiv:1912.06680,

    Berner, C., Brockman, G., Chan, B., Cheung, V ., Debiak, P., Dennison, C., Farhi, D., Fischer, Q., Hashme, S., Hesse, C., et al. Dota 2 with large scale deep reinforcement learning.arXiv preprint arXiv:1912.06680,

    arXiv 1912

Show all 13 references
  1. [2019]

    and Shaked, U

    Fridman, E. and Shaked, U. On reachable sets for linear systems with delay and bounded peak inputs.Automatica, 39(11):2005–2010,

    2005

  2. [2020]

    and Schmidhuber, J

    Ha, D. and Schmidhuber, J. World models.arXiv preprint arXiv:1803.10122,

    arXiv

  3. [2021]

    Rein- forcement learning from delayed observations via world models.arXiv preprint arXiv:2403.12309,

    Karamzade, A., Kim, K., Kalsi, M., and Fox, R. Rein- forcement learning from delayed observations via world models.arXiv preprint arXiv:2403.12309,

    arXiv

  4. [2022]

    Control of a quadrotor with reinforcement learning.IEEE Robotics and Automation Letters, 2(4):2096–2103,

    Hwangbo, J., Sa, I., Siegwart, R., and Hutter, M. Control of a quadrotor with reinforcement learning.IEEE Robotics and Automation Letters, 2(4):2096–2103,

  5. [2023]

    Learning a belief representation for delayed reinforcement learning

    Liotet, P., Venneri, E., and Restelli, M. Learning a belief representation for delayed reinforcement learning. In 2021 International Joint Conference on Neural Networks (IJCNN), pp. 1–8. IEEE,

    2021

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.