Pith. sign in

REVIEW 4 major objections 7 minor 83 references

Closing the Gap between TD Learning and Supervised Learning with $Q$-Conditioned Maximization

T0 review · 4 major / 7 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read GCReinSL conditions a supervised offline RL policy on a flow-estimated, within-dataset maximum Q-value, giving outcome-conditioned behavioral cloning the trajectory-stitching ability of TD learning.

desk verdict A sensible new combination of flows, expectiles, and Q-conditioned policies that gets real empirical stitching gains, but the theoretical identification of the flow with the discounted occupancy Q is under-specified and likely wrong as stated. read the letter →

arxiv 2506.00795 v3 pith:OQNC5CTS submitted 2025-06-01 cs.LG

Xing Lei , Zifeng Zhuang , Shentao Yang , Sheng Xu , Yunhao Luo , Fei Shen , Wenyan Yang , Xuetao Zhang
show 1 more author
Donglin Wang
This is my paper
classification cs.LG
keywords offlinereinforcementlearningtrajectorystitchinggoal-conditionedoutcome-conditionedbehavioralcloningnormalizingflowsexpectileregressionsequencemodeling
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Outcome-conditioned behavioral cloning (OCBC) — supervised-learning methods such as Decision Transformer and RvS that imitate actions given a goal — is known to lack trajectory stitching: it cannot combine fragments of different trajectories into a new path to a distant goal. This paper proposes GCReinSL, which adds a Q-conditioned maximization layer to OCBC: a normalizing flow estimates the goal-reaching probability Q²(s,a,g) of the behavior policy from the offline dataset, an expectile-regression loss drives the model to predict the maximum Q-value present in the data for each state-goal pair, and the policy is trained to act on that predicted maximum. The authors argue that this makes an SL method stitch trajectories the way TD methods do, without bootstrapping or off-policy evaluation. On stitching-oriented goal-conditioned benchmarks and the D4RL Antmaze suite, GCReinSL consistently improves its OCBC backbones and beats goal-data-augmentation baselines, narrowing the gap with TD methods such as IQL and CQL.

What carries the argument

The load-bearing mechanism is the Q-conditioned maximization loop built from three pieces: (1) the probability–reward identity (Theorem 5.1, borrowed from the contrastive-RL literature) that lets a conditional normalizing flow p_ψ(g|s,a), trained by maximum likelihood, serve as the behavior policy's Q-function; (2) expectile regression, whose asymmetric loss L_m converges to $Q^{{max}}$ as m→1 (Theorem 5.2), giving the model a training signal for the maximum in-distribution Q-value without dynamic programming; and (3) the re-conditioned policy π(a|s,g,Q) — the slot in the OCBC objective where the goal used to sit is augmented by the predicted Q-value, so that at inference the model first emits $Q^{{max}}$(s,g) and then acts on it. The flow provides the stitching signal (which future states are actually reachable per the data) and the expectile provides the maximization; neither alone is claimed to suffice.

What would settle it

On a stitching benchmark, estimate Q^β(s,a,g) for held-out (s,a,g) triplets two ways — by the paper's normalizing flow and by Monte Carlo rollouts of the behavior policy — and run GCReinSL with each; if the flow's error against the Monte Carlo values is large on the maze tasks, or the Monte-Carlo-conditioned policy outperforms the flow-conditioned one, the load-bearing density assumption fails exactly where the stitching gains are claimed.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is that the stitching gap between SL and TD offline RL can be closed by replacing the implicit conditioning signal of OCBC with a maximized in-distribution Q-value. Because the reward function r(s,a,g) = (1−γ)γ p(s_{t+1}=g | s_t,a_t) makes the goal-reaching probability under the discounted state-occupancy measure equal the Q-function of the behavior policy (Theorem 5.1), a normalizing flow trained by maximum likelihood on the offline dataset yields Q^β(s,a,g) = p_θ(g|s,a). GCReinSL then trains the OCBC model to output both an action and a Q-value, with an expectile-regression loss whose optimum provably converges to $Q^{{max}}$ = max_{a∼D} Q^β(s,a,g) as the expectile m→1 (Theorem 5.2) — the highest goal-reaching probability the dataset supports, never a global maximum outside the data. At inference the model first predicts this $Q^{{max}}$ for the current (s,g) from the environment, then conditions the policy on it to emit the action. The paper claims this is what equips supervised learning with stitching: the action is chosen to maximize the estimated chance of reaching the goal along paths stitched across separate trajectories, and the whole pipeline avoids TD bootstrapping.

Load-bearing premise

The whole pipeline stands on the assumption that the normalizing flow's maximum-likelihood density genuinely equals the behavior policy's goal-reaching probability Q^β(s,a,g) in the high-dimensional tasks, since a biased estimate makes the maximum-Q condition point toward wrong actions even though the expectile-convergence theorem is correct.

Editorial extensions

If this is right

  • OCBC backbones (DT and RvS) inherit trajectory stitching: on the Pointmaze and Visual-Pointmaze stitching benchmarks, GCReinSL improves both backbones in every task and outperforms the goal-augmentation methods SGDA and TGDA, whose augmented goals can be unreachable.
  • In the return-conditioned setting, swapping Decision Transformer's return-to-go token for the flow-estimated Q-value plus expectile maximization makes GCReinSL the best or runner-up sequence-modeling method on most D4RL Antmaze-v2 tasks, and the authors report this as significantly closing the gap with TD methods such as CQL.
  • Because the max Q-value is taken inside the dataset support, the method inherits the out-of-distribution safety of conservative offline RL by construction rather than by an added penalty term, as argued in the proof of Theorem 5.2.
  • The ablation shows performance tracks estimator accuracy: replacing normalizing flows with CVAE or contrastive estimators degrades results, and success rate rises as the expectile m approaches 1, consistent with the convergence theorem.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper's only direct validation of the flow's density accuracy is a small 5×5 grid world (Appendix G.3), a protocol the authors themselves note is separate from the actual implementation; the same KL-error metric applied to the flow used in the Pointmaze and Antmaze runs would directly test whether flow error explains the performance ceiling in the harder tasks.
  • The expectile parameter m amounts to a one-knob safety-versus-optimality dial (mean at 0.5, maximum as m→1), so failures at m=0.999 read as mild overfitting to extreme dataset Q-values rather than a refutation of the convergence theorem.
  • Because the Q-signal and the policy are trained in two stages, the flow could be pretrained once and reused across different OCBC backbones and unseen goals; nothing in the paper rules out applying the same Q-conditioned maximization to newer sequence-modeling architectures.
  • An implicit testable corollary: if the flow is accurate, the policy conditioned on a Monte-Carlo-estimated Q^{max} for held-out (s,g) pairs should behave almost identically to the flow-conditioned policy, giving a direct way to isolate density error from policy error.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 7 minor

Summary. The paper proposes GCReinSL, a supervised-learning method for offline goal-conditioned RL. It first trains a conditional normalizing flow to estimate the behavior policy's Q-function as the goal-reaching probability in the discounted state-occupancy measure (Eqs. 8-10). It then trains a Q-conditioned OCBC policy with an action MSE loss plus an expectile-regression loss (Eq. 11), so that at inference the policy can be conditioned on a predicted maximum in-distribution Q-value. The method is instantiated with DT and RvS backbones and evaluated on Pointmaze, Visual-Pointmaze, and Antmaze stitching datasets from Ghugare et al. [2024], with an additional return-conditioned extension evaluated on D4RL Antmaze-v2. The reported experiments show consistent improvements over OCBC and goal-data-augmentation baselines, with bootstrap confidence intervals and probability-of-improvement analyses.

Significance. If the mechanism works as claimed, GCReinSL is a useful middle ground between OCBC and TD-based offline RL: it avoids bootstrapping while aiming to provide trajectory stitching, an ability usually associated with TD methods. The paper deserves credit for evaluating on the stitching-oriented datasets of Ghugare et al., for reporting bootstrap confidence intervals and average probability of improvement, and for ablating the expectile level m and the choice of density estimator. However, the central theoretical link between Eq. (9) and the discounted state-occupancy Q-function is not established, the Theorem 5.2 statement and proof are not rigorous as written, and the return-conditioned extension used for Table 1 is underspecified. The empirical contribution is promising, but the theoretical and methodological gaps require substantial revision before the central claims can be accepted.

major comments (4)
  1. [Section 5.3.1, Eqs. (9)-(10), and Appendix G.3] The MLE objective in Eq. (9) trains the flow on the empirical conditional distribution of goals in the offline dataset, not on the discounted state-occupancy measure of Eq. (1). Maximum likelihood over (s,a,g)~D converges to the conditional distribution of g under the dataset's relabeling scheme; Section 3.1 only states that eta_t for 0<=t<=T are used as relabeled goals, with no geometric sampling or gamma^{Delta t} importance weighting. Unless the data construction samples future goals with probability proportional to gamma^t, p_psi(g|s,a) will not equal p_beta_+(g|s,a), and the quantity used as Q_beta in Eq. (7), Theorem 5.2, and the inference procedure is not the discounted occupancy Q-function. The only direct validation in Appendix G.3 is a 5x5 gridworld, and that appendix does not state the relabeling distribution used to train the flow. The authors should either add explicit geometric relabeling or gamma^{Delta t} importance weighting to Eq. (9), or prove that their existing sampling already yields the discounted measure, and they should report Q-estimation checks on the actual maze tasks where the stitching gains are claimed.
  2. [Theorem 5.2 and Appendix A] Theorem 5.2 is not stated as a well-defined mathematical claim. If SG contains a single (s,g,a,Q_beta) tuple, the argmin of the asymmetric loss over that tuple is just Q_beta, not a maximum over actions. If instead the loss is the global expectation in Eq. (7), the minimizer is a function of (s,g) and the theorem needs a dataset-level statement that also accounts for finite model capacity and possibly missing actions. The proof in Appendix A only uses monotonicity and boundedness, which establishes convergence to some limit, not that the limit is Q_max. In addition, the theorem concerns the predicted conditioning value and does not prove anything about the action selected by pi(a|s,g,\hat Q); Section 4's illustrative example is not a proof that conditioning on Q_max makes the policy choose the stitching action. Please restate and prove a well-defined dataset-level version of the theorem, or explicitly label the max-Q conditioning as a heuristic supported by experiments.
  3. [Section B and Table 1] The return-conditioned extension used for the D4RL Antmaze-v2 results is not specified. The normalizing flow in Section 5.3.1 models a density over goals g conditioned on (s,a); Section B does not state what plays the role of the goal when there is no explicit goal state, how Eq. (8) is adapted to returns, how Q_beta is computed and embedded in the DT token sequence, or what the inference-time maximum is taken over. Without this information, the results in Table 1 are not reproducible from the method description, and a significant part of the empirical evaluation is not covered by the algorithmic presentation.
  4. [Section 6.1, Tables 2-3, and Table 1] The evaluation protocol makes the reported gains harder to interpret. The hyperparameters m and K are tuned separately for each dataset, and Appendix F indicates that training steps or learning rate are also adjusted, with no description of a validation-based selection rule; this suggests the possibility of tuning on the final evaluation metric. Table 1 reports the normalized best score for Antmaze-v2 rather than the mean and standard deviation over seeds, which is problematic for a benchmark where D4RL results are normally reported as mean+-std. Please report a fixed or validation-selected hyperparameter protocol and full mean/std for Antmaze-v2, and clarify that the reported comparisons are not the result of per-dataset test-time selection.
minor comments (7)
  1. [Abstract and Section 7] The claim of closing the gap between SL and TD learning is stronger than the evidence: in Table 1, GCReinSL's total is 338.9 compared with IQL's 432.0, so a substantial gap remains; please temper the wording.
  2. [Figures 5-7 and 13-14] In the provided manuscript, several figure axes and labels render as garbled unicode strings rather than readable text; please ensure the final PDF has correctly embedded fonts and labels.
  3. [Eq. (11)] The action MSE and the expectile Q-function loss are combined with equal weights without any ablation or justification; a brief sensitivity analysis over the relative weight would clarify the design choice.
  4. [Algorithms 1 and 2 and Section C] At inference, the model first predicts the Q token with placeholder masks and then feeds that predicted token back to predict the action, while training uses the ground-truth Q token for action prediction; this train/inference discrepancy is not discussed and deserves a few sentences, especially because the predicted Q may be inaccurate.
  5. [Appendix G.3] The gridworld validation of the normalizing flow uses a single dataset of 100 trajectories and reports no error bars or repeated-seed estimates; it is also an on-policy evaluation, whereas the downstream use is offline, so the strength of the validation is more limited than the text suggests.
  6. [Tables 2 and 3] Notation such as "0.9->0.99" and "0.99->0.999" is ambiguous; please clarify whether the arrows indicate the DT-to-RvS setting, a tuning path, or a range of values.
  7. [Footnote in Section C.3] The footnote stating that no strict distinction is made between the V-function and the Q-function is concerning for the RvS variant, since Q_beta(s,a,g) depends on the action and a value model conditioned only on (s,g) would need either an expectation or a maximization over actions; please clarify the exact training target of the value model.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the Q-maximization step is a fitted value used as a conditioning input, but the central performance claims are measured against environment success and the expectile limit is proven rather than imported.

full rationale

The paper's derivation chain is self-contained at the level of construction. Q^β(s,a,g) is defined as the conditional density p_ψ(g|s,a) of a normalizing flow trained by maximum likelihood on the offline dataset (Eqs. 8-10). This is a modeling choice, not an equation that reduces to the target result by definition. Theorem 5.2 then proves, via the monotonicity of m-expectiles, that the expectile regression loss in Eq. 7 drives the predicted Q toward max_{a∼D} Q^β(s,a,g) as m→1; this is a standard consistency property of expectiles, not an imported or self-referential assumption. The policy is trained with the model's own predicted Q as a conditioning input (Eq. 11), but the empirical evaluation of stitching is against environment success rates on Ghugare et al. [2024] and D4RL benchmarks, which are independent of the fitted Q labels. The self-citations to Reinformer [Zhuang et al., 2024] and related one-step RL work are used as inspiration and baselines, not as load-bearing justification for the central claim; the key theoretical statement is proven in the paper itself. The appendix limitation (Section H) explicitly acknowledges that performance depends on the accuracy of the estimated discounted state occupancy distribution, and Appendix G.3 provides a tabular validation of the flow estimator. This is a validation gap or correctness risk, not a circular reduction: nothing in the paper equates the flow's MLE objective with the discounted occupancy measure by construction, and no fitted parameter is renamed as an independent prediction in a way that forces the reported success rates. Therefore the circularity score is 0.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The central method rests on one fitted hyperparameter pair (m, K), on a density-estimation assumption carried over from the normalizing-flow estimator, and on the transfer of conditioning from data Q-values to predicted maximum Q-values. No new physical or conceptual entities are introduced.

free parameters (2)
  • expectile level m = 0.7 to 0.999, chosen per dataset (e.g., 0.99 for most Antmaze-v2, 0.999 for Pointmaze-Large)
    Controls how close the predicted Q-value gets to the maximum in-distribution value; selected per dataset based on performance (Appendix F.1, Table 2).
  • context length K = 2, 3, 5, 10, or 20, chosen per dataset (e.g., 2 for Antmaze-umaze-diverse-v2, 20 for Antmaze-Umaze)
    Sequence context window for the DT backbone; selected per dataset based on stability and performance (Appendix F.2, Table 3).
assumptions (4)
  • standard math Discounted state-occupancy probability equals the Q-function under the goal-conditioned reward definition (Eqs. 3-6).
    Theorem 5.1 restates Proposition 1 of Eysenbach et al. [2022b]; accepted from cited prior work without new proof.
  • domain assumption A normalizing flow trained by maximum likelihood on relabeled offline transitions estimates the discounted future state distribution of the behavior policy.
    Eqs. 9-10 in Section 5.3.1; validated only on a toy grid world in Appendix G.3, and the Limitations section says performance depends on this estimate.
  • standard math The m-expectile of the behavior Q-function converges to the maximum Q-value over actions in the data support as m approaches 1.
    Theorem 5.2 in Section 5.2 and Appendix A; a standard property of expectile regression, though the proof does not fully address conditional function approximation.
  • domain assumption A policy trained with data Q-values as conditioning will act correctly when given the predicted maximum Q-value at inference.
    Section 5.3.2 and C.1: training conditions on Q^beta while inference feeds hat Q_max, a distribution shift that is not directly tested.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Closing the Gap between TD Learning and Supervised Learning with $Q$-Conditioned Maximization." pith.science (2026). https://pith.science/paper/OQNC5CTS

@misc{pith2026250600795,
  author       = {Pith},
  title        = {Pith review of: Closing the Gap between TD Learning and Supervised Learning with $Q$-Conditioned Maximization},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/OQNC5CTS}},
  note         = {Machine review of arXiv:2506.00795}
}
abstract

Recently, supervised learning (SL) methodology has emerged as an effective approach for offline reinforcement learning (RL) due to their simplicity, stability, and efficiency. However, recent studies show that SL methods lack the trajectory stitching capability, typically associated with temporal difference (TD)-based approaches. A question naturally surfaces: \textit{How can we endow SL methods with stitching capability and close its performance gap with TD learning?} To answer this question, we introduce $Q$-conditioned maximization supervised learning for offline goal-conditioned RL, which enhances SL with the stitching capability through $Q$-conditioned policy and $Q$-conditioned maximization. Concretely, we propose \textbf{G}oal-\textbf{C}onditioned \textbf{\textit{Rein}}forced \textbf{S}upervised \textbf{L}earning (\textbf{GC\textit{Rein}SL}), which consists of (1) estimating the $Q$-function by Normalizing Flows from the offline dataset and (2) finding the maximum $Q$-value within the data support by integrating $Q$-function maximization with Expectile Regression. In inference time, our policy chooses optimal actions based on such a maximum $Q$-value. Experimental results from stitching evaluations on offline RL datasets demonstrate that our method outperforms prior SL approaches with stitching capabilities and goal data augmentation techniques.

Figures

Figures reproduced from arXiv: 2506.00795 by the authors.

Figure 1
Figure 1. An illustrative example for stitching analysis. [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. The overview of GCReinSL structure. [s0, s1, s2, s3] ∈ sd, [a0, a1, a2, a3] ∈ ad, [g0, g1, g2, g3] ∈ gd and [Q0, Q1, Q2, Q3] ∈ Qβ come from offline data D. (sr, gr) come from environment. ER denotes Expectile Regression. Qmax denotes in-distribution max Q-value. Qˆ and aˆ represent the predicted Q-value and the output action of the model, respectively. Left: The original offline dataset D. Middle: Normalizing Flows … view at source ↗
Figure 3
Figure 3. Estimating the Q-function of the behavior policy via Normalizing Flows. Left: Original offline trajectory, where the goal g is reachable from the state s. Right: Normalizing Flows are trained to directly estimate the log-likelihood, log p β +(g | s0 = s, a). Note that p β +(g | s0 = s, a) is exactly the goal-reaching probability for the behavior policy β. We employ a conditional Normalizing Flow model fψ : G ×S ×A →… view at source ↗
Figures from the paper (13 more)
Figure 4
Figure 4. Figure 4: Left and Right at the Top: OCBC. Left and Right at the Bottom: GCReinSL. s, g and Qβ are come from offline data D. sr and gr are come from environment. ER denotes Expectile Regression. The red section highlights the differences. To further clarify the differences betwe…
Figure 5
Figure 5. Figure 5: Performance of the original OCBC, as well as OCBC with corresponding goal data [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 6
Figure 6. Figure 6: Qualitative Comparison of DT, TGDA, GCReinSL for DT and IQL on Ghugare et al. [2024] Pointmaze-Medium task. We observe that DT is unable to reach the specified goal (upper right) from start state (bottom left) and lacks stitching capability. Although TGDA can reach the…
Figure 7
Figure 7. Figure 7: Performance of the original OCBC, as well as OCBC with corresponding goal data [PITH_FULL_IMAGE:figures/full_fig_p009_7.png]
Figure 8
Figure 8. Figure 8: Ablation study of different probability estimators and [PITH_FULL_IMAGE:figures/full_fig_p010_8.png]
Figure 9
Figure 9. Figure 9: Left and Right at the Top: DT. Left and Right at the Bottom: GCReinSL. s, a, RTG and Qβ are come from offline data D. sr comes from en￾vironment. ER denotes Expectile Re￾gression. The red section highlights the differences. To further clarify the differences between DT…
Figure 10
Figure 10. Figure 10: , with RvS being similar, differing only in terms of its architecture. C.1 Implementation of GCReinSL for DT 1) 2) 3) GCReinSL for DT [PITH_FULL_IMAGE:figures/full_fig_p016_10.png]
Figure 11
Figure 11. Figure 11: Goal-conditioned RL datasets from Ghugare et al. [2024]: Different colors represent the navigation regions of various data collection policies. During data collection, these policies navigate between randomly selected state-goal pairs within their respective navigatio…
Figure 12
Figure 12. Figure 12: Return-conditioned RL Datasets from Fu et al. [2020]: The AntMaze-v2 datasets involve controlling an 8-DoF quadruped to navigate towards a specified goal state. This benchmark requires value propagation to effectively stitch together sub-optimal trajectories from the …
Figure 13
Figure 13. Figure 13: Average probability of improvement on offline (a) (b) [PITH_FULL_IMAGE:figures/full_fig_p022_13.png]
Figure 14
Figure 14. Figure 14: Performance on high-dimensional Ghugare et al. [2024] Antmaze datasets. GCReinSL can consistently improve the performance of OCBC and surpass goal data augmentation methods on all high-dimensional Antmaze datasets. Error bars denote 95% bootstrap confidence intervals.…
Figure 15
Figure 15. Figure 15: Experiments on the effectiveness of density estimation using Normalizing Flows. Left: We evaluate CVAE, C-learning, CRL and Normalizing Flows for predicting the future state distribution in the on￾policy setting. As anticipated, Normalizing Flows demonstrated the lowe…
Figure 16
Figure 16. Figure 16: Training curves of OCBC and related goal data augmentation methods on [PITH_FULL_IMAGE:figures/full_fig_p025_16.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

83 extracted references · 42 canonical work pages

  1. [1]

    Deep reinforcement learning at the edge of the statistical precipice

    Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron C Courville, and Marc Bellemare. Deep reinforcement learning at the edge of the statistical precipice. Advances in neural information processing systems, 34: 0 29304--29320, 2021

    2021

  2. [2]

    On the estimation of production frontiers: maximum likelihood estimation of the parameters of a discontinuous density function

    Dennis J Aigner, Takeshi Amemiya, and Dale J Poirier. On the estimation of production frontiers: maximum likelihood estimation of the parameters of a discontinuous density function. International economic review, pages 377--396, 1976

    1976

  3. [3]

    Hindsight experience replay

    Marcin Andrychowicz, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, OpenAI Pieter Abbeel, and Wojciech Zaremba. Hindsight experience replay. Advances in neural information processing systems, 30, 2017

    2017

  4. [4]

    Learning successor states and goal-dependent values: A mathematical viewpoint

    L \'e onard Blier, Corentin Tallec, and Yann Ollivier. Learning successor states and goal-dependent values: A mathematical viewpoint. arXiv preprint arXiv:2101.07123, 2021

    arXiv 2021

  5. [5]

    Accelerating goal-conditioned reinforcement learning algorithms and research

    Micha Bortkiewicz, W adys aw Pa ucki, Vivek Myers, Tadeusz Dziarmaga, Tomasz Arczewski, ukasz Kuci \'n ski, and Benjamin Eysenbach. Accelerating goal-conditioned reinforcement learning algorithms and research. In The Thirteenth International Conference on Learning Representations, 2025

    work page 2025

  6. [6]

    Offline rl without off-policy evaluation

    David Brandfonbrener, Will Whitney, Rajesh Ranganath, and Joan Bruna. Offline rl without off-policy evaluation. Advances in neural information processing systems, 34: 0 4933--4946, 2021

    2021

  7. [7]

    When does return-conditioned supervised learning work for offline reinforcement learning? Advances in Neural Information Processing Systems, 35: 0 1542--1553, 2022

    David Brandfonbrener, Alberto Bietti, Jacob Buckman, Romain Laroche, and Joan Bruna. When does return-conditioned supervised learning work for offline reinforcement learning? Advances in Neural Information Processing Systems, 35: 0 1542--1553, 2022

    work page 2022

  8. [8]

    Mamba as decision maker: Exploring multi-scale sequence modeling in offline reinforcement learning

    Jiahang Cao, Qiang Zhang, Ziqing Wang, Jingkai Sun, Jiaxu Wang, Hao Cheng, Yecheng Shao, Wen Zhao, Gang Han, Yijie Guo, et al. Mamba as decision maker: Exploring multi-scale sequence modeling in offline reinforcement learning. arXiv preprint arXiv:2406.02013, 2024

    arXiv 2024

Show all 83 references
  1. [9]

    Goal-conditioned reinforcement learning with imagined subgoals

    Elliot Chane-Sane, Cordelia Schmid, and Ivan Laptev. Goal-conditioned reinforcement learning with imagined subgoals. In International conference on machine learning, pages 1430--1440. PMLR, 2021

    2021

  2. [10]

    On the statistical benefits of temporal difference learning

    David Cheikhi and Daniel Russo. On the statistical benefits of temporal difference learning. In International Conference on Machine Learning, pages 4269--4293. PMLR, 2023

    2023

  3. [11]

    Decision transformer: Reinforcement learning via sequence modeling

    Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Misha Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via sequence modeling. Advances in neural information processing systems, 34: 0 15084--15097, 2021

    2021

  4. [12]

    Goal-conditioned imitation learning

    Yiming Ding, Carlos Florensa, Pieter Abbeel, and Mariano Phielipp. Goal-conditioned imitation learning. Advances in neural information processing systems, 32, 2019

    2019

  5. [13]

    Rvs: What is essential for offline rl via supervised learning? arXiv preprint arXiv:2112.10751, 2021

    Scott Emmons, Benjamin Eysenbach, Ilya Kostrikov, and Sergey Levine. Rvs: What is essential for offline rl via supervised learning? arXiv preprint arXiv:2112.10751, 2021

    2021 arXiv

  6. [14]

    C-learning: Learning to achieve goals via recursive classification

    Benjamin Eysenbach, Ruslan Salakhutdinov, and Sergey Levine. C-learning: Learning to achieve goals via recursive classification. arXiv preprint arXiv:2011.08909, 2020

    2011 arXiv

  7. [15]

    Imitating past successes can be very suboptimal

    Benjamin Eysenbach, Soumith Udatha, Russ R Salakhutdinov, and Sergey Levine. Imitating past successes can be very suboptimal. Advances in Neural Information Processing Systems, 35: 0 6047--6059, 2022 a

    2022

  8. [16]

    Contrastive learning as goal-conditioned reinforcement learning

    Benjamin Eysenbach, Tianjun Zhang, Sergey Levine, and Russ R Salakhutdinov. Contrastive learning as goal-conditioned reinforcement learning. Advances in Neural Information Processing Systems, 35: 0 35603--35620, 2022 b

    2022

  9. [17]

    Inference via interpolation: Contrastive representations provably enable planning and inference

    Benjamin Eysenbach, Vivek Myers, Ruslan Salakhutdinov, and Sergey Levine. Inference via interpolation: Contrastive representations provably enable planning and inference. Advances in Neural Information Processing Systems, 37: 0 58901--58928, 2025

    2025

  10. [18]

    D4rl: Datasets for deep data-driven reinforcement learning

    Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.07219, 2020

    2004 arXiv

  11. [19]

    A minimalist approach to offline reinforcement learning

    Scott Fujimoto and Shixiang Shane Gu. A minimalist approach to offline reinforcement learning. Advances in neural information processing systems, 34: 0 20132--20145, 2021

    2021

  12. [20]

    Learning to reach goals via iterated supervised learning

    Dibya Ghosh, Abhishek Gupta, Ashwin Reddy, Justin Fu, Coline Manon Devin, Benjamin Eysenbach, and Sergey Levine. Learning to reach goals via iterated supervised learning. In International Conference on Learning Representations, 2021

    2021

  13. [21]

    Closing the gap between TD learning and supervised learning - a generalisation point of view

    Raj Ghugare, Matthieu Geist, Glen Berseth, and Benjamin Eysenbach. Closing the gap between TD learning and supervised learning - a generalisation point of view. In The Twelfth International Conference on Learning Representations, 2024

    2024

  14. [22]

    Distance weighted supervised learning for offline interaction data

    Joey Hejna, Jensen Gao, and Dorsa Sadigh. Distance weighted supervised learning for offline interaction data. arXiv preprint arXiv:2304.13774, 2023

    2023 arXiv

  15. [23]

    Diffused task-agnostic milestone planner

    Mineui Hong, Minjae Kang, and Songhwai Oh. Diffused task-agnostic milestone planner. Advances in Neural Information Processing Systems, 36: 0 387--405, 2023

    2023

  16. [24]

    Decision mamba: Reinforcement learning via hybrid selective sequence modeling

    Sili Huang, Jifeng Hu, Zhejian Yang, Liwei Yang, Tao Luo, Hechang Chen, Lichao Sun, and Bo Yang. Decision mamba: Reinforcement learning via hybrid selective sequence modeling. arXiv preprint arXiv:2406.00079, 2024

    2024 arXiv

  17. [25]

    Learning to reach goals via diffusion

    Vineet Jain and Siamak Ravanbakhsh. Learning to reach goals via diffusion. In International Conference on Machine Learning, pages 21170--21195. PMLR, 2024

    2024

  18. [26]

    Efficient planning in a compact latent action space

    Zhengyao Jiang, Tianjun Zhang, Michael Janner, Yueying Li, Tim Rockt \"a schel, Edward Grefenstette, and Yuandong Tian. Efficient planning in a compact latent action space. 2023

    2023

  19. [27]

    Adaptive q -aid for conditional supervised learning in offline reinforcement learning

    Jeonghye Kim, Suyoung Lee, Woojun Kim, and Youngchul Sung. Adaptive q -aid for conditional supervised learning in offline reinforcement learning. Advances in Neural Information Processing Systems, 37: 0 87104--87135, 2024

    2024

  20. [28]

    Imitating graph-based planning with goal-conditioned policies

    Junsu Kim, Younggyo Seo, Sungsoo Ahn, Kyunghwan Son, and Jinwoo Shin. Imitating graph-based planning with goal-conditioned policies. arXiv preprint arXiv:2303.11166, 2023

    2023 arXiv

  21. [29]

    Adam: A method for stochastic optimization

    Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. Computer Science, 2014

    2014

  22. [30]

    An introduction to variational autoencoders

    Diederik P Kingma and Max Welling. An introduction to variational autoencoders. arXiv preprint arXiv:1906.02691, 2019

    1906 arXiv

  23. [31]

    Offline reinforcement learning with implicit q-learning

    Ilya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit q-learning. arXiv preprint arXiv:2110.06169, 2021

    2021 arXiv

  24. [32]

    Stabilizing off-policy q-learning via bootstrapping error reduction

    Aviral Kumar, Justin Fu, Matthew Soh, George Tucker, and Sergey Levine. Stabilizing off-policy q-learning via bootstrapping error reduction. Advances in neural information processing systems, 32, 2019

    2019

  25. [33]

    Conservative q-learning for offline reinforcement learning

    Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. Advances in Neural Information Processing Systems, 33: 0 1179--1191, 2020

    2020

  26. [34]

    Multi-game decision transformers

    Kuang-Huei Lee, Ofir Nachum, Mengjiao Sherry Yang, Lisa Lee, Daniel Freeman, Sergio Guadarrama, Ian Fischer, Winnie Xu, Eric Jang, Henryk Michalewski, et al. Multi-game decision transformers. Advances in Neural Information Processing Systems, 35: 0 27921--27936, 2022 a

    2022

  27. [35]

    Dhrl: a graph-based approach for long-horizon and sparse hierarchical reinforcement learning

    Seungjae Lee, Jigang Kim, Inkyu Jang, and H Jin Kim. Dhrl: a graph-based approach for long-horizon and sparse hierarchical reinforcement learning. Advances in Neural Information Processing Systems, 35: 0 13668--13678, 2022 b

    2022

  28. [36]

    Offline reinforcement learning: Tutorial, review, and perspectives on open problems

    Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint arXiv:2005.01643, 2020

    2005 arXiv

  29. [37]

    Hierarchical planning through goal-conditioned offline reinforcement learning

    Jinning Li, Chen Tang, Masayoshi Tomizuka, and Wei Zhan. Hierarchical planning through goal-conditioned offline reinforcement learning. IEEE Robotics and Automation Letters, 7 0 (4): 0 10216--10223, 2022

    2022

  30. [38]

    Didi: diffusion-guided diversity for offline behavioral generation

    Jinxin Liu, Xinghong Guo, Zifeng Zhuang, and Donglin Wang. Didi: diffusion-guided diversity for offline behavioral generation. arXiv preprint arXiv:2405.14790, 2024 a

    2024 arXiv

  31. [39]

    Beyond ood state actions: Supported cross-domain offline reinforcement learning

    Jinxin Liu, Ziqi Zhang, Zhenyu Wei, Zifeng Zhuang, Yachen Kang, Sibo Gai, and Donglin Wang. Beyond ood state actions: Supported cross-domain offline reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 13945--13953, 2024 b

    2024

  32. [40]

    Least squares quantization in pcm

    Stuart Lloyd. Least squares quantization in pcm. IEEE transactions on information theory, 28 0 (2): 0 129--137, 1982

    1982

  33. [41]

    Decoupled weight decay regularization

    I Loshchilov. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101, 2017

    2017 arXiv

  34. [42]

    Generative trajectory stitching through diffusion composition

    Yunhao Luo, Utkarsh A Mishra, Yilun Du, and Danfei Xu. Generative trajectory stitching through diffusion composition. arXiv preprint arXiv:2503.05153, 2025

    2025 arXiv

  35. [43]

    Decision mamba: A multi-grained state space model with self-evolution regularization for offline rl

    Qi Lv, Xiang Deng, Gongwei Chen, Michael Yu Wang, and Liqiang Nie. Decision mamba: A multi-grained state space model with self-evolution regularization for offline rl. Advances in Neural Information Processing Systems, 37: 0 22827--22849, 2024

    2024

  36. [44]

    Learning latent plans from play

    Corey Lynch, Mohi Khansari, Ted Xiao, Vikash Kumar, Jonathan Tompson, Sergey Levine, and Pierre Sermanet. Learning latent plans from play. In Conference on robot learning, pages 1113--1132. PMLR, 2020

    2020

  37. [45]

    How far i'll go: Offline goal-conditioned reinforcement learning via f -advantage regression

    Yecheng Jason Ma, Jason Yan, Dinesh Jayaraman, and Osbert Bastani. How far i'll go: Offline goal-conditioned reinforcement learning via f -advantage regression. arXiv preprint arXiv:2206.03023, 2022

    2022 arXiv

  38. [46]

    VIP : Towards universal visual reward and representation via value-implicit pre-training

    Yecheng Jason Ma, Shagun Sodhani, Dinesh Jayaraman, Osbert Bastani, Vikash Kumar, and Amy Zhang. VIP : Towards universal visual reward and representation via value-implicit pre-training. In The Eleventh International Conference on Learning Representations, 2023

    2023

  39. [47]

    Human-level control through deep reinforcement learning

    Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. nature, 518 0 (7540): 0 529--533, 2015

    2015

  40. [48]

    Learning temporal distances: Contrastive successor features can provide a metric structure for decision-making

    Vivek Myers, Chongyi Zheng, Anca Dragan, Sergey Levine, and Benjamin Eysenbach. Learning temporal distances: Contrastive successor features can provide a metric structure for decision-making. arXiv preprint arXiv:2406.17098, 2024

    2024 arXiv

  41. [49]

    Asymmetric least squares estimation and testing

    Whitney K Newey and James L Powell. Asymmetric least squares estimation and testing. Econometrica: Journal of the Econometric Society, pages 819--847, 1987

    1987

  42. [50]

    Decision mamba: Reinforcement learning via sequence modeling with selective state spaces

    Toshihiro Ota. Decision mamba: Reinforcement learning via sequence modeling with selective state spaces. arXiv preprint arXiv:2403.19925, 2024

    2024 arXiv

  43. [51]

    Hiql: Offline goal-conditioned rl with latent states as actions

    Seohong Park, Dibya Ghosh, Benjamin Eysenbach, and Sergey Levine. Hiql: Offline goal-conditioned rl with latent states as actions. Advances in Neural Information Processing Systems, 36: 0 34866--34891, 2023

    2023

  44. [52]

    Foundation policies with hilbert representations

    Seohong Park, Tobias Kreiman, and Sergey Levine. Foundation policies with hilbert representations. arXiv preprint arXiv:2402.15567, 2024

    2024 arXiv

  45. [53]

    Ogbench: Benchmarking offline goal-conditioned rl

    Seohong Park, Kevin Frans, Benjamin Eysenbach, and Sergey Levine. Ogbench: Benchmarking offline goal-conditioned rl. In International Conference on Learning Representations (ICLR), 2025

    2025

  46. [54]

    A survey on offline reinforcement learning: Taxonomy, review, and open problems

    Rafael Figueiredo Prudencio, Marcos ROA Maximo, and Esther Luna Colombini. A survey on offline reinforcement learning: Taxonomy, review, and open problems. IEEE Transactions on Neural Networks and Learning Systems, 2023

    2023

  47. [55]

    Goal-conditioned imitation learning using score-based diffusion policies

    Moritz Reuss, Maximilian Li, Xiaogang Jia, and Rudolf Lioutikov. Goal-conditioned imitation learning using score-based diffusion policies. arXiv preprint arXiv:2304.02532, 2023

    2023 arXiv

  48. [56]

    Stochastic backpropagation and approximate inference in deep generative models

    Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and approximate inference in deep generative models. In ICML, 2014

    2014

  49. [57]

    Outcome-driven reinforcement learning via variational inference

    Tim GJ Rudner, Vitchyr Pong, Rowan McAllister, Yarin Gal, and Sergey Levine. Outcome-driven reinforcement learning via variational inference. Advances in Neural Information Processing Systems, 34: 0 13045--13058, 2021

    2021

  50. [58]

    Reinforcement learning upside down: Don't predict rewards -- just map them to actions, 2020

    Juergen Schmidhuber. Reinforcement learning upside down: Don't predict rewards -- just map them to actions, 2020

    2020

  51. [59]

    Rapid exploration for open-world navigation with latent goal models

    Dhruv Shah, Benjamin Eysenbach, Nicholas Rhinehart, and Sergey Levine. Rapid exploration for open-world navigation with latent goal models. In Conference on Robot Learning, pages 674--684. PMLR, 2022

    2022

  52. [60]

    Score models for offline goal-conditioned reinforcement learning

    Harshit Sikchi, Rohan Chitnis, Ahmed Touati, Alborz Geramifard, Amy Zhang, and Scott Niekum. Score models for offline goal-conditioned reinforcement learning. In The Twelfth International Conference on Learning Representations, 2024

    2024

  53. [61]

    Geoadditive expectile regression

    Fabian Sobotka and Thomas Kneib. Geoadditive expectile regression. Computational Statistics & Data Analysis, 56 0 (4): 0 755--767, 2012

    2012

  54. [62]

    Learning structured output representation using deep conditional generative models

    Kihyuk Sohn, Honglak Lee, and Xinchen Yan. Learning structured output representation using deep conditional generative models. Advances in neural information processing systems, 28, 2015

    2015

  55. [63]

    Gymnasium (mar 2023), 2023

    Mark Towers, Jordan K Terry, Ariel Kwiatkowski, JU Balis, Gd Cola, T Deleu, M Goul \ a o, A Kallinteris, A KG, M Krimmel, et al. Gymnasium (mar 2023), 2023

    2023

  56. [64]

    Deep reinforcement learning and the deadly triad

    Hado Van Hasselt, Yotam Doron, Florian Strub, Matteo Hessel, Nicolas Sonnerat, and Joseph Modayil. Deep reinforcement learning and the deadly triad. arXiv preprint arXiv:1812.02648, 2018

    2018 arXiv

  57. [65]

    Attention is all you need

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, ukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017

    2017

  58. [66]

    GOP lan: Goal-conditioned offline reinforcement learning by planning with learned models

    Mianchu Wang, Rui Yang, Xi Chen, Hao Sun, Meng Fang, and Giovanni Montana. GOP lan: Goal-conditioned offline reinforcement learning by planning with learned models. Transactions on Machine Learning Research, 2024 a . ISSN 2835-8856. URL https://openreview.net/forum?id=zOKAmm8R9B

    2024

  59. [67]

    Optimal goal-reaching reinforcement learning via quasimetric learning

    Tongzhou Wang, Antonio Torralba, Phillip Isola, and Amy Zhang. Optimal goal-reaching reinforcement learning via quasimetric learning. In International Conference on Machine Learning, pages 36411--36430. PMLR, 2023

    2023

  60. [68]

    Critic-guided decision transformer for offline reinforcement learning

    Yuanfu Wang, Chao Yang, Ying Wen, Yu Liu, and Yu Qiao. Critic-guided decision transformer for offline reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 15706--15714, 2024 b

    2024

  61. [69]

    Supported policy optimization for offline reinforcement learning

    Jialong Wu, Haixu Wu, Zihan Qiu, Jianmin Wang, and Mingsheng Long. Supported policy optimization for offline reinforcement learning. Advances in Neural Information Processing Systems, 35: 0 31278--31291, 2022

    2022

  62. [70]

    Elastic decision transformer

    Yueh-Hua Wu, Xiaolong Wang, and Masashi Hamaya. Elastic decision transformer. arXiv preprint arXiv:2307.02484, 2023

    2023 arXiv

  63. [71]

    Q-learning decision transformer: Leveraging dynamic programming for conditional sequence modelling in offline rl

    Taku Yamagata, Ahmed Khalil, and Raul Santos-Rodriguez. Q-learning decision transformer: Leveraging dynamic programming for conditional sequence modelling in offline rl. In International Conference on Machine Learning, pages 38989--39007. PMLR, 2023

    2023

  64. [72]

    Rethinking goal-conditioned supervised learning and its connection to offline rl

    Rui Yang, Yiming Lu, Wenzhe Li, Hao Sun, Meng Fang, Yali Du, Xiu Li, Lei Han, and Chongjie Zhang. Rethinking goal-conditioned supervised learning and its connection to offline rl. arXiv preprint arXiv:2202.04478, 2022

    2022 arXiv

  65. [73]

    What is essential for unseen goal generalization of offline goal-conditioned rl? In International Conference on Machine Learning, pages 39543--39571

    Rui Yang, Lin Yong, Xiaoteng Ma, Hao Hu, Chongjie Zhang, and Tong Zhang. What is essential for unseen goal generalization of offline goal-conditioned rl? In International Conference on Machine Learning, pages 39543--39571. PMLR, 2023 a

    2023

  66. [74]

    a m \"a r \

    Wenyan Yang, Huiling Wang, Dingding Cai, Joni Pajarinen, and Joni-Kristen K \"a m \"a r \"a inen. Swapped goal-conditioned offline reinforcement learning. arXiv preprint arXiv:2302.08865, 2023 b

    2023 arXiv

  67. [75]

    Breadth-first exploration on adaptive grid for reinforcement learning

    Youngsik Yoon, Gangbok Lee, Sungsoo Ahn, and Jungseul Ok. Breadth-first exploration on adaptive grid for reinforcement learning. In Forty-first International Conference on Machine Learning, 2024

    2024

  68. [76]

    Goal-conditioned predictive coding for offline reinforcement learning

    Zilai Zeng, Ce Zhang, Shijie Wang, and Chen Sun. Goal-conditioned predictive coding for offline reinforcement learning. Advances in Neural Information Processing Systems, 36: 0 25528--25548, 2023

    2023

  69. [77]

    Stabilizing contrastive rl: Techniques for offline goal reaching

    Chongyi Zheng, Benjamin Eysenbach, Homer Walke, Patrick Yin, Kuan Fang, Ruslan Salakhutdinov, and Sergey Levine. Stabilizing contrastive rl: Techniques for offline goal reaching. arXiv preprint arXiv:2306.03346, 2023

    2023 arXiv

  70. [78]

    Contrastive difference predictive coding

    Chongyi Zheng, Ruslan Salakhutdinov, and Benjamin Eysenbach. Contrastive difference predictive coding. In The Twelfth International Conference on Learning Representations, 2024

    2024

  71. [79]

    Online decision transformer

    Qinqing Zheng, Amy Zhang, and Aditya Grover. Online decision transformer. In international conference on machine learning, pages 27042--27059. PMLR, 2022

    2022

  72. [80]

    Behavior proximal policy optimization

    Zifeng Zhuang, Kun Lei, Jinxin Liu, Donglin Wang, and Yilang Guo. Behavior proximal policy optimization. arXiv preprint arXiv:2302.11312, 2023

    2023 arXiv

  73. [81]

    Reinformer: Max-return sequence modeling for offline rl

    Zifeng Zhuang, Dengyun Peng, Ziqi Zhang, Donglin Wang, et al. Reinformer: Max-return sequence modeling for offline rl. arXiv preprint arXiv:2405.08740, 2024

    2024 arXiv

  74. [82]

    Revisiting the design choices in max-return sequence modeling, 2025

    Zifeng Zhuang, Dengyun Peng, Donglin Wang, Jiacheng Liu, Xing Lei, Diyuan Shi, and Ziqi Zhang. Revisiting the design choices in max-return sequence modeling, 2025

    2025

  75. [83]

    Maximum entropy inverse reinforcement learning

    Brian D Ziebart, Andrew L Maas, J Andrew Bagnell, Anind K Dey, et al. Maximum entropy inverse reinforcement learning. In Aaai, volume 8, pages 1433--1438. Chicago, IL, USA, 2008

    2008

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.