Pith. sign in

REVIEW 4 major objections 6 minor 92 references

Temporal Distance-aware Transition Augmentation for Offline Model-based Reinforcement Learning

T0 review · 4 major / 6 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read TempDATA shows that offline model-based reinforcement learning can solve sparse-reward, long-horizon goal-reaching tasks when transitions are augmented in a temporal-distance-aware latent space.

desk verdict TempDATA is a promising empirical contribution to offline MBRL for sparse-reward long-horizon tasks, but a duplicated table, a shaky theorem proof, and a missing ablation that isolates augmentation from reward shaping mean the core mechanism is not yet nailed down. read the letter →

arxiv 2505.13144 v1 pith:THFVHA74 submitted 2025-05-19 cs.LG cs.AIcs.RO

Dongsu Lee , Minhae Kwon This is my paper · ORCID
classification cs.LGcs.AIcs.RO
keywords offlinereinforcementlearningmodel-basedRLtemporaldistancelatentrepresentationgoal-conditioneddataaugmentationsparserewardlong-horizontasks
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that offline model-based reinforcement learning fails on sparse-reward, long-horizon goal-reaching tasks because it augments transitions in raw state space, where Euclidean closeness has little to do with how long a task actually takes, so synthetic rollouts cross obstacles or miss the goal. TempDATA instead learns a latent space in which the distance $d(f(s),f(s_{\text{goal}}))$ between two encoded states approximates the shortest temporal path between them, enforced at both the trajectory level (how far a goal is) and the transition level (single-step coherence). A dynamics model trained in this latent space generates augmented transitions, and an intrinsic reward $\tilde{r}(s,s') = d(f(s'),f(s_{\text{goal}})) - d(f(s),f(s_{\text{goal}}))$ supplies learning signal to any off-the-shelf offline RL algorithm. The paper reports that TempDATA outperforms prior offline model-based methods, often by large margins, and matches or surpasses diffusion-based trajectory augmentation and goal-conditioned RL on D4RL AntMaze, FrankaKitchen, CALVIN, and pixel-based FrankaKitchen. If correct, this means the temporal metric, rather than a better dynamics model, is the missing ingredient for offline model-based RL in long-horizon tasks.

What carries the argument

The temporal-distance-aware autoencoder is the load-bearing component. Its encoder $f: \mathcal{S} \to \mathcal{Z}$ is trained with three objectives: state reconstruction; a trajectory-level expectile regression that drives $d(f(s),f(s_{\text{goal}}))$ toward the shortest path / optimal value; and a transition-level constraint $d(f(s),f(s')) \le d_0$ that keeps single steps temporally smooth. The latent dynamics model $\zeta(z'|z,a)$ learns one-step transitions in this metric space, so that model rollouts respect the temporal geometry rather than cutting through walls or inventing shortcuts. The intrinsic reward $\tilde r(s,s') = d(f(s'),f(s_{\text{goal}})) - d(f(s),f(s_{\text{goal}}))$ converts latent progress into a learning signal for any off-the-shelf offline RL algorithm; in practice the authors combine IQL-style expectile regression with an AWR-style weighted imitation objective, plus a skill variable selected at test time by pointing along the latent direction to the goal.

What would settle it

Train TempDATA on an AntMaze-medium dataset from which the corridor connecting the start region to the goal region has been excised, then check whether the latent dynamics invent a path through the missing corridor (over-generalization) or the agent fails to reach the goal (distance learning confined to dataset-supported shortcuts). Alternatively, compute the rank correlation between $d(f(s),f(s_{\text{goal}}))$ and true shortest-path length on states sampled from evaluation rollouts that never appeared in the training set; a low correlation would show the temporal metric does not generalize outside its support.

Watch

Extended reading notes

Core claim

TempDATA establishes that the central obstacle for offline model-based RL in long-horizon goal-reaching tasks is the space in which rollouts happen, not the quality of the learned dynamics. Its autoencoder is regularized so that the latent distance approximates the optimal goal-conditioned value, i.e., the shortest temporal path from $s$ to the goal (Theorem 4.2, in the limit as the expectile coefficient $\tau \to 1$), while a transition-level constraint keeps consecutive states temporally coherent. A latent forward model $\zeta(z'|z,a)$ then generates augmented transitions entirely in this representation space, decodes them back to states, and the intrinsic reward $\tilde r(s,s') = d(f(s'),f(s_{\text{goal}})) - d(f(s),f(s_{\text{goal}}))$ provides the learning signal. On D4RL AntMaze (including the Ultra levels where prior MBRL methods score zero), FrankaKitchen, CALVIN, and a pixel-based Kitchen, the augmented data lets a standard offline RL backbone exceed previous MBRL methods and compete with diffusion-based trajectory augmentation and hierarchical goal-conditioned RL.

Load-bearing premise

The learned latent distance and the latent forward model must stay accurate for states and actions that lie outside the fixed dataset's support, because the augmented transitions that carry the policy to the goal are synthesized in that latent space — and the paper itself asks how well model-based rollouts venture beyond the empirical support.

Editorial extensions

If this is right

  • Prior model-based baselines that scored near zero on AntMaze (MOPO, COMBO, RAMBO, ROMI) can be lifted to competitive or better performance purely by changing where augmentation happens.
  • TempDATA matches or surpasses diffusion-based trajectory augmentation (GTA, SynthER, S4RL) without generative modeling of whole trajectories, and with markedly lower training time.
  • The learned temporal metric generalizes to arbitrary test-time goals, as shown by obstacle-aware distance heatmaps, so a single representation supports planning to unseen destinations.
  • The framework is modular: any offline RL algorithm can consume its augmented dataset, and the deterministic autoencoder can be replaced by a variational one for pixel-based inputs.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the intrinsic reward is a difference of a learned potential (the latent distance), TempDATA's augmented rewards act as potential-based shaping; whenever the learned distance equals the true shortest-path value, the optimal policy is unchanged by the augmentation, which would explain why no reward model is needed.
  • The same latent distance could also support zero-shot planning in latent space (for example, model predictive control over $\zeta$), rather than decoding every rollout back to raw states; the paper does not explore this.
  • A direct stress test would be to measure distance-preservation error on states visited during evaluation but absent from the training buffer; if the metric shatters on stochastic or partially observed variants of the same benchmarks, performance should collapse exactly in those regimes.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper introduces TempDATA, an offline model-based RL method that learns a temporal-distance-aware autoencoder, trains a latent dynamics model, generates augmented transitions in latent space, and then extracts a policy using offline RL with a potential-based intrinsic reward. The authors claim that TempDATA outperforms prior offline MBRL methods and matches or exceeds diffusion-based trajectory augmentation and goal-conditioned RL on D4RL AntMaze, FrankaKitchen, CALVIN, and pixel-based FrankaKitchen.

Significance. If the empirical claims are correct, TempDATA would be a notable advance: it would be the first offline MBRL method to succeed on sparse-reward, long-horizon goal-reaching benchmarks where prior model-based methods score near zero. The paper also ships a practical pipeline with code, and the external success-rate benchmark is independent of the learned representation. However, the current manuscript contains a data-reporting inconsistency in the central AntMaze table, and the ablation does not isolate transition augmentation from the dense intrinsic reward, so the main mechanistic claim is not yet established.

major comments (4)
  1. [Table 1 (main text) vs Appendix Table 1 (page 8)] The central AntMaze results are reported inconsistently across the two copies of Table 1. For example, TempDATA's score on medium-play is 74.8±8.3 in the main table but 60.3±16.3 in the appendix version; large-diverse is 44.2±15.3 vs 55.2±15.3; ultra-diverse is 35.3±10.9 vs 62.2±8.0; and the total scores differ (520.2 vs 272.6). Because the headline claim rests on these numbers, the authors must reconcile the tables, clarify which numbers correspond to which experimental configuration, and provide the raw per-seed results.
  2. [Section 5.5, Figure 8; Eq. (6)] The ablation does not isolate the effect of transition augmentation from the effect of the dense intrinsic reward. Baselines labeled 'vanilla RL' and 'skill RL' are trained with the sparse relabeled goal reward, while TempDATA uses the potential-based reward r̃(s,s') of Eq. (6) on all sampled data. A control that keeps Eq. (6) but disables model rollouts (σ_bD=0 in Algorithm 1) is missing, so the reported gains could come entirely from reward shaping rather than from augmented transitions. This is load-bearing for the paper's central claim and should be addressed with an additional ablation.
  3. [Theorem 4.2 and Appendix B] The theorem and its proof are not aligned and do not support the practical configuration. The theorem states equality for τ=1, but the proof argues via a limit τ→1; the main text writes 'lim τ=1' in Eq. (3). The proof also asserts that dθ is 'bounded both above and below by the same shortest-path cost' and therefore 'converges exactly', which assumes the very equality being proved. Moreover, the practical expectile coefficients are 0.95, 0.97, and 0.7, far from 1, so the theorem does not justify the learned distances used in the method. Please provide a correct statement, a correct proof, or finite-τ guarantees, or explicitly position Theorem 4.2 as only an asymptotic motivation.
  4. [Section 4.1, Bellman target for Ltraj] The Bellman target Bd is written with a max over θ, while the proof in Appendix B uses a min over actions. Although a min is appropriate for a distance/cost formulation, the inconsistency between the main text and the proof makes the derivation hard to verify. Please define the target precisely and state the action optimization consistently in both places.
minor comments (6)
  1. [Eq. (3)] The notation 'lim τ=1' should be 'lim_{τ→1}'.
  2. [Section 5.5, Figure 8] Figure 8 reports only bar heights with no numerical values or error bars, and the 'boxed markers' are not defined in the caption; please add numerical results and a description of the marker convention.
  3. [Section 5.6, Figure 9] The text calls the heatmaps 'quantitative' but reports no quantitative metric; please either provide a quantitative evaluation of learned distances against true shortest paths or soften the wording.
  4. [Section 5.1] The phrase 'unlike prior MARL methods' appears to be a typo for 'MBRL methods'; please correct it.
  5. [Acknowledgements] The sentence 'This work was supported by by Institute...' contains a duplicated 'by'; please fix.
  6. [Throughout] There are several typos and formatting issues (e.g., 'pre-defiend' in Section 4.1, the garbled duplication of Table 1 in the appendix). A careful proofreading pass is needed.

Circularity Check

1 steps flagged · score 4.0 of 10

Mechanistic claim conflates transition augmentation with the fitted distance-based reward; central benchmark results remain externally grounded.

  1. fitted input called prediction [Section 4.3 Eq. (6) and Section 5.5 Ablation Study]
    "˜r(s,s′) = d(f(s′;θ), f(s_goal;θ)) − d(f(s;θ), f(s_goal;θ)) (Eq. 6). ... 'Comprehensively, these findings validate that the performance gains arise directly from our augmentation framework' (Sec. 5.4)."

    The ablation in Sec. 5.5 compares sparse-reward baselines ('vanilla RL', 'skill RL') against TempDATA, which simultaneously adds latent rollouts and the dense potential-based intrinsic reward of Eq. (6). That reward is computed from the learned distance d, which is itself fit to the same offline Bellman backups used to define goal-reaching. No condition keeps Eq. (6) while disabling rollouts (σ_bD=0 in Algorithm 1), so the observed improvement cannot be attributed to the augmentation mechanism. The claim that gains 'arise directly from our augmentation framework' therefore reduces, in the reported evidence, to the fitted reward shaping rather than to the synthesized transitions.

full rationale

TempDATA's final success-rate numbers are measured against external D4RL / CALVIN goal-reaching benchmarks, not computed from the learned representation, so the headline performance claim is not circular in the strong sense of fitting the target. The self-citations in the introduction (Lee & Kwon, Eo et al.) are not load-bearing, and the theoretical statement in Theorem 4.2 is an analysis of the training objective rather than an imported uniqueness theorem. The main circularity concern is the confounded ablation: the mechanistic conclusion 'the performance gains arise directly from our augmentation framework' is supported only by comparisons that change both the reward function and the presence of rollouts simultaneously. Since the Eq. (6) reward is a re-expression of the same temporal-distance representation trained on the same offline data, the augmentation's marginal contribution is not identified. This warrants a moderate circularity score of 4: the central benchmark claims are externally grounded, but the paper's own causal attribution for the mechanism is partially self-referential and unverified.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The method rests on a learned temporal-distance representation whose Bellman-like loss is bootstrapped from the same offline data, plus a parametric latent dynamics model. No new physical entities are introduced. Several hyperparameters (expectile tau, balancing weights eta1/eta2, inverse temperature beta, latent dimension, rollout schedule) are chosen by hand and are not covered by a sensitivity analysis.

free parameters (5)
  • Eta1, Eta2 (regularizer weights in Eq. 4) = not reported
    Balance reconstruction, trajectory-level distance, and transition-level consistency; no values or sensitivity study are given in the paper.
  • Expectile coefficient tau for autoencoder = 0.95 (AntMaze), 0.97 (CALVIN/FrankaKitchen), 0.7 (pixel-based)
    Per-environment setting in Appendix C.2; Theorem 4.2's limit tau to 1 is not used in experiments.
  • Inverse temperature beta for AWR policy loss (Eq. 7) = 10 (AntMaze), 3 (CALVIN/FrankaKitchen)
    Controls the weighting of advantage-based policy updates and is domain-specific.
  • Latent dimension of autoencoder = 32 (AntMaze), 10 (CALVIN/FrankaKitchen)
    Hand-chosen compression size; affects the geometry of temporal distances.
  • Rollout length k and augmentation schedule = k=3; rollouts start after 30% of training, then every 10% add half the buffer size
    Manual model-rollout settings that influence whether augmentation helps or hurts.
assumptions (4)
  • domain assumption Value-metric equivalence (Proposition 4.1): optimal goal-conditioned value V*(s,g) equals -d*(f(s), f(g)) for a temporal-distance encoder f.
    Imported from Wang and Isola (2022) and Wang et al. (2023); it justifies the trajectory-level Bellman loss in Section 4.1.
  • domain assumption Deterministic discrete-time MDP (Section 3).
    The shortest-path interpretation of temporal distance and Theorem 4.2 assume deterministic transitions; stochastic dynamics are listed as an open question in the Closing Statements.
  • standard math Expectile limit used in Appendix B: m_tau(X) tends to sup X as tau approaches 1.
    The proof of Theorem 4.2 relies on this property of expectiles, but the implemented loss uses finite tau and a bootstrapped target rather than an observed random variable.
  • domain assumption Latent dynamics zeta(z' | z, a) is Gaussian, so its negative log-likelihood reduces to mean squared error.
    Section 4.2 assumes a Gaussian predictive model; this may under-represent transition uncertainty in the representation space.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Temporal Distance-aware Transition Augmentation for Offline Model-based Reinforcement Learning." pith.science (2026). https://pith.science/paper/THFVHA74

@misc{pith2026250513144,
  author       = {Pith},
  title        = {Pith review of: Temporal Distance-aware Transition Augmentation for Offline Model-based Reinforcement Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/THFVHA74}},
  note         = {Machine review of arXiv:2505.13144}
}
read the original abstract

The goal of offline reinforcement learning (RL) is to extract a high-performance policy from the fixed datasets, minimizing performance degradation due to out-of-distribution (OOD) samples. Offline model-based RL (MBRL) is a promising approach that ameliorates OOD issues by enriching state-action transitions with augmentations synthesized via a learned dynamics model. Unfortunately, seminal offline MBRL methods often struggle in sparse-reward, long-horizon tasks. In this work, we introduce a novel MBRL framework, dubbed Temporal Distance-Aware Transition Augmentation (TempDATA), that generates augmented transitions in a temporally structured latent space rather than in raw state space. To model long-horizon behavior, TempDATA learns a latent abstraction that captures a temporal distance from both trajectory and transition levels of state space. Our experiments confirm that TempDATA outperforms previous offline MBRL methods and achieves matching or surpassing the performance of diffusion-based trajectory augmentation and goal-conditioned RL on the D4RL AntMaze, FrankaKitchen, CALVIN, and pixel-based FrankaKitchen.

Figures

Figures reproduced from arXiv: 2505.13144 by the authors.

Figure 1
Figure 1. Performance comparison overview. (a) Umaze envi￾ronment, the most naive level among AntMaze. The 8-DoF ant robot navigates the maze to reach the goal state, marked as a yellow star. (b) Comparison between the proposed solution and previous MBRL on two D4RL benchmark datasets. TempDATA (proposed) achieves the best performance in two benchmarks. limitations by leveraging the offline dataset without real￾time interacti… view at source ↗
Figure 3
Figure 3. Intuition of state abstraction. (Top) Temporal-aware autoencoder, our main idea, maps into state space into a represen￾tation that preserves temporal distance information. (Down) An empirical result shows that our encoder can map raw state space as latent state space in antmaze-medium environments. Here, we consider an ant position state of raw state space as fixed and use t-SNE to plot latent state space visually. … view at source ↗
Figure 4
Figure 4. Transitions in a representation space Z. Dashed lines are related to representation space. 4.2. Latent Dynamic Transition Model This work aims to construct a forward dynamic model in a representation space, dubbed a latent dynamic model, not a state space. Unlike a dynamic model in a state space, the latent dynamic model can handle high-dimensional domains (e.g., pixel-based environments), enhancing generalization a… view at source ↗
Figures from the paper (5 more)
Figure 5
Figure 5. Figure 5: Selected experimental environments. (a-c) State-based, single goal-reaching, and long-horizon navigation. (d-e) State-based, multi-goal subtasks, and long-horizon manipulation. We reuse the Kitchen environment as a Pixel-based task. Algorithm 1 TempDATA with offline RL…
Figure 6
Figure 6. Figure 6: Further investigation on performance. (a) Learning curve about success rate on AntMaze Ultra task. (b) Wall-clock time for each MBRL implementation. (c) RLiable plots (Agarwal et al., 2021) for D4RL benchmark. 5. Experiments In our experiments, we evaluate TempDATA’s p…
Figure 7
Figure 7. Figure 7: Performance comparison on dense reward tasks of D4RL. We evaluate the TempDATA on dense reward tasks, i.e., Halfcheetah and Walker2D. In x-axes of this plot, MR and ME denote medium-replay and medium-expert datasets. While our primary focus is on long-horizon, sparse-r…
Figure 8
Figure 8. Figure 8: Ablation study by policy and training type on four dataset variants (medium-play, medium-diverse, large-play, large-diverse). Bright bars (blue: vanilla RL; red: Skill-based RL) show baselines, while darker bars add TempDATA augmentation. Boxed markers underneath indic…
Figure 9
Figure 9. Figure 9: Heatmap according to four different goals in learned latent space. This visualization shows the distance between ev￾ery quantized state and goal positions in the learned latent space. Brighter and darker colors imply lower and higher costs [PITH_FULL_IMAGE:figures/ful…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

92 extracted references · 72 canonical work pages

  1. [1]

    Deep reinforcement learning at the edge of the statistical precipice

    Agarwal, R., Schwarzer, M., Castro, P., Courville, A., and Bellemare, M. Deep reinforcement learning at the edge of the statistical precipice. NeurIPS, 2021

    2021

  2. [2]

    OPAL : Offline primitive discovery for accelerating offline reinforcement learning

    Ajay, A., Kumar, A., Agrawal, P., Levine, S., and Nachum, O. OPAL : Offline primitive discovery for accelerating offline reinforcement learning. ICLR, 2021

    2021

  3. [3]

    Learning M arkov state abstractions for deep reinforcement learning

    Allen, C., Parikh, N., Gottesman, O., and Konidaris, G. Learning M arkov state abstractions for deep reinforcement learning. NeurIPS, 2021

    2021

  4. [4]

    Uncertainty-based offline reinforcement learning with diversified Q -ensemble

    An, G., Moon, S., Kim, J., and Song, H. Uncertainty-based offline reinforcement learning with diversified Q -ensemble. NeurIPS, 2021

    2021

  5. [5]

    Hindsight experience replay

    Andrychowicz, M., Wolski, F., Ray, A., Schneider, J., Fong, R., Welinder, P., McGrew, B., Tobin, J., Pieter Abbeel, O., and Zaremba, W. Hindsight experience replay. NeurIPS, 30, 2017

    2017

  6. [6]

    and Arnold, G

    Argenson, A. and Arnold, G. Model-based offline planning. ICLR, 2021

    2021

  7. [7]

    Autoencoders

    Bank, D., Koenigstein, N., and Giryes, R. Autoencoders. Machine learning for data science handbook: data mining and knowledge discovery handbook, pp.\ 353--374, 2023

    2023

  8. [8]

    Successor features for transfer in reinforcement learning

    Barreto, A., Dabney, W., Munos, R., Hunt, J., Schaul, T., Hasselt, H., and Silver, D. Successor features for transfer in reinforcement learning. NeurIPS, 2017

    2017

Show all 92 references
  1. [9]

    OpenAI gym

    Brockman, G., Cheung, V., Pettersson, L., Schneider, J., Schulman, J., Tang, J., and Zaremba, W. OpenAI gym. arXiv preprint arXiv:1606.01540, 2016

    2016 arXiv

  2. [10]

    Improving generalization for temporal difference learning: The successor representation

    Dayan, P. Improving generalization for temporal difference learning: The successor representation. Neural computation, 5 0 (4): 0 613--624, 1993

    1993

  3. [11]

    and Hazan, E

    Dekel, O. and Hazan, E. Better rates for any adversarial deterministic MDP . In ICML, 2013

    2013

  4. [12]

    Diederik, P. K. Adam: A method for stochastic optimization. ICLR, 2015

    2015

  5. [13]

    The impact of dataset on offline reinforcement learning performance in uav-based emergency network recovery tasks

    Eo, J., Lee, D., and Kwon, M. The impact of dataset on offline reinforcement learning performance in uav-based emergency network recovery tasks. IEEE Communications Letters, 28 0 (5): 0 1058--1061, 2023

    2023

  6. [14]

    IMPALA : Scalable distributed deep- RL with importance weighted actor-learner architectures

    Espeholt, L., Soyer, H., Munos, R., Simonyan, K., Mnih, V., Ward, T., Doron, Y., Firoiu, V., Harley, T., Dunning, I., et al. IMPALA : Scalable distributed deep- RL with importance weighted actor-learner architectures. In ICML, pp.\ 1407--1416, 2018

    2018

  7. [15]

    Bisimulation makes analogies in goal-conditioned reinforcement learning

    Estruch, P., Zhang, A., Nair, A., Yin, P., and Levine, S. Bisimulation makes analogies in goal-conditioned reinforcement learning. In ICML, 2022

    2022

  8. [16]

    C-learning: Learning to achieve goals via recursive classification

    Eysenbach, B., Salakhutdinov, R., and Levine, S. C-learning: Learning to achieve goals via recursive classification. ICLR, 2021

    2021

  9. [17]

    Contrastive learning as goal-conditioned reinforcement learning

    Eysenbach, B., Zhang, T., Levine, S., and Salakhutdinov, R. Contrastive learning as goal-conditioned reinforcement learning. NeurIPS, 2022

    2022

  10. [18]

    D4RL : Datasets for deep data-driven reinforcement learning

    Fu, J., Kumar, A., Nachum, O., Tucker, G., and Levine, S. D4RL : Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.07219, 2020

    2004 arXiv

  11. [19]

    and Gu, S

    Fujimoto, S. and Gu, S. A minimalist approach to offline reinforcement learning. NeurIPS, 2021

    2021

  12. [20]

    For SALE : State-action representation learning for deep reinforcement learning

    Fujimoto, S., Chang, W., Smith, E., Gu, S., Precup, D., and Meger, D. For SALE : State-action representation learning for deep reinforcement learning. NeurIPS, 2023

    2023

  13. [21]

    Learning to reach goals via iterated supervised learning

    Ghosh, D., Gupta, A., Reddy, A., Fu, J., Devin, C., Eysenbach, B., and Levine, S. Learning to reach goals via iterated supervised learning. ICLR, 2021

    2021

  14. [22]

    Reinforcement learning from passive data via latent intentions

    Ghosh, D., Bhateja, C., and Levine, S. Reinforcement learning from passive data via latent intentions. In ICML, 2023

    2023

  15. [23]

    Relay policy learning: Solving long-horizon tasks via imitation and reinforcement learning

    Gupta, A., Kumar, V., Lynch, C., Levine, S., and Hausman, K. Relay policy learning: Solving long-horizon tasks via imitation and reinforcement learning. CoRL, 2019

    2019

  16. [24]

    Learning latent dynamics for planning from pixels

    Hafner, D., Lillicrap, T., Fischer, I., Villegas, R., Ha, D., Lee, H., and Davidson, J. Learning latent dynamics for planning from pixels. In ICML, 2019

    2019

  17. [25]

    Distance weighted supervised learning for offline interaction data

    Hejna, J., Gao, J., and Sadigh, D. Distance weighted supervised learning for offline interaction data. In ICML, 2023

    2023

  18. [26]

    Efficient planning in a compact latent action space

    Jiang, Z., Zhang, T., Janner, M., Li, Y., Rockt \"a schel, T., Grefenstette, E., and Tian, Y. Efficient planning in a compact latent action space. ICLR, 2022

    2022

  19. [27]

    Learning to achieve goals

    Kaelbling, L. Learning to achieve goals. In IJCAI, 1993

    1993

  20. [28]

    MOReL : Model-based offline reinforcement learning

    Kidambi, R., Rajeswaran, A., Netrapalli, P., and Joachims, T. MOReL : Model-based offline reinforcement learning. NeurIPS, 2020

    2020

  21. [29]

    and Welling, M

    Kingma, D. and Welling, M. Auto-encoding variational B ayes. arXiv preprint arXiv:1312.6114, 2013

    2013 arXiv

  22. [30]

    Offline reinforcement learning with implicit q -learning

    Kostrikov, I., Nair, A., and Levine, S. Offline reinforcement learning with implicit q -learning. ICLR, 2022

    2022

  23. [31]

    Stabilizing off-policy Q -learning via bootstrapping error reduction

    Kumar, A., Fu, J., Soh, M., Tucker, G., and Levine, S. Stabilizing off-policy Q -learning via bootstrapping error reduction. NeurIPS, 2019

    2019

  24. [32]

    Conservative Q -learning for offline reinforcement learning

    Kumar, A., Zhou, A., Tucker, G., and Levine, S. Conservative Q -learning for offline reinforcement learning. NeurIPS, 2020

    2020

  25. [33]

    CURL : Contrastive unsupervised representations for reinforcement learning

    Laskin, M., Srinivas, A., and Abbeel, P. CURL : Contrastive unsupervised representations for reinforcement learning. In ICML, 2020

    2020

  26. [34]

    Lipschitz lifelong reinforcement learning

    Lecarpentier, E., Abel, D., Asadi, K., Jinnai, Y., Rachelson, E., and Littman, M. Lipschitz lifelong reinforcement learning. In AAAI, 2021

    2021

  27. [35]

    Representation balancing offline model-based reinforcement learning

    Lee, B., Lee, J., and Kim, K. Representation balancing offline model-based reinforcement learning. In ICLR, 2021

    2021

  28. [36]

    and Kwon, M

    Lee, D. and Kwon, M. Foresighted decisions for inter-vehicle interactions: An offline reinforcement learning approach. In ITSC, 2023

    2023

  29. [37]

    and Kwon, M

    Lee, D. and Kwon, M. Episodic future thinking with offline reinforcement learning for autonomous driving. IEEE Internet of Things Journal, 2025

    2025

  30. [38]

    AD4RL : Autonomous driving benchmarks for offline reinforcement learning with value-based dataset

    Lee, D., Eom, C., and Kwon, M. AD4RL : Autonomous driving benchmarks for offline reinforcement learning with value-based dataset. In ICRA. IEEE, 2024 a

    2024

  31. [39]

    K., Choi, W., and Woo, H

    Lee, D., Yoo, M., Kim, W. K., Choi, W., and Woo, H. Incremental learning of retrievable skills for efficient continual task adaptation. NeurIPS, 2024 b

    2024

  32. [40]

    GTA : Generative trajectory augmentation with guidance for offline reinforcement learning

    Lee, J., Yun, S., Yun, T., and Park, J. GTA : Generative trajectory augmentation with guidance for offline reinforcement learning. NeurIPS, 2024 c

    2024

  33. [41]

    Metric residual network for sample efficient goal-conditioned reinforcement learning

    Liu, B., Feng, Y., Liu, Q., and Stone, P. Metric residual network for sample efficient goal-conditioned reinforcement learning. In AAAI, 2023

    2023

  34. [42]

    Synthetic experience replay

    Lu, C., Ball, P., Teh, Y., and Holder, J. Synthetic experience replay. NeurIPS, 2023

    2023

  35. [43]

    Conservative offline distributional reinforcement learning

    Ma, Y., Jayaraman, D., and Bastani, O. Conservative offline distributional reinforcement learning. NeurIPS, 2021

    2021

  36. [44]

    VIP : Towards universal visual reward and representation via value-implicit pre-training

    Ma, Y., Sodhani, S., Jayaraman, D., Bastani, O., Kumar, V., and Zhang, A. VIP : Towards universal visual reward and representation via value-implicit pre-training. ICLR, 2023

    2023

  37. [45]

    Contrastive value learning: Implicit models for simple offline RL

    Mazoure, B., Eysenbach, B., Nachum, O., and Tompson, J. Contrastive value learning: Implicit models for simple offline RL . In CoRL, 2023

    2023

  38. [46]

    CALVIN : A benchmark for language-conditioned policy learning for long-horizon robot manipulation tasks

    Mees, O., Hermann, L., Rosete, E., and Burgard, W. CALVIN : A benchmark for language-conditioned policy learning for long-horizon robot manipulation tasks. IEEE Robotics and Automation Letters, 7 0 (3): 0 7327--7334, 2022

    2022

  39. [47]

    Discovering and achieving goals via world models

    Mendonca, R., Rybkin, O., Daniilidis, K., Hafner, D., and Pathak, D. Discovering and achieving goals via world models. NeurIPS, 34: 0 24379--24391, 2021

    2021

  40. [48]

    Offline meta-reinforcement learning with advantage weighting

    Mitchell, E., Rafailov, R., Peng, X., Levine, S., and Finn, C. Offline meta-reinforcement learning with advantage weighting. In ICML, 2021

    2021

  41. [49]

    Human-level control through deep reinforcement learning

    Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A., Veness, J., Bellemare, M., Graves, A., Riedmiller, M., Fidjeland, A., Ostrovski, G., Petersen, S., Beattie, C., Sadik, A., Antonoglou, I., King, H., Kumaran, D., Wierstra, D., Legg, S., and Hassabis, D. Human-level control throu...

    2015

  42. [50]

    Learning temporal distances: Contrastive successor features can provide a metric structure for decision-making

    Myers, V., Zheng, C., Dragan, A., Levine, S., and Eysenbach, B. Learning temporal distances: Contrastive successor features can provide a metric structure for decision-making. ICML, 2024

    2024

  43. [51]

    AWAC : Accelerating online reinforcement learning with offline datasets

    Nair, A., Gupta, A., Dalal, M., and Levine, S. AWAC : Accelerating online reinforcement learning with offline datasets. arXiv preprint arXiv:2006.09359, 2020

    2006 arXiv

  44. [52]

    Planning with goal-conditioned policies

    Nasiriany, S., Pong, V., Lin, S., and Levine, S. Planning with goal-conditioned policies. NeurIPS, 32, 2019

    2019

  45. [53]

    Geometric autoencoders--what you see is what you decode

    Nazari, P., Damrich, S., and Hamprecht, F. Geometric autoencoders--what you see is what you decode. ICML, 2023

    2023

  46. [54]

    and Powell, J

    Newey, W. and Powell, J. Asymmetric least squares estimation and testing. Econometrica: Journal of the Econometric Society, pp.\ 819--847, 1987

    1987

  47. [55]

    HIQL : Offline goal-conditioned RL with latent states as actions

    Park, S., Ghosh, D., Eysenbach, B., and Levine, S. HIQL : Offline goal-conditioned RL with latent states as actions. NeurIPS, 36, 2023a

  48. [56]

    Foundation policies with H ilbert representations

    Park, S., Kreiman, T., and Levine, S. Foundation policies with H ilbert representations. ICML, 2024

    2024

  49. [57]

    Long-horizon visual planning with goal-conditioned hierarchical predictors

    Pertsch, K., Rybkin, O., Ebert, F., Zhou, S., Jayaraman, D., Finn, C., and Levine, S. Long-horizon visual planning with goal-conditioned hierarchical predictors. NeurIPS, 2020

    2020

  50. [58]

    and Juditsky, A

    Polyak, B. and Juditsky, A. Acceleration of stochastic approximation by averaging. SIAM Journal on Control and Optimization, 30 0 (4): 0 838--855, 1992

    1992

  51. [59]

    Temporal difference models: Model-free deep RL for model-based control

    Pong, V., Gu, S., Dalal, M., and Levine, S. Temporal difference models: Model-free deep RL for model-based control. ICLR, 2018

    2018

  52. [60]

    MOTO : Offline pre-training to online fine-tuning for model-based robot learning

    Rafailov, R., Hatch, K., Kolev, V., Martin, J., Phielipp, M., and Finn, C. MOTO : Offline pre-training to online fine-tuning for model-based robot learning. In CoRL, 2023

    2023

  53. [61]

    Goal-conditioned offline reinforcement learning via metric learning

    Reichlin, A., Vasco, M., Yin, H., and Kragic, D. Goal-conditioned offline reinforcement learning via metric learning. arXiv preprint arXiv:2402.10820, 2024

    2024

  54. [62]

    RAMBO-RL : Robust adversarial model-based offline reinforcement learning

    Rigter, M., Lacerda, B., and Hawes, N. RAMBO-RL : Robust adversarial model-based offline reinforcement learning. NeurIPS, 2022

    2022

  55. [63]

    An overview of gradient descent optimization algorithms

    Ruder, S. An overview of gradient descent optimization algorithms. arXiv preprint arXiv:1609.04747, 2016

    2016 arXiv

  56. [64]

    Universal value function approximators

    Schaul, T., Horgan, D., Gregor, K., and Silver, D. Universal value function approximators. In ICML, 2015

    2015

  57. [65]

    Reinforcement learning with action-free pre-training from videos

    Seo, Y., Lee, K., James, S., and Abbeel, P. Reinforcement learning with action-free pre-training from videos. In ICML, 2022

    2022

  58. [66]

    Skill-based model-based reinforcement learning

    Shi, L., Lim, J., and Lee, Y. Skill-based model-based reinforcement learning. CoRL, 2022

    2022

  59. [67]

    K., and Woo, H

    Shin, S., Lee, D., Yoo, M., Kim, W. K., and Woo, H. One-shot imitation in a non-stationary environment via multi-modal skill. In ICML, 2023

    2023

  60. [68]

    S4RL : Surprisingly simple self-supervision for offline reinforcement learning in robotics

    Sinha, S., Mandlekar, A., and Garg, A. S4RL : Surprisingly simple self-supervision for offline reinforcement learning in robotics. In CoRL, 2022

    2022

  61. [69]

    Offline RL for natural language generation with implicit language Q learning

    Snell, C., Kostrikov, I., Su, Y., Yang, M., and Levine, S. Offline RL for natural language generation with implicit language Q learning. ICLR, 2023

    2023

  62. [70]

    Intrinsic motivation and automatic curricula via asymmetric self-play

    Sukhbaatar, S., Lin, Z., Kostrikov, I., Synnaeve, G., Szlam, A., and Fergus, R. Intrinsic motivation and automatic curricula via asymmetric self-play. ICLR, 2018

    2018

  63. [71]

    Model- B ellman inconsistency for model-based offline reinforcement learning

    Sun, Y., Zhang, J., Jia, C., Lin, H., Ye, J., and Yu, Y. Model- B ellman inconsistency for model-based offline reinforcement learning. In ICML, 2023

    2023

  64. [72]

    Leveraging factored action spaces for efficient offline reinforcement learning in healthcare

    Tang, S., Makar, M., Sjoding, M., Doshi, F., and Wiens, J. Leveraging factored action spaces for efficient offline reinforcement learning in healthcare. NeurIPS, 2022

    2022

  65. [73]

    Revisiting the minimalist approach to offline reinforcement learning

    Tarasov, D., Kurenkov, V., Nikulin, A., and Kolesnikov, S. Revisiting the minimalist approach to offline reinforcement learning. NeurIPS, 2023

    2023

  66. [74]

    CORL : Research-oriented deep offline reinforcement learning library

    Tarasov, D., Nikulin, A., Akimov, D., Kurenkov, V., and Kolesnikov, S. CORL : Research-oriented deep offline reinforcement learning library. NeurIPS, 2024

    2024

  67. [75]

    and Mannor, S

    Tennenholtz, G. and Mannor, S. Uncertainty estimation using R iemannian model dynamics for offline reinforcement learning. NeurIPS, 2022

    2022

  68. [76]

    Offline reinforcement learning with reverse model-based imagination

    Wang, J., Li, W., Jiang, H., Zhu, G., Li, S., and Zhang, C. Offline reinforcement learning with reverse model-based imagination. NeurIPS, 2021

    2021

  69. [77]

    and Isola, P

    Wang, T. and Isola, P. On the learning and learnability of quasimetrics. ICLR, 2022

    2022

  70. [78]

    Optimal goal-reaching reinforcement learning via quasimetric learning

    Wang, T., Torralba, A., Isola, P., and Zhang, A. Optimal goal-reaching reinforcement learning via quasimetric learning. In ICML, 2023

    2023

  71. [79]

    Critic regularized regression

    Wang, Z., Novikov, A., Zolna, K., Merel, J., Springenberg, J., Reed, S., Shahriari, B., Siegel, N., Gulcehre, C., Heess, N., and Freitas, N. Critic regularized regression. NeurIPS, 2020

    2020

  72. [80]

    OCEAN-MBRL : Offline conservative exploration for model-based offline reinforcement learning

    Wu, F., Zhang, R., Yi, Q., Gao, Y., Guo, J., Peng, S., Lan, S., Han, H., Pan, Y., Yuan, K., et al. OCEAN-MBRL : Offline conservative exploration for model-based offline reinforcement learning. In AAAI, 2024

    2024

  73. [81]

    Behavior regularized offline reinforcement learning

    Wu, Y., Tucker, G., and Nachum, O. Behavior regularized offline reinforcement learning. arXiv preprint arXiv:1911.11361, 2019

    1911 arXiv

  74. [82]

    A policy-guided imitation approach for offline reinforcement learning

    Xu, H., Jiang, L., Jianxiong, L., and Zhan, X. A policy-guided imitation approach for offline reinforcement learning. NeurIPS, 2022

    2022

  75. [83]

    Xu, H., Jiang, L., Li, J., Yang, Z., Wang, Z., Chan, V. W. K., and Zhan, X. Offline RL with no OOD actions: In-sample learning via implicit value regularization. ICLR, 2023

    2023

  76. [84]

    Image augmentation is all you need: Regularizing deep reinforcement learning from pixels

    Yarats, D., Kostrikov, I., and Fergus, R. Image augmentation is all you need: Regularizing deep reinforcement learning from pixels. In ICLR, 2021

    2021

  77. [85]

    MOPO : Model-based offline policy optimization

    Yu, T., Thomas, G., Yu, L., Ermon, S., Zou, J., Levine, S., Finn, C., and Ma, T. MOPO : Model-based offline policy optimization. NeurIPS, 2020

    2020

  78. [86]

    COMBO : Conservative offline model-based policy optimization

    Yu, T., Kumar, A., Rafailov, R., Rajeswaran, A., Levine, S., and Finn, C. COMBO : Conservative offline model-based policy optimization. NeurIPS, 2021

    2021

  79. [87]

    BRAC+ : Improved behavior regularized actor critic for offline reinforcement learning

    Zhang, C., Kuppannagari, S., and Viktor, P. BRAC+ : Improved behavior regularized actor critic for offline reinforcement learning. In ACML, 2021

    2021

  80. [88]

    Discriminator-guided model-based offline imitation learning

    Zhang, W., Xu, H., Niu, H., Cheng, P., Li, M., Zhang, H., Zhou, G., and Zhan, X. Discriminator-guided model-based offline imitation learning. In CoRL, 2023

    2023

  81. [89]

    Contrastive difference predictive coding

    Zheng, C., Salakhutdinov, R., and Eysenbach, B. Contrastive difference predictive coding. ICLR, 2024

    2024

  82. [90]

    TACO : Temporal latent action-driven contrastive loss for visual reinforcement learning

    Zheng, R., Wang, X., Sun, Y., Ma, S., Zhao, J., Xu, H., Daum \'e , H., and Huang, F. TACO : Temporal latent action-driven contrastive loss for visual reinforcement learning. NeurIPS, 2023

    2023

  83. [91]

    PLAS : Latent action space for offline reinforcement learning

    Zhou, W., Bajracharya, S., and Held, D. PLAS : Latent action space for offline reinforcement learning. In CoRL, 2021

    2021

  84. [92]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.