Pith. sign in

REVIEW 2 major objections 5 minor 57 references

Normalizing flows plus triangle-slack reweighting fix Gaussian mode collapse and optimistic bias in hierarchical offline goal-conditioned RL.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-10 16:38 UTC pith:WCTW2BVT

load-bearing objection Clean closed-form mode-averaging result for AWR Gaussians plus a practical NF+slack fix that actually moves the needle on HIQL's known failure modes. the 2 major comments →

arxiv 2607.07855 v1 pith:WCTW2BVT submitted 2026-07-08 cs.LG

NFTR: From Provable Mode-Averaging to Geodesic Subgoal Selection in Offline Goal-Conditioned RL

Erdemt Bao , Xing Lei , Jun Chen This is my paper
classification cs.LG
keywords offline goal-conditioned RLhierarchical RLnormalizing flowstriangle inequalitysubgoal selectionadvantage-weighted regressionmode collapsequasimetric
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Hierarchical Implicit Q-Learning picks subgoals from value advantages alone. That rule produces two linked failures: it treats lucky stochastic outcomes as skillful choices, and its unimodal Gaussian collapses multi-modal path distributions into a mean that often sits in unreachable walls. NFTR replaces the Gaussian high-level policy with a conditional normalizing flow and multiplies the advantage weights by a triangle-slack penalty that downweights detours. The paper proves that any AWR-optimal Gaussian mean is a convex combination of tilted mode centers and therefore leaks irreducible mass onto walls when modes straddle them; flows are the minimal generative class that simultaneously supplies multi-modality, exact log-density and single-pass sampling. The resulting RWDR objective keeps AWR’s population-level monotonic improvement and admits a three-term suboptimality bound. On OGBench teleport, stitching and noisy-manipulation tasks the method substantially outperforms HIQL and related baselines precisely where both failure modes co-occur.

Core claim

The unique AWR minimizer of a unimodal Gaussian high-level policy is the convex combination of AWR-tilted modal centers and therefore places irreducible positive mass on unreachable wall regions whenever those centers straddle a wall. Conditional normalizing flows are the minimal generative class that simultaneously satisfies multi-modality, exact log-density and single-pass sampling. The RWDR objective that multiplies AWR weights by exp(−κΔ) admits a three-term suboptimality decomposition while preserving population-level monotonic improvement.

What carries the argument

Triangle-slack reweighting (RWDR): the multiplicative correction w_RWDR = w_HIQL · exp(−κ Δ) where Δ = ReLU(d(s,w) + d(w,g) − d(s,g)) is the residual of an architecturally enforced triangle inequality. Δ vanishes exactly on geodesics in deterministic MDPs and remains a conservative upper bound on composability violation under stochastic dynamics, thereby filtering lucky transitions without requiring accurate distances.

Load-bearing premise

The offline dataset must cover the optimal high-level subgoal distribution well enough that the density ratio of the optimum to the behavior policy stays finite; otherwise both the sampling-error term and the flow’s ability to hit the right modes become uncontrolled.

What would settle it

On a controlled two-corridor maze whose balanced modes are separated by a wall, train a Gaussian AWR high-level policy and measure whether its mean lands inside the wall and its wall mass stays bounded away from zero for every practical variance; if the Gaussian places negligible wall mass or the normalizing-flow version fails to raise success rate, the mode-averaging claim is false.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • AWR-trained subgoal policies cannot safely remain unimodal when offline data contain branching corridors or stochastic landings.
  • Architectural triangle inequality alone supplies a usable geometric filter for lucky transitions, even when the distance network is left untrained.
  • Hierarchical offline GCRL can retain HIQL’s overall structure and still gain from multi-modal generative heads plus slack reweighting.
  • Empirical gains concentrate where mode collapse and optimistic bias co-occur and recede where neither dominates (long-horizon or high-DoF contact-rich deterministic regimes).

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same closed-form mode-averaging argument applies to any unimodal head under exponential-weight MLE, so the NF choice may transfer to other AWR-style hierarchical methods beyond HIQL.
  • Triangle-slack could be applied as a cheap post-hoc filter on already-trained hierarchical policies without retraining the value function.
  • When concentrability fails on very long horizons, the sampling term dominates and temporal-abstraction methods become the higher-priority complement rather than a competitor.
  • The architectural triangle inequality may serve more generally as a structural prior inside any advantage-weighted objective that must reject off-manifold samples.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. The paper identifies two coupled failure modes of HIQL's high-level AWR subgoal selection in offline goal-conditioned RL: optimistic bias from lucky stochastic transitions and mode collapse of a unimodal Gaussian onto unreachable wall regions. It proposes NFTR, which replaces the Gaussian with a conditional Normalizing Flow (motivated by a closed-form mode-averaging result, Theorem 3.1) and multiplies AWR weights by a triangle-slack factor exp(-κΔ) derived from an architecturally inequality-preserving quasimetric (RWDR). The authors characterize triangle-slack on deterministic and stochastic MDPs (Proposition 3.2), prove a three-term suboptimality decomposition for the RWDR population minimizer (Theorem 3.7), and show that RWDR inherits AWR's population-level monotonic improvement (Proposition 3.8). Empirically, NFTR improves over HIQL and several hierarchical/distance/flow baselines on OGBench teleport, stitch, and noisy-manipulation tasks, with component ablations and a full κ sweep.

Significance. If the results hold, the paper supplies a clean, reusable diagnosis of why unimodal Gaussians fail under AWR on multi-modal subgoal data, together with a minimal generative class (conditional NFs) that meets the three simultaneous requirements of multi-modality, exact log-density, and single-pass sampling. The triangle-slack construction is architecturally grounded rather than estimation-dependent, and the three-term bound plus population monotonicity give a transparent account of what RWDR buys and what it costs. Full Appendix D proofs, multi-seed tables, component ablations (NF alone, RWDR alone, trained vs untrained distance), negative results on HumanoidMaze, and a public GitHub repository are concrete strengths that raise the bar for hierarchical offline GCRL papers. The contribution is incremental relative to HIQL/QRL/TMD but well-scoped and practically useful where both failure modes co-occur.

major comments (2)
  1. Assumption 3.5 (single-policy concentrability) is load-bearing for the sampling-error term of Theorem 3.7, yet the paper never reports density-ratio diagnostics or coverage proxies on OGBench. Without even a qualitative check that C_π remains moderate on the teleport/stitch splits where the largest gains appear, the O(√(C_π C(Π)/n)) term is uncontrolled and the realizability claim for the NF target is only formal. A short empirical appendix (e.g., estimated importance weights under the learned high-level policy, or a leave-one-corridor ablation) would make the bound operational rather than purely notational.
  2. Proposition 3.2(4) correctly states that under stochastic dynamics triangle-slack is only a one-sided conservative upper bound on composability violation; the link to lucky-transition filtering is therefore mechanistic rather than tight. The main text and abstract still phrase the contribution as remaining 'stable under stochastic dynamics' and 'provably avoids' the Gaussian collapse. The claim should be tightened to match the one-sided characterization already proved, and the abstract should not over-sell a tight stochastic guarantee that the appendix does not deliver.
minor comments (5)
  1. Figure 2 caption and surrounding text mix μθ* / μ̃m notation inconsistently with the theorem statement; a single consistent symbol set would help.
  2. Table 1 reports 4-seed means while Appendix F.1 Table 9 uses a 3-seed protocol for the FMTR/DMTR comparison; the discrepancy should be flagged or unified.
  3. The κ sweep (Table 12) is informative but buried in the appendix; a one-sentence pointer in the main-text Q5 answer would help readers find the full pattern.
  4. Several related-work citations (e.g., multistep quasimetric estimation, hierarchical entity-centric diffusion) appear only in the appendix; a short main-text sentence acknowledging them would improve positioning.
  5. Typographical: 'Optimistic biastreats' and 'mode collapsereduces' in the abstract lack spaces; 'aquasimetric' / 'astructural' appear in §2.3.

Circularity Check

0 steps flagged

No significant circularity; mode-averaging, slack characterization, and RWDR bounds are elementary closed-form or standard weighted-MLE arguments that do not embed their conclusions in the inputs.

full rationale

Theorem 3.1 follows directly from expanding the Gaussian log-density, changing measure under the bounded AWR weight, and solving the strictly convex first-order condition for the mean; the resulting convex combination of tilted modal centers is a characterization of the Gaussian minimizer, not a re-statement of the multi-modality claim. Proposition 3.2 is immediate from the definition of a quasimetric (architectural triangle inequality of MRN) plus the standard identification d* = -log V* under deterministic dynamics; the stochastic case is correctly stated as one-sided. Corollary 3.4 is pure algebra on the multiplicative reweight. Theorem 3.7 and Proposition 3.8 are textbook decompositions of weighted MLE / AWR under external concentrability and value-error assumptions; the geometric residual R_kappa is bounded by construction from Delta <= 2 D_max and vanishes on geodesics by Proposition 3.2(2). The design-space claim that conditional NFs are minimal simply enumerates three architectural requirements (multi-modality, exact density, single-pass sampling) that the Gaussian fails and that diffusion/flow-matching also fail; it does not smuggle an ansatz or uniqueness theorem. No parameters are fitted and then re-labeled as predictions, no load-bearing uniqueness result is imported from overlapping authors, and the empirical ablations (trained vs untrained d_theta) are independent of the population-level statements. The derivation chain is therefore self-contained.

Axiom & Free-Parameter Ledger

3 free parameters · 4 axioms · 2 invented entities

The central claims rest on standard offline-RL concentrability and value-error assumptions, the architectural triangle inequality of MRN (already proved in prior work), and a small number of free coefficients (chiefly κ) that are tuned per task. No new physical entities are postulated; triangle-slack and RWDR are derived quantities, not free inventions.

free parameters (3)
  • κ (RWDR geometric penalty coefficient) = task-dependent, typically 1.0 for teleport, 0–0.5 for pure stitch, 2.0 for noisy manipulation
    Controls strength of triangle-slack reweighting; tuned per task on a held-out validation split (κ∈{0,0.5,1,2,5}), with different optima for deterministic vs stochastic environments.
  • α_H (high-level AWR temperature) = 3.0
    Exponential temperature on the advantage; fixed at 3.0 across tasks but still a free scale that interacts with κ.
  • Δ_max (slack clipping threshold) = 10.0
    Numerical stability clip on triangle-slack; set to 10.0 by hand.
axioms (4)
  • domain assumption Single-policy concentrability: d_π*/d_β ≤ C_π < ∞ on the support of the optimal high-level policy (Assumption 3.5).
    Standard offline-RL coverage assumption required for the sampling-error term of Theorem 3.7; not verified on OGBench.
  • domain assumption Value estimation quality: E[(V_θ - V*)²] ≤ ε_V (Assumption 3.6).
    Controls the value-error term in the three-term bound; inherited from IQL analysis.
  • standard math MRN architecture satisfies the triangle inequality exactly for any parameters (Assumption 3.3).
    Proved for Metric Residual Networks in Liu et al. (2023); used as a structural prior rather than an estimation target.
  • domain assumption AWR weights are bounded and integrable (enforced by clipping).
    Needed for the change-of-measure argument in the proof of Theorem 3.1.
invented entities (2)
  • triangle-slack Δ(s,w,g) = ReLU(d(s,w)+d(w,g)-d(s,g)) independent evidence
    purpose: Geometric consistency score that vanishes on geodesics and upper-bounds composability violation under stochastic dynamics.
    Defined from the architectural triangle inequality; independent evidence is the ablation showing untrained d_θ already works and the characterization in Proposition 3.2.
  • RWDR objective (AWR weights multiplied by exp(-κΔ)) independent evidence
    purpose: Joint value-and-geometry weighting for high-level policy extraction.
    Constructed in Section 3.2; its population properties follow from non-negativity of the weights, not from circular definition.

pith-pipeline@v1.1.0-grok45 · 34564 in / 3332 out tokens · 36778 ms · 2026-07-10T16:38:04.304049+00:00 · methodology

0 comments
read the original abstract

Hierarchical Implicit Q-Learning (HIQL), an offline goal-conditioned RL method, selects subgoals by value-function advantages alone. This rule has two coupled failure modes. Optimistic bias treats lucky stochastic outcomes as skillful choices, and mode collapse reduces a multi-modal subgoal distribution to a single Gaussian mean that often falls in unreachable regions. We propose NFTR (Normalizing Flows subgoal policies with Triangle-slack Reweighting). A conditional Normalizing Flow replaces the Gaussian policy, and a closed-form mode-averaging result identifies NFs as the minimal generative class for AWR-based subgoal selection. A triangle slack score, built on the architectural triangle inequality without relying on distance accuracy, multiplicatively corrects the AWR weight to downweight subgoals whose detour cost exceeds average reachability. Triangle-slack vanishes on geodesics in deterministic MDPs and remains a conservative upper bound on composability violation under stochastic dynamics. The RWDR objective preserves AWR's population-level monotonic improvement and admits a three-term suboptimality decomposition. Together, these two ingredients yield subgoal selection that provably avoids the Gaussian collapse described above and remains stable under stochastic dynamics. GitHub page: https://github.com/erdemtbao/NFTR

Figures

Figures reproduced from arXiv: 2607.07855 by Erdemt Bao, Jun Chen, Xing Lei.

Figure 1
Figure 1. Figure 1: Overview of NFTR. (a) Offline trajectories. (b) A Normalizing Flow high-level policy models a multi-modal subgoal distribution. (c) A quasimetric dθ induces the slack ∆(s, w, g) = ReLU(d(s, w)+d(w, g)−d(s, g)), which downweights non-composable subgoals. (d) Execution yields a feasible stitched path. RWDR combines Aenv from Vθ and ∆ from dθ to train the high-level subgoal policy. Definition 1 (Quasimetric).… view at source ↗
Figure 2
Figure 2. Figure 2: Geometric illustration of Theorem 3.1. (a) In the symmetric two-mode case πe1 ≈ πe2 ≈ 1 2 , the AWR￾optimal Gaussian mean µ ∗ θ from Equation (8) sits at the midpoint of [µe1, µe2] inside Swall, and the Gaussian (green ellipses at multiple σ) leaks irreducible mass onto Swall. (b) A conditional NF (Equation (9)) places mass on both corridors and drives the wall mass arbitrarily small as ca￾pacity grows [31… view at source ↗
Figure 3
Figure 3. Figure 3: Triangle-slack geometry. A composable waypoint w approximately sat￾isfies the triangle equality, i.e., d(s, w) + d(w, g) ≈ d(s, g), yielding a small trian￾gle slack ∆(s, w, g) ≈ 0 (green edges). In contrast, a non-composable waypoint w˜ vi￾olates this relation, producing a large slack ∆(s, w, g ˜ ) > 0 (red dashed edges), which indicates a detour / poor composability and should be downweighted by RWDR. We … view at source ↗
Figure 5
Figure 5. Figure 5: Ablation: NF vs Gaussian. Com￾paring policy architectures (Gaussian vs NFs) combined with distance learning configurations (with/without trained distance) and RWDR. Re￾sults show success rate (%) on four tasks. that the MRN architecture’s built-in triangle inequality is the primary source of RWDR’s effectiveness rather than precise distance estimation. Removing the Bellman loss (A3) similarly shows minimal… view at source ↗
Figure 6
Figure 6. Figure 6: Training curves of NFTR across four representative environments. Solid curves show mean success rate over 4 random seeds. Shaded bands show ±1 standard deviation. The light gray vertical band marks the 0.3 to 0.5 × 106 convergence window. The coloured number on the right of each curve is the mean success rate at 1 × 106 steps. (a) Deterministic locomotion. (b) High dimensional deterministic locomotion. (c)… view at source ↗
Figure 7
Figure 7. Figure 7: Log-likelihood heatmap on antmaze-teleport-navigate task1. [PITH_FULL_IMAGE:figures/full_fig_p028_7.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

57 extracted references · 57 canonical work pages · 14 internal anchors

  1. [1]

    Option-aware temporally abstracted value for offline goal-conditioned reinforcement learning.arXiv preprint arXiv:2505.12737, 2025

    Hongjoon Ahn, Heewoong Choi, Jisu Han, and Taesup Moon. Option-aware temporally abstracted value for offline goal-conditioned reinforcement learning.arXiv preprint arXiv:2505.12737, 2025

    work page arXiv 2025

  2. [2]

    Let offline RL flow: Training conservative agents in the latent space of normalizing flows

    Dmitry Akimov, Vladislav Kurenkov, Alexander Nikulin, Denis Tarasov, and Sergey Kolesnikov. Let offline RL flow: Training conservative agents in the latent space of normalizing flows. In3rd Offline RL Workshop: Offline RL as a ”Launchpad”, 2022

    work page 2022

  3. [3]

    Hindsight experience replay

    Marcin Andrychowicz, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, OpenAI Pieter Abbeel, and Wojciech Zaremba. Hindsight experience replay. Advances in neural information processing systems, 30, 2017

    work page 2017

  4. [4]

    Graph-assisted stitching for offline hierarchical reinforcement learning

    Seungho Baek, Taegeon Park, Jongchan Park, Seungjun Oh, and Yusung Kim. Graph-assisted stitching for offline hierarchical reinforcement learning. InProceedings of the 42nd International Conference on Machine Learning (ICML), volume 267 ofProceedings of Machine Learning Research, pages 2391–2408. PMLR, 2025

    work page 2025

  5. [5]

    Test-time offline reinforcement learning on goal-related experience

    Marco Bagatella, Mert Albaba, Jonas Hübotter, Georg Martius, and Andreas Krause. Test-time offline reinforcement learning on goal-related experience. InProceedings of the 42nd International Conference on Machine Learning (ICML), volume 267 ofProceedings of Machine Learning Research. PMLR, 2025

    work page 2025

  6. [6]

    Flowpg: action-constrained policy gradient with normalizing flows.Advances in Neural Information Processing Systems, 36:20118–20132, 2023

    Janaka Brahmanage, Jiajing Ling, and Akshat Kumar. Flowpg: action-constrained policy gradient with normalizing flows.Advances in Neural Information Processing Systems, 36:20118–20132, 2023

    work page 2023

  7. [7]

    Maximum entropy reinforcement learning via energy-based normalizing flow.Advances in Neural Information Processing Systems, 37:56136–56165, 2024

    Chen-Hao Chao, Chien Feng, Wei-Fang Sun, Cheng-Kuang Lee, Simon See, and Chun-Yi Lee. Maximum entropy reinforcement learning via energy-based normalizing flow.Advances in Neural Information Processing Systems, 37:56136–56165, 2024

    work page 2024

  8. [8]

    NICE: Non-linear Independent Components Estimation

    Laurent Dinh, David Krueger, and Yoshua Bengio. Nice: Non-linear independent components estimation. arXiv preprint arXiv:1410.8516, 2014

    work page internal anchor Pith review Pith/arXiv arXiv 2014

  9. [9]

    Density estimation using real NVP

    Laurent Dinh, Jascha Sohl-Dickstein, and Samy Bengio. Density estimation using real NVP. InInterna- tional Conference on Learning Representations, 2017

    work page 2017

  10. [10]

    Inference via interpolation: Contrastive representations provably enable planning and inference.Advances in Neural Information Processing Systems, 37:58901–58928, 2025

    Benjamin Eysenbach, Vivek Myers, Ruslan Salakhutdinov, and Sergey Levine. Inference via interpolation: Contrastive representations provably enable planning and inference.Advances in Neural Information Processing Systems, 37:58901–58928, 2025

    work page 2025

  11. [11]

    Contrastive learning as goal-conditioned reinforcement learning.Advances in Neural Information Processing Systems, 35:35603– 35620, 2022

    Benjamin Eysenbach, Tianjun Zhang, Sergey Levine, and Russ R Salakhutdinov. Contrastive learning as goal-conditioned reinforcement learning.Advances in Neural Information Processing Systems, 35:35603– 35620, 2022

    work page 2022

  12. [12]

    Normalizing Flows are Capable Models for Continuous Control

    Raj Ghugare and Benjamin Eysenbach. Normalizing flows are capable models for rl.arXiv preprint arXiv:2505.23527, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  13. [13]

    Physics-informed value learner for offline goal-conditioned reinforcement learning.arXiv preprint arXiv:2509.06782, 2025

    Vittorio Giammarino, Ruiqi Ni, and Ahmed H Qureshi. Physics-informed value learner for offline goal-conditioned reinforcement learning.arXiv preprint arXiv:2509.06782, 2025

    work page arXiv 2025

  14. [14]

    Hierarchical entity- centric reinforcement learning with factored subgoal diffusion

    Dan Haramati, Carl Qi, Tal Daniel, Amy Zhang, Aviv Tamar, and George Konidaris. Hierarchical entity- centric reinforcement learning with factored subgoal diffusion. InInternational Conference on Learning Representations (ICLR), 2026

    work page 2026

  15. [15]

    Diffused task-agnostic milestone planner.Advances in Neural Information Processing Systems, 36:387–405, 2023

    Mineui Hong, Minjae Kang, and Songhwai Oh. Diffused task-agnostic milestone planner.Advances in Neural Information Processing Systems, 36:387–405, 2023

    work page 2023

  16. [16]

    Learning to reach goals via diffusion

    Vineet Jain and Siamak Ravanbakhsh. Learning to reach goals via diffusion. InInternational Conference on Machine Learning, pages 21170–21195. PMLR, 2024

    work page 2024

  17. [17]

    Conservative offline goal-conditioned implicit V-learning

    Kaiqiang Ke, Qian Lin, Zongkai Liu, Shenghong He, and Chao Yu. Conservative offline goal-conditioned implicit V-learning. InProceedings of the 42nd International Conference on Machine Learning (ICML), volume 267 ofProceedings of Machine Learning Research, pages 29591–29607. PMLR, 2025

    work page 2025

  18. [18]

    Glow: Generative flow with invertible 1x1 convolutions.Advances in neural information processing systems, 31, 2018

    Durk P Kingma and Prafulla Dhariwal. Glow: Generative flow with invertible 1x1 convolutions.Advances in neural information processing systems, 31, 2018

    work page 2018

  19. [19]

    Offline reinforcement learning with implicit q-learning

    Ilya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit q-learning. InInternational Conference on Learning Representations, 2022. 10

    work page 2022

  20. [20]

    State-covering trajectory stitching for diffusion planners

    Kyowoon Lee and Jaesik Choi. State-covering trajectory stitching for diffusion planners. InAdvances in Neural Information Processing Systems (NeurIPS), 2025

    work page 2025

  21. [21]

    GCHR : Goal-Conditioned Hindsight Regularization for Sample-Efficient Reinforcement Learning

    Xing Lei, Wenyan Yang, Kaiqiang Ke, Shentao Yang, Xuetao Zhang, Joni Pajarinen, and Donglin Wang. Gchr: Goal-conditioned hindsight regularization for sample-efficient reinforcement learning.arXiv preprint arXiv:2508.06108, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  22. [22]

    Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems

    Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems.arXiv preprint arXiv:2005.01643, 2020

    work page internal anchor Pith review Pith/arXiv arXiv 2005

  23. [23]

    Metric residual network for sample efficient goal- conditioned reinforcement learning

    Bo Liu, Yihao Feng, Qiang Liu, and Peter Stone. Metric residual network for sample efficient goal- conditioned reinforcement learning. InProceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 8799–8806, 2023

    work page 2023

  24. [24]

    Generative Trajectory Stitching through Diffusion Composition

    Yunhao Luo, Utkarsh A Mishra, Yilun Du, and Danfei Xu. Generative trajectory stitching through diffusion composition.arXiv preprint arXiv:2503.05153, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  25. [25]

    How Far I'll Go: Offline Goal-Conditioned Reinforcement Learning via $f$-Advantage Regression

    Yecheng Jason Ma, Jason Yan, Dinesh Jayaraman, and Osbert Bastani. How far i’ll go: Offline goal- conditioned reinforcement learning viaf-advantage regression.arXiv preprint arXiv:2206.03023, 2022

    work page internal anchor Pith review Pith/arXiv arXiv 2022

  26. [26]

    Horizon generalization in reinforcement learning

    Vivek Myers, Catherine Ji, and Benjamin Eysenbach. Horizon generalization in reinforcement learning. In International Conference on Learning Representations (ICLR), 2025

    work page 2025

  27. [27]

    Offline goal-conditioned reinforcement learning with quasimetric representations

    Vivek Myers, Bill Zheng, Benjamin Eysenbach, and Sergey Levine. Offline goal-conditioned reinforcement learning with quasimetric representations. InThe Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025

    work page 2025

  28. [28]

    Offline goal-conditioned reinforcement learning with quasimetric representations

    Vivek Myers, Bill Chunyuan Zheng, Benjamin Eysenbach, and Sergey Levine. Offline goal-conditioned reinforcement learning with quasimetric representations. InAdvances in Neural Information Processing Systems (NeurIPS), 2025

    work page 2025

  29. [29]

    Learning Temporal Distances: Contrastive Successor Features Can Provide a Metric Structure for Decision-Making

    Vivek Myers, Chongyi Zheng, Anca Dragan, Sergey Levine, and Benjamin Eysenbach. Learning temporal distances: Contrastive successor features can provide a metric structure for decision-making.arXiv preprint arXiv:2406.17098, 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  30. [30]

    Test-Time Graph Search for Goal-Conditioned Reinforcement Learning

    Evgenii Opryshko, Junwei Quan, Claas V oelcker, Yilun Du, and Igor Gilitschenski. Test-time graph search for goal-conditioned reinforcement learning.arXiv preprint arXiv:2510.07257, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  31. [31]

    Normalizing flows for probabilistic modeling and inference.Journal of Machine Learning Research, 22(57):1–64, 2021

    George Papamakarios, Eric Nalisnick, Danilo Jimenez Rezende, Shakir Mohamed, and Balaji Lakshmi- narayanan. Normalizing flows for probabilistic modeling and inference.Journal of Machine Learning Research, 22(57):1–64, 2021

    work page 2021

  32. [32]

    Ogbench: Benchmarking offline goal-conditioned rl

    Seohong Park, Kevin Frans, Benjamin Eysenbach, and Sergey Levine. Ogbench: Benchmarking offline goal-conditioned rl. InInternational Conference on Learning Representations (ICLR), 2025

    work page 2025

  33. [33]

    HIQL: Offline goal-conditioned RL with latent states as actions

    Seohong Park, Dibya Ghosh, Benjamin Eysenbach, and Sergey Levine. HIQL: Offline goal-conditioned RL with latent states as actions. InThirty-seventh Conference on Neural Information Processing Systems, 2023

    work page 2023

  34. [34]

    Foundation Policies with Hilbert Representations

    Seohong Park, Tobias Kreiman, and Sergey Levine. Foundation policies with hilbert representations.arXiv preprint arXiv:2402.15567, 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  35. [35]

    Flow Q-Learning

    Seohong Park, Qiyang Li, and Sergey Levine. Flow q-learning.arXiv preprint arXiv:2502.02538, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  36. [36]

    Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learning

    Xue Bin Peng, Aviral Kumar, Grace Zhang, and Sergey Levine. Advantage-weighted regression: Simple and scalable off-policy reinforcement learning.arXiv preprint arXiv:1910.00177, 2019

    work page internal anchor Pith review Pith/arXiv arXiv 1910

  37. [37]

    Bridging offline reinforcement learning and imitation learning: A tale of pessimism

    Paria Rashidinejad, Banghua Zhu, Cong Ma, Jiantao Jiao, and Stuart Russell. Bridging offline reinforcement learning and imitation learning: A tale of pessimism. In A. Beygelzimer, Y . Dauphin, P. Liang, and J. Wortman Vaughan, editors,Advances in Neural Information Processing Systems, 2021

    work page 2021

  38. [38]

    Goal-Conditioned Imitation Learning using Score-based Diffusion Policies

    Moritz Reuss, Maximilian Li, Xiaogang Jia, and Rudolf Lioutikov. Goal-conditioned imitation learning using score-based diffusion policies.arXiv preprint arXiv:2304.02532, 2023

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  39. [39]

    Score models for offline goal-conditioned reinforcement learning

    Harshit Sikchi, Rohan Chitnis, Ahmed Touati, Alborz Geramifard, Amy Zhang, and Scott Niekum. Score models for offline goal-conditioned reinforcement learning. InThe Twelfth International Conference on Learning Representations, 2024

    work page 2024

  40. [40]

    Parrot: Data-Driven Behavioral Priors for Reinforcement Learning

    Avi Singh, Huihan Liu, Gaoyue Zhou, Albert Yu, Nicholas Rhinehart, and Sergey Levine. Parrot: Data- driven behavioral priors for reinforcement learning.arXiv preprint arXiv:2011.10024, 2020. 11

    work page internal anchor Pith review Pith/arXiv arXiv 2011

  41. [41]

    GOPlan: Goal- conditioned offline reinforcement learning by planning with learned models

    Mianchu Wang, Rui Yang, Xi Chen, Hao Sun, Meng Fang, and Giovanni Montana. GOPlan: Goal- conditioned offline reinforcement learning by planning with learned models. InInternational Conference on Learning Representations (ICLR), 2025

    work page 2025

  42. [42]

    Optimal goal-reaching reinforcement learning via quasimetric learning

    Tongzhou Wang, Antonio Torralba, Phillip Isola, and Amy Zhang. Optimal goal-reaching reinforcement learning via quasimetric learning. InInternational Conference on Machine Learning, pages 36411–36430. PMLR, 2023

    work page 2023

  43. [43]

    Improving Exploration in Soft-Actor-Critic with Normalizing Flows Policies

    Patrick Nadeem Ward, Ariella Smofsky, and Avishek Joey Bose. Improving exploration in soft-actor-critic with normalizing flows policies.arXiv preprint arXiv:1906.02771, 2019

    work page internal anchor Pith review Pith/arXiv arXiv 1906

  44. [44]

    A policy-guided imitation approach for offline reinforcement learning.Advances in neural information processing systems, 35:4085–4098, 2022

    Haoran Xu, Li Jiang, Li Jianxiong, and Xianyuan Zhan. A policy-guided imitation approach for offline reinforcement learning.Advances in neural information processing systems, 35:4085–4098, 2022

    work page 2022

  45. [45]

    An optimal discriminator weighted imitation perspective for reinforcement learning

    Haoran Xu, Shuozhe Li, Harshit Sikchi, Scott Niekum, and Amy Zhang. An optimal discriminator weighted imitation perspective for reinforcement learning. InInternational Conference on Learning Representations (ICLR), 2025

    work page 2025

  46. [46]

    Breadth-first exploration on adaptive grid for reinforcement learning

    Youngsik Yoon, Gangbok Lee, Sungsoo Ahn, and Jungseul Ok. Breadth-first exploration on adaptive grid for reinforcement learning. InForty-first International Conference on Machine Learning, 2024

    work page 2024

  47. [47]

    Scaling goal-conditioned reinforcement learning with multistep quasimetric distances

    Bill Zheng, Vivek Myers, Benjamin Eysenbach, and Sergey Levine. Scaling goal-conditioned reinforcement learning with multistep quasimetric distances. InThe Fourteenth International Conference on Learning Representations, 2026

    work page 2026

  48. [48]

    Flattening hierarchies with policy bootstrapping.arXiv preprint arXiv:2505.14975, 2025

    John L Zhou and Jonathan C Kao. Flattening hierarchies with policy bootstrapping.arXiv preprint arXiv:2505.14975, 2025. 12 Appendix A Related Work Offline Goal-Conditioned RL.Offline GCRL aims to learn goal-reaching policies from static datasets without environment interaction. Existing approaches can be categorized into several paradigms: goal-conditione...

    work page arXiv 2025

  49. [49]

    Sampleϵ∼ N(0, I)from the NF base distribution

    work page

  50. [50]

    Compute the subgoal representationz=f −1 θ (ϵ; [s;g])via the inverse NF

    work page

  51. [51]

    Apply length normalizationz←z· ∥ϕ([s;g])∥ 2/(∥z∥2 +ε)for numerical stability

    work page

  52. [52]

    Execute the low-level policya=π L(s, z)forksteps, or until the subgoal is reached

    work page

  53. [53]

    ∆∗ = 0 iff w is on an optimal path

    Repeat from step 1 with the updated state. D Proofs We provide complete proofs for all theoretical results stated in the main text. The proofs establish the geometric properties of triangle-slack, convergence guarantees for RWDR, robustness to stochastic optimism, and monotonic improvement. D.1 Proof of Theorem 3.1 Setup.Fix (s, g) and write AH(z) :=A H(s...

    work page

  54. [54]

    ELBO/ODE estimates.AWR requires evaluating logπ H(z|s, g) for the weighted MLE objective

    Exact density vs. ELBO/ODE estimates.AWR requires evaluating logπ H(z|s, g) for the weighted MLE objective. NFs compute this exactly via change-of-variables in a single pass, whereas diffusion models require ELBO bounds or expensive ODE-based estimates that introduce bias and gradient noise into the AWR weighting. 23 Table 8: Normalizing Flows architectur...

    work page

  55. [55]

    Bijective gradient flow.The bijective transformation in NFs avoids the information-loss bottlenecks of stochastic forward processes in diffusion, stabilizing gradients under heavily- weighted AWR targets. The two cases where FMTR/DMTR do not lose to NFTR ( pointmaze-medium-stitch, scene-noisy) are environments with relatively well-clustered modes where it...

    work page

  56. [56]

    valid corridor

    Physical realizability of multi-modal subgoals.The 21-DoF humanoid’s contact-rich dynamics make many of the multiple subgoal modes generated by the Normalizing Flow not all physically realizable: a "valid corridor" entrance in maze coordinates may be unreachable for a humanoid given balance and footstep constraints. Consequently, multi-modal subgoal propo...

    work page

  57. [57]

    lucky exploration

    Limited environmental stochasticity.HumanoidMaze does not contain teleporters or strong random transitions, so the optimistic-bias issue that triangle-slack is designed to address is largely absent. RWDR therefore provides limited gain while still introducing additional optimization complexity. This is consistent with our scope statement (Section 4): NFTR...

    work page