Pith. sign in

REVIEW 3 major objections 6 minor 1 cited by

Hierarchical Reinforcement Learning with Uncertainty-Guided Diffusional Subgoals

T0 review · 3 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A conditional diffusion model regularized by a Gaussian process prior, plus a GP-mean selection rule, generates achievable subgoals for off-policy hierarchical RL, beating prior HRL methods on continuous control benchmarks.

desk verdict Strong empirical HRL paper undermined by an inconsistent GP loss specification and overclaimed theory. read the letter →

arxiv 2505.21750 v1 pith:WX5S25GQ submitted 2025-05-27 cs.LG

Vivienne Huiling Wang , Tinghuai Wang , Joni Pajarinen This is my paper
classification cs.LG MSC 68T0568T07
keywords hierarchicalreinforcementlearningdiffusionmodelsubgoalgenerationGaussianprocessuncertaintyquantificationcontinuouscontrolhindsightrelabelingoff-policy
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Hierarchical reinforcement learning is unstable because the low-level policy keeps changing, so subgoals that used to be reachable stop being reachable and the high-level policy chases a moving training target. This paper proposes HIDI, which turns the high-level policy into a conditional diffusion model that generates subgoals by denoising noise conditioned on the current state, and regularizes it with a Gaussian process prior that adds uncertainty quantification. A hybrid selection rule occasionally substitutes the GP's predictive mean for a diffusion sample, trading expressiveness for reliability. If the paper is right, off-policy hierarchical RL can be stable and sample-efficient without constraining the subgoal space, and the method beats prior HRL approaches on MuJoCo continuous-control benchmarks, including stochastic and image-observation variants.

What carries the argument

The load-bearing object is the generative high-level policy: a conditional diffusion model treated as a density over subgoals given the state, $\pi_h(g|s) = p_{\theta_h}(g^{0:N}|s)$, trained with the standard noise-prediction objective of denoising diffusion probabilistic models. Around it sits a sparse Gaussian process prior whose predictive mean $\mu_*(s)$ and variance $\sigma_*^2(s)$ define the uncertainty signal; the GP loss gradient takes the explicit form $\nabla_{\theta_h} L_{gp} = \mathbb{E}\left[\frac{g - \mu_*(s)}{\sigma_*^2(s)} \cdot \nabla_{\theta_h} g\right]$, so parameter updates are strongest exactly where the GP is most confident. The third element is the hybrid selection rule $g^* = \mu_*$ with probability $\varepsilon$, otherwise a sample from $\pi_h(\cdot|s)$, which the theory analyzes as an $\varepsilon$-mixed policy whose single-step regret is bounded by $\varepsilon(R^*(s) - R_{\min}) + (1-\varepsilon)\delta$.

What would settle it

Track, during HIDI training on a stochastic Ant Maze task, the divergence between the relabeled-subgoal distribution used to train the diffusion model and the subgoal distribution that maximizes the trained high-level Q-function: if that divergence does not shrink as training progresses, the relabeling conjecture the method rests on is false and the reported gains cannot be credited to learning a near-optimal subgoal distribution.

Watch

Extended reading notes

Core claim

In HIDI, the high-level policy is not a neural actor mapping states to subgoals but the reverse process of a conditional diffusion model, $\pi_h(g|s)$, which iteratively denoises Gaussian noise into a subgoal conditioned on the current state. The generator is trained jointly in three directions: a denoising objective over HIRO-relabeled subgoals, so the generated distribution tracks what the current low-level policy can actually reach; a Gaussian process prior whose negative log marginal likelihood penalizes subgoals inconsistent with the smooth structure of previously successful state-subgoal pairs; and a TD3-style objective that backpropagates through the denoising chain to maximize the high-level Q-function. At decision time, with probability $\varepsilon$ the GP's predictive mean is used as the subgoal instead of a diffusion sample, a hybrid the authors argue combines the diffusion model's expressiveness with the GP's reliability. The paper claims this outperforms HIRO, HRAC, HIGL, SAGA, and HLPS on MuJoCo continuous-control benchmarks, with ablations attributing roughly 15% gains to the diffusion component, 15–16% to the GP regularization, and 7–8% to the selection rule, and it supports the selection rule with a bounded single-step regret result and a single-step policy improvement result.

Load-bearing premise

The whole pipeline learns from HIRO-style relabeled subgoals, and the paper's Section 2 conjecture — that relabeled subgoals are asymptotically drawn from a distribution approximating an optimal high-level policy — is stated but not proved; if it is false, the diffusion model's target is biased and the GP regularization inherits the same bias.

Editorial extensions

If this is right

  • Subgoal generation no longer needs an artificially constrained action space: the diffusion model directly captures complex subgoal distributions, so stability mechanisms like HRAC's adjacency constraint and HIGL's landmarks become unnecessary.
  • The GP supplies principled uncertainty quantification: the gradient analysis shows GP regularization pulls generated subgoals toward the predictive mean with strength inversely proportional to predictive variance, anchoring learning to well-supported regions.
  • The reported ablations are roughly additive: replacing the relabeling baseline with diffusion gains about 15%, adding GP regularization gains about 15–16%, and adding the selection rule gains about 7–8% on the hard tasks.
  • On the image-observation Stochastic Ant FourRooms task, HIDI reaches 0.64 success where HIRO and HRAC score 0.00, so the method scales to settings where adjacency-based subgoal generation collapses.
  • The selection strategy carries theoretical guarantees: under a near-optimal diffusion assumption its single-step regret is bounded by $\varepsilon(R^*(s) - R_{\min}) + (1-\varepsilon)\delta$, and a policy-improvement argument shows mixing in the GP mean does not degrade single-step performance.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The $\varepsilon$-mixture rule is effectively an exploration thermostat: at $\varepsilon = 0$ the high level is pure diffusion expressiveness and at $\varepsilon = 1$ it is a deterministic GP-mean policy, so an adaptive $\varepsilon$ schedule that starts high and decays would be a natural extension the paper does not test.
  • The GP's predictive variance doubles as a coverage measure for the state space, so a direct extension is to use high-uncertainty regions to drive high-level exploration bonuses, converting the regularizer from a passive anchor into an active subgoal-seeking signal.
  • If the relabeling conjecture is correct, the diffusion-plus-GP module is a drop-in replacement for the high-level actor in any goal-conditioned off-policy HRL scheme, which predicts the gains transfer beyond the TD3/HIRO scaffolding used in the experiments.
  • The 0.64-versus-0.00 gap on the image-observation FourRooms variant hints that diffusion subgoal generation tolerates high-dimensional observations that adjacency-constrained methods cannot; re-running the comparison on larger visual navigation domains would stress-test that reading.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. This paper proposes HIDI, a hierarchical reinforcement learning method in which the high-level subgoal policy is a conditional diffusion model. The diffusion policy is trained with a denoising objective on HIRO-relabeled subgoals, an auxiliary Gaussian Process (GP) regularization term intended to quantify uncertainty, and a TD3-style deterministic policy gradient objective. At decision time, subgoals are sampled from the diffusion policy with probability 1−ε and from the sparse GP predictive mean with probability ε. The paper reports experiments on continuous control benchmarks (Reacher, Pusher, Point Maze, Ant Maze variants, Ant Fall, Ant FourRooms, including stochastic and image-based variants) showing that HIDI outperforms HIRO, HRAC, HIGL, SAGA, and HLPS on all twelve reported task/reward settings, with ablations isolating the diffusion model, GP regularization, subgoal selection, diffusion steps, and the weights ψ and η. Appendix A provides proofs for diffusion-distribution validity, a GP gradient formula, and regret and policy-improvement statements for subgoal selection.

Significance. If the empirical results are reproducible, HIDI is a valuable contribution: it demonstrates that a generative high-level policy can outperform strong off-policy HRL baselines across a broad and consistent suite of tasks, with 10 seeds, systematic ablations, and qualitative subgoal-reachability visualization. The sparse-GP machinery is a principled way to add uncertainty information to a diffusion policy, and the proposed subgoal selection strategy is simple and effective. However, the paper's theoretical support is not reliable in its current form, and the main-text specification of the GP loss is inconsistent with the appendix analysis. The empirical story is externally grounded in environment success rates and does not depend on the theory, so the central claim may survive revision, but the method as written cannot be reimplemented without guessing which of two different GP losses was actually used. No code is released, which amplifies the reproducibility risk.

major comments (3)
  1. [§3.2, Eqs. (9)–(11); Appendix A.1.3, Eqs. (30)–(31)] The GP regularization objective is specified inconsistently. Equation (9) defines p(g|s; θ_gp) as an N-dimensional joint prior N(0, K_N + σ²I) over replay-buffer outputs, so the density in Eq. (11) does not condition on the query state s and cannot be evaluated for a d-dimensional generated subgoal g. The gradient claimed in Proposition 3.2 and derived in Eq. (31), (g − μ_*(s))/σ_*², is the gradient of the sparse GP predictive NLL in Eq. (30), not of Eq. (11); for Eq. (11), the gradient through g is (K_N + σ²I)^{-1} g, which pulls g toward zero rather than toward the GP predictive mean. The appendix analyzes a different loss from the one defined in the main text, and since no code is released, the actually implemented objective is not recoverable. This is a load-bearing specification failure: the ablation gain of roughly 15–16% attributed to GP regularization and the theorems in §3.2 are attached to an internally inconsistent description. Please unify the objective, either by using the sparse predictive NLL throughout and removing or replacing Eqs. (9) and (11), or by specifying the implemented loss and proving the gradient formula for that loss.
  2. [Appendix A.2, Theorem 3.3 and Proposition 3.4] The theoretical guarantees for subgoal selection are assumption-laden to the point of near-circularity. Theorem 3.3's regret bound is ε(R*(s) − R_min) + (1−ε)δ, where R_min is only assumed to be a lower bound on R(s, μ(s)) and can be arbitrarily far below R*(s); without an additional condition linking μ(s) to high-reward subgoals, the bound is vacuous. The proof of Proposition 3.4 assumes that high-Q subgoals in the buffer cluster around μ(s), that μ(s) approximates g* within error ϵ, that Q_h is Lipschitz with constant L, and that Lϵ < Δ; these assumptions are essentially the desired conclusion restated as conditions. The near-optimal diffusion policy assumption (Assumption A.4) also assumes the core property the regret bound is meant to quantify. Please reformulate these results with explicit coverage and smoothness conditions that make the bounds non-vacuous, or clearly label them as heuristic motivation rather than formal guarantees.
  3. [Appendix A.1.2, Eq. (27)] The identity in Eq. (27), D_KL(q(g0|s)||p_θ(g0|s)) = L_VLB(θ_h) − H(q(g0|s)), is incorrect as an equality: it drops the nonnegative term E_q[D_KL(q(g1:N|g0,s)||p_θ(g1:N|g0,s))]. The correct relation is an inequality, D_KL ≤ L_VLB − H(q), and consequently D_KL ≤ L_VLB. The intended bound can be repaired because D_KL ≤ L_VLB whenever the L_VLB bound holds, but the proof as written does not establish Theorem A.1. Please correct Eq. (27) and the surrounding argument.
minor comments (6)
  1. [Table 2 and Figure 1] The caption of Table 2 says the results are for policies obtained after 5M steps with sparse rewards, but the table also contains dense-reward rows and Figure 1 shows 10M-step x-axes for Stochastic Ant Fall and Stochastic Ant FourRooms. Please make the training horizon and reward settings consistent between the table and figures.
  2. [Section 4.2, Fig. 2] The labels in Fig. 2(d) and the surrounding text render the ε probability as a blank symbol in the caption (e.g., 'HIDI ( = 0.05)'). Please fix the typography so the subgoal selection probability is displayed correctly.
  3. [Algorithm 1, line 14] Line 14 stores the transition (s_{t−1}, g_{t−1}, a_t, r_t, s_t, g_t), but the low-level transition should be indexed consistently with the notation in Section 2, e.g., (s_t, a_t, r_t, s_{t+1}) with the current subgoal g_t. Please clarify the indexing.
  4. [Appendix A.1.3, Remark A.3] Remark A.3 states that minimizing L_gp reduces the KL divergence between p_θh(·|s) and the GP predictive distribution, but the displayed identity includes an additional entropy term H(p_θh(·|s)). Minimizing L_gp therefore also encourages lower entropy, which is in tension with the claim that the diffusion model retains expressiveness. Please state this trade-off explicitly.
  5. [Section 2 and related work] The paper relies on the conjecture in Section 2 that HIRO-style relabeled subgoals asymptotically approximate an optimal high-level policy. This is an important assumption for the diffusion training target; it is acknowledged as a conjecture, but the paper should explicitly discuss what could go wrong if it fails and whether any experiments test the stationarity of the relabeled target. Also, 'Diffsuion-QL' in Section 4 is a typo for 'Diffusion-QL', and the Mish activation is used in Table 3 without a reference.
  6. [Theorem numbering] The numbering of theoretical results is inconsistent between the main text and the appendix: Theorem 3.1 and Proposition 3.2 in the main text correspond to Theorem A.2 in Appendix A.1.3, but the connection is never stated. Proposition 3.4's proof is in Appendix A.2 without a matching label. Please align the numbering and add cross-references.

Circularity Check

2 steps flagged · score 4.0 of 10

Theorems 3.3 and Proposition 3.4 restate their own assumptions as 'guarantees'; the benchmark-based empirical claims remain externally grounded and non-circular.

  1. other [Appendix A.2, Detailed Proof of Proposition 3.4; main text Section 3.3]
    "If the GP’s predictive mean µ(s) is close to subgoals g with high Qh, then µ(s) will inherit high Q-values through smoothness... If high-Q subgoals in Bh cluster around µ(s), then: Qh(s, µ(s)) ≥ Eg∼p(g|s,Bh)[Qh(s, g)]... Thus, for Lϵ < ∆, we have: Qh(s,µ(s)) ≥ Eg∼πθh[Qh(s,g)]. Therefore: Eg∼eπh(·|s)[Qh(s,g)] ≥ Eg∼πθh(·|s)[Qh(s,g)]."

    The proposition's conclusion is that the ε-mixture eπh = εδ_μ(s) + (1−ε)πθh does not degrade performance. By the proof's own algebra, E_{eπh}[Qh] = εQh(s,µ(s)) + (1−ε)E_{πθh}[Qh], so the conclusion is equivalent to Qh(s,µ(s)) ≥ E_{πθh}[Qh]. The proof obtains this inequality only from unproven conditions (Lϵ<Δ, unimodality/concentration of high-Q subgoals) that assert the GP mean is at least as good as the diffusion policy. The 'guarantee' is therefore the assumed no-degradation property restated by construction.

  2. other [Appendix A.2, Assumption A.4 and Detailed Proof of Theorem 3.3; main text Section 3.3]
    "we assume that after sufficient training and data coverage, for every s ∈ S, Eg∼πθh(·|s)[R(s, g)] ≥ maxg′∈G R(s, g′) − δ ... the single-step regret of the subgoal selection strategy is bounded by ε(R∗(s) − Rmin) + (1 − ε)δ."

    The regret bound is computed by substituting Assumption A.4 into E_{eπh}[R] = εR(s,µ(s)) + (1−ε)E_{πθh}[R]. The result is a convex combination of the assumed suboptimality δ and the assumed baseline gap R∗−Rmin; no property of the GP mean or the selection mix is derived. As ε→0, the 'bound' is exactly the assumption. The theorem's guarantee is the input assumption repackaged, not an independent derivation.

full rationale

The empirical core of the paper is not circular: HIDI's success rates are measured against external baselines (HIRO, HRAC, HIGL, SAGA, HLPS) in MuJoCo environments, and the ablations compare variants of the method on the same external reward signal. These results do not reduce to the method's internal losses or to fitted constants. The circular content is confined to the theoretical support. Theorem 3.3's regret bound is obtained by substituting Assumption A.4 (the diffusion policy is already δ-near-optimal) into the mixture expectation; the bound is just a convex combination of that assumed δ and the assumed baseline gap, so the 'guarantee' restates its input. Proposition 3.4's proof derives the no-degradation conclusion from the inequality Qh(s,µ(s)) ≥ E_{πθh}[Qh], which is exactly the condition needed for the ε-mixture not to degrade; the proof supplies only unproven sufficient conditions (e.g., 'This holds under unimodality or concentration of high-Q subgoals in Bh') that amount to assuming the GP mean is already at least as good as the diffusion policy. The paper's own equations (E_{eπh}[Qh] = εQh(s,µ(s)) + (1−ε)E_{πθh}[Qh]) make this equivalence explicit. Section 2's 'conjecture' that relabeled subgoals asymptotically approximate an optimal high-level policy is an admitted unproved premise, cited partly to the authors' own SAGA paper (Wang et al., 2023a); it is load-bearing for the diffusion objective (Eq. 6), but because it is explicitly conjectural, it is a missing-support issue rather than a hidden circular derivation. Independently of circularity, the GP-regularization theory is weakened by a specification mismatch: Eq. 11 defines Lgp through the joint prior N(g|0, KN + σ²I), while Theorem 3.1/Proposition 3.2 and Eq. 31 analyze the predictive NLL N(g|µ*(s), σ*²(s)I); the gradient formula in Eq. 31 does not follow from Eq. 11. This is a correctness/reproducibility gap, not a circular reduction, so it does not raise the circularity score further.

Assumptions & free parameters 6 free parameters · 6 assumptions · 0 invented entities

The central method depends on the relabeled subgoal distribution being a useful training target (a stated conjecture), on the GP being a suitable prior over subgoals, and on several assumptions in the theoretical sections that essentially assume the desired performance. The hyperparameters psi, eta, epsilon, N, and M are hand-set or learned and are required for the method as described.

free parameters (6)
  • GP kernel hyperparameters (gamma, lengthscale, noise) = learned, not reported
    RBF kernel variance, lengthscale, and noise are learned by maximizing marginal likelihood on relabeled subgoal data; the central GP regularization and predictive mean depend on them.
  • psi (GP loss weight) = 1e-3
    Set by hand; balances diffusion objective and GP regularization in Eq. 4.
  • eta (RL objective weight) = 5
    Set by hand; balances diffusion objective and DPG objective in Eq. 4.
  • epsilon (subgoal selection probability) = 0.1
    Set by hand; controls how often the GP mean replaces the diffusion sample in Eq. 15.
  • N (number of diffusion steps) = 5
    Set by hand; tradeoff between expressiveness and computation, tested in Fig. 2c.
  • M (number of inducing states) = 16
    Set by hand; controls sparse GP approximation quality.
assumptions (6)
  • domain assumption Relabeled subgoals from HIRO-style hindsight relabeling asymptotically approximate samples from an optimal high-level policy.
    Section 2 states this as a conjecture and the diffusion objective Eq. 5 trains on these relabeled subgoals as targets; if false, the learned distribution is biased.
  • ad hoc to paper The diffusion policy is near-optimal for single-step reward after sufficient training (Assumption A.4).
    Used in Theorem 3.3; this assumption essentially contains the conclusion that the diffusion policy achieves high reward.
  • ad hoc to paper High-Q subgoals cluster around the GP predictive mean, and the predictive mean approximates the optimal subgoal within error epsilon (Prop 3.4).
    Required for the policy improvement claim; no evidence is provided for unimodality or concentration.
  • domain assumption A zero-mean GP with RBF kernel is a suitable prior for the subgoal function over raw states.
    Section 3.2 places the GP prior; the smoothness assumption may not hold in maze environments with walls.
  • standard math Standard diffusion ELBO equivalence from Ho et al. (2020).
    Used in Theorem A.1 to connect noise prediction loss to KL divergence; the proof's use is flawed.
  • standard math Sparse GP posterior formulas from Snelson and Ghahramani (2005) and Titsias (2009).
    Appendix A.3 uses the FITC or VFE approximations for computational tractability.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Hierarchical Reinforcement Learning with Uncertainty-Guided Diffusional Subgoals." pith.science (2026). https://pith.science/paper/WX5S25GQ

@misc{pith2026250521750,
  author       = {Pith},
  title        = {Pith review of: Hierarchical Reinforcement Learning with Uncertainty-Guided Diffusional Subgoals},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/WX5S25GQ}},
  note         = {Machine review of arXiv:2505.21750}
}
read the original abstract

Hierarchical reinforcement learning (HRL) learns to make decisions on multiple levels of temporal abstraction. A key challenge in HRL is that the low-level policy changes over time, making it difficult for the high-level policy to generate effective subgoals. To address this issue, the high-level policy must capture a complex subgoal distribution while also accounting for uncertainty in its estimates. We propose an approach that trains a conditional diffusion model regularized by a Gaussian Process (GP) prior to generate a complex variety of subgoals while leveraging principled GP uncertainty quantification. Building on this framework, we develop a strategy that selects subgoals from both the diffusion policy and GP's predictive mean. Our approach outperforms prior HRL methods in both sample efficiency and performance on challenging continuous control benchmarks.

Figures

Figures reproduced from arXiv: 2505.21750 by the authors.

Figure 1
Figure 1. Learning curves of our method and baselines, i.e., HLPS (Wang et al., 2024), SAGA (Wang et al., 2023a), HIGL (Kim et al., 2021), HRAC (Zhang et al., 2020), and HIRO (Nachum et al., 2018). Each curve and its shaded region represent the average success rate and 95% confidence interval respectively, averaged over 10 independent trials. 0.0 0.2 0.4 0.6 0.8 1.0 Time Steps 1e7 0.0 0.2 0.4 0.6 0.8 Success Rate Stochastic A… view at source ↗
Figure 2
Figure 2. (a-b) Learning curves of various baselines: HIDI-A refers to HIDI without subgoal selection, HIDI-B refers to HIDI without subgoal selection and GP priors. (c) HIDI performance with varying diffusion steps. (d) HIDI performance with varying probabilities for subgoal selection. rithm that introduces an adversarially guided framework for generating subgoals; (3) HIGL (Kim et al., 2021): an HRL algorithm that trains a … view at source ↗
Figure 3
Figure 3. Environments used in our experiments. • Pusher: A robotic arm must push a puck-shaped object on a plane to a designated goal position. • Point Maze: A simulation ball starts in the bottom left corner of a “ ⊃”-shaped maze, aiming to reach the top left corner. • Ant Maze (U-shape): A simulated ant starts in the bottom left of a “ ⊃”-shaped maze, targeting the top left corner. • Ant Maze (W-shape): A simulated ant sta… view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: (Left) Impact of η, which balances the diffusion objective and RL objective. (Middle) Impact of ψ, which adjusts the influence of GP prior in learning the distribution of diffusional subgoals. (Right) Visualization of the learned inducing states (2D coordinates) compar…
Figure 5
Figure 5. Figure 5: Visualization of generated subgoals and reached subgoals of HIDI and compared baselines in Ant Maze (W-shape, sparse) with the same starting location. 22 [PITH_FULL_IMAGE:figures/full_fig_p022_5.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. S3: Stable Subgoal Selection by Constraining Uncertainty of Coarse Dynamics in Hierarchical Reinforcement Learning

    cs.LG 2026-07 conditional novelty 6.0 of 10

    S3 adds a high-level intrinsic reward that penalizes the predicted variance of coarse multi-step subgoal outcomes, improving HRL performance on bottleneck-heavy MuJoCo tasks.

Reference graph

Works this paper leans on

47 extracted references · 38 canonical work pages · cited by 1 Pith paper

  1. [1]

    Hindsight experience replay

    Andrychowicz, M., Crow, D., Ray, A., Schneider, J., Fong, R., Welinder, P., McGrew, B., Tobin, J., Abbeel, P., and Zaremba, W. Hindsight experience replay. In Advances in Neural Information Processing Systems, pp.\ 5048--5058, 2017 a

    work page 2017

  2. [2]

    Hindsight experience replay

    Andrychowicz, M., Wolski, F., Ray, A., Schneider, J., Fong, R., Welinder, P., McGrew, B., Tobin, J., Pieter Abbeel, O., and Zaremba, W. Hindsight experience replay. Advances in neural information processing systems, 30, 2017 b

    work page 2017

  3. [3]

    The option-critic architecture

    Bacon, P.-L., Harb, J., and Precup, D. The option-critic architecture. In The AAAI Conference on Artificial Intelligence, volume 31, 2017

    work page 2017

  4. [4]

    and Konidaris, G

    Bagaria, A. and Konidaris, G. Option discovery using deep skill chaining. In International Conference on Learning Representations, 2019

    work page 2019

  5. [5]

    u ttler, H., Tenenbaum, J. B., Rockt \

    Campero, A., Raileanu, R., K \" u ttler, H., Tenenbaum, J. B., Rockt \" a schel, T., and Grefenstette, E. Learning with amigo: Adversarially motivated intrinsic goals. In International Conference on Learning Representations, 2021

    work page 2021

  6. [6]

    Simple hierarchical planning with diffusion

    Chen, C., Deng, F., Kawaguchi, K., G \" u l c ehre, C ., and Ahn, S. Simple hierarchical planning with diffusion. CoRR, abs/2401.02644, 2024

    arXiv 2024

  7. [7]

    and Hinton, G

    Dayan, P. and Hinton, G. E. Feudal reinforcement learning. In Advances in Neural Information Processing Systems, pp.\ 271--278, 1992

    work page 1992

  8. [8]

    P., Fox, D., and Rasmussen, C

    Deisenroth, M. P., Fox, D., and Rasmussen, C. E. Gaussian processes for data-efficient learning in robotics and control. IEEE transactions on pattern analysis and machine intelligence, 37 0 (2): 0 408--423, 2013

    work page 2013

Show all 47 references
  1. [9]

    Bayes meets bellman: The gaussian process approach to temporal difference learning

    Engel, Y., Mannor, S., and Meir, R. Bayes meets bellman: The gaussian process approach to temporal difference learning. In International Conference on Machine Learning, pp.\ 154--161, 2003

    2003

  2. [10]

    Diversity is all you need: Learning skills without a reward function

    Eysenbach, B., Gupta, A., Ibarz, J., and Levine, S. Diversity is all you need: Learning skills without a reward function. In International Conference on Learning Representations (Poster), 2019

    2019

  3. [11]

    Automatic goal generation for reinforcement learning agents

    Florensa, C., Held, D., Geng, X., and Abbeel, P. Automatic goal generation for reinforcement learning agents. In International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp.\ 1514--1523. PMLR , 2018

    2018

  4. [12]

    Multi-level discovery of deep options

    Fox, R., Krishnan, S., Stoica, I., and Goldberg, K. Multi-level discovery of deep options. CoRR, abs/1703.08294, 2017

    2017 arXiv

  5. [13]

    Addressing function approximation error in actor-critic methods

    Fujimoto, S., van Hoof, H., and Meger, D. Addressing function approximation error in actor-critic methods. In International Conference on Machine Learning, pp.\ 1587--1596, 2018

    2018

  6. [14]

    J., and Wierstra, D

    Gregor, K., Rezende, D. J., and Wierstra, D. Variational intrinsic control. In International Conference on Learning Representations (Workshop), 2017

    2017

  7. [15]

    Denoising diffusion probabilistic models

    Ho, J., Jain, A., and Abbeel, P. Denoising diffusion probabilistic models. In Advances in Neural Information Processing Systems, pp.\ 6840--6851, 2020

    2020

  8. [16]

    B., and Levine, S

    Janner, M., Du, Y., Tenenbaum, J. B., and Levine, S. Planning with diffusion for flexible behavior synthesis. In International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp.\ 9902--9915. PMLR , 2022

    2022

  9. [17]

    and Langford, J

    Kakade, S. and Langford, J. Approximately optimal approximate reinforcement learning. In Proceedings of the Nineteenth International Conference on Machine Learning, pp.\ 267--274, 2002

    2002

  10. [18]

    Efficient diffusion policies for offline reinforcement learning

    Kang, B., Ma, X., Du, C., Pang, T., and Yan, S. Efficient diffusion policies for offline reinforcement learning. Advances in Neural Information Processing Systems, 36, 2024

    2024

  11. [19]

    Landmark-guided subgoal generation in hierarchical reinforcement learning

    Kim, J., Seo, Y., and Shin, J. Landmark-guided subgoal generation in hierarchical reinforcement learning. Advances in Neural Information Processing Systems, 34: 0 28336--28349, 2021

    2021

  12. [20]

    and Barto, A

    Konidaris, G. and Barto, A. Efficient skill learning using abstraction selection. In International Joint Conference on Artificial Intelligence, 2009

    2009

  13. [21]

    D., Narasimhan, K., Saeedi, A., and Tenenbaum, J

    Kulkarni, T. D., Narasimhan, K., Saeedi, A., and Tenenbaum, J. Hierarchical deep reinforcement learning: Integrating temporal abstraction and intrinsic motivation. Advances in neural information processing systems, 29: 0 3675--3683, 2016

    2016

  14. [22]

    and Rasmussen, C

    Kuss, M. and Rasmussen, C. Gaussian processes in reinforcement learning. Advances in neural information processing systems, 16, 2003

    2003

  15. [23]

    Nonlinear inverse reinforcement learning with gaussian processes

    Levine, S., Popovic, Z., and Koltun, V. Nonlinear inverse reinforcement learning with gaussian processes. Advances in neural information processing systems, 24, 2011

    2011

  16. [24]

    D., Jr., R

    Levy, A., Konidaris, G. D., Jr., R. P., and Saenko, K. Learning multi-level hierarchies with hindsight. In International Conference on Learning Representations, 2019

    2019

  17. [25]

    Learning subgoal representations with slow dynamics

    Li, S., Zheng, L., Wang, J., and Zhang, C. Learning subgoal representations with slow dynamics. In International Conference on Learning Representations, 2021

    2021

  18. [26]

    Active hierarchical exploration with stable subgoal representation learning

    Li, S., Zhang, J., Wang, J., Yu, Y., and Zhang, C. Active hierarchical exploration with stable subgoal representation learning. In International Conference on Learning Representations, 2022

    2022

  19. [27]

    Hierarchical diffusion for offline decision making

    Li, W., Wang, X., Jin, B., and Zha, H. Hierarchical diffusion for offline decision making. In International Conference on Machine Learning, pp.\ 20035--20064. PMLR, 2023

    2023

  20. [28]

    Data-efficient hierarchical reinforcement learning

    Nachum, O., Gu, S., Lee, H., and Levine, S. Data-efficient hierarchical reinforcement learning. In Advances in Neural Information Processing Systems, pp.\ 3307--3317, 2018

    2018

  21. [29]

    Near-optimal representation learning for hierarchical reinforcement learning

    Nachum, O., Gu, S., Lee, H., and Levine, S. Near-optimal representation learning for hierarchical reinforcement learning. In International Conference on Learning Representations, 2019

    2019

  22. [30]

    Visual reinforcement learning with imagined goals

    Nair, A., Pong, V., Dalal, M., Bahl, S., Lin, S., and Levine, S. Visual reinforcement learning with imagined goals. In Advances in Neural Information Processing Systems, pp.\ 9209--9220, 2018

    2018

  23. [31]

    Exploration via hindsight goal generation

    Ren, Z., Dong, K., Zhou, Y., Liu, Q., and Peng, J. Exploration via hindsight goal generation. In Advances in Neural Information Processing Systems, pp.\ 13464--13474, 2019

    2019

  24. [32]

    and Wahnsiedler, R

    Schmidhuber, J. and Wahnsiedler, R. Planning simple trajectories using neural subgoal generators. In From Animals to Animats 2: Proceedings of the Second International Conference on Simulation of Adaptive Behavior, volume 2, pp.\ 196. MIT Press, 1993

    1993

  25. [33]

    Dynamics-aware unsupervised discovery of skills

    Sharma, A., Gu, S., Levine, S., Kumar, V., and Hausman, K. Dynamics-aware unsupervised discovery of skills. In International Conference on Learning Representations, 2020

    2020

  26. [34]

    and Ghahramani, Z

    Snelson, E. and Ghahramani, Z. Sparse gaussian processes using pseudo-inputs. Advances in neural information processing systems, 18, 2005

    2005

  27. [35]

    A., Maheswaranathan, N., and Ganguli, S

    Sohl-Dickstein, J., Weiss, E. A., Maheswaranathan, N., and Ganguli, S. Deep unsupervised learning using nonequilibrium thermodynamics. Proceedings of the 32nd International Conference on Machine Learning, pp.\ 2256--2265, 2015

    2015

  28. [36]

    P., Kumar, A., Ermon, S., and Poole, B

    Song, Y., Sohl - Dickstein, J., Kingma, D. P., Kumar, A., Ermon, S., and Poole, B. Score-based generative modeling through stochastic differential equations. In International Conference on Learning Representations, 2021

    2021

  29. [37]

    Sutton, R. S. and Barto, A. G. Reinforcement learning: An introduction. MIT press, 2018

    2018

  30. [38]

    Titsias, M. K. Variational learning of inducing variables in sparse gaussian processes. In Artificial intelligence and statistics, pp.\ 567--574. PMLR, 2009

    2009

  31. [39]

    Mujoco: A physics engine for model-based control

    Todorov, E., Erez, T., and Tassa, Y. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ international conference on intelligent robots and systems, pp.\ 5026--5033. IEEE, 2012

    2012

  32. [40]

    S., Osindero, S., Schaul, T., Heess, N., Jaderberg, M., Silver, D., and Kavukcuoglu, K

    Vezhnevets, A. S., Osindero, S., Schaul, T., Heess, N., Jaderberg, M., Silver, D., and Kavukcuoglu, K. Feudal networks for hierarchical reinforcement learning. In International Conference on Machine Learning, pp.\ 3540--3549. PMLR, 2017

    2017

  33. [41]

    I2hrl: Interactive influence-based hierarchical reinforcement learning

    Wang, R., Yu, R., An, B., and Rabinovich, Z. I2hrl: Interactive influence-based hierarchical reinforcement learning. In International Joint Conference on Artificial Intelligence, pp.\ 3131--3138, 2020

    2020

  34. [42]

    a m \"a r \

    Wang, V. H., Pajarinen, J., Wang, T., and K \"a m \"a r \"a inen, J.-K. State-conditioned adversarial subgoal generation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pp.\ 10184--10191, 2023 a

    2023

  35. [43]

    a m \" a r \

    Wang, V. H., Wang, T., Yang, W., K \" a m \" a r \" a inen, J., and Pajarinen, J. Probabilistic subgoal representations for hierarchical reinforcement learning. In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024 , 2024

    2024

  36. [44]

    J., and Zhou, M

    Wang, Z., Hunt, J. J., and Zhou, M. Diffusion policies as an expressive policy class for offline reinforcement learning. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023 . OpenReview.net, 2023 b . URL https://openre...

    2023

  37. [45]

    Williams, C. K. and Rasmussen, C. E. Gaussian processes for machine learning, volume 2. MIT press Cambridge, MA, 2006

    2006

  38. [46]

    Generating adjacency-constrained subgoals in hierarchical reinforcement learning

    Zhang, T., Guo, S., Tan, T., Hu, X., and Chen, F. Generating adjacency-constrained subgoals in hierarchical reinforcement learning. In Advances in Neural Information Processing Systems, 2020

    2020

  39. [47]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.