pith. sign in

arxiv: 2506.21039 · v3 · pith:V56ZQ5PWnew · submitted 2025-06-26 · 💻 cs.LG · cs.AI

Strict Subgoal Execution: Reliable Long-Horizon Planning in Hierarchical Reinforcement Learning

Jaebak Hwang , Sanghyeon Lee , Jeongmo Kim , Seungyul Han This is my paper

Pith reviewed 2026-05-22 00:48 UTC · model grok-4.3

classification 💻 cs.LG cs.AI
keywords hierarchical reinforcement learninglong-horizon planninggoal-conditioned RLsubgoal executionfrontier experience replaysparse rewardsgraph-based methods
0
0 comments X

The pith

Strict Subgoal Execution filters unreachable subgoals to raise success rates in long-horizon hierarchical reinforcement learning.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper introduces Strict Subgoal Execution as a graph-based hierarchical RL method that adds Frontier Experience Replay to mark which subgoals are actually reachable. Conventional hindsight relabeling leaves many subgoals infeasible, so high-level policies waste steps on impossible plans in sparse-reward settings. By storing and replaying only failure and partial-success transitions, FER draws a clear reachability frontier that raises the fraction of usable subgoals and cuts unnecessary high-level calls. A decoupled exploration policy fills gaps in the goal space while path refinement updates edge costs from observed low-level failures. The resulting system records higher success rates and better sample efficiency than prior goal-conditioned and hierarchical baselines on standard long-horizon benchmarks.

Core claim

SSE integrates Frontier Experience Replay to separate unreachable from admissible subgoals by replaying failure and partial-success transitions. This delineation identifies unreliable subgoals, raises overall subgoal reliability, and reduces unnecessary high-level decisions. A decoupled exploration policy covers underexplored goal regions, and path refinement adjusts edge costs from observed low-level failures. Across diverse long-horizon benchmarks the method produces higher success rates and greater efficiency than existing goal-conditioned and hierarchical RL approaches.

What carries the argument

Frontier Experience Replay (FER), which stores failure and partial-success transitions to delineate the reachability frontier and thereby separates unreachable subgoals from admissible ones.

If this is right

  • Higher subgoal reliability directly reduces the number of wasted high-level planning steps.
  • Fewer high-level decisions free low-level policies to focus on execution rather than recovery from bad subgoals.
  • Decoupled exploration improves coverage of the goal space without altering the main learning loop.
  • Path refinement that incorporates low-level failure costs produces more accurate high-level graphs over time.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same frontier-labeling idea could be tested in non-hierarchical goal-conditioned RL to see whether it reduces dead-end trajectories.
  • Path refinement based on low-level failures might transfer to classical motion-planning graphs that lack learned policies.
  • If the reachability frontier proves stable across random seeds, it could lower the variance of hierarchical training runs in practice.

Load-bearing premise

That labeling subgoals by failure and partial-success transitions will cleanly separate reachable from unreachable ones without adding new biases to the replay buffer or exploration policy.

What would settle it

A controlled run on the same long-horizon benchmarks in which SSE shows no improvement in success rate or number of high-level decisions over a matched hierarchical baseline without FER would falsify the central performance claim.

Figures

Figures reproduced from arXiv: 2506.21039 by Jaebak Hwang, Jeongmo Kim, Sanghyeon Lee, Seungyul Han.

Figure 1
Figure 1. Figure 1: Agent trajectory comparison in goal space [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Initial subgoals at t = 0 selected by π h and π exp, with corresponding Ant agent trajectories at (a) early, (b) intermediate, and (c) final training stages in the U-maze task. The goal space (agent positions in the map) is partitioned into grid cells C m G . π h selects between g˜max and g˜rand to encourage broad coverage, while π exp samples from g˜novel, g˜max, and g to visit underexplored regions and t… view at source ↗
Figure 3
Figure 3. Figure 3: illustrates the effect of the proposed path refinement in a bottleneck environment. In (a), without refinement, the agent repeatedly fol￾lows the shortest path through a narrow corri￾dor near a wall, often resulting in failure. In (b), with refinement applied, increased edge costs in high-failure regions steer Dijkstra’s algorithm toward safer detours. When no alternatives exist (e.g., in the bottleneck), … view at source ↗
Figure 4
Figure 4. Figure 4: The proposed SSE framework. Algorithm 1 Strict Subgoal Execution (SSE) Initialize policies π h , π exp , π l , and graph G for each iteration do for each episode do for each high-level selection step do Sample a subgoal g˜t from π h or π exp with ratio η : (1−η) Plan the waypoint path from ϕ(st) to g˜t Roll out low-level policy π l (st, wpi ) ∀i if ∥ϕ(st+kt ) − g˜t∥ < λ (success) then Store (˜st, g, g˜t, P… view at source ↗
Figure 5
Figure 5. Figure 5: Considered long-horizon environments: 5 AntMaze, 2 KeyChest, and 2 Reacher tasks [PITH_FULL_IMAGE:figures/full_fig_p007_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Performance comparison on various long-horizon environments [PITH_FULL_IMAGE:figures/full_fig_p008_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Trajectory analysis for SSE subgoals g˜max in AntDoubleKeyChest at: (a) early stage, (b) reaches first key, (c) collects both keys, (d) reaches goal after collecting both keys (task success). 00 1M 2M 3M 4M 5M 0.2 0.4 0.6 Success Rate Timestep SSE (ours) SSE w/o PR SSE w/o gmax SSE w/o gnovel SSE w/o g SSE w/o exp SSE w FLS (a) Component evaluation 00 1M 2M 3M 4M 5M 0.2 0.4 0.6 Success Rate Timestep cdist=… view at source ↗
Figure 8
Figure 8. Figure 8: Ablation study on AntDoubleKeyChest environment failing to learn entirely, emphasizing the importance of strict subgoal execution and exploratory diversity. For hyperparameter sensitivity, setting cdist = 5 achieves the optimal balance between penalizing failure-prone paths and avoiding excessive detours. Smaller values have little effect, while larger ones result in inefficient routing. Grid resolution re… view at source ↗
Figure 9
Figure 9. Figure 9: Considered long-horizon environments: 5 AntMaze, 2 KeyChest, and 2 Reacher tasks [PITH_FULL_IMAGE:figures/full_fig_p017_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Performance comparison in random goal setting [PITH_FULL_IMAGE:figures/full_fig_p018_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: Trajectory analysis of SSE subgoals g˜max in AntMazeBottleneck across: (a) early stage, (b) mid training, and (c) task success [PITH_FULL_IMAGE:figures/full_fig_p019_11.png] view at source ↗
Figure 12
Figure 12. Figure 12: Trajectory analysis of SSE subgoals g˜max in AntMazeDoubleBottleneck across: (a) early stage, (b) mid training, and (c) task success. A similar trend is observed in AntMazeDoubleBottleneck as [PITH_FULL_IMAGE:figures/full_fig_p020_12.png] view at source ↗
Figure 13
Figure 13. Figure 13: Trajectory analysis of SSE subgoals g˜max in AntKeyChest across: (a) early stage, (b) after reaching the key, and (c) reaching the goal with the key (task success). In [PITH_FULL_IMAGE:figures/full_fig_p020_13.png] view at source ↗
Figure 14
Figure 14. Figure 14: Component evaluation results in various maps [PITH_FULL_IMAGE:figures/full_fig_p021_14.png] view at source ↗
Figure 15
Figure 15. Figure 15: Distance scail cdist analysis in various maps [PITH_FULL_IMAGE:figures/full_fig_p021_15.png] view at source ↗
Figure 16
Figure 16. Figure 16: Grid size dG analysis in various maps [PITH_FULL_IMAGE:figures/full_fig_p022_16.png] view at source ↗
Figure 17
Figure 17. Figure 17: Exploration ratio η analysis in various maps [PITH_FULL_IMAGE:figures/full_fig_p022_17.png] view at source ↗
read the original abstract

Long-horizon goal-conditioned tasks pose fundamental challenges for reinforcement learning (RL), particularly when goals are distant and rewards are sparse. While hierarchical and graph-based methods offer partial solutions, their reliance on conventional hindsight relabeling often fails to correct subgoal infeasibility, leading to inefficient high-level planning. To address this, we propose Strict Subgoal Execution (SSE), a graph-based hierarchical RL framework that integrates Frontier Experience Replay (FER) to separate unreachable from admissible subgoals and streamline high-level decision making. FER delineates the reachability frontier using failure and partial-success transitions, which identifies unreliable subgoals, increases subgoal reliability, and reduces unnecessary high-level decisions. Additionally, SSE employs a decoupled exploration policy to cover underexplored regions of the goal space and a path refinement that adjusts edge costs using observed low-level failures. Experimental results across diverse long-horizon benchmarks show that SSE consistently outperforms existing goal-conditioned and hierarchical RL methods in both efficiency and success rate. Our code is available at https://jaebak1996.github.io/SSE/

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

3 major / 2 minor

Summary. The paper proposes Strict Subgoal Execution (SSE), a graph-based hierarchical RL method that incorporates Frontier Experience Replay (FER) to separate failure and partial-success transitions, thereby delineating the reachability frontier, increasing subgoal reliability, and reducing unnecessary high-level decisions. SSE further adds a decoupled exploration policy and path refinement based on observed low-level failures. The central empirical claim is that SSE consistently outperforms existing goal-conditioned and hierarchical RL baselines on diverse long-horizon benchmarks in both success rate and efficiency.

Significance. If the performance gains prove robust and generalizable, the work could meaningfully improve reliability in hierarchical RL for sparse-reward, long-horizon tasks by directly addressing subgoal infeasibility. The public release of code is a clear strength that supports reproducibility and future extensions.

major comments (3)
  1. [Experimental Results] Experimental Results section: The abstract asserts consistent outperformance without reporting the number of independent runs, standard errors, statistical significance tests, or precise baseline implementations and hyperparameter matching. These omissions are load-bearing for the central empirical claim and prevent verification that reported gains exceed variance or implementation differences.
  2. [§3.2] §3.2 (FER description): Storing and replaying failure and partial-success transitions separately risks altering the experience distribution seen by both low- and high-level policies. The manuscript provides no quantitative analysis of buffer statistics, subgoal reliability estimates, or policy divergence before versus after FER, leaving open the possibility that selection or exploration bias inflates the reported improvements.
  3. [§4] §4 (Benchmark results): Without tables or figures showing per-task variance, failure-mode breakdowns, or ablation isolating FER from the decoupled exploration and path-refinement components, it is impossible to attribute gains specifically to the reachability-frontier mechanism rather than other design choices.
minor comments (2)
  1. [§3] Notation for the reachability frontier and edge-cost adjustment should be introduced with explicit definitions and an accompanying diagram to improve readability for readers unfamiliar with graph-based HRL.
  2. [Abstract / Introduction] The abstract mentions 'diverse long-horizon benchmarks' but does not list the exact environments or task suites; this should be stated explicitly in the introduction or experimental setup.

Simulated Author's Rebuttal

3 responses · 0 unresolved

We thank the referee for the constructive and detailed feedback, which highlights important aspects of empirical rigor and mechanistic clarity in our work on Strict Subgoal Execution. We have revised the manuscript to strengthen the reporting of experimental results, provide additional quantitative analyses, and include ablations and breakdowns as suggested. Our point-by-point responses follow.

read point-by-point responses

  1. Referee: [Experimental Results] Experimental Results section: The abstract asserts consistent outperformance without reporting the number of independent runs, standard errors, statistical significance tests, or precise baseline implementations and hyperparameter matching. These omissions are load-bearing for the central empirical claim and prevent verification that reported gains exceed variance or implementation differences.

    Authors: We agree that comprehensive statistical reporting is essential to substantiate the central empirical claims. In the revised manuscript, we now report results over 5 independent random seeds, include standard errors in all tables and figures, and present statistical significance tests (paired t-tests with p-values) comparing SSE against baselines. We also provide precise details on baseline implementations (including code references where available) and confirm hyperparameter matching to the extent possible given the original papers. These additions demonstrate that the reported gains in success rate and efficiency exceed variance and are not due to implementation differences. revision: yes

  2. Referee: [§3.2] §3.2 (FER description): Storing and replaying failure and partial-success transitions separately risks altering the experience distribution seen by both low- and high-level policies. The manuscript provides no quantitative analysis of buffer statistics, subgoal reliability estimates, or policy divergence before versus after FER, leaving open the possibility that selection or exploration bias inflates the reported improvements.

    Authors: The separation of failure and partial-success transitions in Frontier Experience Replay (FER) is a deliberate design choice to explicitly delineate the reachability frontier and thereby increase subgoal reliability, which is the core contribution for addressing infeasible subgoals in long-horizon tasks. While this does induce a controlled change in the replay distribution, it is counterbalanced by the decoupled exploration policy that ensures broad coverage. In the revision, we have added quantitative analyses including buffer statistics (e.g., proportions of failure vs. success transitions), subgoal reliability estimates (success rates conditioned on frontier proximity), and policy divergence metrics before/after FER. These results indicate that performance gains derive from improved frontier identification rather than selection bias. revision: yes

  3. Referee: [§4] §4 (Benchmark results): Without tables or figures showing per-task variance, failure-mode breakdowns, or ablation isolating FER from the decoupled exploration and path-refinement components, it is impossible to attribute gains specifically to the reachability-frontier mechanism rather than other design choices.

    Authors: We concur that isolating the contribution of each component is necessary to attribute improvements specifically to the reachability-frontier mechanism. The revised manuscript now includes additional tables and figures reporting per-task variance across seeds, failure-mode breakdowns (categorizing failures by unreachable subgoals versus execution or exploration issues), and ablation studies that systematically disable FER, the decoupled exploration policy, and path refinement individually. These ablations confirm that the primary gains in both success rate and sample efficiency on the long-horizon benchmarks stem from the strict subgoal execution enabled by FER. revision: yes

Circularity Check

0 steps flagged

No significant circularity in empirical hierarchical RL framework

full rationale

The paper proposes SSE with FER as a practical engineering solution for long-horizon goal-conditioned RL, validated via benchmark experiments and released code. No mathematical derivation chain, equations, or self-referential definitions are present that reduce claimed improvements (e.g., subgoal reliability or decision reduction) to fitted parameters or inputs by construction. The reachability frontier delineation is a stated design mechanism, not a prediction forced by prior fits or self-citations. This is a standard non-circular empirical contribution.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 0 invented entities

Abstract-only review yields no explicit free parameters, axioms, or invented entities; the method appears to rely on standard RL assumptions about Markovian transitions and reward sparsity.

pith-pipeline@v0.9.0 · 5719 in / 1086 out tokens · 42306 ms · 2026-05-22T00:48:39.850853+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Lean theorems connected to this paper

Citations machine-checked in the Pith Canon. Every link opens the source theorem in the public Lean library.

  • IndisputableMonolith/Cost/FunctionalEquation.lean washburn_uniqueness_aczel unclear
    ?
    unclear

    Relation between the paper passage and the cited Recognition theorem.

    We introduce a failure-aware path refinement strategy that increases edge costs in unreliable regions... ratiofail(CmG) = Nfail(CmG)/N(CmG), and refine the edge distance... ˜d(v1 → v2) = d(v1 → v2) × max(1, cdist · ratiofail(CmG))

What do these tags mean?
matches
The paper's claim is directly supported by a theorem in the formal canon.
supports
The theorem supports part of the paper's argument, but the paper may add assumptions or extra steps.
extends
The paper goes beyond the formal theorem; the theorem is a base layer rather than the whole result.
uses
The paper appears to rely on the theorem as machinery.
contradicts
The paper's claim conflicts with a theorem or certificate in the canon.
unclear
Pith found a possible connection, but the passage is too broad, indirect, or ambiguous to say the theorem truly supports the claim.

Reference graph

Works this paper leans on

62 extracted references · 62 canonical work pages · 5 internal anchors

  1. [1]

    Playing Atari with Deep Reinforcement Learning

    V olodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller. Playing atari with deep reinforcement learning. arXiv preprint arXiv:1312.5602, 2013

    work page internal anchor Pith review Pith/arXiv arXiv 2013

  2. [2]

    Mas- tering the game of go with deep neural networks and tree search

    David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den Driess- che, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, et al. Mas- tering the game of go with deep neural networks and tree search. nature, 529(7587):484–489, 2016

    work page 2016

  3. [3]

    Universal value function ap- proximators

    Tom Schaul, Daniel Horgan, Karol Gregor, and David Silver. Universal value function ap- proximators. In International conference on machine learning , pages 1312–1320. PMLR, 2015

    work page 2015

  4. [4]

    Learning Multi-Level Hi- erarchies with Hindsight, September 2019

    Andrew Levy, George Konidaris, Robert Platt, and Kate Saenko. Learning multi-level hierar- chies with hindsight. arXiv preprint arXiv:1712.00948, 2017

    work page arXiv 2017

  5. [5]

    Planning with goal- conditioned policies

    Soroush Nasiriany, Vitchyr Pong, Steven Lin, and Sergey Levine. Planning with goal- conditioned policies. Advances in neural information processing systems, 32, 2019

    work page 2019

  6. [6]

    The option-critic architecture

    Pierre-Luc Bacon, Jean Harb, and Doina Precup. The option-critic architecture. In Proceedings of the AAAI conference on artificial intelligence, volume 31, 2017

    work page 2017

  7. [7]

    Feudal networks for hierarchical reinforcement learning

    Alexander Sasha Vezhnevets, Simon Osindero, Tom Schaul, Nicolas Heess, Max Jaderberg, David Silver, and Koray Kavukcuoglu. Feudal networks for hierarchical reinforcement learning. In International conference on machine learning, pages 3540–3549. PMLR, 2017

    work page 2017

  8. [8]

    Composable planning with attributes

    Amy Zhang, Sainbayar Sukhbaatar, Adam Lerer, Arthur Szlam, and Rob Fergus. Composable planning with attributes. In International Conference on Machine Learning, pages 5842–5851. Pmlr, 2018

    work page 2018

  9. [9]

    Near-Optimal Representation Learning for Hierarchical Reinforcement Learning

    Ofir Nachum, Shixiang Gu, Honglak Lee, and Sergey Levine. Near-optimal representation learning for hierarchical reinforcement learning. arXiv preprint arXiv:1810.01257, 2018

    work page internal anchor Pith review Pith/arXiv arXiv 2018

  10. [10]

    Mapping state space using landmarks for universal goal reaching

    Zhiao Huang, Fangchen Liu, and Hao Su. Mapping state space using landmarks for universal goal reaching. Advances in Neural Information Processing Systems, 32, 2019

    work page 2019

  11. [11]

    Search on the replay buffer: Bridging planning and reinforcement learning

    Ben Eysenbach, Russ R Salakhutdinov, and Sergey Levine. Search on the replay buffer: Bridging planning and reinforcement learning. Advances in neural information processing systems, 32, 2019

    work page 2019

  12. [12]

    Landmark-guided subgoal generation in hi- erarchical reinforcement learning

    Junsu Kim, Younggyo Seo, and Jinwoo Shin. Landmark-guided subgoal generation in hi- erarchical reinforcement learning. Advances in neural information processing systems , 34: 28336–28349, 2021

    work page 2021

  13. [13]

    World model as a graph: Learning latent landmarks for planning

    Lunjun Zhang, Ge Yang, and Bradly C Stadie. World model as a graph: Learning latent landmarks for planning. In International conference on machine learning, pages 12611–12620. PMLR, 2021

    work page 2021

  14. [14]

    Cqm: Curriculum reinforcement learning with a quantized world model

    Seungjae Lee, Daesol Cho, Jonghae Park, and H Jin Kim. Cqm: Curriculum reinforcement learning with a quantized world model. Advances in Neural Information Processing Systems, 36:78824–78845, 2023

    work page 2023

  15. [15]

    Breadth-first exploration on adaptive grid for reinforcement learning

    Youngsik Yoon, Gangbok Lee, Sungsoo Ahn, and Jungseul Ok. Breadth-first exploration on adaptive grid for reinforcement learning. In Forty-first International Conference on Machine Learning, 2024

    work page 2024

  16. [16]

    Generating adjacency- constrained subgoals in hierarchical reinforcement learning

    Tianren Zhang, Shangqi Guo, Tian Tan, Xiaolin Hu, and Feng Chen. Generating adjacency- constrained subgoals in hierarchical reinforcement learning. Advances in neural information processing systems, 33:21579–21590, 2020

    work page 2020

  17. [17]

    Hierarchical rein- forcement learning: A comprehensive survey

    Shubham Pateria, Budhitama Subagdja, Ah-hwee Tan, and Chai Quek. Hierarchical rein- forcement learning: A comprehensive survey. ACM Computing Surveys (CSUR), 54(5):1–35, 2021. 10

    work page 2021

  18. [18]

    Hierarchical reinforcement learning: A survey and open research challenges

    Matthias Hutsebaut-Buysse, Kevin Mets, and Steven Latré. Hierarchical reinforcement learning: A survey and open research challenges. Machine Learning and Knowledge Extraction, 4(1): 172–221, 2022

    work page 2022

  19. [19]

    Dhrl: a graph-based approach for long-horizon and sparse hierarchical reinforcement learning

    Seungjae Lee, Jigang Kim, Inkyu Jang, and H Jin Kim. Dhrl: a graph-based approach for long-horizon and sparse hierarchical reinforcement learning. Advances in Neural Information Processing Systems, 35:13668–13678, 2022

    work page 2022

  20. [20]

    Novelty-aware graph traversal and expansion for hierarchical reinforcement learning

    Jongchan Park, Seungjun Oh, and Yusung Kim. Novelty-aware graph traversal and expansion for hierarchical reinforcement learning. In Proceedings of the 33rd ACM International Conference on Information and Knowledge Management, pages 1846–1855, 2024

    work page 2024

  21. [21]

    A note on two problems in connexion with graphs

    Edsger W Dijkstra. A note on two problems in connexion with graphs. Numerische mathematik, 1(1):269–271, 1959

    work page 1959

  22. [22]

    Learning to achieve goals

    Leslie Pack Kaelbling. Learning to achieve goals. In IJCAI, volume 2, pages 1094–1098. Citeseer, 1993

    work page 1993

  23. [23]

    Goal-conditioned reinforcement learning: Problems and solutions

    Minghuan Liu, Menghui Zhu, and Weinan Zhang. Goal-conditioned reinforcement learning: Problems and solutions. arXiv preprint arXiv:2201.08299, 2022

    work page arXiv 2022

  24. [24]

    Autotelic agents with intrinsically motivated goal-conditioned reinforcement learning: a short survey

    Cédric Colas, Tristan Karch, Olivier Sigaud, and Pierre-Yves Oudeyer. Autotelic agents with intrinsically motivated goal-conditioned reinforcement learning: a short survey. Journal of Artificial Intelligence Research, 74:1159–1199, 2022

    work page 2022

  25. [25]

    Multi-Goal Reinforcement Learning: Challenging Robotics Environments and Request for Research

    Matthias Plappert, Marcin Andrychowicz, Alex Ray, Bob McGrew, Bowen Baker, Glenn Powell, Jonas Schneider, Josh Tobin, Maciek Chociej, Peter Welinder, et al. Multi-goal reinforcement learning: Challenging robotics environments and request for research. arXiv preprint arXiv:1802.09464, 2018

    work page internal anchor Pith review Pith/arXiv arXiv 2018

  26. [26]

    Hindsight experience replay

    Marcin Andrychowicz, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, OpenAI Pieter Abbeel, and Wojciech Zaremba. Hindsight experience replay. Advances in neural information processing systems, 30, 2017

    work page 2017

  27. [27]

    Dher: Hindsight experience replay for dynamic goals

    Meng Fang, Cheng Zhou, Bei Shi, Boqing Gong, Jia Xu, and Tong Zhang. Dher: Hindsight experience replay for dynamic goals. In International Conference on Learning Representations, 2018

    work page 2018

  28. [28]

    Visual reinforcement learning with imagined goals

    Ashvin V Nair, Vitchyr Pong, Murtaza Dalal, Shikhar Bahl, Steven Lin, and Sergey Levine. Visual reinforcement learning with imagined goals. Advances in neural information processing systems, 31, 2018

    work page 2018

  29. [29]

    Curriculum-guided hindsight experience replay

    Meng Fang, Tianyi Zhou, Yali Du, Lei Han, and Zhengyou Zhang. Curriculum-guided hindsight experience replay. Advances in neural information processing systems, 32, 2019

    work page 2019

  30. [30]

    Ahegc: Adaptive hindsight experience replay with goal-amended curiosity module for robot control

    Hongliang Zeng, Ping Zhang, Fang Li, Chubin Lin, and Junkang Zhou. Ahegc: Adaptive hindsight experience replay with goal-amended curiosity module for robot control. IEEE Transactions on Neural Networks and Learning Systems, 2023

    work page 2023

  31. [31]

    Unsupervised Control Through Non-Parametric Discriminative Rewards

    David Warde-Farley, Tom Van de Wiele, Tejas Kulkarni, Catalin Ionescu, Steven Hansen, and V olodymyr Mnih. Unsupervised control through non-parametric discriminative rewards.arXiv preprint arXiv:1811.11359, 2018

    work page internal anchor Pith review Pith/arXiv arXiv 2018

  32. [32]

    arXiv preprint arXiv:1903.03698 , year=

    Vitchyr H Pong, Murtaza Dalal, Steven Lin, Ashvin Nair, Shikhar Bahl, and Sergey Levine. Skew-fit: State-covering self-supervised reinforcement learning. arXiv preprint arXiv:1903.03698, 2019

    work page arXiv 1903

  33. [33]

    Exploration via hindsight goal generation

    Zhizhou Ren, Kefan Dong, Yuan Zhou, Qiang Liu, and Jian Peng. Exploration via hindsight goal generation. Advances in Neural Information Processing Systems, 32, 2019

    work page 2019

  34. [34]

    Dis- covering and achieving goals via world models

    Russell Mendonca, Oleh Rybkin, Kostas Daniilidis, Danijar Hafner, and Deepak Pathak. Dis- covering and achieving goals via world models. Advances in Neural Information Processing Systems, 34:24379–24391, 2021. 11

    work page 2021

  35. [35]

    First return, then explore

    Adrien Ecoffet, Joost Huizinga, Joel Lehman, Kenneth O Stanley, and Jeff Clune. First return, then explore. Nature, 590(7847):580–586, 2021

    work page 2021

  36. [36]

    Goal-conditioned reinforcement learning with imagined subgoals

    Elliot Chane-Sane, Cordelia Schmid, and Ivan Laptev. Goal-conditioned reinforcement learning with imagined subgoals. In International conference on machine learning, pages 1430–1440. PMLR, 2021

    work page 2021

  37. [37]

    Recent advances in hierarchical reinforcement learning

    Andrew G Barto and Sridhar Mahadevan. Recent advances in hierarchical reinforcement learning. Discrete event dynamic systems, 13:341–379, 2003

    work page 2003

  38. [38]

    Hierarchical deep reinforcement learning: Integrating temporal abstraction and intrinsic motivation

    Tejas D Kulkarni, Karthik Narasimhan, Ardavan Saeedi, and Josh Tenenbaum. Hierarchical deep reinforcement learning: Integrating temporal abstraction and intrinsic motivation. Advances in neural information processing systems, 29, 2016

    work page 2016

  39. [39]

    Why does hierarchy (sometimes) work so well in reinforcement learning? arXiv preprint arXiv:1909.10618, 2019

    Ofir Nachum, Haoran Tang, Xingyu Lu, Shixiang Gu, Honglak Lee, and Sergey Levine. Why does hierarchy (sometimes) work so well in reinforcement learning? arXiv preprint arXiv:1909.10618, 2019

    work page arXiv 1909

  40. [40]

    Data-efficient hierarchical reinforcement learning

    Ofir Nachum, Shixiang Shane Gu, Honglak Lee, and Sergey Levine. Data-efficient hierarchical reinforcement learning. Advances in neural information processing systems, 31, 2018

    work page 2018

  41. [41]

    Planning-augmented hierarchical reinforcement learning

    Robert Gieselmann and Florian T Pokorny. Planning-augmented hierarchical reinforcement learning. IEEE Robotics and Automation Letters, 6(3):5097–5104, 2021

    work page 2021

  42. [42]

    Imitating graph-based planning with goal-conditioned policies

    Junsu Kim, Younggyo Seo, Sungsoo Ahn, Kyunghwan Son, and Jinwoo Shin. Imitating graph-based planning with goal-conditioned policies. arXiv preprint arXiv:2303.11166, 2023

    work page arXiv 2023

  43. [43]

    Mujoco: A physics engine for model-based control

    Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ international conference on intelligent robots and systems, pages 5026–5033. IEEE, 2012

    work page 2012

  44. [44]

    Addressing function approximation error in actor-critic methods

    Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error in actor-critic methods. In International conference on machine learning, pages 1587–1596. PMLR, 2018

    work page 2018

  45. [45]

    Adam: A Method for Stochastic Optimization

    Diederik P Kingma. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014. 12 A Broader Impact The proposed SSE framework enhances the practicality and robustness of HRL in sparse-reward, long- horizon environments by eliminating fixed subgoal intervals and integrating failure-aware planning. This design reduces the dependency on...

    work page internal anchor Pith review Pith/arXiv arXiv 2014

  46. [46]

    Qh(˜st, ˜gt) − rh t + γh min i=1,2 Qh ¯ψh i (˜st+kt , πh θh(˜st+kt , g)) 2# LQl(ψl) = EBl

    directly use the low-level value function Ql, which is trained with a step-based reward of −1, to define the distance as the expected number of steps required for the low-level policy to navigate from v1 to v2. Although this method reflects actual navigation costs, it is sensitive to instability during Ql training, resulting in fluctuating edge distances....

    work page 2000

  47. [47]

    Guidelines: • The answer NA means that the abstract and introduction do not include the claims made in the paper

    Claims Question: Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope? Answer: [Yes] Justification: The abstract and introduction clearly state the proposed framework (SSE), its key features and highlight consistent improvements across benchmark tasks in Section 1. Guidelines: • The answer NA mean...

    work page

  48. [48]

    Limitations

    Limitations Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes] Justification: Section 6 describes limitations in high-dimensional goal spaces and in repre- senting tasks that require abstract skills not decomposable into spatial subgoals. Guidelines: • The answer NA means that the paper has no limitation w...

    work page

  49. [49]

    Guidelines: • The answer NA means that the paper does not include theoretical results

    Theory assumptions and proofs Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? 23 Answer: [NA] Justification: This work is empirical in nature and does not include formal theoretical claims that require proof. Guidelines: • The answer NA means that the paper does not include theo...

    work page

  50. [50]

    These descriptions are sufficient to reproduce the main experimental results without requiring access to the source code

    Experimental result reproducibility Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)? Answer: [Yes] Justification: The proposed method is des...

    work page

  51. [51]

    The code is organized for immediate execution with provided environment configurations

    Open access to data and code Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material? Answer: [Yes] Justification: We provide the full codebase and experiment scripts in the supplemental material, including detailed instru...

    work page

  52. [52]

    These descriptions are sufficient to fully interpret and reproduce the experimental results

    Experimental setting/details Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results? Answer: [Yes] Justification: All relevant training and evaluation details, including optimizer configurations, hyperparameter settings, an...

    work page

  53. [53]

    Guidelines: • The answer NA means that the paper does not include experiments

    Experiment statistical significance Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments? Answer: [Yes] Justification: All experimental results are averaged over 5 independent runs with different random seeds, and shaded regions in all figures repre...

    work page

  54. [54]

    Guidelines: • The answer NA means that the paper does not include experiments

    Experiments compute resources Question: For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments? Answer: [Yes] Justification: Details of the computational setup, including GPU types, memory, and per-run training time, are provide...

    work page

  55. [55]

    Guidelines: • The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics

    Code of ethics Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines? Answer: [Yes] Justification: The research is based on standard benchmark environments, poses no foresee- able societal or ethical risks, and adheres fully to the NeurIPS Code of Ethics. Gu...

    work page

  56. [56]

    26 Guidelines: • The answer NA means that there is no societal impact of the work performed

    Broader impacts Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed? Answer: [Yes] Justification: Broader Impact is discussed in Appendix A. 26 Guidelines: • The answer NA means that there is no societal impact of the work performed. • If the authors answer NA or No, they should exp...

    work page

  57. [57]

    Guidelines: • The answer NA means that the paper poses no such risks

    Safeguards Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [NA] Justification: The paper does not involve the release of models or data with potential for misuse or dual-use ...

    work page

  58. [58]

    Guidelines: • The answer NA means that the paper does not use existing assets

    Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected? Answer: [Yes] Justification: All environments and baseline implementations are used under their official licenses and appropr...

    work page

  59. [59]

    Guidelines: • The answer NA means that the paper does not release new assets

    New assets Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets? Answer: [Yes] Justification: All introduced methods and environments are provided with detailed docu- mentation to support reproducibility and understanding. Guidelines: • The answer NA means that the paper does not release n...

    work page

  60. [60]

    Guidelines: • The answer NA means that the paper does not involve crowdsourcing nor research with human subjects

    Crowdsourcing and research with human subjects Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)? Answer: [NA] Justification: The research does not involve any human subjects or c...

    work page

  61. [61]

    Guidelines: 28 • The answer NA means that the paper does not involve crowdsourcing nor research with human subjects

    Institutional review board (IRB) approvals or equivalent for research with human subjects Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or ...

    work page

  62. [62]

    Answer: [NA] Justification: No LLMs were used in the development of the core methodology; usage was limited to writing and editing support

    Declaration of LLM usage Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, decla...

    work page 2025