Pith. sign in

REVIEW 2 major objections 5 minor 35 references

S3 stabilizes hierarchical RL by making the high-level agent avoid subgoals with uncertain outcomes, improving success on long-horizon bottleneck tasks.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 13:00 UTC pith:SMHGVMY2

load-bearing objection The core idea—penalizing MDN variance of coarse dynamics as a manager-level intrinsic reward—is new and worth testing, but the advertised policy-invariance proof doesn't match the algorithm, and the empirical edge is mostly one environment. the 2 major comments →

arxiv 2607.19232 v1 pith:SMHGVMY2 submitted 2026-07-21 cs.LG cs.MA

S3: Stable Subgoal Selection by Constraining Uncertainty of Coarse Dynamics in Hierarchical Reinforcement Learning

Kshitij Kumar Srivastava , Kshitij Jerath This is my paper
classification cs.LG cs.MA
keywords Hierarchical Reinforcement LearningIntrinsic MotivationCoarse DynamicsPredictive UncertaintyMixture Density NetworkPotential-Based Reward ShapingSubgoal SelectionNon-stationary environments
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Hierarchical reinforcement learning splits a task between a manager that picks subgoals every c steps and a worker that executes them, but the manager gets sparse, delayed rewards and must plan around a worker whose abilities change during training. This paper proposes S3, an intrinsic reward for the manager that penalizes subgoals with high predictive uncertainty in the coarse dynamics—the distribution of where the worker actually ends up c steps later, given the subgoal. The uncertainty is estimated by a Mixture Density Network trained online, and the penalty is a potential-based shaping term, so it preserves the optimal policy while providing dense feedback. Empirically, S3 outperforms HIRO and HRAC on Ant Fall and Ant Push, where contact dynamics create bottlenecks and irreversible mistakes, and matches them on Ant Maze. The intended contribution is a general, dynamics-aware way to select stable subgoals that reduces the non-stationarity from the evolving worker.

Core claim

S3's central claim is that subgoal reliability can be operationalized as the dispersion of the c-step terminal state distribution p(s_{t+c}|s_t,g_t): a subgoal is 'stable' if the worker's landing states are tightly clustered. The paper models this distribution with a Mixture Density Network and uses the trace of the total predictive covariance (within-mode plus between-mode) as the dispersion metric. This quantity becomes a potential-based intrinsic reward for the manager, Φ = -β tr(Σ_mix), so the manager is shaped to prefer subgoals whose outcomes are predictable. Because the shaping is potential-based, the set of optimal policies is unchanged, but the manager now receives dense, capability

What carries the argument

The coarse-dynamics predictive model: a Mixture Density Network (MDN) trained on (s_t, g_t, s_{t+c}) tuples that outputs a K-component Gaussian mixture for the c-step terminal state distribution. S3's intrinsic reward is the trace of the mixture's total covariance, a scalar dispersion estimate that decomposes into within-mode and between-mode variance; this is plugged into a potential-based shaping function (Φ = -β tr(Σ_mix)) to give the manager dense feedback without changing the optimal policy.

Load-bearing premise

The intrinsic reward assumes that the MDN's predicted covariance trace accurately reflects how reliably the current worker will land near the assigned subgoal, even though the MDN is trained on data from older versions of both policies and the potential function is treated as policy-invariant while the MDN keeps changing.

What would settle it

Train S3 in an environment with a known 'trap' region that temporarily makes the worker's outcomes highly variable (e.g., ice or actuator noise). If S3's manager does not consistently avoid issuing subgoals inside that region—or if success rates do not degrade when the MDN's predicted variance is replaced by a constant—the central claim that the covariance trace drives risk-averse subgoal selection is falsified.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Share X Bluesky LinkedIn Reddit HN

If this is right

  • Managers that shape subgoal selection by coarse-dynamics uncertainty will concentrate exploration near bottleneck states and avoid irreversible failures.
  • S3's intrinsic reward is policy-invariant, so it can be layered on existing HRL algorithms like HRAC without altering their optimal policies.
  • In environments with long-horizon preconditions and contact-induced multi-modality, S3 improves sample efficiency and final success rate versus HIRO and HRAC.
  • Manager horizon c interacts with calibration: c=10 yields a stability region in Ant Maze, while c≥50 degrades learning.
  • S3 is most beneficial near bottlenecks and irreversible transitions; in purely geometric navigation (Ant Maze) it offers limited gain.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The shaping signal is only as good as the MDN's calibration; a testable extension is to replace the trace of covariance with other dispersion metrics (e.g., spectral radius or CVaR) and compare on the same benchmarks.
  • Because the MDN is trained on transitions from older policies, S3 implicitly relies on the replay buffer containing a curriculum from high-variance early trajectories to sharp later ones; explicit experience replay prioritization might make the uncertainty estimate more responsive to sudden worker capability changes.
  • Risk-averse subgoal selection could be combined with curiosity-driven exploration at the worker level: the manager avoids uncertain subgoals while the worker actively seeks novel states, potentially resolving exploration-stability trade-offs in sparse-reward tasks.
  • The potential-function interpretation suggests S3's benefit will diminish in environments where outcome variance is not state-dependent; a natural falsification experiment is to run S3 on a task with additive, state-independent noise where the manager cannot learn to avoid variance.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. The paper proposes S3, a high-level intrinsic reward for hierarchical reinforcement learning. A Mixture Density Network (MDN) is trained on (s_t, g_t, s_{t+c}) tuples to model the c-step terminal-state distribution of the low-level worker; the trace of the mixture covariance is used as a negative potential that discourages the manager from selecting subgoals with highly dispersed outcomes. S3 is layered on top of HRAC and evaluated on Ant Maze, Ant Push, and Ant Fall. The paper claims that the shaping term is potential-based and therefore preserves optimality, and that it outperforms state-of-the-art HRL methods in non-stationary long-horizon environments.

Significance. The idea of providing dense, dynamics-aware feedback at the manager level is timely and underexplored. Modeling multi-modal coarse dynamics with an MDN is a sensible way to capture contact-rich, bottleneck-dependent outcomes, and the reported Ant Fall improvement (0.374 vs. 0.059/0.000) is substantial. The qualitative analysis showing terminal-state concentration away from hazardous regions is also compelling. However, the central theoretical claim—policy invariance through potential-based shaping—is not supported by the implementation as written, and the empirical support is partial: the gain is clear only on Ant Fall. Since the invariant-optimality argument is the main justification for the method's design, this needs to be repaired or substantially qualified before the paper can be accepted.

major comments (2)
  1. [Section 4.1, Eq. (2)/(4)/(5), Algorithm 1 lines 14–17] The theoretical guarantee and the implementation do not match. Eq. (4) defines the 'after' potential as Φ_{t+c}(s_{t+c}) = −βΨ(·|s_t,g_t), i.e., as a function of the interval's initial state and goal, not of the next state/action. Algorithm 1 then queries the MDN with (s_{t−c},g_{t−c}) to set Φ_t and computes Φ_t − Φ_{t−c}. This is not a potential difference of the form γΦ(s',a')−Φ(s,a); it is a delayed, action-dependent penalty. Moreover, with the discount factor γ appearing in Eq. (2)/(5) but not in the algorithm, the cumulative discounted shaping does not telescope to boundary terms—interior terms of the form (1−γ)Σγ^n Φ_n survive. Consequently, the claim in Section 4.1 that the optimal high-level policy is preserved is unsupported. The authors should either change Algorithm 1 so that the MDN is queried at the current (s_t,g_t) and the shaping term is γ^c Φ_{t+c}(s_{t+c},g_{t+c}) − Φ_
  2. [Table 1, Section 6.2, Figure 2] The empirical evidence does not uniformly support the abstract's claim of outperforming state-of-the-art HRL methods. On Ant Push, S3 (0.186±0.107) is within one standard error of HRAC (0.180±0.180) and the margin over HIRO is not shown to be significant. On Ant Maze, HRAC (0.832±0.039) is nominally better than S3 (0.827±0.024). Only Ant Fall shows a clear, decisive improvement. The authors should report per-seed results, confidence intervals, or a formal significance test, and temper the global performance claim accordingly. As written, the claim 'outperform state-of-the-art HRL methods in non-stationary long-horizon environments' is supported on one of three benchmarks.
minor comments (5)
  1. [Notation, Eqs. (2), (4), (9)] The potential is written as Φ(s,a) in Eq. (2), as Φ(s) in Eq. (4), and as Φ(s,g) in Eq. (9). Unify the notation and make the decision-time indexing explicit (e.g., t, t+c at the manager's timescale).
  2. [Algorithm 1 line 17 vs. Eq. (5)] Eq. (5) includes a discount factor γ multiplying Φ_{t+c}, but Algorithm 1 computes Φ_t − Φ_{t−c} with no discount. Align the discounting between the theory and the implementation.
  3. [Section 6.3, Figure 3] The text says 'by 1M episodes samples remain fairly dispersed,' but training is described in environment steps. Please correct the unit to 'environment steps' and specify the checkpoint scale in Figure 3.
  4. [Eq. (3)] The summation bounds in Eq. (3) are malformed; the product over c steps and the nested sums over actions and intermediate states need clear indexing.
  5. [Figure 2] The caption says 'Percentage learning curves' but the axes are not fully labeled. Please make the y-axis unit (success rate) explicit and ensure the legend is visible.

Circularity Check

0 steps flagged

No circularity found: S3's headline result is an independent empirical benchmark outcome; the learned MDN uncertainty is a training signal, not a fitted prediction of the target result.

full rationale

S3's central claim is empirical: it reports success rates on Ant Maze/Push/Fall measured from environment rollouts (Table 1, Figures 2-5). These values are not constructed by the MDN or by the paper's shaping equations, so the headline result is not equivalent to its inputs. The MDN is trained on B_MDN transitions generated by the agent's own policies and its trace-covariance is used as an intrinsic reward; this is a self-referential learned model, but the paper does not present the MDN's dispersion as an external ground truth or as a prediction of final performance, so it does not fit the 'fitted input called prediction' pattern. The only author self-citation ([12], Haeri, Ahmadzadeh, and Jerath) appears in Related Work as an example of multi-agent reward shaping and is not load-bearing. The closest concern is that the implemented shaping in Algorithm 1 (lines 14-17: Φ_t = -β_t·tr(Σ_mix) from (s_{t-c},g_{t-c}); r̃_hi = r_env + Φ_t − Φ_{t-c}) does not literally instantiate Eq. 2's γΦ_{t+c} − Φ_t as a potential difference over the interval [t−c,t], with Eq. 4 defining Φ_{t+c}(s_{t+c}) = −βΨ(·|s_t,g_t). That is a soundness issue for the optimality-preservation claim, not a circularity: the derivation does not reduce to its own assumptions by construction; it fails to meet the external theorem's conditions. Because the paper is self-contained against external benchmarks and no load-bearing step is definitionally equivalent to its inputs, the circularity score is 0.

Axiom & Free-Parameter Ledger

3 free parameters · 3 axioms · 0 invented entities

The central mechanism rests on a learned uncertainty model and two tuning coefficients; there are no new physical entities. The main extra structure is the coarse-dynamics MDN, which is trained rather than postulated. The free parameters are β, the horizon c, and the unspecified MDN hyperparameters—all of which directly affect the reported performance.

free parameters (3)
  • β (intrinsic reward weight) = not reported (tuned)
    Controls the magnitude of the dispersion penalty in Φ (Eq. 4/9). The paper says β is tuned (Section 5) and Algorithm 1 includes a schedule β_t, but no value or schedule is given.
  • Manager horizon c = 10 (c*=10)
    Algorithm 1 input; Figure 3 sweep over {5,10,20,50,100} selects c=10 as best. This is chosen based on performance and then used in the headline results.
  • MDN hyperparameters (K, network sizes, learning rates) = not reported
    The MDN in Eq. 7/8 is used to approximate the coarse dynamics; the number of mixture components K and optimization details are unspecified, and the intrinsic reward directly depends on the MDN's output covariance.
axioms (3)
  • domain assumption The c-step high-level transition forms an MDP and the PBRS theorem applies with discount γ rather than γ^c.
    Section 4.1 applies Eq. (1) directly to c-step transitions; no derivation of the semi-MDP discount or state-action extension is given. If the discount should be γ^c or the state is not Markov, policy invariance is not established.
  • domain assumption The MDN's predicted covariance is a valid estimate of the true c-step terminal-state distribution under the current low-level policy.
    Section 5 relies on the MDN trained on B_MDN to stand in for the intractable Eq. (3); no calibration or validation of the uncertainty estimates is reported.
  • domain assumption The learned potential can change over time without breaking optimal-policy invariance.
    The potential Φ is updated as θ_MDN changes (Algorithm 1 lines 14–20). The paper cites [7] for dynamic PBRS but does not prove the conditions hold for a potential that depends on the previous subgoal/state pair rather than the current one.

pith-pipeline@v1.3.0-alltime-deepseek · 12171 in / 18148 out tokens · 191031 ms · 2026-08-01T13:00:55.804028+00:00 · methodology

0 comments
Cite this review

Pith. "Pith review of S3: Stable Subgoal Selection by Constraining Uncertainty of Coarse Dynamics in Hierarchical Reinforcement Learning." pith.science (2026). https://pith.science/paper/SMHGVMY2

@misc{pith2026260719232,
  author       = {Pith},
  title        = {Pith review of: S3: Stable Subgoal Selection by Constraining Uncertainty of Coarse Dynamics in Hierarchical Reinforcement Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/SMHGVMY2}},
  note         = {Machine review of arXiv:2607.19232}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Hierarchical Reinforcement Learning (HRL) intends to separate strategic planning from primitive execution. It has been widely successful in solving long-horizon and complex tasks, where flat-RL algorithms have difficulty in learning. However, while the low-level agent in HRL benefits from dense feedback and abundant trial opportunities, the high-level agent receives sparse, delayed feedback from the environment and its performance depends on the low-level execution capability. In this paper, we study whether subgoal selection by the high-level agent can be performed more strategically, by providing it with dynamics-aware intrinsic motivation. Since motivation based on primitive transition dynamics would require broad coverage of the state-action space, we propose to use coarse dynamics, i.e., environment transitions aggregated over multiple steps at the temporal scale at which the high-level agent operates. This approach stabilizes the high-level policy by learning to minimize the predictive uncertainty associated with the coarse dynamics, and provides a guided structure for navigation. We model the predictive uncertainty by evaluating different dispersion metrics as approximated by a Mixture Density Network (MDN). Empirically, we observe that a dense, dynamics-aware intrinsic reward leads to risk-averse subgoal selection, enabling it to outperform state-of-the-art HRL methods in non-stationary long-horizon environments.

Figures

Figures reproduced from arXiv: 2607.19232 by Kshitij Jerath, Kshitij Kumar Srivastava.

Figure 1
Figure 1. Figure 1: HRL framework with S3 implemented by using MDN Network. (b) S3 module implementation. During the training, S3 [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Percentage Learning curves on Ant Fall, Ant Push, and Ant Maze comparing S3, HIRO, and HRAC. All methods are [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: S3 learning curves under different Manager hori [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Scatter plot designed to show the spread of worker capability at different points in the training. Blue dots are terminal [PITH_FULL_IMAGE:figures/full_fig_p007_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Manager-guided trajectories for S3 (blue) and HRAC (green) over 50 evaluation runs in (a) Ant Fall, (b) Ant Push, [PITH_FULL_IMAGE:figures/full_fig_p008_5.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

35 extracted references · 9 linked inside Pith

  1. [1]

    Marcin Andrychowicz, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, Pieter Abbeel, and Wojciech Zaremba

  2. [2]

    Andrew G Barto and Sridhar Mahadevan. 2003. Recent advances in hierarchical reinforcement learning.Discrete event dynamic systems13, 4 (2003), 341–379

    2003

  3. [3]

    Christopher M. Bishop. 1994. Mixture density networks. (1994)

    1994

  4. [4]

    Taylor, and Ann Nowé

    Tim Brys, Anna Harutyunyan, Matthew E. Taylor, and Ann Nowé. 2015. Policy Transfer using Reward Shaping. InProceedings of the 2015 International Confer- ence on Autonomous Agents and Multiagent Systems(Istanbul, Turkey)(AAMAS ’15). International Foundation for Autonomous Agents and Multiagent Systems, Richland, SC, 181–188

    2015

  5. [5]

    Peter Dayan and Geoffrey E Hinton. 1992. Feudal Reinforcement Learning. In Advances in Neural Information Processing Systems, S. Hanson, J. Cowan, and C. Giles (Eds.), Vol. 5. Morgan-Kaufmann. https://proceedings.neurips.cc/paper_ files/paper/1992/file/d14220ee66aeec73c49038385428ec4c-Paper.pdf

    1992

  6. [6]

    Sam Devlin, Logan Yliniemi, Daniel Kudenko, and Kagan Tumer. 2014. Potential- based difference rewards for multiagent reinforcement learning. InProceedings of the 2014 international conference on Autonomous agents and multi-agent systems. 165–172

    2014

  7. [7]

    Sam Michael Devlin and Daniel Kudenko. 2012. Dynamic potential-based reward shaping. In11th International Conference on Autonomous Agents and Multiagent Systems (AAMAS 2012). IFAAMAS, 433–440

    2012

  8. [8]

    Benjamin Eysenbach, Abhishek Gupta, Julian Ibarz, and Sergey Levine. 2018. Diversity is all you need: Learning skills without a reward function.arXiv preprint arXiv:1802.06070(2018)

    Pith/arXiv arXiv 2018

  9. [9]

    Fred E Fiedler. 1967. A THEORY OF LEADERSHIP EFFECTIVENESS. MCGRAW- HILL SERIES IN MANAGEMENT. (1967)

    1967

  10. [10]

    Xiaozhu Gao, Jinhui Liu, Bo Wan, and Lingling An. 2024. Hierarchical reinforce- ment learning from demonstration via reachability-based reward shaping.Neural Processing Letters56, 3 (2024), 184

    2024

  11. [11]

    Karol Gregor, Danilo Jimenez Rezende, and Daan Wierstra. 2016. Variational intrinsic control.arXiv preprint arXiv:1611.07507(2016)

    Pith/arXiv arXiv 2016

  12. [12]

    Hossein Haeri, Reza Ahmadzadeh, and Kshitij Jerath. 2022. Reward-sharing relational networks in multi-agent reinforcement learning as a framework for emergent behavior.arXiv preprint arXiv:2207.05886(2022)

    Pith/arXiv arXiv 2022

  13. [13]

    Junsu Kim, Younggyo Seo, and Jinwoo Shin. 2021. Landmark-guided subgoal generation in hierarchical reinforcement learning.Advances in neural information processing systems34 (2021), 28336–28349

    2021

  14. [14]

    Alexander S Klyubin, Daniel Polani, and Chrystopher L Nehaniv. 2005. Empow- erment: A universal agent-centric measure of control. In2005 ieee congress on evolutionary computation, Vol. 1. IEEE, 128–135

    2005

  15. [15]

    Tejas D Kulkarni, Karthik Narasimhan, Ardavan Saeedi, and Josh Tenenbaum

  16. [16]

    Andrew Levy, George Konidaris, Robert Platt, and Kate Saenko. 2017. Learning multi-level hierarchies with hindsight.arXiv preprint arXiv:1712.00948(2017)

    Pith/arXiv arXiv 2017

  17. [17]

    Siyuan Li, Jin Zhang, Jianhao Wang, Yang Yu, and Chongjie Zhang. 2021. Active hierarchical exploration with stable subgoal representation learning.arXiv preprint arXiv:2105.14750(2021)

    Pith/arXiv arXiv 2021

  18. [18]

    Siyuan Li, Lulu Zheng, Jianhao Wang, and Chongjie Zhang. 2021. Learning subgoal representations with slow dynamics. InInternational Conference on Learning Representations

    2021

  19. [19]

    Willie McClinton, Andrew Levy, and George Konidaris. 2021. Hac explore: Accelerating exploration with hierarchical reinforcement learning.arXiv preprint arXiv:2108.05872(2021)

    Pith/arXiv arXiv 2021

  20. [20]

    Amy McGovern and Andrew G Barto. 2001. Automatic discovery of subgoals in reinforcement learning using diverse density. (2001)

    2001

  21. [21]

    Ishai Menache, Shie Mannor, and Nahum Shimkin. 2002. Q-cut—dynamic discov- ery of sub-goals in reinforcement learning. InEuropean conference on machine learning. Springer, 295–306

    2002

  22. [22]

    Ofir Nachum, Michael Ahn, Hugo Ponte, Shixiang Gu, and Vikash Kumar. 2019. Multi-agent manipulation via locomotion using hierarchical sim2real.arXiv preprint arXiv:1908.05224(2019)

    Pith/arXiv arXiv 2019

  23. [23]

    Ofir Nachum, Shixiang Shane Gu, Honglak Lee, and Sergey Levine. 2018. Data- efficient hierarchical reinforcement learning.Advances in neural information processing systems31 (2018)

    2018

  24. [24]

    Andrew Y Ng, Daishi Harada, and Stuart Russell. 1999. Policy invariance under reward transformations: Theory and application to reward shaping. InIcml, Vol. 99. Citeseer, 278–287

    1999

  25. [25]

    Jürgen Schmidhuber and Reiner Wahnsiedler. 1992. Planning simple trajectories using neural subgoal generators. InProceedings of the 2nd International Conference on Simulation of Adaptive Behavior. 196–202

    1992

  26. [26]

    Richard S Sutton, Doina Precup, and Satinder Singh. 1999. Between MDPs and semi-MDPs: A framework for temporal abstraction in reinforcement learning. Artificial intelligence112, 1-2 (1999), 181–211

    1999

  27. [27]

    Emanuel Todorov, Tom Erez, and Yuval Tassa. 2012. MuJoCo: A physics engine for model-based control. In2012 IEEE/RSJ International Conference on Intelligent Robots and Systems. 5026–5033. https://doi.org/10.1109/IROS.2012.6386109

    arXiv 2012

  28. [28]

    Alexander Sasha Vezhnevets, Simon Osindero, Tom Schaul, Nicolas Heess, Max Jaderberg, David Silver, and Koray Kavukcuoglu. 2017. Feudal networks for hier- archical reinforcement learning. InInternational conference on machine learning. PMLR, 3540–3549

    2017

  29. [29]

    Vivienne Huiling Wang, Tinghuai Wang, and Joni Pajarinen. 2025. Hierarchical Reinforcement Learning with Uncertainty-Guided Diffusional Subgoals.arXiv preprint arXiv:2505.21750(2025)

    Pith/arXiv arXiv 2025

  30. [30]

    Kandai Watanabe, Mathew Strong, and Omer Eldar. 2022. SHIRO: Soft hierarchical reinforcement learning.arXiv preprint arXiv:2212.12786(2022)

    Pith/arXiv arXiv 2022

  31. [31]

    Eric Wiewiora, Garrison W Cottrell, and Charles Elkan. 2003. Principled methods for advising reinforcement learning agents. InProceedings of the 20th international conference on machine learning (ICML-03). 792–799

    2003

  32. [32]

    Gary Yukl and David D Van Fleet. 1992. Theory and research on leadership in organizations. (1992)

    1992

  33. [33]

    Tianren Zhang, Shangqi Guo, Tian Tan, Xiaolin Hu, and Feng Chen. 2020. Gen- erating adjacency-constrained subgoals in hierarchical reinforcement learning. Advances in neural information processing systems33 (2020), 21579–21590

    2020

  34. [2016]

    Hierarchical deep reinforcement learning: Integrating temporal abstraction and intrinsic motivation.Advances in neural information processing systems29 (2016)

    2016

  35. [2017]

    Hindsight experience replay.Advances in neural information processing systems30 (2017)

    2017