Pith. sign in

REVIEW 5 major objections 5 minor 47 references

Hierarchical Reinforcement Learning in Multi-Goal Spatial Navigation with Autonomous Mobile Robots

T0 review · 5 major / 5 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The Option-Critic hierarchical architecture converges about 2.6 times faster than PPO on a four-rooms navigation maze, and the authors argue the speedup comes from sub-goals created jointly by the termination function and the critic.

desk verdict A solid OC-vs-PPO comparison is buried under an unfalsifiable sub-goal story and a shaky convergence metric. read the letter →

arxiv 2504.18794 v3 pith:5JDRRJEG submitted 2025-04-26 cs.AI cs.RO

Brendon Johnson , Alfredo Weitzenfeld This is my paper · ORCID
classification cs.AIcs.RO
keywords hierarchicalreinforcementlearningOption-Criticarchitectureproximalpolicyoptimizationspatialnavigationsub-goalsterminationfunctionsparserewardsmobilerobots
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that hierarchical reinforcement learning—specifically the Option-Critic architecture—converges faster than standard proximal policy optimization (PPO) on multi-goal robot navigation problems with sparse rewards, and that this advantage grows as mazes get harder. Its central quantitative evidence is the four-rooms maze, where Option-Critic converges in 175,055.6 steps versus PPO's 459,396.8 steps ($p = 0.036$), with similar final path lengths. The paper argues that the advantage comes from sub-goals that the option termination function and the critic create together: removing sub-goals by terminating every step, or letting one option cover the whole task, both degrade performance. It also reports that automatically learned sub-goals outperform hand-set doorway sub-goals, because hand-set goals restrict exploration. If the claim holds, it matters because it identifies why hierarchy helps in sparse-reward navigation rather than treating hierarchy as a black-box improvement.

What carries the argument

The central object is the Option-Critic Architecture, a hierarchical extension of the policy-gradient actor-critic method in which actions are replaced by options—temporally extended action sequences, each with its own policy and termination function. The load-bearing identity is the termination-function update (Equation 5), where a regularization term $\phi$ penalizes termination, and the critic's value comparison $Q_\Omega(s', \omega) - V_\Omega(s')$ decides whether ending the option is beneficial. Together with the critic's evaluation (Equations 2 and 3), this mechanism creates sub-goals: the termination function sets how often options end, and the critic repeatedly selects the same option until a sub-goal state is reached. The paper uses the termination regulation value $\phi$ as the experimental dial that controls how many sub-goals exist, and the convergence-time metric (steps until the algorithm stops improving on the optimal path) as the yardstick used across all four experiments.

What would settle it

Re-run the termination-regulation sweep on the four-rooms maze without the 500,000-step cutoff, recording the first step at which each run reaches a fixed path-length threshold for a fixed number of consecutive episodes. If $\phi$ values of 0.08 and 0.10 eventually converge at speeds comparable to $\phi = 0.01$, then the reported monotonic worsening in Table 8 is an artifact of the cap, and the conclusion that frequent termination drives the advantage would be undercut.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that the Option-Critic architecture, a hierarchical algorithm built on temporally extended options, outperforms a flat PPO baseline in sparse-reward maze navigation, and that the mechanism is the joint creation of sub-goals by the termination function and the critic. The termination function decides when an option ends (Equation 5), the critic evaluates option-state-action triples and drives which option is chosen (Equations 2 and 3), and the policy over options repeatedly picks options, so the combination effectively selects intermediate states to reach. In the authors' account, this is why Experiment 1 shows OC converging about 2.6 times faster than PPO on the four-rooms maze, why Experiment 2 shows that termination-created sub-goals beat critic-only sub-goals (118,489.85 vs 172,863.95 steps), and why Experiment 4 shows that a termination regulation of $\phi = 0.00$–$0.01$, which keeps options short, converges faster than larger $\phi$ values that let options grow. The paper concludes that sub-goals, and the hierarchy they define, are the source of HRL's improved performance in these tasks.

Load-bearing premise

The load-bearing premise is the paper's definition of convergence time as the step count at which the algorithm stops improving on the optimal path, because that count is the primary measure in all four experiments; the paper gives no criterion for detecting the optimal path, reports no standard deviations, and truncates runs at 500,000 steps, so several reported convergence times are censored values rather than true measurements.

Editorial extensions

If this is right

  • In sparse-reward navigation, switching from a flat PPO policy to an Option-Critic policy can cut convergence steps by more than half (four-rooms: 175,056 vs 459,397 steps) while keeping final path length essentially unchanged.
  • Sub-goals are load-bearing: both ways of removing them—terminating every step so no sub-goal forms, or letting one option run the whole task—raise convergence time and path length.
  • Automatic sub-goal discovery beats hand-set sub-goals when the hand-set goals divide the task evenly, because automatic sub-goals permit more exploration of different options.
  • The termination regulation $\phi$ is a practical control knob: small values (0.00–0.01) give fast convergence, while values of 0.06 and above make the algorithm approach or hit the 500,000-step limit, indicating option degeneration.
  • A well-tuned Option-Critic model (25,872 parameters) converges faster than every PPO variant tested, including a PPO about 50 times larger (1,317,897 parameters).

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper's convergence-time metric is right-censored at 500,000 steps, so the ANOVA in Table 8 partly reflects cap effects; a cleaner test would record uncapped learning curves or time-to-threshold.
  • A direct causal test of the sub-goal explanation would ablate the critic's role in option selection (e.g., random option switching at termination) while keeping the termination function; the paper's Experiment 2 ablates termination, not the critic, so the joint mechanism is inferred rather than isolated.
  • The findings suggest a design heuristic for HRL: tune the termination regularization so options last a handful of states—long enough to be temporally abstract, short enough that the critic frequently influences which option is active—and re-evaluate in continuous-action or physical-robot settings, where the finite 8-action maze may not transfer.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper compares the Option-Critic (OC) hierarchical reinforcement learning architecture with PPO in simulated multi-goal spatial navigation tasks performed by a mobile robot in Webots. Four experiments are reported: (1) OC versus PPO on three mazes of increasing difficulty; (2) termination-created versus critic-created sub-goals, implemented by disabling the termination function so that options terminate every step; (3) automatically set versus manually set sub-goals; and (4) a sweep over the termination regulation parameter φ. The authors report faster convergence for OC on the harder mazes, faster convergence for termination-created sub-goals than for critic-created sub-goals, faster convergence for model-set than hand-set sub-goals, and generally worse performance as φ increases. The paper's stated primary contribution is to demonstrate the importance of sub-goals in HRL.

Significance. If the empirical results were fully supported, the paper would provide useful evidence about the relative performance of OC and PPO in sparse-reward robot navigation, a domain where such comparisons are of interest. The authors should be credited for making the code and data available through FAIRIS, for selecting a PPO baseline conservatively from several model sizes, and for reporting statistically significant convergence-time differences on the two harder mazes. The strongest part of the paper is therefore the raw OC-versus-PPO comparison. However, the significance of the paper as it stands is limited because the central mechanistic claim about sub-goals is not supported by the experiments as designed: the paper's own conclusions are unfalsifiable as stated, and the primary performance metric is not operationally defined. The contribution could become publishable after substantial revision, including better metric definitions, censoring-aware statistics, and at least one direct ablation that removes sub-goals without changing the base algorithm.

major comments (5)
  1. [§4.3, Tables 3, 6, 7, 8] The convergence-time metric is not operationally defined. Section 4.3 defines it as 'the number of steps the algorithm takes to find the optimal path, i.e. when the algorithm stops learning and improving upon the optimal path,' but no criterion is given for detecting the optimal path or for deciding when learning has stopped. This matters because convergence time is the primary measure used in all four experiments. The problem is compounded by the 500,000-step cap: Table 8 reports convergence times greater than the stated cap (500,578.8 for φ=0.08 and 500,335.6 for φ=0.10), so those observations are censored rather than true convergence times, yet the ANOVA treats them as ordinary values. As a result, the p-values in Tables 3, 6, and 8 do not rest on a well-defined outcome variable.
  2. [§5.3, §6.4, §6.5] The central claim that sub-goals cause OC's advantage is unfalsifiable as tested. In Experiment 2, removing the termination function so that options terminate every step is interpreted as 'critic created sub-goals' that perform worse. In Experiment 4, the φ=0.00 condition, where the average option length is 1.033 steps, is interpreted as the critic repeatedly picking the same option and therefore still creating sub-goals. Under this pair of interpretations, no possible experimental outcome can disconfirm the hypothesis that sub-goals are important. The paper needs an independent, measurable definition of a sub-goal and at least one condition in which sub-goals are genuinely absent, together with a stated prediction for that condition.
  3. [§6.5, §5.2, Tables 4 and 5] The 'remove the terminations and have one option' condition is never run. Section 6.5 identifies this condition with the PPO baseline from Experiment 1, but PPO differs from OC in network size, layer count, and exploration schedule (Tables 4 and 5), so the comparison does not isolate the role of the hierarchy. A one-option OC ablation with the termination function removed is needed to test the claim that the performance difference comes from sub-goals rather than from other architectural differences between OC and PPO.
  4. [§6.3, §6.5] The paper's own discussion undercuts the causal interpretation of Experiment 3. Section 6.3 states that automatic sub-goals perform better 'because they allow the algorithm to explore more not because the sub-goals are necessarily better.' This directly contradicts the conclusion in Section 6.5 that the experiments demonstrate the importance of sub-goals, and the contradiction is not reconciled anywhere in the manuscript.
  5. [All experiments, Tables 3, 6, 7, 8] The statistical reporting is incomplete for the claims made. No standard deviations, confidence intervals, or error bars are reported anywhere, all comparisons use only five runs, and no correction is applied for the multiple comparisons across the three mazes in Experiment 1 or across the eleven φ values in Experiment 4. Given that the underlying convergence-time values are censored for several φ values, the reported p-values should not be taken at face value; at minimum, the censoring must be handled explicitly and variance estimates should be reported.
minor comments (5)
  1. [Table 6 caption] The caption reads 'Experiment Four Results' but the table reports Experiment 2; the caption should be corrected to 'Experiment Two Results.'
  2. [§6.4] The sentence 'as seen in experiment 3, this can also lead to poor performance' appears to refer to terminating every state, which is Experiment 2, not Experiment 3; please correct the cross-reference.
  3. [Figures 8, 11, 12] The learning curves in Figures 8, 11, and 12 lack axis labels, legends explaining the colors, and any indication of run-to-run variability; adding these would make the figures interpretable without referring to the table text.
  4. [§3.3, Eq. (2)] Equation (2) is typeset with a line break that makes it hard to parse; presenting it as a multi-line display and defining each symbol immediately after the equation would improve readability.
  5. [§5.5, §6.4] The term 'degerenating' in Section 5.5 is a typo for 'degenerating,' and Section 6.4 defines the degenerate case only qualitatively; since average option length is used as the degeneration diagnostic, a clear threshold or quantitative definition would help.

Circularity Check

3 steps flagged · score 5.0 of 10

The reported OC-vs-PPO convergence advantage is not circular, but the paper's central mechanistic claim — that sub-goals drive the advantage — is non-falsifiable as executed because every condition that removes terminations is reinterpreted as critic-created sub-goals.

  1. self definitional [Section 5.3 / Table 6 and Section 6.5]
    "To compare to the critic created sub-goals, the termination function has been removed completely, and instead, the option will be terminated at every step. This is done to remove the termination function and prevent it from creating sub-goals, so the critic will set the sub-goals instead."

    The experiment is designed to remove sub-goals by terminating every step, yet Section 6.5 reclassifies this same condition as still containing sub-goals: 'even if we remove the termination functions and terminate every step instead; the critic will still create sub-goals.' The claim 'removing sub-goals hurts performance' is therefore tested only by a condition that the authors define as not removing sub-goals. No observation can falsify the claim that sub-goals are important, because the condition meant to remove them is redefined as retaining them.

  2. self definitional [Section 5.5 / Table 8 (phi=0.00) and Section 6.4]
    "Additionally whenϕ is 0.00 the average option lengths are about one. This would suggest that the options terminate every state, but as shown in Figure 13, the options still can last longer than one state, resulting in the critic repeatedly picking the same option in those states. This suggests the critic assists the termination function in creating the sub-goals."

    The paper uses the phi=0.00 condition as evidence of good performance through critic-created sub-goals, but this is the same mechanism used to explain the worse-performing critic-only condition in Experiment 2. Average option length of 1.03 means options almost never extend beyond a single state; calling 'the critic repeatedly picking the same option' a sub-goal makes 'sub-goal' unfalsifiable, because any repeated choice of the same option over multiple states qualifies regardless of whether a distinct sub-goal state was ever learned.

1 more flagged steps
  1. other [Section 6.5 and Section 6.3]
    "These experiments show the importance of sub-goals because if we remove them by not terminating or terminating every step then the performance decreases. ... In summary the automatic sub-goals perform better because they allow the algorithm to explore more not because the sub-goals are necessarily better."

    The paper's stated primary contribution is to 'demonstrate the importance of sub-goals' (Section 6), but its own Experiment 3 discussion explicitly disclaims the causal interpretation: automatic sub-goals win 'because they allow the algorithm to explore more not because the sub-goals are necessarily better.' Meanwhile, the first way of removing sub-goals listed in Section 6.5 — 'remove the terminations and have one option' — is never run as an OC ablation; it is identified with the PPO baseline from Experiment 1, which differs in architecture, network size, and exploration schedule. The central mechanistic conclusion is thus supported only by conditions that are either redefined as still having sub-goals or not directly executed.

full rationale

The raw empirical comparison in Experiment 1 (OC vs PPO convergence times, Table 3) is not circular: it is an external benchmark against a standard baseline, and the p-values are reported. Experiments 3 and 4 also provide concrete quantitative results (Tables 7 and 8) that are not constructed to equal their inputs. However, the paper's central claim — stated in the introduction and conclusions as 'demonstrate the importance of sub-goals' — is tested in a way that cannot fail. Experiment 2 removes the termination function and terminates every step, but Section 6.5 says the critic 'will still create sub-goals' in that condition. Experiment 4's phi=0.00 condition, with average option length 1.03, is credited to 'the critic repeatedly picking the same option,' again a critic-created sub-goal. Thus every condition that removes terminations is reinterpreted as still containing sub-goals, making the hypothesis 'sub-goals are important' immune to disconfirmation within this paper. The paper itself further undercuts the mechanistic reading in Section 6.3: automatic sub-goals win 'because they allow the algorithm to explore more not because the sub-goals are necessarily better.' The suggested ablation that would cleanly remove sub-goals — one option with no terminations — is never run; the paper instead equates it with the architecturally different PPO baseline. Because the empirical OC-vs-PPO result is real but the paper's explanation of that result is non-falsifiable as executed, the circularity score is 5 rather than higher: the data are not fabricated or fitted, but the stated mechanism is protected by definitional reclassification.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The paper has no mathematical derivation with fitted constants. The hand-set hyperparameters that carry the central claim are the termination regularization phi (the independent variable of Experiment 4) and the fixed number of options. The model also assumes standard RL convergence behavior and the validity of the Webots simulation as a proxy for real robots.

free parameters (2)
  • Termination regulation value phi = 0.01 in the base model; swept from 0.00 to 0.10 in Experiment 4
    Hand-chosen hyperparameter that directly controls the frequency of option termination; the paper's Experiment 4 treats it as the independent variable and concludes that smaller values improve convergence, so the central sub-goal claim depends on this choice.
  • Number of options = 4
    Set by hand from Bacon et al. [3]; the number of sub-policies is not tuned or varied, yet it affects the hierarchy's expressiveness and the comparison with PPO.
assumptions (4)
  • domain assumption The Webots/FAIRIS simulation is a faithful proxy for real mobile-robot navigation such that conclusions transfer to physical robots.
    Invoked throughout Section 4.2; the paper uses simulation to draw conclusions about HRL advantages for autonomous robots without hardware validation.
  • domain assumption The convergence time can be reliably measured as the number of steps until the algorithm 'stops learning and improving upon the optimal path' without an explicit detection criterion.
    Defined in Section 4.3 and used for all comparisons; no algorithm for detecting the optimal path or for deciding convergence is given.
  • standard math Student's t-test and ANOVA are valid with five independent runs per condition and no verification of normality or homogeneity of variance.
    Used in Sections 5.2-5.5; with n=5 and no variance reporting, the reported p-values are not robust.
  • standard math Standard convergence properties of policy gradient methods and the Option-Critic architecture hold in this setting despite the authors noting OC has no convergence guarantee.
    Section 3.3 acknowledges OC lacks a convergence guarantee; the experiments assume practical convergence of the trained policies.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Hierarchical Reinforcement Learning in Multi-Goal Spatial Navigation with Autonomous Mobile Robots." pith.science (2026). https://pith.science/paper/5JDRRJEG

@misc{pith2026250418794,
  author       = {Pith},
  title        = {Pith review of: Hierarchical Reinforcement Learning in Multi-Goal Spatial Navigation with Autonomous Mobile Robots},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/5JDRRJEG}},
  note         = {Machine review of arXiv:2504.18794}
}
read the original abstract

Hierarchical reinforcement learning (HRL) is hypothesized to be able to leverage the inherent hierarchy in learning tasks where traditional reinforcement learning (RL) often fails. In this research, HRL is evaluated and contrasted with traditional RL in complex robotic navigation tasks. We evaluate unique characteristics of HRL, including its ability to create sub-goals and the termination functions. We constructed a number of experiments to test: 1) the differences between RL proximal policy optimization (PPO) and HRL, 2) different ways of creating sub-goals in HRL, 3) manual vs automatic sub-goal creation in HRL, and 4) the effects of the frequency of termination on performance in HRL. These experiments highlight the advantages of HRL over RL and how it achieves these advantages.

Figures

Figures reproduced from arXiv: 2504.18794 by the authors.

Figure 1
Figure 1. Difference Between MDP, SMDP, and MDP with Options where [PITH_FULL_IMAGE:figures/full_fig_p005_1.png] view at source ↗
Figure 2
Figure 2. The Option Critic Architecture: Showing the Critic, the Options, and the [PITH_FULL_IMAGE:figures/full_fig_p009_2.png] view at source ↗
Figure 3
Figure 3. The Neural Network used for the Option Critic Architecture, with two shared [PITH_FULL_IMAGE:figures/full_fig_p012_3.png] view at source ↗
Figures from the paper (10 more)
Figure 4
Figure 4. Figure 4: Degenerated Option Example: a) non-degenerated sample path where each color [PITH_FULL_IMAGE:figures/full_fig_p014_4.png]
Figure 5
Figure 5. Figure 5: The Neural Network used for PPO with four fully connected layers and ReLU [PITH_FULL_IMAGE:figures/full_fig_p015_5.png]
Figure 6
Figure 6. Figure 6: The four-rooms maze in the Webots simulator with the robot at the starting [PITH_FULL_IMAGE:figures/full_fig_p017_6.png]
Figure 7
Figure 7. Figure 7: All of the mazes used in the experiments: a) the one-room multiple small [PITH_FULL_IMAGE:figures/full_fig_p019_7.png]
Figure 8
Figure 8. Figure 8: The results for the PPO vs OC for each maze: a) one-room multiple small obsta [PITH_FULL_IMAGE:figures/full_fig_p021_8.png]
Figure 9
Figure 9. Figure 9: The figure shows the path generated when termination created sub-goals are [PITH_FULL_IMAGE:figures/full_fig_p024_9.png]
Figure 10
Figure 10. Figure 10: This figure shows the sub-goals for each option. The start point is shown in [PITH_FULL_IMAGE:figures/full_fig_p025_10.png]
Figure 11
Figure 11. Figure 11: Termination Regulation Option Lengths during training for each of the tested [PITH_FULL_IMAGE:figures/full_fig_p028_11.png]
Figure 12
Figure 12. Figure 12: Termination Regulation Path Lengths during training for each of the tested [PITH_FULL_IMAGE:figures/full_fig_p029_12.png]
Figure 13
Figure 13. Figure 13: Termination Regulation Paths for each of the tested Termination Regulation [PITH_FULL_IMAGE:figures/full_fig_p030_13.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

47 extracted references · 22 canonical work pages

  1. [1]

    Maximum a Posteriori Policy Optimisation

    Abbas Abdolmaleki et al. Maximum a Posteriori Policy Optimisation

    work page

  2. [2]

    Marcin Andrychowicz et al.Hindsight Experience Replay. 2018. arXiv: 1707.01495 [cs.LG]. url: https://arxiv.org/abs/1707.01495

    arXiv 2018

  3. [3]

    The Option-Critic Architecture

    Pierre-Luc Bacon, Jean Harb, and Doina Precup. The Option-Critic Architecture. 2016. arXiv:1609.05140 [cs.AI]. url: https://arxiv. org/abs/1609.05140

    arXiv 2016

  4. [4]

    Option Discovery using Deep Skill Chaining

    Akhil Bagaria and George Konidaris. “Option Discovery using Deep Skill Chaining”. In:International Conference on Learning Representa- tions. 2020. url: https://openreview.net/forum?id=B1gqipNYwH

    work page 2020

  5. [5]

    Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio.Neural Ma- chine Translation by Jointly Learning to Align and Translate. 2016. arXiv: 1409 . 0473 [cs.CL]. url: https : / / arxiv . org / abs / 1409 . 0473

    work page 2016

  6. [6]

    FAIRIS: Framework for Autonomous Intelligent Robotic Interaction and Simulation

    BioRobotics and Artificial Intelligence Lab. FAIRIS: Framework for Autonomous Intelligent Robotic Interaction and Simulation. https : //github.com/biorobaw/FAIRIS. 2025. 36

    work page 2025

  7. [7]

    Reservoir computing model of prefrontal cortex creates novel combinations of previous navigation sequences from hip- pocampal place-cell replay with spatial reward propagation

    Nicolas Cazin et al. “Reservoir computing model of prefrontal cortex creates novel combinations of previous navigation sequences from hip- pocampal place-cell replay with spatial reward propagation”. In:PLOS Computational Biology 15.7 (July 2019), pp. 1–32. doi: 10 . 1371 / journal.pcbi.1006624. url: https://doi.org/10.1371/journal. pcbi.1006624

    doi:10.1371/journal 2019

  8. [8]

    Raviteja Chunduru and Doina Precup.Attention Option-Critic. 2022. arXiv: 2201.02628 [cs.LG] . url: https://arxiv.org/abs/2201. 02628

    arXiv 2022

Show all 47 references
  1. [9]

    Feudal Reinforcement Learn- ing

    Peter Dayan and Geoffrey E Hinton. “Feudal Reinforcement Learn- ing”. In: Advances in Neural Information Processing Systems. Ed. by S. Hanson, J. Cowan, and C. Giles. Vol. 5. Morgan-Kaufmann, 1992. url: https : / / proceedings . neurips . cc / paper _ files / paper / 1992/file...

    1992

  2. [10]

    Dietterich

    Thomas G. Dietterich. Hierarchical Reinforcement Learning with the MAXQ Value Function Decomposition.1999.arXiv: cs/9905014 [cs.LG]. url: https://arxiv.org/abs/cs/9905014

    1999 arXiv

  3. [11]

    Implementation Matters in Deep RL: A Case Study on PPO and TRPO

    Logan Engstrom et al. “Implementation Matters in Deep RL: A Case Study on PPO and TRPO”. In:International Conference on Learning Representations. 2020. url: https://openreview.net/forum?id= r1etN1rtPB

    2020

  4. [12]

    Reverse Curriculum Generation for Reinforce- ment Learning

    Carlos Florensa et al. Reverse Curriculum Generation for Reinforce- ment Learning. 2018. arXiv: 1707 . 05300 [cs.AI]. url: https : / / arxiv.org/abs/1707.05300

    2018 arXiv

  5. [13]

    Soft Actor-Critic: Off-Policy Maximum En- tropy Deep Reinforcement Learning with a Stochastic Actor.2018.arXiv: 1801.01290 [cs.LG]

    Tuomas Haarnoja et al. Soft Actor-Critic: Off-Policy Maximum En- tropy Deep Reinforcement Learning with a Stochastic Actor.2018.arXiv: 1801.01290 [cs.LG]. url: https://arxiv.org/abs/1801.01290

    2018 arXiv

  6. [14]

    TheTerminationCritic

    AnnaHarutyunyanetal.“TheTerminationCritic”.In: CoRR abs/1902.09996 (2019). arXiv: 1902 . 09996. url: http : / / arxiv . org / abs / 1902 . 09996

    2019 arXiv

  7. [15]

    Double Q-learning

    Hado Hasselt. “Double Q-learning”. In: Advances in Neural Informa- tion Processing Systems. Ed. by J. Lafferty et al. Vol. 23. Curran As- sociates, Inc., 2010.url: https://proceedings.neurips.cc/paper_ files / paper / 2010 / file / 091d584fced301b442654dd8c23b3fc9 - Paper.pdf. 37

    2010

  8. [16]

    Khimya Khetarpal et al.Options of Interest: Temporal Abstraction with Interest Functions. 2020. arXiv: 2001 . 00271 [cs.LG]. url: https : //arxiv.org/abs/2001.00271

    2020 arXiv

  9. [17]

    Learnings Options End-to-End for Continu- ous Action Tasks

    Martin Klissarov et al. Learnings Options End-to-End for Continu- ous Action Tasks. 2017. arXiv: 1712 . 00004 [cs.LG]. url: https : //arxiv.org/abs/1712.00004

    2017 arXiv

  10. [18]

    Actor-Critic Algorithms

    Vijay Konda and John Tsitsiklis. “Actor-Critic Algorithms”. In: Ad- vances in Neural Information Processing Systems. Ed. by S. Solla, T. Leen, and K. Müller. Vol. 12. MIT Press, 1999

    1999

  11. [19]

    Skill Discovery in Continuous ReinforcementLearningDomainsusingSkillChaining

    George Konidaris and Andrew Barto. “Skill Discovery in Continuous ReinforcementLearningDomainsusingSkillChaining”.In: Advances in Neural Information Processing Systems. Ed. by Y. Bengio et al. Vol. 22. Curran Associates, Inc., 2009.url: https://proceedings.neurips. cc/paper_fi...

    2009

  12. [20]

    UAV Trajectory Optimization for Spectrum Cartogra- phy: A PPO Approach

    Li Li et al. “UAV Trajectory Optimization for Spectrum Cartogra- phy: A PPO Approach”. In:IEEE Communications Letters27.6 (2023), pp. 1575–1579. doi: 10.1109/LCOMM.2023.3265214

    2023

  13. [21]

    Lillicrap et al.Continuous control with deep reinforcement learning

    Timothy P. Lillicrap et al.Continuous control with deep reinforcement learning. 2019. arXiv: 1509 .02971 [cs.LG]. url: https :/ / arxiv. org/abs/1509.02971

    2019 arXiv

  14. [22]

    Automatic Discovery of Sub- goals in Reinforcement Learning using Diverse Density

    Amy McGovern and Andrew G. Barto. “Automatic Discovery of Sub- goals in Reinforcement Learning using Diverse Density”. In:Proceed- ings of the Eighteenth International Conference on Machine Learning. ICML ’01. San Francisco, CA, USA: Morgan Kaufmann Publishers Inc., 2001, 361–...

    2001

  15. [23]

    A Place Cell Model for Spatio-Temporal Navigation Learning with LSTM

    Thiago Medeiros and Alfredo Weitzenfeld. “A Place Cell Model for Spatio-Temporal Navigation Learning with LSTM”. In:2024 Interna- tional Joint Conference on Neural Networks (IJCNN). 2024, pp. 1–8. doi: 10.1109/IJCNN60899.2024.10650241

    2024

  16. [24]

    Q-Cut—Dynamic Discovery of Sub-goals in Reinforcement Learning

    Ishai Menache, Shie Mannor, and Nahum Shimkin. “Q-Cut—Dynamic Discovery of Sub-goals in Reinforcement Learning”. In:Machine Learn- ing: ECML 2002. Ed. by Tapio Elomaa, Heikki Mannila, and Hannu Toivonen.Berlin,Heidelberg:SpringerBerlinHeidelberg,2002,pp.295–

    2002

  17. [25]

    Volodymyr Mnih et al.Asynchronous Methods for Deep Reinforcement Learning. 2016. arXiv: 1602.01783 [cs.LG] . url: https://arxiv. org/abs/1602.01783

    2016 arXiv

  18. [26]

    Human-level control through deep reinforce- ment learning

    Volodymyr Mnih et al. “Human-level control through deep reinforce- ment learning”. In:Nature 518.7540 (2015), pp. 529–533. issn: 1476-

    2015

  19. [27]

    Ofir Nachum et al.Data-Efficient Hierarchical Reinforcement Learning

  20. [28]

    Why Does Hierarchy (Sometimes) Work So Well in Reinforcement Learning?

    Ofir Nachum et al. “Why Does Hierarchy (Sometimes) Work So Well in Reinforcement Learning?” In:CoRR abs/1909.10618 (2019). arXiv: 1909.10618. url: http://arxiv.org/abs/1909.10618

    2019 arXiv

  21. [29]

    Rectified linear units improve restricted boltzmann machines

    Vinod Nair and Geoffrey E. Hinton. “Rectified linear units improve restricted boltzmann machines”. In: Proceedings of the 27th Interna- tional Conference on International Conference on Machine Learning. ICML’10.Haifa,Israel:Omnipress,2010,807–814. isbn:9781605589077

  22. [30]

    Brendan O’Donoghue et al.Combining policy gradient and Q-learning

  23. [31]

    url: https://arxiv.org/abs/ 1805.08296

    arXiv: 1805.08296 [cs.LG] . url: https://arxiv.org/abs/ 1805.08296

    arXiv

  24. [32]

    Hindsight policy gradients

    Paulo Rauber et al. Hindsight policy gradients. 2019. arXiv: 1711 . 06006 [cs.LG]. url: https://arxiv.org/abs/1711.06006

    2019 arXiv

  25. [33]

    Comparative Analysis of A3C and PPO Algorithms in Reinforcement Learning: A SurveyonGeneralEnvironments

    Alberto del Rio, David Jimenez, and Javier Serrano. “Comparative Analysis of A3C and PPO Algorithms in Reinforcement Learning: A SurveyonGeneralEnvironments”.In: IEEE Access12(2024),pp.146795– 146806. doi: 10.1109/ACCESS.2024.3472473

    2024

  26. [34]

    Proximal Policy Optimization Algorithms

    John Schulman et al. Proximal Policy Optimization Algorithms. 2017. arXiv: 1707.06347 [cs.LG] . url: https://arxiv.org/abs/1707. 06347

    2017 arXiv

  27. [35]

    Trust Region Policy Optimization

    John Schulman et al. Trust Region Policy Optimization. 2017. arXiv: 1502.05477 [cs.LG]. url: https://arxiv.org/abs/1502.05477. 39

    2017 arXiv

  28. [36]

    Hierarchical Reinforcement Learning: A Com- prehensive Survey

    Shubham Pateria et al. “Hierarchical Reinforcement Learning: A Com- prehensive Survey”. In:ACM Comput. Surv. 54.5 (June 2021). issn: 0360-0300. doi: 10.1145/3453160. url: https://doi.org/10.1145/ 3453160

    2021 doi

  29. [37]

    Optimizing Autonomous Driving with Advanced Reinforcement Learning: Evaluating DQN and PPO

    Rishabh Sharma and Prateek Garg. “Optimizing Autonomous Driving with Advanced Reinforcement Learning: Evaluating DQN and PPO”. In: 2024 5th International Conference on Smart Electronics and Com- munication (ICOSEC).2024,pp.910–914. doi: 10.1109/ICOSEC61587. 2024.10722344

    2024

  30. [38]

    Optimizing Autonomous Vehi- cle Navigation with DQN and PPO: A Reinforcement Learning Ap- proach

    Rishabh Sharma and Prateek Garg. “Optimizing Autonomous Vehi- cle Navigation with DQN and PPO: A Reinforcement Learning Ap- proach”. In:2024 Asia Pacific Conference on Innovation in Technology (APCIT). 2024, pp. 1–5.doi: 10.1109/APCIT62007.2024.10673440

    2024

  31. [39]

    Sutton and Andrew G

    Richard S. Sutton and Andrew G. Barto.Reinforcement Learning: An Introduction. Cambridge, MA, USA: A Bradford Book, 2018. isbn: 0262039249

    2018

  32. [40]

    Between MDPs and semi-MDPs: A framework for temporal abstraction in reinforce- ment learning

    Richard S. Sutton, Doina Precup, and Satinder Singh. “Between MDPs and semi-MDPs: A framework for temporal abstraction in reinforce- ment learning”. In: Artificial Intelligence 112.1 (1999), pp. 181–211. issn: 0004-3702. doi: https://doi.org/10.1016/S0004-3702(99) 00052-1. url...

    1999 doi

  33. [41]

    Integration of velocity-dependent spatio- temporal structure of place cell activation during navigation in a reser- voir model of prefrontal cortex

    Pablo Scleidorovich et al. “Integration of velocity-dependent spatio- temporal structure of place cell activation during navigation in a reser- voir model of prefrontal cortex”. In:Biological Cybernetics116.5 (Dec. 2022), pp. 585–610

    2022

  34. [42]

    FeUdal Networks for Hierarchi- cal Reinforcement Learning

    Alexander Sasha Vezhnevets et al. FeUdal Networks for Hierarchi- cal Reinforcement Learning. 2017. arXiv: 1703.01161 [cs.AI] . url: https://arxiv.org/abs/1703.01161. 40

    2017 arXiv

  35. [46]

    Policy Gradient Methods for Reinforcement Learning with Function Approximation

    Richard S Sutton et al. “Policy Gradient Methods for Reinforcement Learning with Function Approximation”. In:Advances in Neural Infor- mation Processing Systems. Ed. by S. Solla, T. Leen, and K. Müller. Vol. 12. MIT Press, 1999.url: https://proceedings.neurips.cc/ paper_files/...

    1999

  36. [306]

    isbn: 978-3-540-36755-0. 38

  37. [2017]

    url: https://arxiv.org/abs/ 1611.01626

    arXiv: 1611.01626 [cs.LG] . url: https://arxiv.org/abs/ 1611.01626

    arXiv

  38. [2018]

    url: https://arxiv.org/abs/ 1806.06920

    arXiv: 1806.06920 [cs.LG] . url: https://arxiv.org/abs/ 1806.06920

    arXiv

  39. [4687]

    url: https://doi.org/10.1038/ nature14236

    doi: 10.1038/nature14236. url: https://doi.org/10.1038/ nature14236

    doi

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.