Pith. sign in

REVIEW 5 major objections 6 minor 27 references

A two-level hierarchy with Soft Actor-Critic at both levels beats flat SAC on sparse-reward, long-horizon search.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-30 14:44 UTC pith:QG73PAQQ

load-bearing objection Incremental hierarchical SAC on a custom SAR sim; the headline reward gains are arithmetically impossible under the paper’s own reward table and may mix in intrinsic terms only HRL receives. the 5 major comments →

arxiv 2607.23726 v1 pith:QG73PAQQ submitted 2026-07-26 cs.RO cs.LG

Hierarchical Soft Actor-Critic for Sparse-Reward Long-Horizon Reinforcement Learning

Zahra Abdalla Elashaal , Afef Hfaiedh , Nahla Khraief , Issmail Ellabib , Giansalvo Cirrincione This is my paper
classification cs.RO cs.LG
keywords hierarchical reinforcement learningSoft Actor-Criticsparse-reward environmentslong-horizon decision-makingentropy-regularized controlsearch and rescuecontinuous control
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Long-horizon tasks with rare rewards are hard for ordinary reinforcement learning because credit for a late success never reaches the early choices that made it possible. This paper argues that the fix is a two-level hierarchy in which a high-level policy picks sub-goals every few dozen steps and a low-level Soft Actor-Critic policy executes continuous actions toward those sub-goals, both levels using entropy regularization. On a continuous search-and-rescue environment built from the SAR-2 imagery set, the hierarchical agent reaches the target more often, covers more of the map, and finishes episodes sooner than a flat SAC baseline. The authors present the result as evidence that dual-level entropy-regularized policies are a practical way to give agents both strategic patience and stable local control when rewards arrive only at the end of long trajectories.

Core claim

Hierarchical Soft Actor-Critic (HRL-SAC) that applies maximum-entropy Soft Actor-Critic optimization at both the meta-controller and the low-level continuous controller solves sparse-reward, long-horizon search better than flat SAC, raising training success from 28.7 % to 50.9 % and test success from 30 % to 40 % while improving coverage and shortening trajectories on the SAR-2-derived environment.

What carries the argument

HRL-SAC: a meta-policy that samples a sub-goal every K steps and a goal-conditioned low-level SAC policy that maximizes entropy-regularized return under an intrinsic L2 distance reward to that sub-goal, with extrinsic sparse task reward accumulated only for the meta-level update.

Load-bearing premise

That the custom continuous search environment built from SAR-2 images, together with its particular sparse reward numbers and the single flat-SAC baseline, is a fair enough test to conclude that dual-level entropy regularization is what produces the gains.

What would settle it

Train both HRL-SAC and a carefully matched flat SAC (same network size, same total environment steps, same reward) on the identical SAR-2 environment and check whether the hierarchical agent still shows statistically higher success rate, coverage, and lower episode length on the held-out test split.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Share X Bluesky LinkedIn Reddit HN

If this is right

  • Sparse-reward continuous search tasks can be attacked by adding a lightweight meta-controller that issues sub-goals every few dozen steps rather than by redesigning the reward.
  • Entropy regularization kept at both hierarchy levels supplies enough local exploration that the low-level skills remain reusable even when the meta-policy is still learning.
  • Sample efficiency improves because the effective planning horizon at the meta-level shrinks from T to roughly T/K.
  • The same dual-entropy hierarchy is positioned by the authors as ready for resource-limited search-and-rescue drones once sim-to-real gaps are closed.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If dual-level entropy is the real driver, ablating entropy only at the meta-level or only at the low level should erase most of the reported gain over flat SAC.
  • The same architecture should transfer to other continuous sparse settings (warehouse inspection, long-horizon manipulation) that share delayed terminal rewards and a natural spatial sub-goal space.
  • Variable K learned online, rather than the fixed K=30 used here, is a direct next experiment suggested by the paper’s own future-work list.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. The manuscript proposes HRL-SAC, a two-level hierarchical reinforcement learning framework in which a meta-controller selects one of four discrete high-level goals every K=30 steps and a goal-conditioned low-level SAC policy executes continuous actions, with entropy regularization at both levels and an L2-distance intrinsic reward for sub-goal execution (Eq. 8). The method is evaluated on a custom continuous-control search-and-rescue environment derived from the SAR-2 image dataset (276-D observation, 4-D action). The authors report that HRL-SAC outperforms a flat SAC baseline on cumulative reward (mean 421,330 vs. 138,547), training success rate (50.9% vs. 28.7%), coverage (0.200 vs. 0.096), and sample efficiency (0.40M vs. 4.80M total steps), with test-set success of 40% vs. 30% on 199 held-out images.

Significance. If the results hold, the paper offers a clean demonstration that dual-level entropy-regularized hierarchical SAC improves over flat SAC in a sparse-reward, long-horizon continuous-control task, with a fully specified implementation (network architectures in Tables 7-8, hyperparameters in Table 4, pseudocode in Algorithm 1) that is in principle reproducible. The SAR-2-derived environment could be a useful benchmark contribution. However, the significance is currently capped by three factors: (i) the headline reward numbers are arithmetically inconsistent with the paper's own reward function, so the central quantitative claim is unverifiable as written; (ii) the only baseline is flat SAC, so no claim about superiority over existing HRL methods (HAC, HIRO, SHIRO, all cited) is supported; and (iii) the test-set gap is modest (40% vs. 30% success, and the baseline actually wins on test coverage, Table 11), which tempers the conclusion that the method is "production-ready" (§5.1).

major comments (5)
  1. [§4.1, Figure 2a, Table 10 vs. Table 5 and Table 9] The reported reward figures are arithmetically impossible under the reward function stated in Table 5. The extrinsic components are +100 (target reached), -50 (collision), -0.1/step, -10 (boundary), and -0.01·||a||². With a mean episode length of 4,253 steps (Fig. 2a) and a 10,000-step cap (Table 3), the maximum achievable per-episode extrinsic return is roughly 100 - 0.1·4253 ≈ -325, and any episode with a collision or boundary penalty is more negative still. Yet Figure 2a reports a mean 'Extrinsic Reward per Episode' of 421,330 with an axis extending to 2×10⁶, and Table 11 reports a Standard SAC test reward of -365,453.61, which is likewise unreachable (a maximally bad 8,054-step episode costs at most about -865 under Table 5). Table 9 defines 'Cumulative Reward' as environmental reward plus intrinsic reward r_i, which only the hierarchical agent receives (Eq. 8); if that is what is pl
  2. [Table 10 vs. Table 6 vs. Figure 2] The total environment-step counts are mutually inconsistent in three places. Table 6 states 1,000 episodes with 'Total Environment Steps ∼30,000' (1,000×30); Table 10 reports 0.40M total steps for HRL-SAC and 4.80M for Std SAC; and Figure 2 reports mean episode lengths of 4,253 and 4,803 steps, which over 1,000 episodes implies ≈4.25M and ≈4.80M steps respectively. So the HRL-SAC 0.40M figure contradicts its own episode-length data by an order of magnitude, and Table 6's 30,000-step schedule contradicts both. Table 6 also conflates episodes with steps ('Meta-Decisions per Episode ∼33.3 = 1,000÷30' divides the episode count, not an episode length). The sample-efficiency claim ('requires only 0.40M environment steps', §4.2.4) is load-bearing for the abstract's 'convergence' claim and must be corrected and made consistent across all three locations, ideally with a single, explicit accountin
  3. [Table 10, §3.5, §4] The statistical reporting does not support the superiority claims. §3.5 states 'each experiment is repeated in multiple random seeds' but the number of seeds is never given anywhere in the paper, and all tables and figures appear to report a single training run with per-episode variability rather than cross-seed aggregates. Table 10 reports success rate as 50.9%±50.0% for HRL-SAC and 28.7%±45.2% for SAC — standard deviations equal to or exceeding the mean, consistent with per-episode binary outcomes, not seed-level means. On those numbers the success-rate difference is not established as statistically significant. The test results in Table 11 (40% vs. 30% on 199 episodes, 10 percentage points) would require roughly a McNemar or binomial analysis to interpret. The paper needs: number of seeds, seed-aggregated means and standard deviations (or confidence intervals) for each metric, and a s
  4. [§2.1, Eq. (8), Table 7] The sub-goal space is defined inconsistently. §2.1 and Eq. (8) treat g as a point in a continuous space (the intrinsic reward is the L2 distance between a state embedding f(s_{t+1}) and g_t, and §3.1 says 'the sub-goal space G is defined in the same representation space as a subset of the state variables'), but Table 7 specifies a meta-actor with a categorical output over exactly four discrete goals {ExploreArea, SearchSpecificLocation, TrackObject, ReturnToBase} with a one-hot encoding in the observation. An L2 distance between a state coordinate and a one-hot goal label like 'ExploreArea' is not defined as written. Please specify precisely what g_t is (discrete mode label, continuous target location, or both), what f(·) extracts in each case, and how Eq. (8) is computed when the meta-policy outputs 'ExploreArea'. This matters because the intrinsic reward is the mechanism claimed to 'de
  5. [§4.1, Table 11, §1 (Contribution), §5.1] The evaluation supports only a claim relative to flat SAC, but the manuscript repeatedly asserts superiority over, or advantages beyond, existing hierarchical methods ('superior sample efficiency and convergence' relative to SHIRO and others, §1; 'consistently outperforms flat RL methods', §2.1.3). No comparison to any hierarchical baseline (HAC, HIRO, SHIRO, HADS — all cited) or to SAC with dense shaping or HER is provided. Either the claims must be narrowed to the single baseline actually run, or at least one hierarchical baseline must be added. Relatedly, the test-set results partially contradict the narrative: Standard SAC beats HRL-SAC on test coverage (0.123 vs. 0.078, Table 11), and the abstract's 'coverage efficiency' claim is based on training curves only. The 'production-ready for resource-constrained SAR drones' statement in §5.1 is unsupported by any computational-cost or har
minor comments (6)
  1. [§3.4, Table 4] Learning rate inconsistency: Table 4 gives 3×10⁻⁴, but the text of §3.4 states 'the 3e−3 learning rate'. Please fix.
  2. [§1 (Contribution), §2.2, Table 5] The claim that the method works 'without auxiliary rewards or task-specific heuristics' is contradicted by the intrinsic reward of Eq. (8), the five-component shaped reward of Table 5 (step penalty, energy term, boundary penalty), and the dataset-derived detection logic (detection radius 0.35, §3.3). These are legitimate design choices, but the framing should acknowledge them.
  3. [§1.1] Two citation problems: (i) HAC is attributed to 'Nachum et al. (2018)' in the Contribution paragraph but to 'Levy et al. (2017)' elsewhere — Levy et al. is HAC, Nachum et al. is HIRO; please make consistent. (ii) Xu and Yang (2023), 'Hierarchically regularized entropy balancing' (Political Analysis), is a causal-inference weighting method, not 'hierarchical reinforcement learning with entropy-regularized objectives' as the text states; this reference appears to be a mistaken lookup and should be replaced.
  4. [Table 6] The warm-up arithmetic is inconsistent with the rest of the paper: 'Warm-up Episodes ∼200 = 2,000÷10 avg. steps per episode' assumes 10-step episodes, while reported mean episode lengths are ~4,000+ steps (Fig. 2). Also 'Learning Rate (λ)' should use the conventional symbol (α or η); λ usually denotes TD(λ).
  5. [Figures 2, 3, 6] Figure 2's y-axis label 'Extrinsic Reward per Episode' with a 1e6 multiplier needs unit explanation (see Major Comment 1). Figure 3a/3b are described as 'evaluation episodes' in §4.2.1 but appear to be training curves; Figure 6 ('Evaluation Success over Training') is never quantified and its axis tops out at 80% while the text claims peaks near 100% (§4.2.1). Please align captions, text, and axes.
  6. [Throughout] Typographical and grammatical issues that should be cleaned up: 'its outperforming' (Abstract); 'Soft SAC' (§2); 'tow diminution (2D)' (§3.3); 'across categories if person or other objects' (§3.2.2); the malformed tuple 'M = (S, A, p, r, γ]' with a closing bracket (§2); 'they hard to handle' (§1); the stray footnote 'This is a claim 1.' (§2.1.3). Equation (14) states a variance inequality as 'requirements' without derivation or measurement; either cite supporting evidence precisely or soften the statement.

Circularity Check

1 steps flagged

Empirical methods paper; reward metric may mix in HRL-only intrinsic terms (Table 9 vs ‘extrinsic’ plots), a mild by-construction comparison flaw, but success/coverage claims are independent external benchmarks.

specific steps
  1. self definitional [Table 9; §4.1 / Fig. 2a; Eq. 8]
    "Table 9: ‘Cumulative Reward Total environmental reward (Renv) and intrinsic reward (ri) per episode.’ §4.1: ‘HRL-SAC achieves a final 10-episode moving average of 421,330 extrinsic reward which 3.04× higher than Standard SAC’s’. Eq. 8: rint(st,at,gt)=−∥f(st+1)−gt∥2"

    The evaluation metric named in Table 9 includes the intrinsic sub-goal reward that exists only inside the hierarchical agent. Flat SAC has no ri channel. Labeling the same quantity ‘extrinsic’ in Fig. 2/§4.1 and using the 3.04× gap as a centerpiece superiority claim therefore makes part of the reward win true by construction of the metric (HRL receives an extra dense term the baseline never sees), rather than solely by better task return under Table 5’s extrinsic schedule.

full rationale

This is a standard empirical HRL methods paper. The SAC/HRL objectives (Eqs. 1–11), intrinsic sub-goal reward (Eq. 8), and hierarchical SMDP reduction are textbook constructions, not algebraic self-predictions. No uniqueness theorem is imported from the authors; self-citations (e.g., Hfaiedh et al. 2024, Al Homsi/Cirrincione) are peripheral related-work, not load-bearing premises. The central superiority claim is an external comparison to flat SAC on SAR-2 (success rate, coverage, episode length). The only mild circularity is metric contamination on the headline reward numbers: Table 9 explicitly defines ‘Cumulative Reward’ as environmental plus intrinsic ri, while §4.1/Fig. 2 label the same curves ‘Extrinsic Reward’ and report means (~421k) impossible under Table 5’s bounded extrinsic schedule. If the plotted quantity includes the HRL-only intrinsic channel, then part of the 3.04× reward gap is true by metric definition rather than task performance. Success rate (50.9% vs 28.7%) and coverage remain non-circular external metrics. Proportionate score is therefore 2, not a derivation collapse.

Axiom & Free-Parameter Ledger

5 free parameters · 5 axioms · 3 invented entities

Load-bearing content is mostly standard MDP/SAC/HRL plus engineering choices for a custom SAR simulator. Free parameters are the usual deep-RL knobs and the hand-designed reward and hierarchy interval. No new physical entities; invented pieces are architectural (meta sub-goal interface, intrinsic L2 shaping, SAR-2-to-MDP mapping).

free parameters (5)
  • sub-goal interval K = 30
    Meta action every K environment steps; set to 30 and controls temporal abstraction strength.
  • reward weights (Rg, Pc, Ps, energy, boundary) = +100 / -50 / -0.1 / -0.01 / -10
    Hand-chosen sparse task shaping (+100, −50, −0.1, −0.01, −10) that defines what success and efficiency mean.
  • SAC temperature α = auto
    Entropy trade-off; paper says auto-tuned each step, still a learned free coefficient.
  • network width/depth, lr, τ, batch sizes, dropout = see Table 4
    Architecture and Adam settings (512-unit MLPs, lr 3e-4, τ=0.005, batches 128/64, dropout 0.1) chosen for stability on this task.
  • detection radius and grid resolution = 0.35; 32x32
    0.35 detection radius and 32×32 coverage grid define success and coverage metrics.
axioms (5)
  • domain assumption Environment is an MDP/SMDP where hierarchical decomposition with period K shortens effective horizon and reduces policy-gradient variance.
    Stated in §2.2–2.3 and Eq. 13–14 as intuition supported by surveys, not derived for this MDP.
  • domain assumption Soft Actor-Critic maximum-entropy objective and soft Bellman backup are valid off-policy learners for the continuous low-level controller.
    Imported from Haarnoja et al. 2018; used in Eqs. 1–7 and Algorithm 1.
  • domain assumption Intrinsic reward r_int = −∥f(s_{t+1})−g_t∥² is a valid dense shaping signal that does not invalidate the extrinsic task optimum in practice.
    Eq. 8; cites reward-shaping intuition (Ng et al.) without a potential-based proof for this f and G.
  • ad hoc to paper SAR-2 imagery and YOLO boxes can be turned into a fair continuous drone search MDP without domain-specific heuristics that favor the hierarchical agent.
    §3.1–3.2 construct the benchmark from a CV dataset; fairness vs flat SAC depends on this mapping.
  • standard math Standard math of discounted returns, Gaussian squashed policies, and Polyak target updates.
    Background throughout §2 and implementation §3.4.
invented entities (3)
  • HRL-SAC dual-level entropy-regularized agent (meta SAC + goal-conditioned low-level SAC) no independent evidence
    purpose: Name the proposed architecture that applies entropy-regularized actor-critic at both hierarchy levels for sparse long-horizon search.
    Framed as novel integration versus HAC/SHIRO; functionally a configuration of known HRL+SAC parts.
  • SAR-2-derived continuous maritime search MDP (276-D obs, 4-D action, 48 condition combos) no independent evidence
    purpose: Provide the sparse-reward long-horizon benchmark used for all claims.
    Built ad hoc from a detection dataset; not a community RL benchmark with independent baselines.
  • Four discrete high-level goals (ExploreArea, SearchSpecificLocation, TrackObject, ReturnToBase) no independent evidence
    purpose: Meta-action set for strategic planning.
    Introduced in Tables 2 and 7; tension with continuous sub-goal L2 intrinsic reward elsewhere.

pith-pipeline@v1.2.0-grok45-kimik3 · 23463 in / 4150 out tokens · 97361 ms · 2026-07-30T14:44:11.051652+00:00 · methodology

0 comments
Cite this review

Pith. "Pith review of Hierarchical Soft Actor-Critic for Sparse-Reward Long-Horizon Reinforcement Learning." pith.science (2026). https://pith.science/paper/QG73PAQQ

@misc{pith2026260723726,
  author       = {Pith},
  title        = {Pith review of: Hierarchical Soft Actor-Critic for Sparse-Reward Long-Horizon Reinforcement Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/QG73PAQQ}},
  note         = {Machine review of arXiv:2607.23726}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Exploration in sparse-reward long-horizon tasks poses significant challenges for reinforcement learning. To address these challenges, we propose a two-level Hierarchical Reinforcement Learning (HRL) framework. The first level handles high-level strategic planning, while the low-level uses the continuous-control Soft Actor-Critic (SAC) algorithm, and they utilize entropy-regularized policy optimization. The proposed framework was trained and evaluated using the Search-and-Rescue-2 (SAR-2) dataset. HRL-SAC effectively addresses sparse-reward long-horizon search problems characterized by delayed rewards and continuous control, and its outperforming the flat SAC baseline reinforcement learning in terms of success rates, coverage efficiency, and convergence. These findings indicate that hierarchical entropy-regularized policies are a promising solution to tackle long-horizon sparse-reward reinforcement learning tasks.

Figures

Figures reproduced from arXiv: 2607.23726 by Afef Hfaiedh, Giansalvo Cirrincione, Issmail Ellabib, Nahla Khraief, Zahra Abdalla Elashaal.

Figure 1
Figure 1. Figure 1: HRL-SAC Agent Architecture. The meta-controller generates high-level sub￾goals based on environment observations, while the low-level controller executes entropy-regularized actions conditioned on these sub-goals. signed sub-goal. The environment provides observations and rewards to both hierarchical levels. Low-level actions influence the environment dynamics, while accumulated task feed￾back is used to u… view at source ↗
Figure 2
Figure 2. Figure 2: Comparison of training performance between the proposed HRL-SAC and Stan￾dard SAC over 1000 Ep. 18 [PITH_FULL_IMAGE:figures/full_fig_p018_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Comparison of HRL-SAC and standard SAC over 1000 Ep. HRL-SAC exhibits higher median success rates and more stable performance across episodes. 19 [PITH_FULL_IMAGE:figures/full_fig_p019_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Reward comparison and distribution for HRL-SAC and standard SAC. [PITH_FULL_IMAGE:figures/full_fig_p020_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: reports the coverage ratio achieved by HRL-SAC and standard SAC over 1000 episodes, smoothed using a 10-episode moving average. The results show that HRL-SAC consistently attains higher coverage throughout training, indicating more effective explo￾ration of the state space. In contrast, standard SAC exhibits lower and less stable coverage, with a pronounced decline as training progresses, suggesting premat… view at source ↗
Figure 6
Figure 6. Figure 6: HRL-SAC vs Standard SAC Evaluation Success over Training [PITH_FULL_IMAGE:figures/full_fig_p022_6.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

27 extracted references · 8 canonical work pages

  1. [3]

    doi: 10.3389/frobt.2025.1567211

    ISSN 2296-9144. doi: 10.3389/frobt.2025.1567211. URLhttps://www.frontiersin.org/journals/ robotics-and-ai/articles/10.3389/frobt.2025.1567211. Arthur Allshire, Roberto Mart ´ ın-Mart ´ ın, Charles Lin, Shawn Manuel, Silvio Savarese, and Animesh Garg. Laser: Learning a latent action space for efficient reinforcement learn- ing. In2021 IEEE International Co...

    arXiv 2025

  2. [7]

    ISBN 978-981-97-5035-1

    Springer Nature Singapore. ISBN 978-981-97-5035-1. doi: https://doi.org/10.1007/978-981-97-5035-1

    work page doi:10.1007/978-981-97-5035-1

  3. [10]

    Zhimin Hou, Kuangen Zhang, Yi Wan, Dongyu Li, Chenglong Fu, and Haoyong Yu

    URLhttps://doi.org/10.1177/ 17298806241279777. Zhimin Hou, Kuangen Zhang, Yi Wan, Dongyu Li, Chenglong Fu, and Haoyong Yu. Off-policy maximum entropy reinforcement learning: Soft actor-critic with advantage weighted mixture policy (sac-awmp).arXiv preprint arXiv:2002.02829,

    Pith/arXiv arXiv 2002

  4. [15]

    Chenghao Liu, Fei Zhu, Quan Liu, and Yuchen Fu

    URLhttp://arxiv.org/abs/1712.00948. Chenghao Liu, Fei Zhu, Quan Liu, and Yuchen Fu. Hierarchical reinforcement learning with automatic sub-goal identification.IEEE/CAA journal of automatica sinica, 8(10): 1686–1696,

    Pith/arXiv arXiv

  5. [16]

    Chao Lv, Ming Zhu, Xiao Guo, Jiajun Ou, and Wenjie Lou

    doi: 10.1109/JAS.2021.1004141. Chao Lv, Ming Zhu, Xiao Guo, Jiajun Ou, and Wenjie Lou. Hierarchical reinforcement learning method for long-horizon path planning of stratospheric airship.Aerospace Sci- ence and Technology, 160:110075,

    arXiv 2021

  6. [17]

    doi: https://doi.org/10.1016/j

    ISSN 1270-9638. doi: https://doi.org/10.1016/j. ast.2025.110075. Ofir Nachum, Shixiang Shane Gu, Honglak Lee, and Sergey Levine. Data-efficient hierarchi- cal reinforcement learning. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa- Bianchi, and R. Garnett, editors,Advances in Neural Information Processing Systems, volume

    arXiv 2025

  7. [18]

    Andrew Y Ng, Daishi Harada, and Stuart Russell

    URLhttps://proceedings.neurips.cc/ paper_files/paper/2018/file/e6384711491713d29bc63fc5eeb5ba4f-Paper.pdf. Andrew Y Ng, Daishi Harada, and Stuart Russell. Policy invariance under reward transfor- mations: Theory and application to reward shaping. InIcml, volume 99, pages 278–287. Citeseer,

    2018

  8. [19]

    RescuedBy

    URLhttps://doi.org/10.1145/345316. RescuedBy. SARD Dataset v1, May

    work page doi:10.1145/345316

  9. [20]

    Version 1, 1980 images, CC BY 4.0, Accessed: February 12,

    URLhttps://universe.roboflow.com/ rescuedby/sard-peykp-lxuf9. Version 1, 1980 images, CC BY 4.0, Accessed: February 12,

    1980

  10. [21]

    26 Hierarchical Soft Actor-Critic for Sparse-Reward Long-Horizon Reinforcement Learning Qianli Shen, Yan Li, Haoming Jiang, Zhaoran Wang, and Tuo Zhao

    URLhttps://arxiv.org/abs/1707.06347. 26 Hierarchical Soft Actor-Critic for Sparse-Reward Long-Horizon Reinforcement Learning Qianli Shen, Yan Li, Haoming Jiang, Zhaoran Wang, and Tuo Zhao. Deep reinforcement learning with robust and smooth policy. In Hal Daum´ e III and Aarti Singh, editors, Proceedings of the 37th International Conference on Machine Lear...

    Pith/arXiv arXiv

  11. [22]

    Richard S Sutton, Doina Precup, and Satinder Singh

    doi: 10.4018/978-1-60960-165-2.ch004. Richard S Sutton, Doina Precup, and Satinder Singh. Between mdps and semi-mdps: A framework for temporal abstraction in reinforcement learning.Artificial intelligence, 112 (1-2):181–211,

    work page doi:10.4018/978-1-60960-165-2.ch004

  12. [24]

    Yiqing Xu and Eddie Yang

    URLhttps://arxiv.org/abs/2212.12786. Yiqing Xu and Eddie Yang. Hierarchically regularized entropy balancing.Political Analysis, 31(3):457–464,

    Pith/arXiv arXiv

  13. [26]

    Jiarui Yang, Bin Zhu, Jingjing Chen, and Yu-Gang Jiang

    URLhttps://doi.org/10.1007/s10462-024-10794-3. Jiarui Yang, Bin Zhu, Jingjing Chen, and Yu-Gang Jiang. Actor-critic for continuous action chunks: A reinforcement learning framework for long-horizon robotic manipulation with sparse reward.arXiv preprint arXiv:2508.11143,

    arXiv

  14. [27]

    48550/arXiv.2508.11143

    URLhttps://doi.org/10. 48550/arXiv.2508.11143. Xianghua Zeng, Hao Peng, Dingli Su, and Angsheng Li. Hierarchical decision making based on structural information principles.Journal of Machine Learning Research, 26(182): 1–55,

    doi:10.48550/arxiv.2508.11143

  15. [28]

    doi: 10.1109/ICMA65362.2025.11120778. 27

    arXiv 2025

  16. [1999]

    URLhttps://doi.org/10.1016/S0004-3702(99) 00052-1

    ISSN 0004-3702. URLhttps://doi.org/10.1016/S0004-3702(99) 00052-1. Alexander Sasha Vezhnevets, Simon Osindero, Tom Schaul, Nicolas Heess, Max Jader- berg, David Silver, and Koray Kavukcuoglu. FeUdal networks for hierarchical rein- forcement learning. In Doina Precup and Yee Whye Teh, editors,Proceedings of the 34th International Conference on Machine Lear...

    work page doi:10.1016/s0004-3702(99

  17. [2000]

    Botao Dong, Longyang Huang, Ning Pang, Hongtian Chen, and Weidong Zhang

    doi: https: //doi.org/10.1613/jair.639. Botao Dong, Longyang Huang, Ning Pang, Hongtian Chen, and Weidong Zhang. His- torical decision-making regularized maximum entropy reinforcement learning.IEEE Transactions on Neural Networks and Learning Systems, 36(7):13446–13459,

    doi:10.1613/jair.639

  18. [2015]

    Mohammad Al Homsi, Maja Trumi´ c, Adriano Fagiolini, and Giansalvo Cirrincione

    URLhttp://dx.doi.org/10.12785/ijcds/ 040207. Mohammad Al Homsi, Maja Trumi´ c, Adriano Fagiolini, and Giansalvo Cirrincione. Com- parative analysis of deep q-learning algorithms for object throwing using a robot ma- nipulator.Frontiers in Robotics and AI, Volume 12 - 2025,

    work page doi:10.12785/ijcds/ 2025

  19. [2016]

    Michael Laskin, Aravind Srinivas, and Pieter Abbeel

    URLhttps://proceedings.neurips.cc/paper_files/paper/2016/file/ f442d33fa06832082290ad8544a8da27-Paper.pdf. Michael Laskin, Aravind Srinivas, and Pieter Abbeel. CURL: Contrastive unsupervised representations for reinforcement learning. In Hal Daum´ e III and Aarti Singh, editors, Proceedings of the 37th International Conference on Machine Learning, volume ...

    2016

  20. [2017]

    v31i1.10916

    doi: 10.1609/aaai. v31i1.10916. URLhttps://ojs.aaai.org/index.php/AAAI/article/view/10916. Yuri Burda, Harri Edwards, Deepak Pathak, Amos Storkey, Trevor Darrell, and Alexei A Efros. Large-scale study of curiosity-driven learning.arXiv preprint arXiv:1808.04355,

    Pith/arXiv arXiv

  21. [2018]

    Timothy Chang, Kourosh Neshatian, and James Atlas

    URLhttps://doi.org/10.48550/arXiv.1808.04355. Timothy Chang, Kourosh Neshatian, and James Atlas. Sample efficient hierarchical rein- forcement learning for the game of othello. In Xin-She Yang, R. Simon Sherratt, Nilanjan Dey, and Amit Joshi, editors,Proceedings of Ninth International Congress on Informa- tion and Communication Technology, pages 419–430, ...

    Pith/arXiv arXiv doi:10.48550/arxiv.1808.04355

  22. [2020]

    Matthias Hutsebaut-Buysse, Kevin Mets, and Steven Latr´ e

    URL https://doi.org/10.48550/arXiv.2002.02829. Matthias Hutsebaut-Buysse, Kevin Mets, and Steven Latr´ e. Hierarchical reinforcement learning: A survey and open research challenges.Machine Learning and Knowledge Extraction, 4(1):172–221,

    Pith/arXiv arXiv doi:10.48550/arxiv.2002.02829 2002

  23. [2021]

    24 Hierarchical Soft Actor-Critic for Sparse-Reward Long-Horizon Reinforcement Learning Pierre-Luc Bacon, Jean Harb, and Doina Precup

    doi: 10.1109/ICRA48506.2021.9561232. 24 Hierarchical Soft Actor-Critic for Sparse-Reward Long-Horizon Reinforcement Learning Pierre-Luc Bacon, Jean Harb, and Doina Precup. The option-critic architecture.Proceedings of the AAAI Conference on Artificial Intelligence, 31(1), Feb

    arXiv 2021

  24. [2022]

    doi: 10.3390/make4010009

    ISSN 2504-4990. doi: 10.3390/make4010009. URLhttps: //www.mdpi.com/2504-4990/4/1/9. 25 Elashaal, Hfaiedh, Khraief, Ellabib and Cirrincione Kazumi Kasaura. Generation of geodesics with actor-critic reinforcement learning to predict midpoints.Journal of Machine Learning Research, 26(212):1–36,

    work page doi:10.3390/make4010009

  25. [2023]

    Jiangyue Yan, Biao Luo, and Xiaodong Xu

    doi: 10.1017/pan.2022.12. Jiangyue Yan, Biao Luo, and Xiaodong Xu. Hierarchical reinforcement learning for handling sparse rewards in multi-goal navigation.Artificial Intelligence Review, 57(6):156,

    work page doi:10.1017/pan.2022.12 2022

  26. [2024]

    Nikolas Gegenava

    doi: 10.1109/TNNLS.2024.3481887. Nikolas Gegenava. SAR-2 Search-and-Rescue Dataset (Extra Classes), May

    arXiv 2024

  27. [2025]

    Mostafa Al-Emran

    URLhttps://doi.org/10.1142/S219688882550023X. Mostafa Al-Emran. Hierarchical reinforcement learning: a survey.International journal of computing and digital systems, 4(02),

    work page doi:10.1142/s219688882550023x