{"id":"21a09449-6684-4456-9e6a-0f31b56452d3","arxiv_id":"2607.23726","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":3.5,"correctness_risk":"high","formal_verification":"none","parameter_count":5,"one_line_summary":"A two-level entropy-regularized hierarchical SAC agent beats flat SAC on a SAR-2-inspired sparse-reward continuous search task in reported success rate and coverage.","lead":"The paper combines hierarchical RL with Soft Actor-Critic so a high-level planner picks sub-goals and a low-level SAC controller acts in continuous space on a search-and-rescue task. It reports better success and coverage than flat SAC on the SAR-2 setup, aiming at sparse-reward drone search.","discovery_kind":"incremental","skeptic_critique":{"model":"moonshotai/kimi-k3","headline":"The reported \"extrinsic reward\" figures (mean 421,330 vs 138,547) are arithmetically impossible under the paper's own reward function, and Table 9 admits the reward metric mixes in intrinsic rewards that only HRL-SAC receives — the headline reward comparison may be apples-to-oranges by construction.","rationale":"The reader's weakest_assumption already identified the reward-magnitude inconsistency (mean ~421k despite a +100 success bonus) and the fairness of the benchmark; my analysis confirms this is indeed the load-bearing point and sharpens it in one respect: Table 9 explicitly defines the cumulative-reward metric as including intrinsic reward, which only the hierarchical agent possesses — so the reward comparison may be confounded by construction, not merely miscalibrated. This strengthens the documentary case but does not change the appropriate verdict: the reader's CONDITIONAL already hinges on \"cleaned metrics, matched step budgets,\" which is exactly the remedy. I keep CONDITIONAL (UNCHANGED relative to the reader) because the success-rate gap, while noisy, is not yet shown to be invalid — the concrete test would settle whether any genuine task-performance advantage remains once the reward metric is purified. Additional secondary issues (test-set coverage reversal in Table 11 where flat SAC beats HRL-SAC on coverage, and the 0.40M vs 4.80M step mismatch undercutting the sample-efficiency claim) reinforce the conditionality but are subordinate to the reward-metric problem. No code or environment release means none of this is independently checkable today, which the reader also noted.","tokens_in":19243,"tokens_out":1682,"duration_ms":77584,"concrete_test":"Recompute per-episode cumulative extrinsic reward from logged trajectories using only Table 5 components and verify the bound reward ≤ 100 − 0.1·(steps) per episode. If the Figure 2/Table 10 values (e.g., 421,330) violate this bound, the metric is mislabeled or includes intrinsic terms; then re-run the HRL-SAC vs flat-SAC comparison reporting pure extrinsic return (excluding Eq. 8's r_int), with ≥5 seeds and equal total environment steps, and check whether the success-rate and reward gaps survive.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The strongest claim rests on quantitative superiority over flat SAC, and the reward numbers are its centerpiece (3.04× higher reward, §4.1, Table 10, Figure 2). But the reward function in Table 5 is bounded: +100 on success, −0.1 per step, −50 collision, −10 boundary, −0.01·‖a‖². With mean episode length 4,253 steps (Fig. 2a), the maximum achievable cumulative extrinsic reward per episode is roughly 100 − 0.1·4253 ≈ −325, and a successful episode scores well below zero net. Yet Figure 2a reports a mean extrinsic reward of 421,330 with a y-axis up to 2×10⁶. No rescaling, accumulation convention, or per-step bonus is documented that could produce this. Two non-exclusive explanations, both damaging: (a) the plotted \"reward\" is not the Table 5 reward at all (e.g., it includes intrinsic shaping terms or is a running sum), meaning the headline reward improvement is unverifiable as stated; (b) Table 9's own definition — \"Cumulative Reward: Total environmental reward (Renv) and intrinsic reward (ri) per episode\" — indicates the comparison metric includes the intrinsic sub-goal reward (Eq. 8), which only the hierarchical agent receives. If so, the reward comparison is structurally rigged: flat SAC has no intrinsic channel, so \"HRL-SAC achieves 3.04× reward\" would follow from the metric definition rather than from task performance. The success-rate gap (50.9% vs 28.7%) is the cleaner metric, but with std ≈ mean (Table 10: 50.9%±50.0%) and no seed count reported, even that is statistically fragile. Since the entire central claim is empirical, an invalid or confounded primary metric is the load-bearing weakness: if the reward numbers cannot be reproduced from the stated reward function, the \"outperforms flat SAC\" claim currently rests only on noisy success rates from an unreleased custom environment.","agreement_with_reader":"agree"},"referee_report":{"model":"moonshotai/kimi-k3","summary":"The manuscript proposes HRL-SAC, a two-level hierarchical reinforcement learning framework in which a meta-controller selects one of four discrete high-level goals every K=30 steps and a goal-conditioned low-level SAC policy executes continuous actions, with entropy regularization at both levels and an L2-distance intrinsic reward for sub-goal execution (Eq. 8). The method is evaluated on a custom continuous-control search-and-rescue environment derived from the SAR-2 image dataset (276-D observation, 4-D action). The authors report that HRL-SAC outperforms a flat SAC baseline on cumulative reward (mean 421,330 vs. 138,547), training success rate (50.9% vs. 28.7%), coverage (0.200 vs. 0.096), and sample efficiency (0.40M vs. 4.80M total steps), with test-set success of 40% vs. 30% on 199 held-out images.","tokens_in":19724,"tokens_out":3596,"duration_ms":138260,"significance":"If the results hold, the paper offers a clean demonstration that dual-level entropy-regularized hierarchical SAC improves over flat SAC in a sparse-reward, long-horizon continuous-control task, with a fully specified implementation (network architectures in Tables 7-8, hyperparameters in Table 4, pseudocode in Algorithm 1) that is in principle reproducible. The SAR-2-derived environment could be a useful benchmark contribution. However, the significance is currently capped by three factors: (i) the headline reward numbers are arithmetically inconsistent with the paper's own reward function, so the central quantitative claim is unverifiable as written; (ii) the only baseline is flat SAC, so no claim about superiority over existing HRL methods (HAC, HIRO, SHIRO, all cited) is supported; and (iii) the test-set gap is modest (40% vs. 30% success, and the baseline actually wins on test coverage, Table 11), which tempers the conclusion that the method is \"production-ready\" (§5.1).","major_comments":[{"comment":"The reported reward figures are arithmetically impossible under the reward function stated in Table 5. The extrinsic components are +100 (target reached), -50 (collision), -0.1/step, -10 (boundary), and -0.01·||a||². With a mean episode length of 4,253 steps (Fig. 2a) and a 10,000-step cap (Table 3), the maximum achievable per-episode extrinsic return is roughly 100 - 0.1·4253 ≈ -325, and any episode with a collision or boundary penalty is more negative still. Yet Figure 2a reports a mean 'Extrinsic Reward per Episode' of 421,330 with an axis extending to 2×10⁶, and Table 11 reports a Standard SAC test reward of -365,453.61, which is likewise unreachable (a maximally bad 8,054-step episode costs at most about -865 under Table 5). Table 9 defines 'Cumulative Reward' as environmental reward plus intrinsic reward r_i, which only the hierarchical agent receives (Eq. 8); if that is what is pl","section":"§4.1, Figure 2a, Table 10 vs. Table 5 and Table 9"},{"comment":"The total environment-step counts are mutually inconsistent in three places. Table 6 states 1,000 episodes with 'Total Environment Steps ∼30,000' (1,000×30); Table 10 reports 0.40M total steps for HRL-SAC and 4.80M for Std SAC; and Figure 2 reports mean episode lengths of 4,253 and 4,803 steps, which over 1,000 episodes implies ≈4.25M and ≈4.80M steps respectively. So the HRL-SAC 0.40M figure contradicts its own episode-length data by an order of magnitude, and Table 6's 30,000-step schedule contradicts both. Table 6 also conflates episodes with steps ('Meta-Decisions per Episode ∼33.3 = 1,000÷30' divides the episode count, not an episode length). The sample-efficiency claim ('requires only 0.40M environment steps', §4.2.4) is load-bearing for the abstract's 'convergence' claim and must be corrected and made consistent across all three locations, ideally with a single, explicit accountin","section":"Table 10 vs. Table 6 vs. Figure 2"},{"comment":"The statistical reporting does not support the superiority claims. §3.5 states 'each experiment is repeated in multiple random seeds' but the number of seeds is never given anywhere in the paper, and all tables and figures appear to report a single training run with per-episode variability rather than cross-seed aggregates. Table 10 reports success rate as 50.9%±50.0% for HRL-SAC and 28.7%±45.2% for SAC — standard deviations equal to or exceeding the mean, consistent with per-episode binary outcomes, not seed-level means. On those numbers the success-rate difference is not established as statistically significant. The test results in Table 11 (40% vs. 30% on 199 episodes, 10 percentage points) would require roughly a McNemar or binomial analysis to interpret. The paper needs: number of seeds, seed-aggregated means and standard deviations (or confidence intervals) for each metric, and a s","section":"Table 10, §3.5, §4"},{"comment":"The sub-goal space is defined inconsistently. §2.1 and Eq. (8) treat g as a point in a continuous space (the intrinsic reward is the L2 distance between a state embedding f(s_{t+1}) and g_t, and §3.1 says 'the sub-goal space G is defined in the same representation space as a subset of the state variables'), but Table 7 specifies a meta-actor with a categorical output over exactly four discrete goals {ExploreArea, SearchSpecificLocation, TrackObject, ReturnToBase} with a one-hot encoding in the observation. An L2 distance between a state coordinate and a one-hot goal label like 'ExploreArea' is not defined as written. Please specify precisely what g_t is (discrete mode label, continuous target location, or both), what f(·) extracts in each case, and how Eq. (8) is computed when the meta-policy outputs 'ExploreArea'. This matters because the intrinsic reward is the mechanism claimed to 'de","section":"§2.1, Eq. (8), Table 7"},{"comment":"The evaluation supports only a claim relative to flat SAC, but the manuscript repeatedly asserts superiority over, or advantages beyond, existing hierarchical methods ('superior sample efficiency and convergence' relative to SHIRO and others, §1; 'consistently outperforms flat RL methods', §2.1.3). No comparison to any hierarchical baseline (HAC, HIRO, SHIRO, HADS — all cited) or to SAC with dense shaping or HER is provided. Either the claims must be narrowed to the single baseline actually run, or at least one hierarchical baseline must be added. Relatedly, the test-set results partially contradict the narrative: Standard SAC beats HRL-SAC on test coverage (0.123 vs. 0.078, Table 11), and the abstract's 'coverage efficiency' claim is based on training curves only. The 'production-ready for resource-constrained SAR drones' statement in §5.1 is unsupported by any computational-cost or har","section":"§4.1, Table 11, §1 (Contribution), §5.1"}],"minor_comments":[{"comment":"Learning rate inconsistency: Table 4 gives 3×10⁻⁴, but the text of §3.4 states 'the 3e−3 learning rate'. Please fix.","section":"§3.4, Table 4"},{"comment":"The claim that the method works 'without auxiliary rewards or task-specific heuristics' is contradicted by the intrinsic reward of Eq. (8), the five-component shaped reward of Table 5 (step penalty, energy term, boundary penalty), and the dataset-derived detection logic (detection radius 0.35, §3.3). These are legitimate design choices, but the framing should acknowledge them.","section":"§1 (Contribution), §2.2, Table 5"},{"comment":"Two citation problems: (i) HAC is attributed to 'Nachum et al. (2018)' in the Contribution paragraph but to 'Levy et al. (2017)' elsewhere — Levy et al. is HAC, Nachum et al. is HIRO; please make consistent. (ii) Xu and Yang (2023), 'Hierarchically regularized entropy balancing' (Political Analysis), is a causal-inference weighting method, not 'hierarchical reinforcement learning with entropy-regularized objectives' as the text states; this reference appears to be a mistaken lookup and should be replaced.","section":"§1.1"},{"comment":"The warm-up arithmetic is inconsistent with the rest of the paper: 'Warm-up Episodes ∼200 = 2,000÷10 avg. steps per episode' assumes 10-step episodes, while reported mean episode lengths are ~4,000+ steps (Fig. 2). Also 'Learning Rate (λ)' should use the conventional symbol (α or η); λ usually denotes TD(λ).","section":"Table 6"},{"comment":"Figure 2's y-axis label 'Extrinsic Reward per Episode' with a 1e6 multiplier needs unit explanation (see Major Comment 1). Figure 3a/3b are described as 'evaluation episodes' in §4.2.1 but appear to be training curves; Figure 6 ('Evaluation Success over Training') is never quantified and its axis tops out at 80% while the text claims peaks near 100% (§4.2.1). Please align captions, text, and axes.","section":"Figures 2, 3, 6"},{"comment":"Typographical and grammatical issues that should be cleaned up: 'its outperforming' (Abstract); 'Soft SAC' (§2); 'tow diminution (2D)' (§3.3); 'across categories if person or other objects' (§3.2.2); the malformed tuple 'M = (S, A, p, r, γ]' with a closing bracket (§2); 'they hard to handle' (§1); the stray footnote 'This is a claim 1.' (§2.1.3). Equation (14) states a variance inequality as 'requirements' without derivation or measurement; either cite supporting evidence precisely or soften the statement.","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":"The arithmetic impossibility of the headline reward figures (Major Comment 1) and the three-way inconsistency in step counts (Major Comment 2) suggest the results pipeline was not carefully audited before submission; I would recommend the editor require corrected tables/figures plus code or detailed logging evidence at revision. The mis-citation of Xu & Yang (2023) — a political-science entropy-balancing paper cited as hierarchical entropy-regularized RL — looks like a reference-manager or automated-search error rather than a deliberate misrepresentation, but it is worth flagging. The paper is within scope for an RL/robotics venue only if the empirical claims are repaired; if the reward metric turns out to include intrinsic rewards for only one agent, the central comparison would need to be redone wholesale rather than merely relabeled, in which case rejection and resubmission would be the more appropriate path."},"author_rebuttal":null,"desk_editor":{"model":"grok-4.5","letter":"The one thing to know: this is a standard two-level goal-conditioned SAC stack (discrete meta goals, continuous low-level SAC, L2 intrinsic) run on a custom SAR-2-derived drone search MDP, claiming big wins over flat SAC. The algorithmic idea is not new—HAC, HIRO-style, SHIRO, and goal-conditioned SAC already cover this ground—but the specific SAR-2 continuous environment and the dual-entropy implementation are a real empirical artifact.\n\nWhat they do well is the engineering write-up. Architecture tables, observation/action specs, K=30, network sizes, and the training loop are concrete enough that you can see what they built. Literature coverage is broad if a bit scattered. Success rate (roughly 51% vs 29% train, 40% vs 30% test) and shorter trajectories are the cleaner signals, and hierarchical decomposition for long-horizon search is a sensible direction for SAR-style robotics.\n\nThe soft spot is load-bearing, not cosmetic. Table 5’s extrinsic reward (+100 success, −0.1/step, collisions, etc.) cannot produce mean “extrinsic” returns of ~421k over ~4k-step episodes; max successful return is deeply negative. Table 9 explicitly defines cumulative reward as environmental plus intrinsic, and only the hierarchical agent gets the intrinsic channel—so the 3× reward claim is either a plotting/scale error or apples-to-oranges by construction. Step budgets also disagree across tables (~30k vs 0.40M vs 4.80M). Meta actions are four discrete goals while the prose sells continuous hierarchical control. Only baseline is flat SAC; no HAC/HIRO/SHIRO. Success std ≈ mean, no seed count, no code. Theory sections restate SAC/HRL intuition without new bounds.\n\nWho it’s for: people already building hierarchical continuous controllers for search who want a worked SAR-flavored example—and who will re-implement and re-measure before trusting the curves. It is not a field-moving result.\n\nI would still put it in front of a referee rather than desk-reject: the setup is complete enough that major revision (fix metrics, matched budgets, stronger HRL baselines, release env) could salvage an honest workshop/journal contribution. I would not cite it or bring it to reading group until the numbers are cleaned.","headline":"Incremental hierarchical SAC on a custom SAR sim; the headline reward gains are arithmetically impossible under the paper’s own reward table and may mix in intrinsic terms only HRL receives.","tokens_in":20728,"tokens_out":595,"would_cite":false,"duration_ms":32342,"reading_group":"no","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.5","headline":"A two-level hierarchy with Soft Actor-Critic at both levels beats flat SAC on sparse-reward, long-horizon search.","keywords":["hierarchical reinforcement learning","Soft Actor-Critic","sparse-reward environments","long-horizon decision-making","entropy-regularized control","search and rescue","continuous control"],"falsifier":"Train both HRL-SAC and a carefully matched flat SAC (same network size, same total environment steps, same reward) on the identical SAR-2 environment and check whether the hierarchical agent still shows statistically higher success rate, coverage, and lower episode length on the held-out test split.","tokens_in":20279,"feed_emoji":"🚁","tokens_out":902,"duration_ms":20791,"temperature":0.7,"pith_summary":"Long-horizon tasks with rare rewards are hard for ordinary reinforcement learning because credit for a late success never reaches the early choices that made it possible. This paper argues that the fix is a two-level hierarchy in which a high-level policy picks sub-goals every few dozen steps and a low-level Soft Actor-Critic policy executes continuous actions toward those sub-goals, both levels using entropy regularization. On a continuous search-and-rescue environment built from the SAR-2 imagery set, the hierarchical agent reaches the target more often, covers more of the map, and finishes episodes sooner than a flat SAC baseline. The authors present the result as evidence that dual-level entropy-regularized policies are a practical way to give agents both strategic patience and stable local control when rewards arrive only at the end of long trajectories.","feed_headline":"Hierarchy plus dual entropy lifts sparse-search success over flat SAC","feed_subtitle":"On SAR-2 drone search, the two-level agent finds targets more often and covers more ground with fewer steps.","key_machinery":"HRL-SAC: a meta-policy that samples a sub-goal every K steps and a goal-conditioned low-level SAC policy that maximizes entropy-regularized return under an intrinsic L2 distance reward to that sub-goal, with extrinsic sparse task reward accumulated only for the meta-level update.","core_discovery":"Hierarchical Soft Actor-Critic (HRL-SAC) that applies maximum-entropy Soft Actor-Critic optimization at both the meta-controller and the low-level continuous controller solves sparse-reward, long-horizon search better than flat SAC, raising training success from 28.7 % to 50.9 % and test success from 30 % to 40 % while improving coverage and shortening trajectories on the SAR-2-derived environment.","pith_inferences":["If dual-level entropy is the real driver, ablating entropy only at the meta-level or only at the low level should erase most of the reported gain over flat SAC.","The same architecture should transfer to other continuous sparse settings (warehouse inspection, long-horizon manipulation) that share delayed terminal rewards and a natural spatial sub-goal space.","Variable K learned online, rather than the fixed K=30 used here, is a direct next experiment suggested by the paper’s own future-work list."],"forward_implications":["Sparse-reward continuous search tasks can be attacked by adding a lightweight meta-controller that issues sub-goals every few dozen steps rather than by redesigning the reward.","Entropy regularization kept at both hierarchy levels supplies enough local exploration that the low-level skills remain reusable even when the meta-policy is still learning.","Sample efficiency improves because the effective planning horizon at the meta-level shrinks from T to roughly T/K.","The same dual-entropy hierarchy is positioned by the authors as ready for resource-limited search-and-rescue drones once sim-to-real gaps are closed."],"fun_headline_variants":["HRL-SAC dual entropy beats flat SAC on sparse long-horizon search","Two-level max-entropy SAC raises SAR-2 success 28.7% to 50.9%","Hierarchy plus SAC entropy cuts steps, lifts coverage vs flat baseline","HRL-SAC meta plus low-level entropy solves delayed-reward drone search","Hierarchical entropy policies top flat SAC in SAR-2 success and efficiency"],"cache_read_input_tokens":16512,"weakest_assumption_plain":"That the custom continuous search environment built from SAR-2 images, together with its particular sparse reward numbers and the single flat-SAC baseline, is a fair enough test to conclude that dual-level entropy regularization is what produces the gains.","fun_headline_variants_meta":{"raw":{"variants":["HRL-SAC dual entropy beats flat SAC on sparse long-horizon search","Two-level max-entropy SAC raises SAR-2 success 28.7% to 50.9%","Hierarchy plus SAC entropy cuts steps, lifts coverage vs flat baseline","HRL-SAC meta plus low-level entropy solves delayed-reward drone search","Hierarchical entropy policies top flat SAC in SAR-2 success and efficiency"]},"model":"grok-4.5","effort":"low","cost_usd":0.004081,"raw_usage":{"total_tokens":1228,"prompt_tokens":713,"num_sources_used":0,"completion_tokens":88,"cost_in_usd_ticks":40808000,"prompt_tokens_details":{"text_tokens":713,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":427,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":713,"tokens_out":88,"duration_ms":6596,"temperature":1.0,"reasoning_tokens":427,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-30T14:44:11.051652+00:00","model_set":{"reader":"grok-4.5"},"falsifier":"Train both HRL-SAC and a carefully matched flat SAC (same network size, same total environment steps, same reward) on the identical SAR-2 environment and check whether the hierarchical agent still shows statistically higher success rate, coverage, and lower episode length on the held-out test split.","supporting_citations":[],"review_version":1}