{"id":"024c7602-5a46-4230-936f-4598f7b4815c","arxiv_id":"2507.04373","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"HRC learns the causal structure among subgoals and prioritizes interventions on the subgoals that matter most for the final goal, lowering training cost in hierarchical RL.","lead":"This paper proposes a hierarchical reinforcement learning method that learns a causal graph of subgoals and uses it to choose which subgoals to practice first, reducing training cost relative to random exploration. A reader might care because the approach adds formal cost guarantees to a line of work that was previously purely empirical.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Theorem 7.4's tree lower bound is invalid for shallow trees: Eq. (21) gives a non-positive α for depth D≤2, and for D=1 the claimed Ω(n^2 b) is actually false.","rationale":"The reader's weakest assumption was the monotonicity Assumption 4.2, which limits the framework's scope but is stated explicitly. My stress-test instead targets an internal correctness issue in the main formal claim: the tree lower-bound proof in Appendix E.2 is not merely incomplete but actually false in the boundary case D=1, where HRCb's cost is Θ(n^2) rather than Ω(n^2 b). This is load-bearing because Theorem 7.4 and Table 1 are the paper's headline theoretical justification for 'remarkable improvements' of targeted over random intervention selection. The upper bound for HRCh may survive, and the qualitative conclusion may still hold, but the claimed lower bound for the baseline is not established as stated. I agree with the reader's conditional posture: the issue is local and addressable by restricting the theorem or correcting the boundary analysis, so the paper need not be rejected outright, but the formal claims should be revised before acceptance.","tokens_in":43099,"tokens_out":12709,"duration_ms":165140,"concrete_test":"Enumerate the cost MDP of Appendix E.1 exactly for depth-1 trees: for b = 2, 3, 4, take a root with b children and one final child, set non-final leaves to have no reachable children, and compute Cgn({}) from Eq. (11) under HRCb's uniform selection. If the exact cost grows as Θ(b^2), as predicted, then Theorem 7.4's tree row must be restricted to D ≥ 3 or replaced with a corrected lower bound.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"Appendix E.2's lower bound for HRCb on trees does not cover shallow trees. The proof defines α = 1 − 2/2^{D−1} in Eq. (21) and then uses Q_NG^(k) ≥ α to conclude Cgn({}) ≥ α/2·n(n+3)w(T+bT′). For D=1, α = −1; for D=2, α = 0, so the conclusion is vacuous. The problem is not just a loose constant. In a depth-1 tree (root plus b children, one of which is the final subgoal; n = b+1), HRCb selects uniformly without replacement among the children. The expected number of selections before hitting the final child is Θ(b), each transition costs Θ(k) probes, so the expected cost is Θ(b^2) = Θ(n^2), not Ω(n^2 b) = Ω(b^3). The proof's f(h) = h(b−1)+1 overcounts grandchildren for depth-1 leaves, which have zero children. Therefore the tree row of Table 1 and the corresponding claim in Theorem 7.4 overstate the random-strategy cost and need either a restricted statement (e.g., D ≥ 3) or a corrected boundary analysis.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes HRC, a hierarchical reinforcement learning framework that models dependencies among subgoals as a causal graph, performs interventions on controllable subgoals, and uses the estimated causal model to prioritize which subgoal to add to the intervention set. The authors define a training-cost MDP and claim that, under Assumptions 7.1–7.3, a targeted strategy HRCh has cost O(log^2(n)b) on trees and O(n^{4/3+2c/3} log n) on semi-Erdős–Rényi graphs, while a random strategy HRCb has cost Ω(n^2 b) and Ω(n^2), respectively (Theorem 7.4). The paper also introduces a causal discovery method, SSD, with a theoretical guarantee (Theorem 8.4) for recovering parents from an abstracted SCM, and reports experiments on synthetic graphs and a 2D-Minecraft environment showing improved training cost over several HRL baselines.","tokens_in":43454,"tokens_out":9128,"duration_ms":106132,"significance":"If the theoretical results were fully valid, the cost separation between targeted and random exploration would be a valuable first formal analysis of causally-guided exploration in hierarchical RL. The paper has several strengths: it proposes a concrete cost model (Section 5 and Appendix E.1), provides a public codebase, evaluates on a nontrivial Minecraft environment, and compares against CDHRL, HAC, HER, OHRL, and PPO. The experimental evidence for a training-cost advantage of HRCh over HRCb and over prior methods is suggestive. However, the theoretical claims currently contain load-bearing gaps: the tree lower bound is not valid for shallow trees, Theorem 8.4 is not stated with the positivity/discoverability conditions needed by its proof, and the experiments use an l1 loss while the theorem is proved for l0. These issues must be repaired before the formal claims can be accepted.","major_comments":[{"comment":"The tree lower bound for HRCb is invalid for shallow trees. The proof defines α = 1 − 2/2^{D−1}; for D=2 this gives α=0 and for D=1 it gives α=−1, so the step from Q_NG(k) ≥ α to the Ω(n^2 b) conclusion is vacuous. This is not merely a loose constant: in a depth-1 tree (root plus b children, one of which is the final subgoal), HRCb selects uniformly without replacement among the b children, so the expected number of selections before hitting the final child is Θ(b), each transition costs Θ(k) probes, and the expected cost is Θ(b^2) = Θ(n^2), not Ω(n^2 b) = Ω(b^3). The counting function f(h) = h(b−1)+1 also overcounts grandchildren for depth-1 leaves, which have no children. The theorem and the tree row of Table 1 therefore overstate the random-strategy cost; either restrict the statement to D ≥ 3 with a corrected boundary analysis, or repair the lower-bound derivation.","section":"Appendix E.2, Eq. (21), Theorem 7.4, Table 1"},{"comment":"Theorem 8.4 claims that positive coefficients in any optimal β* correspond exactly to the parents of Xi, but the proof requires certain assignments in D_My to have positive probability, and no positivity condition appears in the theorem statement. The manuscript's own Example G.2 (Figure 11) describes a valid A-SCM in which X2=1 always implies X1=1; for the AND subgoal X3 with parents X1 and X2, the configuration needed to detect X1 (X1=0, X2=1) has probability zero, making X1 undiscoverable. Proposition 8.3 correctly limits identifiability to discoverable parents, but Theorem 8.4 as stated contradicts this by promising recovery of all parents. The theorem should be restated for discoverable parents under an explicit positivity assumption, or the proof should be reconciled with G.2.","section":"Section 8, Theorem 8.4, Appendix G.2, Appendix G.3"},{"comment":"The experiments for SSD use the l1 norm in the loss (Remark 8.5), while Theorem 8.4 is proved only for the l0 norm. The l1 relaxation is not equivalent to l0 sparsity for the nonlinear threshold model in (3), so the reported empirical performance of SSD does not inherit the theorem's guarantee. Please either provide a corresponding theoretical result for the estimator actually used (which would require additional assumptions, e.g., irrepresentability-type conditions), or explicitly present the l1 variant as a heuristic motivated by the l0 result.","section":"Remark 8.5, Section 9, Table 3"},{"comment":"The cost separation for HRCh is driven by Assumption 7.2, which states that the estimated causal effect of a subgoal on the final subgoal is nonzero exactly when that subgoal is an ancestor. This assumption directly encodes the success of the causal-effect ranking rule, so the theorem is conditional rather than fully explanatory: it shows that if the ECE estimator is perfect in a specific sense, then targeted exploration is cheaper. The paper should state this limitation explicitly, and ideally provide a finite-sample or error-propagating analysis showing how estimation error in the ECE affects the cost guarantees. As written, the claimed 'provable improvement' rests entirely on an assumption that is close to the desired conclusion.","section":"Section 7, Assumption 7.2, Theorem 7.4"}],"minor_comments":[{"comment":"The condition PA_hat^t_gi ⊂ ISt is stated in line 8, but Remark E.1 says it can be removed under Assumption 7.1; the relation between the algorithm's stated condition and the assumption should be clarified in the main text.","section":"Algorithm 1, line 8 and Remark E.1"},{"comment":"The notation wτj_int,g′, wτj_exp, and wτj_train,g′′ is dense and the definition of system probes is spread over several sentences; a short table or a glossary would improve readability.","section":"Appendix E.1, Eq. (5)"},{"comment":"The SHD comparison reports means but no standard deviations or number of runs; adding these would strengthen the claim that SSD has fewer extra edges.","section":"Table 2"},{"comment":"The hyperparameter labeled 'LASSO L1-RATIO / OTHERS' is set to 0.0001 for HRCh (SSD) and HRCb (SSD); please clarify whether this is the λ in the l1 loss and how it was chosen.","section":"Table 3"},{"comment":"There are minor typos, e.g., 'Controllabe Set' in Algorithm 1 and inconsistent spacing in 'HRC h (SSD)' in Figure 6(a); a copyedit pass would be helpful.","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":"The paper has a useful experimental core and a plausible conditional framework, but the central theoretical claims need substantial repair. In particular, the tree lower bound for random exploration is false as stated for depth-1 and depth-2 trees, and Theorem 8.4 overclaims what the proof and the paper's own identifiability example allow. Both issues are addressable by restricting or correcting the statements, but they are load-bearing for the paper's headline results. I would not recommend reject, because the experimental comparison and the cost-modeling framework remain valuable, but the theoretical claims cannot be accepted in their current form."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's my take. This paper is a genuine contribution to causal HRL. It formalizes the cost of hierarchical training as an MDP over intervention sets, proposes two targeted ranking rules plus a hybrid, and gives a new causal discovery algorithm (SSD) with a formal recovery guarantee. The Minecraft comparison is impressive: HRCh reaches 0.5 success in 33.7 minutes versus 352.3 for CDHRL, and the code is public. The synthetic results track the theory. That alone justifies taking it seriously.\n\nThe soft spots are real but containable. The tree lower bound in Theorem 7.4 is overstated. Appendix E.2 defines alpha = 1 - 2/2^{D-1}, which is negative for D=1 and zero for D=2. For a depth-1 tree (n = b+1), the random strategy's expected cost is Theta(b^2) = Theta(n^2), not Omega(n^2 b) = Omega(b^3) as claimed. The proof's counting of controllable subgoals, f(h) = h(b-1)+1, also overcounts for shallow leaves. The theorem and Table 1 need either an explicit D >= 3 restriction or a corrected boundary analysis. That's a load-bearing fix because the claimed gap between O(log^2(n) b) and Omega(n^2 b) is what makes the tree result dramatic.\n\nThe SSD guarantee has a similar gap: Theorem 8.4 is proved for the l0 penalty, while Remark 8.5 says experiments use l1. The proof also implicitly assumes a positivity/coverage condition for the relevant assignments; without it, the AND case does not go through. Stating the condition and either proving the l1 case or presenting the l1 method as a heuristic with only empirical support would be honest.\n\nThe framework is also narrower than the abstract suggests. Assumption 4.2 - once a subgoal is achieved it stays achieved - is fine for skills or permanent resources, but it fails for consumables or expiring effects. That's a scope limit, not an error, but it should be in the main text.\n\nI'd still accept this for peer review. The empirical advantage is strong, the cost model is new, and representing the exploration process as an MDP over intervention sets is a useful way to think about HRL. The fixes are: restrict Theorem 7.4 to D >= 3 (or fix the boundary), make the l0/l1 and positivity caveats explicit, and pin down Assumption 7.2 as what it is - an assumption that installs the optimality of the ranking rule, not a derivation of it. A serious referee should see the revision.","headline":"Solid HRL+causality paper with real empirical gains, but the tree lower bound in Theorem 7.4 does not cover shallow trees and the SSD guarantee is proved for l0 while the experiments use l1; a revision should restrict the claims.","tokens_in":43939,"tokens_out":3181,"would_cite":true,"duration_ms":34354,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["68T05","62D20","68T37","05C80"],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that turning subgoal selection in hierarchical RL into targeted causal interventions provably reduces training cost, with logarithmic guarantees on tree-structured tasks.","keywords":["hierarchical reinforcement learning","causal discovery","targeted interventions","subgoal structure","training cost analysis","AND/OR subgoals","sparse regression","random graphs"],"falsifier":"Run the targeted strategy in an environment where a subgoal can be undone after being achieved, for example a tool that breaks after a fixed number of uses; if the learned causal graph misidentifies parents or the claimed $O(\\log^2(n)b)$-versus-$\\Omega(n^2b)$ cost gap disappears, then Assumption 4.2 is violated and the theoretical bounds do not hold. A sharper check on the discovery claim: construct an AND subgoal whose parent never appears in a separating valid assignment, and verify that the sparse-loss optimum omits it, which is exactly the 'discoverable parents' limitation stated by Proposition 8.3.","tokens_in":42895,"feed_emoji":"🎯","tokens_out":12362,"duration_ms":126292,"temperature":0.7,"pith_summary":"Hierarchical reinforcement learning decomposes long-horizon tasks into subgoals, but exploring subgoals that do not matter wastes samples. The paper claims that modeling subgoal dependencies as a causal graph and treating the achievement of each controllable subgoal as an intervention turns subgoal selection into a planning problem with provable efficiency. Under a monotonicity assumption (once a subgoal is achieved it stays achieved), a targeted strategy selects subgoals by estimated causal effect on the final goal or by a shortest-path heuristic, and the paper bounds the resulting training cost. The core result is a worst-case cost gap: on tree-structured subgoal graphs the targeted strategy costs $O(\\log^2(n)b)$ system probes while random selection costs $\\Omega(n^2b)$, and on semi-Erdős–Rényi graphs with $p = c\\log(n)/(n-1)$ and $0<c<1$ the gap is $O(n^{4/3+2c/3}\\log n)$ versus $\\Omega(n^2)$. A new causal-discovery subroutine, which fits a sparse predictor to an abstracted causal model with AND/OR gates, comes with a guarantee that positive coefficients correspond exactly to the discoverable parents of each subgoal.","feed_headline":"Causal targeting provably cuts hierarchical RL training cost","feed_subtitle":"Guided subgoal intervention costs O(log-squared n) on trees; random exploration costs at least n-squared.","key_machinery":"The machinery has three pieces. The subgoal structure graph: nodes are subgoals, a directed edge from $g_j$ to $g_i$ marks a causal path from one resource variable to another through only non-subgoal environment variables, and each subgoal is labeled AND or OR depending on whether all parents or at least one parent must be achieved first. The cost MDP over intervention sets: its value function (2) counts expected system probes, so any strategy for choosing the next intervention has a well-defined expected training cost, and the ranking rules plug into this MDP as the transition policy. The causal-discovery subroutine: it builds an abstracted structural causal model $X^{t+1}_i = \\theta_i(X^t) \\oplus \\epsilon^{t+1}_i$ with an AND/OR gate $\\theta_i$ and sub-critical Bernoulli noise, then fits a sparse linear score $S(X^t, \\beta)$ by minimizing the loss (4); the nonzero coefficients of the optimal $\\beta$ are provably the parents of $X_i$, up to parents that are undiscoverable because they never vary independently in valid assignments.","core_discovery":"The paper's central claim is that a hierarchical RL agent can learn which subgoals matter and in what order by representing subgoal dependencies as a causal graph, intervening on subgoals it already controls, and using the recovered graph to pick the next intervention. The HRC framework runs a loop: select a controllable subgoal, intervene on it, collect interventional trajectories, run a causal-discovery subroutine, and train the newly reachable subgoals; the selection step uses either the estimated causal effect of each candidate subgoal on the final subgoal or an A*-style shortest-path ranking. The paper proves that on trees this targeted strategy costs $O(\\log^2(n)b)$ versus a lower bound of $\\Omega(n^2b)$ for random selection, and on semi-Erdős–Rényi graphs with $p = c\\log(n)/(n-1)$ and $0<c<1$, the costs are $O(n^{4/3+2c/3}\\log n)$ versus $\\Omega(n^2)$. It also proves that minimizing the sparse loss (4) over an abstracted structural causal model returns positive coefficients exactly on the discoverable parents of each subgoal, giving the discovery step a formal correctness guarantee that prior causal HRL methods lacked.","pith_inferences":["Editorial extension: the same cost gap suggests a general recipe for task decomposition: whenever a task has prerequisite structure and subgoals are irreversible once achieved, exploration should be prioritized by causal effect on the final goal, a principle that could transfer to curriculum learning with prerequisite skills.","Editorial extension: the abstracted causal model assumes binary resource variables, but the paper's own experiments already include non-binary settings; a natural testable extension is to apply the sparse-loss discovery step to thresholded multi-level resources and check whether the parent-recovery guarantee survives.","Editorial extension: the framework's boundary is the monotonicity assumption, so environments with consumable items, expiring status effects, or tools that break are outside the theory; extending the guarantees to reversible subgoals would need a different identifiability notion or a partially observed Markov model."],"forward_implications":["On tree-structured subgoal graphs, targeted subgoal selection costs $O(\\log^2(n)b)$ system probes in the worst case while random selection costs $\\Omega(n^2b)$, so the gap grows with both task size and branching factor.","On sparse semi-Erdős–Rényi subgoal graphs, the targeted strategy costs $O(n^{4/3+2c/3}\\log n)$ probes versus $\\Omega(n^2)$ for random selection, and the gap widens as the density constant $c$ decreases.","The proposed causal-discovery subroutine, trained with a sparse linear loss, has a provable guarantee that positive coefficients land exactly on the discoverable parents of each subgoal, so the discovered hierarchy is more than a heuristic.","In the 2D-Minecraft benchmark, all versions of the proposed HRC algorithm reach the final subgoal with fewer system probes than the compared hierarchical and causal baselines, and using the proposed discovery method beats using the prior off-the-shelf causal-discovery subroutine under the same random strategy.","Because the recovered causal graph drives both intervention selection and the structure of the multi-level policy, the framework connects causal discovery to policy training in one pipeline."],"supporting_citations":[{"why":"Establishes the causal HRL baseline that the paper extends and improves upon, and supplies the prior approach it compares to in the Minecraft experiments.","marker":"Hu et al. (2022)"},{"why":"Represents the other prior causal HRL method that the paper criticizes for relying on random achievement of the final goal and lacking theoretical guarantees.","marker":"Nguyen et al. (2024)"},{"why":"Provides the off-the-shelf causal discovery method used as the baseline discovery subroutine that the paper's proposed method is compared against.","marker":"Ke et al. (2019)"},{"why":"Supplies the A* shortest-path search that the shortest-path ranking rule is built on.","marker":"Hart et al. (1968)"},{"why":"Supplies the notion of admissible heuristics used in the analysis of the shortest-path ranking rule.","marker":"Pearl (1984)"},{"why":"Supplies the definition of structural causal models that the SCM formalism and the abstracted causal model follow.","marker":"Peters et al. (2017)"},{"why":"Provides the 2D-Minecraft environment used for the real-world-style HRL experiments.","marker":"Sohn et al. (2018)"},{"why":"Supplies the Chernoff bound used in the random-graph cost analysis.","marker":"Mitzenmacher & Upfal (2017)"}],"fun_headline_variants":["Causal targeting provably trims HRL training cost","Provable HRL speedup via targeted causal interventions","Causal subgoal selection cuts HRL cost by theory","Guided causal interventions lower HRL training expense","Formal proof: causal HRL beats random subgoal exploration"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that a subgoal, once achieved, stays achieved forever; if resources can be consumed, tools can break, or status effects can expire, achieving a subgoal can no longer be treated as a permanent causal intervention, and the framework's guarantees no longer apply.","fun_headline_variants_meta":{"raw":{"variants":["Causal targeting provably trims HRL training cost","Provable HRL speedup via targeted causal interventions","Causal subgoal selection cuts HRL cost by theory","Guided causal interventions lower HRL training expense","Formal proof: causal HRL beats random subgoal exploration"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000292,"raw_usage":{"total_tokens":1719,"prompt_tokens":978,"completion_tokens":741,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":594,"completion_tokens_details":{"reasoning_tokens":662}},"tokens_in":594,"tokens_out":741,"duration_ms":8977,"temperature":1.0,"reasoning_tokens":662,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T19:50:31.041150+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the targeted strategy in an environment where a subgoal can be undone after being achieved, for example a tool that breaks after a fixed number of uses; if the learned causal graph misidentifies parents or the claimed $O(\\log^2(n)b)$-versus-$\\Omega(n^2b)$ cost gap disappears, then Assumption 4.2 is violated and the theoretical bounds do not hold. A sharper check on the discovery claim: construct an AND subgoal whose parent never appears in a separating valid assignment, and verify that the sparse-loss optimum omits it, which is exactly the 'discoverable parents' limitation stated by Proposition 8.3.","supporting_citations":[],"review_version":1}