REVIEW 4 major objections 5 minor 22 references
Hierarchical Reinforcement Learning with Targeted Causal Interventions
T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read This paper claims that turning subgoal selection in hierarchical RL into targeted causal interventions provably reduces training cost, with logarithmic guarantees on tree-structured tasks.
desk verdict Solid HRL+causality paper with real empirical gains, but the tree lower bound in Theorem 7.4 does not cover shallow trees and the SSD guarantee is proved for l0 while the experiments use l1; a revision should restrict the claims. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The machinery has three pieces. The subgoal structure graph: nodes are subgoals, a directed edge from $g_j$ to $g_i$ marks a causal path from one resource variable to another through only non-subgoal environment variables, and each subgoal is labeled AND or OR depending on whether all parents or at least one parent must be achieved first. The cost MDP over intervention sets: its value function (2) counts expected system probes, so any strategy for choosing the next intervention has a well-defined expected training cost, and the ranking rules plug into this MDP as the transition policy. The causal-discovery subroutine: it builds an abstracted structural causal model $X^{t+1}_i = \theta_i(X^t) \oplus \epsilon^{t+1}_i$ with an AND/OR gate $\theta_i$ and sub-critical Bernoulli noise, then fits a sparse linear score $S(X^t, \beta)$ by minimizing the loss (4); the nonzero coefficients of the optimal $\beta$ are provably the parents of $X_i$, up to parents that are undiscoverable because they never vary independently in valid assignments.
What would settle it
Run the targeted strategy in an environment where a subgoal can be undone after being achieved, for example a tool that breaks after a fixed number of uses; if the learned causal graph misidentifies parents or the claimed $O(\log^2(n)b)$-versus-$\Omega(n^2b)$ cost gap disappears, then Assumption 4.2 is violated and the theoretical bounds do not hold. A sharper check on the discovery claim: construct an AND subgoal whose parent never appears in a separating valid assignment, and verify that the sparse-loss optimum omits it, which is exactly the 'discoverable parents' limitation stated by Proposition 8.3.
Extended reading notes
Core claim
The paper's central claim is that a hierarchical RL agent can learn which subgoals matter and in what order by representing subgoal dependencies as a causal graph, intervening on subgoals it already controls, and using the recovered graph to pick the next intervention. The HRC framework runs a loop: select a controllable subgoal, intervene on it, collect interventional trajectories, run a causal-discovery subroutine, and train the newly reachable subgoals; the selection step uses either the estimated causal effect of each candidate subgoal on the final subgoal or an A*-style shortest-path ranking. The paper proves that on trees this targeted strategy costs $O(\log^2(n)b)$ versus a lower bound of $\Omega(n^2b)$ for random selection, and on semi-Erdős–Rényi graphs with $p = c\log(n)/(n-1)$ and $0<c<1$, the costs are $O(n^{4/3+2c/3}\log n)$ versus $\Omega(n^2)$. It also proves that minimizing the sparse loss (4) over an abstracted structural causal model returns positive coefficients exactly on the discoverable parents of each subgoal, giving the discovery step a formal correctness guarantee that prior causal HRL methods lacked.
Load-bearing premise
The load-bearing premise is that a subgoal, once achieved, stays achieved forever; if resources can be consumed, tools can break, or status effects can expire, achieving a subgoal can no longer be treated as a permanent causal intervention, and the framework's guarantees no longer apply.
Editorial extensions
If this is right
- On tree-structured subgoal graphs, targeted subgoal selection costs $O(\log^2(n)b)$ system probes in the worst case while random selection costs $\Omega(n^2b)$, so the gap grows with both task size and branching factor.
- On sparse semi-Erdős–Rényi subgoal graphs, the targeted strategy costs $O(n^{4/3+2c/3}\log n)$ probes versus $\Omega(n^2)$ for random selection, and the gap widens as the density constant $c$ decreases.
- The proposed causal-discovery subroutine, trained with a sparse linear loss, has a provable guarantee that positive coefficients land exactly on the discoverable parents of each subgoal, so the discovered hierarchy is more than a heuristic.
- In the 2D-Minecraft benchmark, all versions of the proposed HRC algorithm reach the final subgoal with fewer system probes than the compared hierarchical and causal baselines, and using the proposed discovery method beats using the prior off-the-shelf causal-discovery subroutine under the same random strategy.
- Because the recovered causal graph drives both intervention selection and the structure of the multi-level policy, the framework connects causal discovery to policy training in one pipeline.
Reading between the lines
- Editorial extension: the same cost gap suggests a general recipe for task decomposition: whenever a task has prerequisite structure and subgoals are irreversible once achieved, exploration should be prioritized by causal effect on the final goal, a principle that could transfer to curriculum learning with prerequisite skills.
- Editorial extension: the abstracted causal model assumes binary resource variables, but the paper's own experiments already include non-binary settings; a natural testable extension is to apply the sparse-loss discovery step to thresholded multi-level resources and check whether the parent-recovery guarantee survives.
- Editorial extension: the framework's boundary is the monotonicity assumption, so environments with consumable items, expiring status effects, or tools that break are outside the theory; extending the guarantees to reversible subgoals would need a different identifiability notion or a partially observed Markov model.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes HRC, a hierarchical reinforcement learning framework that models dependencies among subgoals as a causal graph, performs interventions on controllable subgoals, and uses the estimated causal model to prioritize which subgoal to add to the intervention set. The authors define a training-cost MDP and claim that, under Assumptions 7.1–7.3, a targeted strategy HRCh has cost O(log^2(n)b) on trees and O(n^{4/3+2c/3} log n) on semi-Erdős–Rényi graphs, while a random strategy HRCb has cost Ω(n^2 b) and Ω(n^2), respectively (Theorem 7.4). The paper also introduces a causal discovery method, SSD, with a theoretical guarantee (Theorem 8.4) for recovering parents from an abstracted SCM, and reports experiments on synthetic graphs and a 2D-Minecraft environment showing improved training cost over several HRL baselines.
Significance. If the theoretical results were fully valid, the cost separation between targeted and random exploration would be a valuable first formal analysis of causally-guided exploration in hierarchical RL. The paper has several strengths: it proposes a concrete cost model (Section 5 and Appendix E.1), provides a public codebase, evaluates on a nontrivial Minecraft environment, and compares against CDHRL, HAC, HER, OHRL, and PPO. The experimental evidence for a training-cost advantage of HRCh over HRCb and over prior methods is suggestive. However, the theoretical claims currently contain load-bearing gaps: the tree lower bound is not valid for shallow trees, Theorem 8.4 is not stated with the positivity/discoverability conditions needed by its proof, and the experiments use an l1 loss while the theorem is proved for l0. These issues must be repaired before the formal claims can be accepted.
major comments (4)
- [Appendix E.2, Eq. (21), Theorem 7.4, Table 1] The tree lower bound for HRCb is invalid for shallow trees. The proof defines α = 1 − 2/2^{D−1}; for D=2 this gives α=0 and for D=1 it gives α=−1, so the step from Q_NG(k) ≥ α to the Ω(n^2 b) conclusion is vacuous. This is not merely a loose constant: in a depth-1 tree (root plus b children, one of which is the final subgoal), HRCb selects uniformly without replacement among the b children, so the expected number of selections before hitting the final child is Θ(b), each transition costs Θ(k) probes, and the expected cost is Θ(b^2) = Θ(n^2), not Ω(n^2 b) = Ω(b^3). The counting function f(h) = h(b−1)+1 also overcounts grandchildren for depth-1 leaves, which have no children. The theorem and the tree row of Table 1 therefore overstate the random-strategy cost; either restrict the statement to D ≥ 3 with a corrected boundary analysis, or repair the lower-bound derivation.
- [Section 8, Theorem 8.4, Appendix G.2, Appendix G.3] Theorem 8.4 claims that positive coefficients in any optimal β* correspond exactly to the parents of Xi, but the proof requires certain assignments in D_My to have positive probability, and no positivity condition appears in the theorem statement. The manuscript's own Example G.2 (Figure 11) describes a valid A-SCM in which X2=1 always implies X1=1; for the AND subgoal X3 with parents X1 and X2, the configuration needed to detect X1 (X1=0, X2=1) has probability zero, making X1 undiscoverable. Proposition 8.3 correctly limits identifiability to discoverable parents, but Theorem 8.4 as stated contradicts this by promising recovery of all parents. The theorem should be restated for discoverable parents under an explicit positivity assumption, or the proof should be reconciled with G.2.
- [Remark 8.5, Section 9, Table 3] The experiments for SSD use the l1 norm in the loss (Remark 8.5), while Theorem 8.4 is proved only for the l0 norm. The l1 relaxation is not equivalent to l0 sparsity for the nonlinear threshold model in (3), so the reported empirical performance of SSD does not inherit the theorem's guarantee. Please either provide a corresponding theoretical result for the estimator actually used (which would require additional assumptions, e.g., irrepresentability-type conditions), or explicitly present the l1 variant as a heuristic motivated by the l0 result.
- [Section 7, Assumption 7.2, Theorem 7.4] The cost separation for HRCh is driven by Assumption 7.2, which states that the estimated causal effect of a subgoal on the final subgoal is nonzero exactly when that subgoal is an ancestor. This assumption directly encodes the success of the causal-effect ranking rule, so the theorem is conditional rather than fully explanatory: it shows that if the ECE estimator is perfect in a specific sense, then targeted exploration is cheaper. The paper should state this limitation explicitly, and ideally provide a finite-sample or error-propagating analysis showing how estimation error in the ECE affects the cost guarantees. As written, the claimed 'provable improvement' rests entirely on an assumption that is close to the desired conclusion.
minor comments (5)
- [Algorithm 1, line 8 and Remark E.1] The condition PA_hat^t_gi ⊂ ISt is stated in line 8, but Remark E.1 says it can be removed under Assumption 7.1; the relation between the algorithm's stated condition and the assumption should be clarified in the main text.
- [Appendix E.1, Eq. (5)] The notation wτj_int,g′, wτj_exp, and wτj_train,g′′ is dense and the definition of system probes is spread over several sentences; a short table or a glossary would improve readability.
- [Table 2] The SHD comparison reports means but no standard deviations or number of runs; adding these would strengthen the claim that SSD has fewer extra edges.
- [Table 3] The hyperparameter labeled 'LASSO L1-RATIO / OTHERS' is set to 0.0001 for HRCh (SSD) and HRCb (SSD); please clarify whether this is the λ in the l1 loss and how it was chosen.
- [Throughout] There are minor typos, e.g., 'Controllabe Set' in Algorithm 1 and inconsistent spacing in 'HRC h (SSD)' in Figure 6(a); a copyedit pass would be helpful.
Circularity Check
Theorem 7.4's HRCh upper bound largely restates Assumption 7.2; the HRCb lower bound and Theorem 8.4 are independent.
-
other
[Section 7, Assumption 7.2 and Appendix E.2 (tree analysis)]
"Assumption 7.2. We assume that the estimated causal effect of a subgoal gi on the final subgoal gn is not zero if gi is an ancestor of gn in G . ... since they add only the ancestors of the final subgoal to the intervention set in the worst-case scenario (based on Assumption 7.2). Therefore, the complexity of our targeted strategies is O(log2(n)b)."
The upper-bound half of Theorem 7.4 is derived by assuming the exact property the targeted ranking rule is supposed to deliver. HRCh selects gsel by estimated causal effect (Section 6), so Assumption 7.2 asserts the estimator is perfect on ancestors and zero elsewhere. Appendix E.2 then uses that assumption to conclude only ancestors are added, immediately giving O(log^2(n)b). The claimed cost reduction for HRCh is thus imported as an assumption rather than derived. The random-strategy lower bound and the SSD identifiability analysis (Theorem 8.4) are independent, so the circularity is partial.
full rationale
The main advertised result, the cost gap in Table 1, is split: the HRCb Omega bounds are derived through the MDP cost recursion, the tree combinatorial argument, and the random-graph path-counting inequalities, which are independent analyses. The HRCh upper bound, however, reduces to Assumption 7.2, which asserts that the estimated causal effect is nonzero exactly for ancestors of the final subgoal. Since the targeted strategy is defined by ranking with that same estimated causal effect, the conclusion that HRCh adds only ancestors is a restatement of the assumption rather than a derived consequence. No load-bearing self-citations appear; the causal discovery theorem (Theorem 8.4) is a self-contained proof within the A-SCM and does not import its conclusion. Shallow-tree boundary issues raised for Eq. (21) concern correctness of a lower-bound calculation, not circularity, and are not scored here. Overall, the paper contains one significant assumption-driven circular component in the upper-bound claim, with independent content elsewhere, giving a score of 4.
Assumptions & free parameters
free parameters (5)
- lambda (l1 regularization in SSD) =
0.0001 (Table 3)
- p (probabilistic subgoal selection in training) =
0.1 (Appendix D.4)
- phi_causal (success ratio threshold) =
0.5 (Table 3)
- Delta (ECE horizon) =
20 (Appendix H.3)
- T and T' (trajectory counts) =
not specified
assumptions (7)
- domain assumption Assumption 4.2: if X_t^i = 1 then X_{t'}^i = 1 for all t' >= t (subgoals persist once achieved).
- ad hoc to paper Assumption 7.1: causal discovery recovers exactly the children of the intervention set (with stated conditions) and every reachable subgoal becomes controllable after training.
- ad hoc to paper Assumption 7.2: estimated ECE of gi on the final subgoal is nonzero iff gi is an ancestor of the final subgoal.
- ad hoc to paper Assumption 7.3: expected probes between consecutive subgoals are equal.
- domain assumption A-SCM (3): X_{t+1}^i = theta_i(X_t) XOR epsilon_{t+1}^i with Bernoulli(rho), rho < 1/2.
- standard math Discoverable-parent positivity (unstated): there exist one-sided valid assignments isolating each parent, as in Definition 8.2.
- domain assumption All environment variables are observed as disentangled factors.
Cite this review
Pith. "Pith review of Hierarchical Reinforcement Learning with Targeted Causal Interventions." pith.science (2026). https://pith.science/paper/3X4PHGRE
@misc{pith2026250704373,
author = {Pith},
title = {Pith review of: Hierarchical Reinforcement Learning with Targeted Causal Interventions},
year = {2026},
howpublished = {\url{https://pith.science/paper/3X4PHGRE}},
note = {Machine review of arXiv:2507.04373}
}
read the original abstract
Hierarchical reinforcement learning (HRL) improves the efficiency of long-horizon reinforcement-learning tasks with sparse rewards by decomposing the task into a hierarchy of subgoals. The main challenge of HRL is efficient discovery of the hierarchical structure among subgoals and utilizing this structure to achieve the final goal. We address this challenge by modeling the subgoal structure as a causal graph and propose a causal discovery algorithm to learn it. Additionally, rather than intervening on the subgoals at random during exploration, we harness the discovered causal model to prioritize subgoal interventions based on their importance in attaining the final goal. These targeted interventions result in a significantly more efficient policy in terms of the training cost. Unlike previous work on causal HRL, which lacked theoretical analysis, we provide a formal analysis of the problem. Specifically, for tree structures and, for a variant of Erd\H{o}s-R\'enyi random graphs, our approach results in remarkable improvements. Our experimental results on HRL tasks also illustrate that our proposed framework outperforms existing work in terms of training cost.
Figures
Figures from the paper (10 more)
Reference graph
Works this paper leans on
-
[1]
For” loop, it records a trajectory τj. The “While
Figure 7b shows another H = (H , L) where H is not an exact estimate of G . The level of W is zero, S is 1 and P is 2. C.2. More Definitions Definition C.1 (Structural Causal Model (SCM)(Peters et al., 2017)) . A Structural Causal Model is defined as a tuple M = ⟨U, V, F, P(U )⟩, where: • U is a set of exogenous variables whose values are determined by fa...
work page 2017
-
[2]
Hence: q1 ≤ 2nc. As we mentioned earlier, q1 is the upper bound on the expected number of the nodes that are ancestors of the node n. Hence: E[ANCn(G)] ≤ n−1X j=1 P (Aj) = q1 ≤ 2nc, (46) where G is the generated random graph and ANCn(G) represents the number of ancestors of the node n in the graph G. The total cost is then computed as the expected sum of ...
work page 2017
-
[3]
At this point, we have constructed Ijr
Continue this process until gn is in the set. At this point, we have constructed Ijr. If i = r, the process is complete. Otherwise, we continue a similar replacement process starting from set Ijr. In Ijr, we replace the node gr′ where r′ is the second maximum index among nodes in set Ij (r′ = max{z : gz ∈ Ij \ {gr}}) with its successive node gr′+1 until w...
work page 1984
-
[4]
Heuristic Update: After computing the shortest path, set h(gi) = dist(gn). F.8. Intuitive Examples In Figure 9a, to achieve the final subgoal g8, all the green subgoals must be achieved and these are precisely the subgoals with a non-zero causal effect on g8. Similarly in Figure 9b, we achieve the final subgoal g8 if we only achieve the green subgoals dur...
work page 2019
-
[5]
First, we replace node gr with gr+1 and construct Ij(r)
-
[6]
Then replace gr+1 in Ij(r) by gr+2
-
[8]
We define a function parent pointer : Φ → Φ, which assigns the corresponding parent of a node gi
Initialize two node sets: a closed set (already evaluated nodes) and an open set (yet to be evaluated, starting with the initial node). We define a function parent pointer : Φ → Φ, which assigns the corresponding parent of a node gi. Nodes are evaluated based on three key functions: • g(gi): the cost from the start node to node gi, • h(gi): an admissible ...
-
[9]
• If gi is the goal, backtrack to construct the path
While the open set is not empty: • Select node gi with the lowest f(gi) from the open set. • If gi is the goal, backtrack to construct the path. 30 HRL with Targeted Causal Interventions • Move gi to the closed set and evaluate its neighbors: – For each neighbor gm of gi, if gm is not in the closed set, add it to the open set if not already present. – Cal...
Show all 22 references
-
[10]
If no path to the goal is found and the open set is empty, conclude no path exists. F.3. Backtrack function We define the function back track(gi) that backtracks through the parent pointers of the node gi and returns the set of ancestors of gi during the backtracking process: ...
-
[11]
We define a function parent pointer : Φ → Φ, which assigns the corresponding parent of a node gi
Initialization: Set the tentative distance dist(gi) = 0 and dist(g) = ∞ (the distance of gi to g) for all g ∈ E \ {gi}. We define a function parent pointer : Φ → Φ, which assigns the corresponding parent of a node gi
-
[12]
Priority Queue: Insert all nodes into a priority queue Q based on their tentative distances
-
[13]
– For each neighbor gv of gu in ˆGt: * Set w(gu, gv) = |C HˆGt gu \ back track(gu)| + 1
Algorithm Execution: • While Q is not empty: – Extract the node gu with the smallest dist(gu) from Q. – For each neighbor gv of gu in ˆGt: * Set w(gu, gv) = |C HˆGt gu \ back track(gu)| + 1. * If dist(gv) ¿ dist(gu) + w(gu, gv), then: update dist(gv) = dist(gu) + w(gu, gv) and...
-
[15]
This implies that in all possible assignments, at least one parent of gi is always 1
There is no valid assignment X ∈ {0, 1}n such that ∀ gk ∈ PAgi , X k = 0. This implies that in all possible assignments, at least one parent of gi is always 1. Therefore, the OR operation can be replaced by 1: θi(Xt) = _ gj ∈PAgi X t j = 1. Thus, the SCM simplifies to: X t+1 i...
-
[16]
This means that in the collected data, if Xj is one, at least of some other parent Xk is also one
If there is no valid assignment X ∈ {0, 1}n such that Xj = 1, ∀ gk ∈ PAgi \ {gj}, X k = 0. This means that in the collected data, if Xj is one, at least of some other parent Xk is also one. Now consider an alternative SCM where gj is not a parent of gi: θ′ i(Xt) = _ gk∈PAgi \{...
-
[17]
Thus, the AND operator equals to 0: θi(Xt) = ^ gj ∈PAgi X t j = 0
There is no valid assignment X ∈ {0, 1}n such that ∀ gk ∈ PAgi , X k = 1. Thus, the AND operator equals to 0: θi(Xt) = ^ gj ∈PAgi X t j = 0. The SCM simplifies to: X t+1 i = X 0 i . Now consider an alternative SCM where gi is isolated and only dependent on its initial state, t...
-
[18]
This means that if Xj is zero, at least of some other parent Xk is one
There is no valid assignment where Xj = 0, ∀ gk ∈ PAgi \ {gj}, X k = 1. This means that if Xj is zero, at least of some other parent Xk is one. Now consider an alternative SCM where gj is not a parent of gi: θ′ i(Xt) = ^ gk∈PAgi \{gj } X t k. Both the original SCM and the alte...
-
[19]
For j /∈ Py, β∗ j = 0,
-
[20]
For j ∈ Py, β∗ j > 0. Let us rewrite the loss function in (67) for any β satisfying the above conditions as follows: L(β) = E[(y − ˆyβ)2] + λ∥β∥0, (68) (a) = X X∈{0,1}n P (X)E[(y − ˆyβ)2|X] + λ|Py| (69) (b) = X X∈{0,1}n P (X)ρ + λ|Py| (70) = ρ + λ|Py|. (71) (a) We marginalize ...
-
[21]
For j /∈ Py, β∗ j = 0
-
[22]
oracle goal space
∀U ⊊ Py : X j∈U β∗ j ≤ −β0 (92) X j∈Py β∗ j > −β0, (93) The value of the loss function in (67) for any β satisfying the above conditions is equal to 71. Now, we show that if some β fails to satisfy either of the aforementioned conditions, it will result in an increase in loss ...
2022
-
[1528]
Forestier, S., Portelas, R., Mollard, Y ., and Oudeyer, P.- Y
PMLR, 2018. Forestier, S., Portelas, R., Mollard, Y ., and Oudeyer, P.- Y . Intrinsically motivated goal exploration processes with automatic curriculum learning. Journal of Machine Learning Research, 23(152):1–41, 2022. Guss, W. H., Houghton, B., Topin, N., Wang, P., Codel, C...
2018 arXiv
-
[3119]
PMLR, 2019. Ke, N. R., Bilaniuk, O., Goyal, A., Bauer, S., Larochelle, H., Sch¨olkopf, B., Mozer, M. C., Pal, C., and Bengio, Y . Learning neural causal models from unknown interven- tions. arXiv preprint arXiv:1910.01075, 2019. Kimura, D., Ono, M., Chaudhury, S., Kohita, R., ...
2019 arXiv
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.