{"id":"73ea3606-70d7-4483-a741-9fa6ae8c4263","arxiv_id":"2505.01979","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"D3HRL combines distributed causal discovery over multiple time lags with conditional independence testing to learn delayed action effects and build hierarchical policies in long-horizon tasks.","lead":"The paper presents D3HRL, a hierarchical reinforcement learning method that discovers cause-and-effect links between actions and later changes in the environment, then filters out accidental correlations. The authors report that in modified 2D-Minecraft and MiniGrid tasks, D3HRL learns subgoals faster and reconstructs the task's causal graph more accurately than four existing HRL methods.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Equation 8's gradient update for the causal graph matrix is not a valid score-function estimator; as written, the distributed SCM training does not optimize the marginal likelihood that the reported SHD results depend on.","rationale":"The reader's weakest assumption concerns the Causal Factored-SMDP assumptions (independent sub-states, causal sufficiency). That is a real and acknowledged limitation, but the paper explicitly scopes its claim to decouplable tasks, so it does not by itself break the central claim inside that scope. The more load-bearing issue is internal to the described algorithm: Section 5.1.2's gradient update in Eq. 8 is not a standard score-function estimator and, taken literally, would not be optimizing the marginal data likelihood under the sampled causal graphs. Since the causal graph matrix σ(η_h) is the sole source of candidate edges for CIT and the hierarchy, a wrong update would undermine the central claim even in a perfectly decouplable environment with all causes observed. This is a concrete, checkable technical concern rather than a scope limitation. I retain the reader's CONDITIONAL verdict: the paper should either correct and derive Eq. 8, provide the implementation, or include a synthetic test that validates the estimator. If the test shows Eq. 8 fails, the verdict should move toward REJECT; if it passes, the concern is resolved.","tokens_in":12949,"tokens_out":19541,"duration_ms":203988,"concrete_test":"Implement Eq. 8 exactly as written and compare it with the correct softmax score-function estimator Σ_k Σ_n (σ(η)−P_n) · e^{L(P_n,D_k)} / Σ_{n′} e^{L(P_{n′},D_k)} on a small synthetic SCM with a known time-lagged graph (e.g., A_{t-1}→E_t, with B_{t-1} unrelated to E_t). Run both updates for the same number of iterations with identical data and initialization, and check whether the Eq. 8 update drives σ(η) toward the true edge and away from the false edge. If it does not, the core causal discovery mechanism as described is not what produces the SHD results in Table 1.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim rests on Module A (§5.1.2) learning the correct causal graph parameter matrix η_h. Equation 8 defines the update as ∇η_h = Σ_k [Σ_n (σ(η)−P_n^h)] · exp(Σ_n L(P_n^h,D_k)) / Σ_{k′} exp(Σ_n L(P_n^h,D_{k′})). As written, this is not the REINFORCE/softmax gradient of the marginal data likelihood. For samples P_n∼σ(η), the per-sample score (σ(η)−P_n) must be weighted by that sample's own likelihood, e.g., e^{L(P_n,D_k)}/Σ_{n′} e^{L(P_{n′},D_k)}, or by L(P_n,D_k) with a baseline. Here the exponent sums over all n and the denominator normalizes over data batches, so the same weight multiplies every P_n in the batch and does not depend on which graph was sampled. The update therefore does not maximize the likelihood of the data under the sampled causal graphs; it appears to target a softmax-weighted batch score instead. Because the candidate edges feeding CIT (§5.2) and the hierarchical construction (§5.3) are read from σ(η_h), an incorrect update invalidates the causal-discovery claim even when the Causal Factored-SMDP assumptions hold. The paper gives no derivation, synthetic validation, or code for Eq. 8, so a reader cannot tell whether this is a typographical error or the actual procedure that produced Table 1.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces D3HRL, a hierarchical reinforcement learning method that combines distributed causal discovery with spurious-correlation filtering. Its three modules are: (A) reverse data collection with parallel SCM training over time spans 1..tau_max (Eqs. 4-8); (B) conditional independence testing via CMI to filter spurious edges and select true time spans (Eqs. 9-11); and (C) construction and training of hierarchical DQN sub-goal networks based on the confirmed causal chain (Section 5.3). The experiments compare D3HRL against CDHRL, HAC, Option-Critic, and LESSON on modified 2D-Minecraft and MiniGrid tasks, reporting ASR, ADC, SHD against CDHRL, an ablation of reverse vs forward collection, CMI curves, and scalability for tau_max up to 16.","tokens_in":13240,"tokens_out":5492,"duration_ms":54727,"significance":"If the causal discovery step is correct, the paper proposes a meaningful integration of time-series causal discovery with HRL, with a concrete mechanism for delay effects and spurious correlations, and the reported SHD values are internally consistent with the claimed advantage over CDHRL. The strengths include the reverse data-collection ablation, the use of SHD with hand-specified ground-truth graphs, and the evaluation across multiple task configurations. However, the significance is conditional: the central update in Eq. (8) is not a valid maximum-likelihood estimator as written, the CIT threshold is tuned in a pre-experiment without reported sensitivity, and the baselines are modified with hand-designed curricula, so the current evidence does not establish the headline claim.","major_comments":[{"comment":"Equation (8) does not define a valid stochastic-gradient estimator for the marginal data likelihood under the graph prior sigma(eta). In a REINFORCE/softmax estimator, the score (sigma(eta)-P_n) for a sampled graph P_n must be weighted by that sample's own likelihood exp(L(P_n,D_k)), or by a baseline, so that low-likelihood samples are down-weighted relative to high-likelihood ones. As written, the exponential weight is exp(sum_n L(P_n^h,D_k)) and is constant across the N_p sampled graphs inside the inner sum, while the normalizer runs over data batches k'. Consequently, the update does not maximize the likelihood of the observed data under the sampled causal graphs, and the causal structure read from sigma(eta_h) after thresholding (Section 5.1.2, threshold 0.8) is not a maximum-likelihood graph. This is load-bearing because Module A's output feeds CIT (Section 5.2) and the hierarchical construction (Section 5.3). Please provide a derivation of Eq. (8), correct it to a valid estimator, and validate it on a synthetic graph with known ground truth; releasing the code would also settle whether the reported SHD results come from this update.","section":"5.1.2, Eq. (8)"},{"comment":"The spurious-correlation filter uses epsilon_cmi = 0.05 to declare genuine causal relationships, and the authors state that this threshold was selected in a separate statistical experiment before the main experiments. Since the threshold directly controls which edges survive into the SHD computation and which time spans are accepted, and since the experiments are conducted on the same task family (GetIron, Wood2Wet), the reported SHD and success curves are at risk of selection bias. The manuscript should report the pre-experiment procedure and show sensitivity of Table 1 and Figure 11 to epsilon_cmi (e.g., a sweep), or justify that the threshold is task-independent.","section":"6.2.5, Eq. (11)"},{"comment":"The baselines are modified with hand-designed curricula or with tau_max-step transition collection to adapt to variable-length transitions, which makes the comparison in Figure 3 hard to interpret. For example, HAC is said to use a pre-defined curriculum to acquire causality, and Option-Critic/LESSON receive curriculum modifications, so it is unclear whether the observed gap reflects D3HRL's causal-discovery modules or the authors' enhancements. In addition, all results are averaged over only 5 seeds and no variance or confidence intervals are shown for ASR/ADC (Figure 3) or sub-goal efficiency (Figure 10), so the significance of the improvements cannot be assessed. Please report per-seed variability and clearly separate the baseline versions used from the original algorithms.","section":"6.1, Figure 3"},{"comment":"The rule 'if Xj->Xi is judged valid across multiple different time spans, the shortest one is identified as true' is stated without proof and is load-bearing for the time-span matrix T used in hierarchical training. Under the Causal Factored-SMDP assumptions, with autocorrelated effect variables and multiple causes, it is not generally true that only the shortest valid span is causal; a longer span could reflect a genuinely distinct delayed mechanism. Please provide a proof or an empirical validation (e.g., synthetic TSGM with known spans) that this rule holds for the class of tasks considered.","section":"5.2.2"}],"minor_comments":[{"comment":"The heading reads 'DEHRL' but should be 'D3HRL'.","section":"6.2.4"},{"comment":"The numeric columns under the combined header 'Task (tau_max)' lack individual column labels, so the reader must infer which values belong to GetIron-R0 versus Wood2Wet and which belong to each tau_max setting; please make the header explicit.","section":"Table 1"},{"comment":"The text refers to 'the second subplot in Fig 4' when discussing CMI values, but the CMI results appear in Figures 8 and 9; the figure reference should be corrected.","section":"6.2.5"},{"comment":"The phrase 'causality stone <- A -> wood' uses an informal notation that is inconsistent with the directed-edge notation used elsewhere; please re-express it with the standard arrow notation.","section":"5.2.1"},{"comment":"The statement that spurious correlations have CMI values of 0 should be quantified, since the plotted values in Figure 9 may be small but not exactly zero; please report the numerical range or a tolerance.","section":"6.2.5"}],"recommendation":"major_revision","confidential_remarks":"The manuscript does not currently include code or a synthetic causal-discovery experiment. Given that Eq. (8) is not a recognizable maximum-likelihood gradient and that epsilon_cmi is tuned in a pre-experiment on the same benchmark family, I would encourage the editor to require code release or synthetic validation before considering the paper for publication. The contribution relative to CDHRL is incremental but potentially real if the causal graph learning is corrected and validated."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: D3HRL is a credible attempt to add causal discovery to hierarchical RL, specifically to handle variable-length state transitions and spurious correlations, and it shows promising numbers on two custom benchmark tasks. What I found new is the combination of distributed SCM training across time spans, a CMI-based spurious-correlation filter, and a reverse data collection strategy. That combination is not in the CDHRL or SDI work they build on. The paper is also unusually honest about its assumptions: it explicitly restricts itself to decouplable state variables and notes that random time spans are out of scope.\n\nThe main problem is Equation 8. This is the gradient update for the causal graph parameter matrix η_h, and everything downstream—the candidate edges, the CIT, the subgoal construction—reads from σ(η_h). As written, the update weights the score term (σ(η)−P_n) by a softmax over data batches, not by the likelihood of each sampled graph. That is not a valid score-function estimator, and no derivation or synthetic validation is given. I can't tell if it's a typo or the actual procedure, but either way the causal-discovery claim is unverifiable without a corrected equation or code. That is a load-bearing gap, not a cosmetic one.\n\nThe empirical section has the usual issues: ε_cmi=0.05 was tuned in a pre-experiment on the same tasks; the baselines were modified with hand-designed curricula; the 5-seed averages come without error bars; and there is no code or data release. Those are addressable, and they don't by themselves sink the paper. The SHD numbers favor D3HRL and the CMI plots make sense qualitatively. The reverse-vs-forward comparison is a nice ablation.\n\nWho is this for? People doing causal RL or HRL for long-horizon tasks will want to read it, especially if they care about delayed effects. It deserves a serious referee, but the referee should push hard on Eq. 8. I would not cite it myself until the gradient issue is cleared up. My recommendation: send to peer review, but with the expectation that the authors either fix Eq. 8 with a proper derivation or release code that reproduces Table 1.","headline":"The paper's causal graph learning step is under-specified and likely wrong as written; the rest is a reasonable HRL+causality combination with addressable empirical gaps.","tokens_in":13833,"tokens_out":3890,"would_cite":false,"duration_ms":35535,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Causal discovery plus independence testing lets hierarchical reinforcement learning learn delayed effects and discard spurious correlations.","keywords":["Hierarchical Reinforcement Learning","Causal discovery","Spurious correlation detection","Conditional independence testing","Temporal abstraction","Semi-MDP","Causal Factored-SMDP","Long-horizon tasks"],"falsifier":"Run D3HRL on a long-horizon task where a hidden confounder deterministically affects both a material's availability and the task reward, or where two supposedly independent sub-states interact. If the learned causal graph still shows low Structural Hamming Distance while the success ratio drops, the causal sufficiency or independence assumption is violated; if the graph contains extra edges pointing to the hidden cause, the CMI filter has not actually removed spurious correlations.","tokens_in":12714,"feed_emoji":"🧩","tokens_out":5632,"duration_ms":56627,"temperature":0.7,"pith_summary":"The paper argues that the two main obstacles to long-horizon hierarchical reinforcement learning—delayed effects, where an action changes the state only after several steps, and spurious correlations, where two variables move together without a causal link—can be overcome by making the hierarchy itself a causal model. D3HRL treats each possible time delay as a separate causal relationship, trains structural causal models in parallel on data collected backwards from state changes, and then filters the discovered edges by conditional independence testing. Only the surviving causal relationships become sub-goals and policy layers. If correct, this yields accurate causal chains and faster training in tasks like gathering and crafting in 2D-Minecraft and MiniGrid.","feed_headline":"Distributed causal discovery teaches RL to handle delayed effects","feed_subtitle":"A new filter removes spurious correlations, so the learned hierarchy mirrors the true cause chain and trains faster.","key_machinery":"Causal Factored-SMDP—a five-component formalization $\\langle C, E, P, T, F \\rangle$ that turns state transitions into SCM generating functions over causal relationships with time spans—and the three-module loop built on it. Distributed SCM training assigns one process per candidate time span $h$, uses reverse data collection from intervention points to assemble $2\\tau_{\\max}+1$-length windows, and learns both a generative network and a causal-graph probability matrix $\\sigma(\\eta^h)$. Spurious correlation detection then estimates the conditional mutual information of each candidate edge and keeps only edges above threshold $\\epsilon_{\\text{cmi}}$; if an edge survives at several time spans, the shortest span is chosen as true. The resulting hierarchy grows one DQN sub-goal network per verified effect, with hindsight transitions of length $h+1$ for training.","core_discovery":"On its own terms, the paper's central claim is that variable-length state transitions are causal relationships across time spans, so they can be learned by distributed structural causal model training, and that spurious correlations produced by common causes and autocorrelation can be removed by conditional independence testing via conditional mutual information. The method formalizes this as Causal Factored-SMDPs, in which the state is decomposed into independent sub-states, the transition rules are the generating functions of an SCM, and the hierarchical policy mirrors the discovered causal chain. In the experiments, D3HRL reaches higher average success ratios than CDHRL, HAC, Option-Critic, and LESSON on modified tasks, and its learned causal graphs have lower Structural Hamming Distance on those tasks.","pith_inferences":["A direct extension would replace the DQN base with any off-policy value method and test whether the causal-chain construction transfers; the paper does not claim this.","A hidden-confounder experiment would probe the causal sufficiency assumption directly: adding an unobserved variable that drives both a material's appearance and task reward should leave the discovered graph missing that variable, a failure no CMI threshold can repair.","The shortest-valid-span heuristic could be tested on a task with two independent causal paths of different lengths between the same pair of variables; under the stated assumptions only the shorter should be a true edge.","Since the paper says early spurious correlations may become genuine later, an online variant that periodically re-tests already-mastered edges is an untested but natural extension."],"forward_implications":["Temporally extended actions become explicit, learnable objects: an agent can plan over what causes what, and after how many steps, instead of only over abstract skills.","The reverse data collection strategy should reduce the number of environment interactions needed to identify a cause, because it samples exactly the pre-change window implied by $\\tau_{\\max}$.","CMI filtering should make causal discovery stable against common causes and autocorrelated variables, the two sources of spurious correlation the paper identifies.","The iterative loop provides a natural stopping condition: once the top-level network can recursively reach a target through its sub-goals, the causal chain is complete and no further discovery is needed.","The reported insensitivity to $\\tau_{\\max}$ across tasks suggests the method can be applied to very long delays at the cost of more parallel processes rather than worse accuracy."],"supporting_citations":[{"why":"CDHRL supplies the progressive causal-chain learning and hierarchical-policy construction that D3HRL adapts.","marker":"[25]"},{"why":"SDI provides the structural causal model training and causal-graph parameter learning procedure that D3HRL adapts.","marker":"[38]"},{"why":"Runge et al. supply the conditional independence testing framework and the $\\tau_{\\max}$ time-span notion used to define candidate edges.","marker":"[7]"},{"why":"This reference establishes that conditional independence is equivalent to zero conditional mutual information, the test D3HRL implements.","marker":"[39]"},{"why":"Options and semi-MDPs provide the temporal-abstraction foundation for modeling variable-length state transitions.","marker":"[1]"},{"why":"Factored-MDPs supply the state-decomposition principle used in Causal Factored-SMDPs.","marker":"[35]"},{"why":"Hindsight Experience Replay supplies the hindsight transition mechanism adapted for variable-length sub-goal training.","marker":"[42]"},{"why":"DQN is the base value-network algorithm used for the multi-level hierarchical policy.","marker":"[41]"},{"why":"2D-Minecraft provides one of the two long-horizon task environments used to evaluate the method.","marker":"[8]"},{"why":"MiniGrid provides the second long-horizon task environment used to evaluate the method.","marker":"[9]"}],"fun_headline_variants":["Causal RL: filtering spurious links, mastering delayed tasks","D3HRL: causal discovery cleans RL's spurious correlations","RL hierarchy built on true causal chains, spurious links dropped","Causal discovery filters spurious correlations for hierarchical RL"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method assumes that all causes of every relevant state variable are observed and that the state can be split into sub-states that do not influence each other; if a hidden cause drives two observed variables or sub-states interact, the discovered causal graph and the hierarchy built on it will be wrong.","fun_headline_variants_meta":{"raw":{"variants":["Causal RL: filtering spurious links, mastering delayed tasks","D3HRL: causal discovery cleans RL's spurious correlations","RL hierarchy built on true causal chains, spurious links dropped","Causal discovery filters spurious correlations for hierarchical RL"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00053,"raw_usage":{"total_tokens":2499,"prompt_tokens":837,"completion_tokens":1662,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":453,"completion_tokens_details":{"reasoning_tokens":1592}},"tokens_in":453,"tokens_out":1662,"duration_ms":11390,"temperature":1.0,"reasoning_tokens":1592,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T04:04:08.603417+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run D3HRL on a long-horizon task where a hidden confounder deterministically affects both a material's availability and the task reward, or where two supposedly independent sub-states interact. If the learned causal graph still shows low Structural Hamming Distance while the success ratio drops, the causal sufficiency or independence assumption is violated; if the graph contains extra edges pointing to the hidden cause, the CMI filter has not actually removed spurious correlations.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"CDHRL supplies the progressive causal-chain learning and hierarchical-policy construction that D3HRL adapts."},{"cited_title":"Runge, P","cited_arxiv_id":null,"evidence_quote":"Runge et al. supply the conditional independence testing framework and the $\\tau_{\\max}$ time-span notion used to define candidate edges."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Options and semi-MDPs provide the temporal-abstraction foundation for modeling variable-length state transitions."},{"cited_title":"Boutilier, T","cited_arxiv_id":null,"evidence_quote":"Factored-MDPs supply the state-decomposition principle used in Causal Factored-SMDPs."},{"cited_title":"Andrychowicz, F","cited_arxiv_id":null,"evidence_quote":"Hindsight Experience Replay supplies the hindsight transition mechanism adapted for variable-length sub-goal training."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"DQN is the base value-network algorithm used for the multi-level hierarchical policy."}],"review_version":1}