D3HRL: A Distributed Hierarchical Reinforcement Learning Approach Based on Causal Discovery and Spurious Correlation Detection

Chenran Zhao; Chunping Qiu; Dianxi Shi; Huanhuan Yang; Jianqiang Xia; Mengzhu Wang; Shaowu Yang; Songchang Jin

REVIEW 4 major objections 5 minor 45 references

D3HRL: A Distributed Hierarchical Reinforcement Learning Approach Based on Causal Discovery and Spurious Correlation Detection

T0 review · 4 major / 5 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Causal discovery plus independence testing lets hierarchical reinforcement learning learn delayed effects and discard spurious correlations.

desk verdict The paper's causal graph learning step is under-specified and likely wrong as written; the rest is a reasonable HRL+causality combination with addressable empirical gaps. read the letter →

arxiv 2505.01979 v1 pith:DU7ZSUAV submitted 2025-05-04 cs.LG

Chenran Zhao , Dianxi Shi , Mengzhu Wang , Jianqiang Xia , Huanhuan Yang , Songchang Jin , Shaowu Yang , Chunping Qiu This is my paper · ORCID

classification cs.LG

keywords HierarchicalReinforcementLearningCausaldiscoverySpuriouscorrelationdetectionConditionalindependencetestingTemporalabstractionSemi-MDPFactored-SMDPLong-horizontasks

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that the two main obstacles to long-horizon hierarchical reinforcement learning—delayed effects, where an action changes the state only after several steps, and spurious correlations, where two variables move together without a causal link—can be overcome by making the hierarchy itself a causal model. D3HRL treats each possible time delay as a separate causal relationship, trains structural causal models in parallel on data collected backwards from state changes, and then filters the discovered edges by conditional independence testing. Only the surviving causal relationships become sub-goals and policy layers. If correct, this yields accurate causal chains and faster training in tasks like gathering and crafting in 2D-Minecraft and MiniGrid.

What carries the argument

Causal Factored-SMDP—a five-component formalization $\langle C, E, P, T, F \rangle$ that turns state transitions into SCM generating functions over causal relationships with time spans—and the three-module loop built on it. Distributed SCM training assigns one process per candidate time span $h$, uses reverse data collection from intervention points to assemble $2\tau_{\max}+1$-length windows, and learns both a generative network and a causal-graph probability matrix $\sigma(\eta^h)$. Spurious correlation detection then estimates the conditional mutual information of each candidate edge and keeps only edges above threshold $\epsilon_{\text{cmi}}$; if an edge survives at several time spans, the shortest span is chosen as true. The resulting hierarchy grows one DQN sub-goal network per verified effect, with hindsight transitions of length $h+1$ for training.

What would settle it

Run D3HRL on a long-horizon task where a hidden confounder deterministically affects both a material's availability and the task reward, or where two supposedly independent sub-states interact. If the learned causal graph still shows low Structural Hamming Distance while the success ratio drops, the causal sufficiency or independence assumption is violated; if the graph contains extra edges pointing to the hidden cause, the CMI filter has not actually removed spurious correlations.

Watch

Extended reading notes

Core claim

On its own terms, the paper's central claim is that variable-length state transitions are causal relationships across time spans, so they can be learned by distributed structural causal model training, and that spurious correlations produced by common causes and autocorrelation can be removed by conditional independence testing via conditional mutual information. The method formalizes this as Causal Factored-SMDPs, in which the state is decomposed into independent sub-states, the transition rules are the generating functions of an SCM, and the hierarchical policy mirrors the discovered causal chain. In the experiments, D3HRL reaches higher average success ratios than CDHRL, HAC, Option-Critic, and LESSON on modified tasks, and its learned causal graphs have lower Structural Hamming Distance on those tasks.

Load-bearing premise

The method assumes that all causes of every relevant state variable are observed and that the state can be split into sub-states that do not influence each other; if a hidden cause drives two observed variables or sub-states interact, the discovered causal graph and the hierarchy built on it will be wrong.

Editorial extensions

If this is right

Temporally extended actions become explicit, learnable objects: an agent can plan over what causes what, and after how many steps, instead of only over abstract skills.
The reverse data collection strategy should reduce the number of environment interactions needed to identify a cause, because it samples exactly the pre-change window implied by $\tau_{\max}$.
CMI filtering should make causal discovery stable against common causes and autocorrelated variables, the two sources of spurious correlation the paper identifies.
The iterative loop provides a natural stopping condition: once the top-level network can recursively reach a target through its sub-goals, the causal chain is complete and no further discovery is needed.
The reported insensitivity to $\tau_{\max}$ across tasks suggests the method can be applied to very long delays at the cost of more parallel processes rather than worse accuracy.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

A direct extension would replace the DQN base with any off-policy value method and test whether the causal-chain construction transfers; the paper does not claim this.
A hidden-confounder experiment would probe the causal sufficiency assumption directly: adding an unobserved variable that drives both a material's appearance and task reward should leave the discovered graph missing that variable, a failure no CMI threshold can repair.
The shortest-valid-span heuristic could be tested on a task with two independent causal paths of different lengths between the same pair of variables; under the stated assumptions only the shorter should be a true edge.
Since the paper says early spurious correlations may become genuine later, an online variant that periodically re-tests already-mastered edges is an untested but natural extension.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Short version: D3HRL is a credible attempt to add causal discovery to hierarchical RL, specifically to handle variable-length state transitions and spurious correlations, and it shows promising numbers on two custom benchmark tasks. What I found new is the combination of distributed SCM training across time spans, a CMI-based spurious-correlation filter, and a reverse data collection strategy. That combination is not in the CDHRL or SDI work they build on. The paper is also unusually honest about its assumptions: it explicitly restricts itself to decouplable state variables and notes that random time spans are out of scope.

The main problem is Equation 8. This is the gradient update for the causal graph parameter matrix η_h, and everything downstream—the candidate edges, the CIT, the subgoal construction—reads from σ(η_h). As written, the update weights the score term (σ(η)−P_n) by a softmax over data batches, not by the likelihood of each sampled graph. That is not a valid score-function estimator, and no derivation or synthetic validation is given. I can't tell if it's a typo or the actual procedure, but either way the causal-discovery claim is unverifiable without a corrected equation or code. That is a load-bearing gap, not a cosmetic one.

The empirical section has the usual issues: ε_cmi=0.05 was tuned in a pre-experiment on the same tasks; the baselines were modified with hand-designed curricula; the 5-seed averages come without error bars; and there is no code or data release. Those are addressable, and they don't by themselves sink the paper. The SHD numbers favor D3HRL and the CMI plots make sense qualitatively. The reverse-vs-forward comparison is a nice ablation.

Who is this for? People doing causal RL or HRL for long-horizon tasks will want to read it, especially if they care about delayed effects. It deserves a serious referee, but the referee should push hard on Eq. 8. I would not cite it myself until the gradient issue is cleared up. My recommendation: send to peer review, but with the expectation that the authors either fix Eq. 8 with a proper derivation or release code that reproduces Table 1.

Referee Report

4 major / 5 minor

Summary. The paper introduces D3HRL, a hierarchical reinforcement learning method that combines distributed causal discovery with spurious-correlation filtering. Its three modules are: (A) reverse data collection with parallel SCM training over time spans 1..tau_max (Eqs. 4-8); (B) conditional independence testing via CMI to filter spurious edges and select true time spans (Eqs. 9-11); and (C) construction and training of hierarchical DQN sub-goal networks based on the confirmed causal chain (Section 5.3). The experiments compare D3HRL against CDHRL, HAC, Option-Critic, and LESSON on modified 2D-Minecraft and MiniGrid tasks, reporting ASR, ADC, SHD against CDHRL, an ablation of reverse vs forward collection, CMI curves, and scalability for tau_max up to 16.

Significance. If the causal discovery step is correct, the paper proposes a meaningful integration of time-series causal discovery with HRL, with a concrete mechanism for delay effects and spurious correlations, and the reported SHD values are internally consistent with the claimed advantage over CDHRL. The strengths include the reverse data-collection ablation, the use of SHD with hand-specified ground-truth graphs, and the evaluation across multiple task configurations. However, the significance is conditional: the central update in Eq. (8) is not a valid maximum-likelihood estimator as written, the CIT threshold is tuned in a pre-experiment without reported sensitivity, and the baselines are modified with hand-designed curricula, so the current evidence does not establish the headline claim.

major comments (4)

[5.1.2, Eq. (8)] Equation (8) does not define a valid stochastic-gradient estimator for the marginal data likelihood under the graph prior sigma(eta). In a REINFORCE/softmax estimator, the score (sigma(eta)-P_n) for a sampled graph P_n must be weighted by that sample's own likelihood exp(L(P_n,D_k)), or by a baseline, so that low-likelihood samples are down-weighted relative to high-likelihood ones. As written, the exponential weight is exp(sum_n L(P_n^h,D_k)) and is constant across the N_p sampled graphs inside the inner sum, while the normalizer runs over data batches k'. Consequently, the update does not maximize the likelihood of the observed data under the sampled causal graphs, and the causal structure read from sigma(eta_h) after thresholding (Section 5.1.2, threshold 0.8) is not a maximum-likelihood graph. This is load-bearing because Module A's output feeds CIT (Section 5.2) and the hierarchical construction (Section 5.3). Please provide a derivation of Eq. (8), correct it to a valid estimator, and validate it on a synthetic graph with known ground truth; releasing the code would also settle whether the reported SHD results come from this update.
[6.2.5, Eq. (11)] The spurious-correlation filter uses epsilon_cmi = 0.05 to declare genuine causal relationships, and the authors state that this threshold was selected in a separate statistical experiment before the main experiments. Since the threshold directly controls which edges survive into the SHD computation and which time spans are accepted, and since the experiments are conducted on the same task family (GetIron, Wood2Wet), the reported SHD and success curves are at risk of selection bias. The manuscript should report the pre-experiment procedure and show sensitivity of Table 1 and Figure 11 to epsilon_cmi (e.g., a sweep), or justify that the threshold is task-independent.
[6.1, Figure 3] The baselines are modified with hand-designed curricula or with tau_max-step transition collection to adapt to variable-length transitions, which makes the comparison in Figure 3 hard to interpret. For example, HAC is said to use a pre-defined curriculum to acquire causality, and Option-Critic/LESSON receive curriculum modifications, so it is unclear whether the observed gap reflects D3HRL's causal-discovery modules or the authors' enhancements. In addition, all results are averaged over only 5 seeds and no variance or confidence intervals are shown for ASR/ADC (Figure 3) or sub-goal efficiency (Figure 10), so the significance of the improvements cannot be assessed. Please report per-seed variability and clearly separate the baseline versions used from the original algorithms.
[5.2.2] The rule 'if Xj->Xi is judged valid across multiple different time spans, the shortest one is identified as true' is stated without proof and is load-bearing for the time-span matrix T used in hierarchical training. Under the Causal Factored-SMDP assumptions, with autocorrelated effect variables and multiple causes, it is not generally true that only the shortest valid span is causal; a longer span could reflect a genuinely distinct delayed mechanism. Please provide a proof or an empirical validation (e.g., synthetic TSGM with known spans) that this rule holds for the class of tasks considered.

minor comments (5)

[6.2.4] The heading reads 'DEHRL' but should be 'D3HRL'.
[Table 1] The numeric columns under the combined header 'Task (tau_max)' lack individual column labels, so the reader must infer which values belong to GetIron-R0 versus Wood2Wet and which belong to each tau_max setting; please make the header explicit.
[6.2.5] The text refers to 'the second subplot in Fig 4' when discussing CMI values, but the CMI results appear in Figures 8 and 9; the figure reference should be corrected.
[5.2.1] The phrase 'causality stone <- A -> wood' uses an informal notation that is inconsistent with the directed-edge notation used elsewhere; please re-express it with the standard arrow notation.
[6.2.5] The statement that spurious correlations have CMI values of 0 should be quantified, since the plotted values in Figure 9 may be small but not exactly zero; please report the numerical range or a tolerance.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: causal-graph discovery and hierarchical-policy results are tested against externally specified ground truth and external baselines.

full rationale

The paper's central claims are empirical: D3HRL learns causal relationships from environment interaction and then builds hierarchical policies. The SHD metric in Table 1 is computed against hand-specified ground-truth causal graphs for the modified 2D-Minecraft and MiniGrid tasks, not against the output of the algorithm's own fitting procedure, so the causal-graph accuracy claim is not self-defined. The success-rate comparisons in Figure 3 are against external baselines (CDHRL, HAC, Option-Critic, LESSON), and the policy construction uses the learned graph as an input rather than as the evaluation target. The spurious-correlation filter (Eqs. 10-11) is a conditional-independence test with an explicit threshold epsilon_cmi; even though epsilon_cmi was chosen in a separate statistical experiment and the conditioning sets PA(...) come from the same learned SCM, this is a standard hyperparameter/estimation choice and does not make the inferred graph equal to the input by construction. References [25] and [38] are external works adapted for SCM training, not self-citations, and no uniqueness theorem is invoked. The suspicious form of the gradient update in Eq. 8 and the potential for threshold overfitting are correctness/robustness concerns, not circularity; they do not reduce the claimed prediction to its own inputs.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The paper introduces a new learning procedure and formal objects (Causal Factored-SMDP, causal relationship matrices, time span matrices) but no new physical or ontological entities, so the invented-entities ledger is empty.

free parameters (3)

tau_max (maximum time span) = 4 in most experiments; 1, 8, 16 in Wood2Wet scalability experiments
Section 3.2 calls tau_max 'an empirical value that incorporates prior knowledge'. It controls the number of parallel causal-discovery processes and the data collection window.
epsilon_cmi (CIT threshold) = 0.05
Section 6.2.5 states epsilon_cmi was determined through a separate statistical experiment before the main experiments. It directly decides which correlations are kept, so it is a fitted threshold.
sigma threshold for causal edges = 0.8
Section 5.1.2 says if sigma(eta_ij^h) >= 0.8 then a causal edge exists, following CDHRL [25]. It is inherited from prior work but is still a hand-set threshold.

assumptions (4)

domain assumption The state can be decomposed into disjoint sub-states and the transitions of each sub-state are independent.
Assumption (1) in Section 4. D3HRL's factored causal discovery and hierarchical policy construction rely on this decomposition; interacting sub-states would break the per-effect SCM training.
domain assumption Causal sufficiency: all relevant causal factors are observed.
Assumption (3) in Section 4. If a hidden confounder exists, the CMI-based CIT can mistake a spurious correlation for a causal edge.
domain assumption Causal Markov condition: true causes are sufficient to predict the effect, and all variables except actions exhibit autocorrelation.
Assumption (4) in Section 4. This justifies conditioning on parents to block backdoor paths in Section 3.3.
ad hoc to paper If a causal relationship appears valid across multiple time spans, the shortest span is the true one.
Section 5.2.2 asserts indirect causation through autocorrelation makes longer spans spurious, so the shortest span is chosen. The conclusion admits the method struggles with randomly distributed time spans, so this heuristic is load-bearing and not generally valid.

how reviews work

0 comments

Cite this review

Pith. "Pith review of D3HRL: A Distributed Hierarchical Reinforcement Learning Approach Based on Causal Discovery and Spurious Correlation Detection." pith.science (2026). https://pith.science/paper/DU7ZSUAV

@misc{pith2026250501979,
  author       = {Pith},
  title        = {Pith review of: D3HRL: A Distributed Hierarchical Reinforcement Learning Approach Based on Causal Discovery and Spurious Correlation Detection},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/DU7ZSUAV}},
  note         = {Machine review of arXiv:2505.01979}
}

read the original abstract

Current Hierarchical Reinforcement Learning (HRL) algorithms excel in long-horizon sequential decision-making tasks but still face two challenges: delay effects and spurious correlations. To address them, we propose a causal HRL approach called D3HRL. First, D3HRL models delayed effects as causal relationships across different time spans and employs distributed causal discovery to learn these relationships. Second, it employs conditional independence testing to eliminate spurious correlations. Finally, D3HRL constructs and trains hierarchical policies based on the identified true causal relationships. These three steps are iteratively executed, gradually exploring the complete causal chain of the task. Experiments conducted in 2D-MineCraft and MiniGrid show that D3HRL demonstrates superior sensitivity to delay effects and accurately identifies causal relationships, leading to reliable decision-making in complex environments.

Figures

Figures reproduced from arXiv: 2505.01979 by the authors.

**Figure 2.** Overview of D3HRL framework. 5.1. Distributed Causal Discovery To capture the variable-length state transitions, we propose a distributed causal discovery method consisting of distributed data collection and distributed SCM training. Based on the maximum time span τmax, multiple processes {rankh} τmax h=1 are set up for distributed learning. 5.1.1. Distributed Data Collection. Each process rankh interacts with the e… view at source ↗

**Figure 3.** ASR and ADC under MiniGrid and MineCraft tasks. [PITH_FULL_IMAGE:figures/full_fig_p014_3.png] view at source ↗

Figures from the paper (9 more)

**Figure 4.** Figure 4: Each figure contains up to 8 curves, representing the causality existence [PITH_FULL_IMAGE:figures/full_fig_p015_4.png]

**Figure 4.** Figure 4: The comparison between reverse and forward data collection strategy in GetIron [PITH_FULL_IMAGE:figures/full_fig_p016_4.png]

**Figure 5.** Figure 5: The causal graph matrices of CDHRL in GetIron-R0-T0 task (up) and GetIron [PITH_FULL_IMAGE:figures/full_fig_p017_5.png]

**Figure 6.** Figure 6: The causal graph matrices of D3HRL in GetIron-R0-T0 task (up) and GetIron [PITH_FULL_IMAGE:figures/full_fig_p017_6.png]

**Figure 7.** Figure 7: Experimental Results. The results indicate that D3HRL consistently produces accurate causal graphs across different configurations. This clearly demonstrates D3HRL’s robust generalization in causal graph identification [PITH_FULL_IMAGE:figures/full_fig_p018_7.png]

**Figure 8.** Figure 8: The CMI of different time spans of causal relationships in GetIron-R0-T1 task with τmax = 4 [PITH_FULL_IMAGE:figures/full_fig_p019_8.png]

**Figure 9.** Figure 9: The CMI of different time spans of spurious correlations in GetIron-R0-T1 task with τmax = 4. 19 [PITH_FULL_IMAGE:figures/full_fig_p019_9.png]

**Figure 10.** Figure 10: Comparison of sub-goal training efficiency in GetIron-R0-T0 with [PITH_FULL_IMAGE:figures/full_fig_p020_10.png]

**Figure 11.** Figure 11: D3HRL’s ASR in MiniGrid-Wood2Wet task with varying [PITH_FULL_IMAGE:figures/full_fig_p021_11.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

45 extracted references · 26 canonical work pages

[1]

R. S. Sutton, D. Precup, S. Singh, Between mdps and semi-mdps: A frame- work for temporal abstraction in reinforcement learning, Artificial intelligence 112 (1-2) (1999) 181–211

work page 1999
[2]

Schaul, D

T. Schaul, D. Horgan, K. Gregor, D. Silver, Universal value function approxi- mators, in: International conference on machine learning, PMLR, 2015, pp. 1312–1320

work page 2015
[3]

Röder, M

F. Röder, M. Eppe, P. D. Nguyen, S. Wermter, Curious hierarchical actor- critic reinforcement learning, in: Artificial Neural Networks and Machine Learning–ICANN 2020: 29th International Conference on Artificial Neural Networks, Bratislava, Slovakia, September 15–18, 2020, Proceedings, Part II 29, Springer, 2020, pp. 408–419

work page 2020
[4]

Campos, A

V . Campos, A. Trott, C. Xiong, R. Socher, X. Giró-i Nieto, J. Torres, Ex- plore, discover and learn: Unsupervised discovery of state-covering skills, in: International Conference on Machine Learning, PMLR, 2020, pp. 1317–1327

work page 2020
[5]

Willig, M

M. Willig, M. Zecevic, D. S. Dhami, K. Kersting, Causal parrots: Large language models may talk causality but are not causal, preprint 8 (2023)

work page 2023
[6]

Z. Deng, J. Jiang, G. Long, C. Zhang, Causal reinforcement learning: A survey, arXiv preprint arXiv:2307.01452 (2023)

arXiv 2023
[7]

Runge, P

J. Runge, P. Nowack, M. Kretschmer, S. Flaxman, D. Sejdinovic, Detecting and quantifying causal associations in large nonlinear time series datasets, Science advances 5 (11) (2019) eaau4996

work page 2019
[8]

S. Sohn, J. Oh, H. Lee, Hierarchical reinforcement learning for zero-shot generalization with subtask dependencies, Advances in neural information processing systems 31 (2018)

2018

Show all 45 references

[9]

Chevalier-Boisvert, B

M. Chevalier-Boisvert, B. Dai, M. Towers, R. de Lazcano, L. Willems, S. Lahlou, S. Pal, P. S. Castro, J. Terry, Minigrid & miniworld: Modu- lar & customizable reinforcement learning environments for goal-oriented tasks, CoRR abs/2306.13831 (2023)

2023 arXiv
[10]

Eysenbach, A

B. Eysenbach, A. Gupta, J. Ibarz, S. Levine, Diversity is all you need: Learn- ing skills without a reward function, arXiv preprint arXiv:1802.06070 (2018). 22

2018 arXiv
[11]

Achiam, H

J. Achiam, H. Edwards, D. Amodei, P. Abbeel, Variational option discovery algorithms, arXiv preprint arXiv:1807.10299 (2018)

2018 arXiv
[12]

Frans, J

K. Frans, J. Ho, X. Chen, P. Abbeel, J. Schulman, Meta learning shared hierarchies, arXiv preprint arXiv:1710.09767 (2017)

2017 arXiv
[13]

Y . Song, J. Wang, T. Lukasiewicz, Z. Xu, M. Xu, Diversity-driven extensible hierarchical reinforcement learning, in: Proceedings of the AAAI conference on artificial intelligence, V ol. 33, 2019, pp. 4992–4999

2019
[14]

S. Song, J. Weng, H. Su, D. Yan, H. Zou, J. Zhu, Playing fps games with environment-aware hierarchical reinforcement learning., in: IJCAI, 2019, pp. 3475–3482

2019
[15]

S. Park, D. Ghosh, B. Eysenbach, S. Levine, Hiql: Offline goal-conditioned rl with latent states as actions, Advances in Neural Information Processing Systems 36 (2024)

2024
[16]

Q. Zou, E. Suzuki, Sample-efficient goal-conditioned reinforcement learning via predictive information bottleneck for goal representation learning, in: 2023 IEEE International Conference on Robotics and Automation (ICRA), IEEE, 2023, pp. 9523–9529

2023
[17]

Y . Li, Y . Wang, X. Tan, Highly valued subgoal generation for efficient goal- conditioned reinforcement learning, Neural Networks 181 (2025) 106825

2025
[18]

W. Ou, B. Luo, B. Wang, Y . Zhao, Modular hierarchical reinforcement learning for multi-destination navigation in hybrid crowds, Neural Networks 171 (2024) 474–484

2024
[19]

Foerster, G

J. Foerster, G. Farquhar, T. Afouras, N. Nardelli, S. Whiteson, Counterfactual multi-agent policy gradients, in: Proceedings of the AAAI conference on artificial intelligence, V ol. 32, 2018

2018
[20]

Buesing, T

L. Buesing, T. Weber, Y . Zwols, S. Racaniere, A. Guez, J.-B. Lespiau, N. Heess, Woulda, coulda, shoulda: Counterfactually-guided policy search, arXiv preprint arXiv:1811.06272 (2018)

2018 arXiv
[21]

Madumal, T

P. Madumal, T. Miller, L. Sonenberg, F. Vetere, Explainable reinforcement learning through a causal lens, in: Proceedings of the AAAI conference on artificial intelligence, V ol. 34, 2020, pp. 2493–2500. 23

2020
[22]

Herlau, R

T. Herlau, R. Larsen, Reinforcement learning of causal variables using medi- ation analysis, in: Proceedings of the AAAI Conference on Artificial Intelli- gence, V ol. 36, 2022, pp. 6910–6917

2022
[23]

W. Ding, H. Lin, B. Li, D. Zhao, Generalizing goal-conditioned reinforcement learning with variational causal reasoning, Advances in Neural Information Processing Systems 35 (2022) 26532–26548

2022
[24]

Corcoll, R

O. Corcoll, R. Vicente, Disentangling controlled effects for hierarchical reinforcement learning, in: Conference on Causal Learning and Reasoning, PMLR, 2022, pp. 178–200

2022
[25]

X. Hu, R. Zhang, K. Tang, J. Guo, Q. Yi, R. Chen, Z. Du, L. Li, Q. Guo, Y . Chen, et al., Causality-driven hierarchical structure discovery for reinforce- ment learning, Advances in Neural Information Processing Systems 35 (2022) 20064–20076

2022
[26]

T. E. Lee, S. Vats, S. Girdhar, O. Kroemer, Scale: Causal learning and discovery of robot manipulation skills using simulation, in: CoRL 2023 Workshop on Learning Effective Abstractions for Planning (LEAP), 2023

2023
[27]

Chuck, K

C. Chuck, K. Black, A. Arjun, Y . Zhu, S. Niekum, Granger causal interaction skill chains, Transactions on Machine Learning Research
[28]

B. Chen, Z. Cao, W. Mayer, M. Stumptner, R. Kowalczyk, Hcpi-hrl: Human causal perception and inference-driven hierarchical reinforcement learning, Neural Networks 187 (2025) 107318

2025
[29]

M. H. Nguyen, H. Le, S. Venkatesh, Variable-agnostic causal exploration for reinforcement learning, in: Joint European Conference on Machine Learning and Knowledge Discovery in Databases, Springer, 2024, pp. 216–232

2024
[30]

Pearl, Models, reasoning and inference, Cambridge, UK: CambridgeUni- versityPress 19 (2) (2000) 3

J. Pearl, Models, reasoning and inference, Cambridge, UK: CambridgeUni- versityPress 19 (2) (2000) 3

2000
[31]

Runge, S

J. Runge, S. Bathiany, E. Bollt, G. Camps-Valls, D. Coumou, E. Deyle, C. Glymour, M. Kretschmer, M. D. Mahecha, J. Muñoz-Marí, et al., Inferring causation from time series in earth system sciences, Nature communications 10 (1) (2019) 2553. 24

2019
[32]

Peters, D

J. Peters, D. Janzing, B. Schölkopf, Elements of causal inference: foundations and learning algorithms, The MIT Press, 2017

2017
[33]

C. W. Granger, Investigating causal relations by econometric models and cross-spectral methods, Econometrica: journal of the Econometric Society (1969) 424–438

1969
[34]

Eichler, Graphical modelling of multivariate time series, Probability Theory and Related Fields 153 (2012) 233–268

M. Eichler, Graphical modelling of multivariate time series, Probability Theory and Related Fields 153 (2012) 233–268

2012
[35]

Boutilier, T

C. Boutilier, T. Dean, S. Hanks, Decision-theoretic planning: Structural assumptions and computational leverage, Journal of Artificial Intelligence Research 11 (1999) 1–94

1999
[36]

Boutilier, R

C. Boutilier, R. Dearden, M. Goldszmidt, Stochastic dynamic programming with factored representations, Artificial intelligence 121 (1-2) (2000) 49–107

2000
[37]

Spirtes, C

P. Spirtes, C. Glymour, R. Scheines, Causation, prediction, and search, MIT press, 2001

2001
[38]

N. R. Ke, O. Bilaniuk, A. Goyal, S. Bauer, H. Larochelle, B. Schölkopf, M. C. Mozer, C. Pal, Y . Bengio, Learning neural causal models from unknown interventions, arXiv preprint arXiv:1910.01075 (2019)

2019 arXiv
[39]

Z. Wang, X. Xiao, Z. Xu, Y . Zhu, P. Stone, Causal dynamics learning for task-independent state abstraction, arXiv preprint arXiv:2206.13452 (2022)

2022 arXiv
[40]

Runge, Causal network reconstruction from time series: From theoretical assumptions to practical estimation, Chaos: An Interdisciplinary Journal of Nonlinear Science 28 (7) (2018)

J. Runge, Causal network reconstruction from time series: From theoretical assumptions to practical estimation, Chaos: An Interdisciplinary Journal of Nonlinear Science 28 (7) (2018)

2018
[41]

V . Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. Riedmiller, A. K. Fidjeland, G. Ostrovski, et al., Human- level control through deep reinforcement learning, nature 518 (7540) (2015) 529–533

2015
[42]

Andrychowicz, F

M. Andrychowicz, F. Wolski, A. Ray, J. Schneider, R. Fong, P. Welinder, B. McGrew, J. Tobin, O. Pieter Abbeel, W. Zaremba, Hindsight experience replay, Advances in neural information processing systems 30 (2017)

2017
[43]

A. Levy, G. Konidaris, R. Platt, K. Saenko, Learning multi-level hierarchies with hindsight, arXiv preprint arXiv:1712.00948 (2017). 25

2017 arXiv
[44]

Bacon, J

P.-L. Bacon, J. Harb, D. Precup, The option-critic architecture, in: Proceed- ings of the AAAI conference on artificial intelligence, V ol. 31, 2017

2017
[45]

W. Kim, J. Kim, Y . Sung, Lesson: learning to integrate exploration strate- gies for reinforcement learning via an option framework, arXiv preprint arXiv:2310.03342 (2023). 26

2023 arXiv

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.

[1] [1]

R. S. Sutton, D. Precup, S. Singh, Between mdps and semi-mdps: A frame- work for temporal abstraction in reinforcement learning, Artificial intelligence 112 (1-2) (1999) 181–211

work page 1999

[2] [2]

Schaul, D

T. Schaul, D. Horgan, K. Gregor, D. Silver, Universal value function approxi- mators, in: International conference on machine learning, PMLR, 2015, pp. 1312–1320

work page 2015

[3] [3]

Röder, M

F. Röder, M. Eppe, P. D. Nguyen, S. Wermter, Curious hierarchical actor- critic reinforcement learning, in: Artificial Neural Networks and Machine Learning–ICANN 2020: 29th International Conference on Artificial Neural Networks, Bratislava, Slovakia, September 15–18, 2020, Proceedings, Part II 29, Springer, 2020, pp. 408–419

work page 2020

[4] [4]

Campos, A

V . Campos, A. Trott, C. Xiong, R. Socher, X. Giró-i Nieto, J. Torres, Ex- plore, discover and learn: Unsupervised discovery of state-covering skills, in: International Conference on Machine Learning, PMLR, 2020, pp. 1317–1327

work page 2020

[5] [5]

Willig, M

M. Willig, M. Zecevic, D. S. Dhami, K. Kersting, Causal parrots: Large language models may talk causality but are not causal, preprint 8 (2023)

work page 2023

[6] [6]

Z. Deng, J. Jiang, G. Long, C. Zhang, Causal reinforcement learning: A survey, arXiv preprint arXiv:2307.01452 (2023)

arXiv 2023

[7] [7]

Runge, P

J. Runge, P. Nowack, M. Kretschmer, S. Flaxman, D. Sejdinovic, Detecting and quantifying causal associations in large nonlinear time series datasets, Science advances 5 (11) (2019) eaau4996

work page 2019

[8] [8]

S. Sohn, J. Oh, H. Lee, Hierarchical reinforcement learning for zero-shot generalization with subtask dependencies, Advances in neural information processing systems 31 (2018)

2018