Pith. sign in

REVIEW 3 major objections 5 minor 1 cited by

Towards Empowerment Gain through Causal Structure Learning in Model-Based RL

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that empowerment-driven exploration under a learned causal mask improves both causal discovery and policy learning in model-based RL.

desk verdict A plausible causal-MBRL loop with a real gap: the implemented exploration objective is a KL disagreement, not empowerment gain, so the central mechanism is unverified. read the letter →

arxiv 2502.10077 v1 pith:FGA7PEOX submitted 2025-02-14 cs.AI cs.LG

Hongye Cao , Fan Feng , Meng Fang , Shaokang Dong , Tianpei Yang , Jing Huo , Yang Gao This is my paper
classification cs.AIcs.LG
keywords model-basedreinforcementlearningcausalstructureempowermentintrinsicmotivationdiscoveryexplorationstateabstraction
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

ECL (Empowerment through Causal Learning) is a framework for model-based reinforcement learning that couples empowerment, the mutual information between an agent's actions and its next state, with a learned causal mask over state and action dimensions. The paper argues that an agent can bootstrap: use the causal structure to seek controllable, informative experience, and use that experience to correct the causal structure. If correct, this closes a loop that existing causal model-based RL methods leave open, because those methods treat the causal structure as given rather than as something exploration should actively improve. Across six environments, including pixel-based ones, ECL reports more accurate causal graphs, better sample efficiency, and higher asymptotic task reward than dense dynamics models and prior causal approaches.

What carries the argument

The central object is the causal mask, a binary adjacency matrix over state and action dimensions that encodes which variables causally influence each next-state dimension and supplies a state abstraction for the reward model. The argument runs on a three-step loop: first train a dense dynamics model and learn the mask, then maximize the empowerment gain between the causal and dense models to collect controllable trajectories that update the mask and reward model while the dynamics model stays fixed, and finally learn the downstream task policy with a curiosity reward that balances causal and dense predictions to prevent overfitting.

What would settle it

Run ECL on an environment whose transition function changes after Step 1, for example a state dimension that was inert becomes action-controlled partway through training, with the true graph held out; if the fixed dense dynamics model cannot represent the new dependence, the causal mask should fail to recover the new edge and downstream reward should drop, while a variant that retrains the dynamics model throughout should not.

Watch

Extended reading notes

Core claim

The central claim is that the gain in empowerment between a causal and a dense dynamics model is the right signal to drive exploration for causal discovery: maximizing the difference in empowerment, approximated through a KL term, focuses data collection on dimensions the agent can actually influence, and the collected data then refines the causal mask and reward model. The paper states that this loop yields better causal discovery accuracy, sample efficiency, and asymptotic policy performance than other causal model-based RL methods, including in latent-state pixel tasks. The framework is method-agnostic: it can integrate both constraint-based conditional-independence testing and score-based sparse-mask learning.

Load-bearing premise

The load-bearing premise is that the dense dynamics model trained in Step 1 stays accurate enough for the whole mask-empowerment loop, since the paper fixes that model and updates only the causal mask and reward model; if it is misspecified, especially with latent encoders in pixel tasks, the exploration policy and mask can chase model error rather than true causal structure.

Editorial extensions

If this is right

  • Causal discovery scores improve, with the paper reporting F1 and ROC AUC above 0.90 across all chemical environments and perfect scores on the physical environment.
  • Learning curves show higher episodic reward at fixed episode counts, indicating better sample efficiency, with the largest gains in manipulation tasks.
  • Task success rises faster than with prior causal model-based RL in collider and manipulation-reach settings, consistent with the empowerment bonus steering the agent away from uninformative regions.
  • One-step and multi-step prediction accuracy is maintained under out-of-distribution states, while dense dynamics models degrade sharply.
  • The framework transfers to pixel-based tasks by operating on latent states from visual encoders, outperforming the visual baseline in average return on the Robodesk task.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The practical empowerment objective drops two entropy terms when deriving Eq. 10; retaining H(st+1|st;M) - H(st+1|st) could push exploration toward higher-diversity regions, giving a direct testable variant.
  • Because the paper claims ECL is method-agnostic, a natural stress test is to plug in a nonlinear, high-dimensional score-based discovery method and check whether the mask-empowerment loop still beats running that discovery method alone.
  • The out-of-distribution prediction gains suggest the causal mask acts as a sparsity regularizer; ablating the mask and measuring OOD error as a function of mask sparsity would separate the empowerment bonus from the mask's own effect.
  • The fixed dynamics model in Step 2 implies the framework's robustness depends on the initial model's coverage; a variant that periodically retrains the dense model could test how much of the reported gain comes from the fixed-model assumption.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes ECL, a framework for model-based RL that couples causal structure learning with empowerment-driven exploration. The method operates in three steps: (1) learn a dense dynamics model and a causal mask via constraint-based or score-based causal discovery, together with a reward model over the masked state abstraction; (2) alternate between optimizing an empowerment-driven exploration policy that maximizes the difference between the causal and dense dynamics models, and updating the causal mask and reward model with the collected data; (3) learn a downstream task policy using the refined causal dynamics together with a curiosity-shaped reward. Experiments on Chemical, Manipulation, Physical, and three pixel-based environments report improvements over several causal and dense MBRL baselines in episodic reward, sample efficiency, and causal graph accuracy, and the paper claims that ECL is method-agnostic across causal discovery algorithms.

Significance. The idea of using learned causal structure to focus empowerment-driven exploration, and then using the collected higher-quality data to refine the causal model, is a valuable and timely direction for MBRL. The paper provides code, a project page, and experiments across six environments, and it demonstrates that two different causal discovery backends can be plugged into the framework. The main significance is conditional on whether the implemented exploration objective actually corresponds to empowerment gain: as written, the entropy terms of the empowerment objective are omitted, so the reported gains may stem from a curiosity-style model-disagreement bonus rather than from the proposed causal-empowerment mechanism. If the authors can close this gap with ablations and additional evidence, the framework would be a solid contribution to causal MBRL and intrinsically motivated exploration.

major comments (3)
  1. [Section 3.2, Eq. (10), Appendix A] The implemented Step-2 objective is not empowerment gain as claimed. Eq. (10) decomposes the empowerment difference into H(s'|s;M) - H(s'|s) plus a KL term, but the text immediately states "For simplicity, we update πe by optimizing the KL term," and Appendix A concedes "we currently omit two entropy terms." These entropy terms are precisely the empowerment-relevant parts of the objective; dropping them reduces the exploration policy to maximizing a KL disagreement between a fixed dense model and its masked version. Such a policy can be maximized by seeking states where the two predictors disagree due to model error, without increasing the mutual information between actions and future states. Since the paper's headline claim is that ECL boosts empowerment gain to improve exploration and causal discovery, the reported gains may be attributable to a curiosity-like disagreement bonus rather than to empowerment. Please provide an ablation comparing the KL-only objective with the full Eq. (10) (or a direct MI estimate), and report the actual empowerment values achieved by the exploration policy.
  2. [Section 3.2, Step 2, Appendix D.2.1] The Step-2 loop fixes the dynamics model Pφc learned in Step 1 and updates only the causal mask and reward model. If Pφc is misspecified, particularly in pixel-based tasks where the dynamics operate on IFactor latent encoders, the KL term in the Step-2 objective can be maximized by exploiting regions where the dense and masked models disagree because of model error, rather than by identifying true causal structure. The paper does not test the sensitivity of the Step-2 alternating procedure to the quality of the frozen dynamics model, nor does it compare against a variant that retrains Pφc during Step 2. Please add such an ablation or present empirical evidence that the fixed-model assumption does not dominate the reported gains.
  3. [Section 5.2.2, Table 1] The causal graph metrics are near ceiling for both ECL and the baselines: for example, all F1 scores in Table 1 are at least 0.93, and many values overlap within one standard error. The claim that ECL "outperforms other causal MBRL methods" in causal discovery accuracy is therefore only weakly supported by this table; the differences are small and may not be statistically significant. Please report significance tests or effect sizes, or use a benchmark where causal discovery is not saturated, so that the comparative claim can be properly assessed.
minor comments (5)
  1. [Abstract and Section 5.1] The abstract states that ECL is evaluated across "6 environments," but Section 5.1 lists three basic environments (Chemical, Manipulation, Physical) and three pixel-based environments (Modified Cartpole, Robodesk, DMC); clarify whether DMC's three tasks count as separate environments or as one environment with multiple tasks.
  2. [Equations (8) and (9)] The notation "max_{a∼πe(a|s)}" in Eqs. (8) and (9) is awkward because the maximum should be over the policy πe, not over actions sampled from πe; please rephrase the notation to make the optimization variable explicit.
  3. [Section 4] There is a typo in the opening sentence of Section 4: "casual dynamics learning" should read "causal dynamics learning."
  4. [Figure 7(d)] The y-axis label of Figure 7(d) is "log-likelihood," but the values range from 86 down to -295; please clarify the units or the normalization used, since a raw log-likelihood of 86 is unusual for this scale.
  5. [Equation (11)] In Eq. (11), Penv is described as the "ground truth dynamics collected from the environment"; in model-based RL the true dynamics are generally unavailable, so please specify how Penv is estimated in practice and whether this introduces a bias in the curiosity reward.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: ECL's causal mask is fit to external environment data, benchmarked against ground-truth graphs, and compared with external baselines, so its reported predictions are not equivalent to its inputs by construction.

full rationale

The central derivation chain is externally grounded rather than self-referential. Step 1 trains a dense dynamics model by likelihood on collected transitions (Eq. 4); Step 2 fits the causal mask by maximizing Lc-dyn (Eq. 5) with dynamics parameters fixed, and trains the exploration policy on the empowerment-gain difference (Eq. 7). The resulting causal graph is then scored against ground-truth causal graphs (Table 1, Figures 12-16) and against external baselines CDL, REG, GRADER, and IFactor, so the causal-discovery and policy claims are not forced by the paper's own definitions. The score-based variant adopts the sparse-regularization objective from Huang et al. (2022), which includes co-author Fan Feng, but this is a published, externally falsifiable component and is not load-bearing: the framework's contribution is the alternating empowerment-exploration loop, and the constraint-based variant uses CDL's conditional-independence testing instead. The paper itself flags the one substantive caveat: Section 3.2 states "For simplicity, we update pi_e by optimizing the KL term," and Appendix A concedes "In the empowerment maximization described by Eq. 10, we currently omit two entropy terms." That means the implemented exploration objective is a KL disagreement between the masked and dense versions of the same frozen dynamics model, not the full mutual-information empowerment gain, so the reported gains could plausibly come from a curiosity-like disagreement bonus rather than empowerment. This is a real correctness/attribution gap, but it is not circularity: the KL term is not the paper's input, the mask is still updated with new environment transitions, and the final causal and policy results are validated externally. No equation reduces a predicted quantity to a fitted parameter, and no result is imported solely from a self-citation chain.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central claim rests on standard causal discovery assumptions, a factored Gaussian dynamics model, and the frozen dynamics model used in Step 2. Per-environment hyperparameters control mask sparsity and thresholds; no new physical or ontological entities are introduced.

free parameters (4)
  • CMI threshold for ECL-Con = 0.02 (Chemical), 0.01 (Physical), 0.002 (Manipulation)
    Hand-set per environment for constraint-based causal discovery; directly controls which conditional independence tests create edges, so it shapes the causal mask used by exploration and planning.
  • L1 regularization coefficient lambda_M for ECL-Sco = 0.002 (Chemical), 0.02 (Physical), 0.001 (Manipulation)
    Hand-set per environment for score-based mask learning; controls the sparsity of the causal mask and hence the entire empowerment-driven exploration signal.
  • Curiosity reward weight lambda = Not reported in the main text.
    Balances task reward and curiosity reward in Eq. 12; the main text does not give values, and the ablation is delegated to Appendix D.8.
  • Regularization start step for ECL-Sco = 100K (Chemical/Physical), 750K (Manipulation)
    Determines when score-based mask regularization begins; affects the trajectory of mask learning and the final causal graph.
assumptions (4)
  • domain assumption Markov condition, faithfulness, no simultaneous or backward edges, and self-edges s_i^t -> s_i^{t+1}
    Invoked in Theorem 1 and Propositions 1-2 (Appendix C.3) to justify conditional independence tests for causal masks.
  • domain assumption Factored transition model with i.i.d. Gaussian noise (Eq. 2)
    Underlies the likelihood objectives Ldyn and Lc-dyn and the Gaussian form of the empowerment estimators in Eqs. 8-9.
  • domain assumption Dense dynamics model Pφc learned in Step 1 remains fixed and sufficiently accurate during Step 2
    The entire mask-policy alternation in Step 2 optimizes against this frozen model; if the model is misspecified, mask updates and exploration can follow model error.
  • domain assumption IFactor latent encoders provide identifiable true-state representations for pixel-based tasks
    Used as the substrate for causal discovery in pixel environments; identifiability is inherited from IFactor, not established in this paper.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Towards Empowerment Gain through Causal Structure Learning in Model-Based RL." pith.science (2026). https://pith.science/paper/FGA7PEOX

@misc{pith2026250210077,
  author       = {Pith},
  title        = {Pith review of: Towards Empowerment Gain through Causal Structure Learning in Model-Based RL},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/FGA7PEOX}},
  note         = {Machine review of arXiv:2502.10077}
}
read the original abstract

In Model-Based Reinforcement Learning (MBRL), incorporating causal structures into dynamics models provides agents with a structured understanding of the environments, enabling efficient decision. Empowerment as an intrinsic motivation enhances the ability of agents to actively control their environments by maximizing the mutual information between future states and actions. We posit that empowerment coupled with causal understanding can improve controllability, while enhanced empowerment gain can further facilitate causal reasoning in MBRL. To improve learning efficiency and controllability, we propose a novel framework, Empowerment through Causal Learning (ECL), where an agent with the awareness of causal dynamics models achieves empowerment-driven exploration and optimizes its causal structure for task learning. Specifically, ECL operates by first training a causal dynamics model of the environment based on collected data. We then maximize empowerment under the causal structure for exploration, simultaneously using data gathered through exploration to update causal dynamics model to be more controllable than dense dynamics model without causal structure. In downstream task learning, an intrinsic curiosity reward is included to balance the causality, mitigating overfitting. Importantly, ECL is method-agnostic and is capable of integrating various causal discovery methods. We evaluate ECL combined with 3 causal discovery methods across 6 environments including pixel-based tasks, demonstrating its superior performance compared to other causal MBRL methods, in terms of causal discovery, sample efficiency, and asymptotic performance.

Figures

Figures reproduced from arXiv: 2502.10077 by the authors.

Figure 1
Figure 1. (a). An example of a robot manipulation task with three trajectories and three nodes: one [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. The framework overview of ECL. Gold lines: model learning. Blue lines: model optimiza￾tion alternating with empowerment-driven exploration (yellow lines). Green lines: policy learning. 3 EMPOWERMENT THROUGH CAUSAL LEARNING An illustration of the ECL framework is shown in [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. The task learning of episodic reward in three environments of [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (28 more)
Figure 4
Figure 4. Figure 4: The learning curves of episodic reward in three different environments and the shadow is [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: Success rate in collider and manipulation [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 6
Figure 6. Figure 6: The causal graph comparison in the chemical collider environment. [PITH_FULL_IMAGE:figures/full_fig_p009_6.png]
Figure 7
Figure 7. Figure 7: Prediction performance (%) on ID and OOD states of [PITH_FULL_IMAGE:figures/full_fig_p010_7.png]
Figure 8
Figure 8. Figure 8: The compared results with IFactor and visualized trajectories in Robodesk environment. [PITH_FULL_IMAGE:figures/full_fig_p010_8.png]
Figure 9
Figure 9. Figure 9: Three basic experimental environments. Manipulation In the manipulation environment, we aim to prove the learned dynamics and policy for difficult settings with spurious correlations and multi-dimension action causal influence. The state space consists of the robot end…
Figure 10
Figure 10. Figure 10: 3 pixel-based experimental environments with [PITH_FULL_IMAGE:figures/full_fig_p021_10.png]
Figure 11
Figure 11. Figure 11: Multi-step prediction performance for four basic environments. ( [PITH_FULL_IMAGE:figures/full_fig_p023_11.png]
Figure 12
Figure 12. Figure 12: Causal graph for the chemical chain environment learned by the [PITH_FULL_IMAGE:figures/full_fig_p025_12.png]
Figure 13
Figure 13. Figure 13: Causal graph for the chemical collider environment learned by the [PITH_FULL_IMAGE:figures/full_fig_p026_13.png]
Figure 14
Figure 14. Figure 14: Causal graph for the chemical full environment learned by the [PITH_FULL_IMAGE:figures/full_fig_p027_14.png]
Figure 15
Figure 15. Figure 15: Causal graph for the manipulation environment learned by the true graph and [PITH_FULL_IMAGE:figures/full_fig_p028_15.png]
Figure 16
Figure 16. Figure 16: Causal graph for the manipulation environment learned by CDL and REG. [PITH_FULL_IMAGE:figures/full_fig_p029_16.png]
Figure 17
Figure 17. Figure 17: The task learning of episodic reward in three environments with [PITH_FULL_IMAGE:figures/full_fig_p030_17.png]
Figure 18
Figure 18. Figure 18: The task learning of episodic reward in three manipulation and physical environments. [PITH_FULL_IMAGE:figures/full_fig_p030_18.png]
Figure 19
Figure 19. Figure 19: The task learning curves of episodic reward in three chemical environments and the [PITH_FULL_IMAGE:figures/full_fig_p030_19.png]
Figure 20
Figure 20. Figure 20: The task learning curves of episodic reward in four environments and the shadow is the [PITH_FULL_IMAGE:figures/full_fig_p031_20.png]
Figure 21
Figure 21. Figure 21: The task learning of episodic reward in three chemical environments. [PITH_FULL_IMAGE:figures/full_fig_p031_21.png]
Figure 22
Figure 22. Figure 22: The results of average return compared with IFactor and visualized trajectories in Modified [PITH_FULL_IMAGE:figures/full_fig_p031_22.png]
Figure 23
Figure 23. Figure 23: The results of visualization in three pixel-based tasks of DMC environment. [PITH_FULL_IMAGE:figures/full_fig_p032_23.png]
Figure 24
Figure 24. Figure 24: The results of average return compared with IFactor in three pixel-based tasks of DMC [PITH_FULL_IMAGE:figures/full_fig_p032_24.png]
Figure 25
Figure 25. Figure 25: The results of average return compared with Dreamer in three pixel-based tasks of DMC [PITH_FULL_IMAGE:figures/full_fig_p032_25.png]
Figure 26
Figure 26. Figure 26: The mean accuracy of prediction with different training steps in chemical environments. [PITH_FULL_IMAGE:figures/full_fig_p033_26.png]
Figure 27
Figure 27. Figure 27: The episodic reward with different hyperparameter [PITH_FULL_IMAGE:figures/full_fig_p033_27.png]
Figure 28
Figure 28. Figure 28: The computation time in two chemical environments. [PITH_FULL_IMAGE:figures/full_fig_p033_28.png]
Figure 29
Figure 29. Figure 29: Learning curves of ablation studies in three chemical environments and the shadow is the [PITH_FULL_IMAGE:figures/full_fig_p034_29.png]
Figure 30
Figure 30. Figure 30: Learning curves of ablation studies in three DMC tasks and the shadow is the standard [PITH_FULL_IMAGE:figures/full_fig_p034_30.png]
Figure 31
Figure 31. Figure 31: Learning curves of ECL with and without curiosity reward in manipulation reach task, [PITH_FULL_IMAGE:figures/full_fig_p035_31.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Learning Task-Sufficient World Models by Synergizing Agentic Exploration and Structured Modeling

    cs.LG 2026-07 conditional novelty 6.0 of 10

    Closed-loop agentic probing plus minimality/sufficiency masking recovers compact task-sufficient world-model latents that improve sample-efficient policy learning and cross-task generalization.

Reference graph

Works this paper leans on

28 extracted references · 17 canonical work pages · cited by 1 Pith paper

  1. [1]

    Computation cost

    In our future work, we will further optimize the improvement scheme for the reward function. Computation cost. To consider the computation cost, we calculate the computation time for two chemical tasks of Chain, and Collider. The experimental results shown in Figure 28 demonstrate that ECL achieves its performance improvements with minimal additional comp...

    work page 2025

  2. [3]

    Open x-embodiment: Robotic learning datasets and rt-x models

    Open-X Embodiment Collaboration, A Padalkar, A Pooley, A Jain, A Bewley, A Herzog, A Irpan, A Khazatsky, A Rai, A Singh, et al. Open x-embodiment: Robotic learning datasets and rt-x models. arXiv preprint arXiv:2310.08864, 1(2),

    arXiv

  3. [4]

    Moreover, The policyπcollect is trained with a reward function r = tanh(PdS j=1 log p(sj t+1|st,at) p(sj t+1|PAsj ) )

    Regarding the other parameter settings, we adhered to the parameter configurations established in CDL (Wang et al., 2022c) and ASR (Huang et al., 2022). Moreover, The policyπcollect is trained with a reward function r = tanh(PdS j=1 log p(sj t+1|st,at) p(sj t+1|PAsj ) ). This reward function measures the prediction difference between the dense predictor a...

    work page 2022

  4. [5]

    For pixel-based task learning, we leverage the four distinct categories of latent state variables by IFactor to conduct empowerment maximization for policy learning

    We outline the parameter configurations for the reward predictor, as well as the settings employed for the cross-entropy method that is applied. For pixel-based task learning, we leverage the four distinct categories of latent state variables by IFactor to conduct empowerment maximization for policy learning. Moreover, we follow the same parameter setting...

    work page 2022

  5. [7]

    Action-sufficient state representation learning for control with structural constraints

    11 Published as a conference paper at ICLR 2025 Biwei Huang, Chaochao Lu, Liu Leqi, Jos´e Miguel Hern´andez-Lobato, Clark Glymour, Bernhard Sch¨olkopf, and Kun Zhang. Action-sufficient state representation learning for control with structural constraints. In International Conference on Machine Learning, pp. 9260–9279. PMLR,

    work page 2025

  6. [8]

    Systematic evaluation of causal discovery in visual model based reinforcement learning

    Nan Rosemary Ke, Aniket Didolkar, Sarthak Mittal, Anirudh Goyal, Guillaume Lajoie, Stefan Bauer, Danilo Rezende, Yoshua Bengio, Michael Mozer, and Christopher Pal. Systematic evaluation of causal discovery in visual model based reinforcement learning. arXiv preprint arXiv:2107.00848,

    arXiv

  7. [9]

    Empowerment: A universal agent-centric measure of control

    Alexander S Klyubin, Daniel Polani, and Chrystopher L Nehaniv. Empowerment: A universal agent-centric measure of control. In 2005 ieee congress on evolutionary computation, volume 1, pp. 128–135. IEEE,

    work page 2005

  8. [11]

    Dreaming: Model-based reinforcement learning by la- tent imagination without reconstruction

    12 Published as a conference paper at ICLR 2025 Masashi Okada and Tadahiro Taniguchi. Dreaming: Model-based reinforcement learning by la- tent imagination without reconstruction. In 2021 ieee international conference on robotics and automation (icra), pp. 4209–4215. IEEE,

    work page 2025

Show all 28 references
  1. [12]

    Robust agents learn causal world models

    Jonathan Richens and Tom Everitt. Robust agents learn causal world models. arXiv preprint arXiv:2402.10877,

    arXiv

  2. [13]

    Composing pre-trained object- centric representations for robotics from” what” and” where” foundation models

    Junyao Shi, Jianing Qian, Yecheng Jason Ma, and Dinesh Jayaraman. Composing pre-trained object- centric representations for robotics from” what” and” where” foundation models. arXiv preprint arXiv:2404.13474,

    arXiv

  3. [15]

    Causal dynamics learning for task-independent state abstraction

    Zizhao Wang, Xuesu Xiao, Zifan Xu, Yuke Zhu, and Peter Stone. Causal dynamics learning for task-independent state abstraction. arXiv preprint arXiv:2206.13452, 2022c. 13 Published as a conference paper at ICLR 2025 Zizhao Wang, Caroline Wang, Xuesu Xiao, Yuke Zhu, and Peter St...

    2025 arXiv

  4. [16]

    Learning invariant representations for reinforcement learning without reconstruction

    Amy Zhang, Rowan McAllister, Roberto Calandra, Yarin Gal, and Sergey Levine. Learning invariant representations for reinforcement learning without reconstruction. arXiv preprint arXiv:2006.10742,

    2006 arXiv

  5. [17]

    3 2.2 Empowerment

    14 Published as a conference paper at ICLR 2025 CONTENTS 1 Introduction 1 2 Preliminaries 3 2.1 MDP with Causal Structures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 2.2 Empowerment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 3 ...

    2025

  6. [18]

    dSX i=1 log Pϕc (si t+1|st, at; ϕc) # (13) Lc−dyn = E(st,at,st+1)∼D

    is a general framework to model four distinct categories of latent state variables, capturing various aspects of information. CDL (Wang et al., 2022c) is a causal dynamics learning method based on conditional independence testing. CDL employs conditional mutual information to ...

    2025

  7. [19]

    Meanwhile, in the downstream tasks, we evaluate the proposed methods by episodic reward and success rate

    Chemical In chemical environment, we aim to discover the causal relationship (Chain, Collider & Full) of chemical items which will prove the learned dynamics and explain the behavior without spurious correlations. Meanwhile, in the downstream tasks, we evaluate the proposed me...

    2025

  8. [20]

    Subsequently, we apply the proposed ECL framework for policy learning

    as our baseline method and used its encoders to process visual inputs. Subsequently, we apply the proposed ECL framework for policy learning. The parameter settings for these three environments are kept consistent with the default configurations of IFactor. Modified Cartpole W...

    2024

  9. [21]

    D.6 P IXEL -BASED TASKS LEARNING We evaluate ECL on 5 pixel-input tasks across 3 latent state environments

    These findings demonstrate that ECL-FCIT, achieves improved policy learning performance than GRADER, further validating the effectiveness of our proposed learning framework ECL. D.6 P IXEL -BASED TASKS LEARNING We evaluate ECL on 5 pixel-input tasks across 3 latent state envir...

    2023

  10. [23]

    ECL-Con and CDL exhibit smaller declines in accuracy 22 Published as a conference paper at ICLR 2025 Table 6: Compared results of causal graph learning on three chemical and physical environments. Metrics Methods Chain Collider Full Physical Accuracy ECL-Con 1.00±0.00 1.00±0.0...

    2025

  11. [24]

    Moreover, we achieve extensive elimination of causality between irrelevant factors

    Compared 23 Published as a conference paper at ICLR 2025 to CDL and REG, ECL-Con accurately uncovers the causal relationships among crucial elements, such as all different dimensions between states and actions, outperforming the other two methods. Moreover, we achieve extensiv...

    2025

  12. [25]

    Compared to CDL shown in Figure 16, ECL-Con learns more causal associations from relevant causal components related to the gripper, movable states, and actions

    From the results in Figure 6, we can discern that ECL-Con achieves around 90% overall fitting degree with the true causal graph and accurately learns the causal association between state and action. Compared to CDL shown in Figure 16, ECL-Con learns more causal associations fr...

    2025

  13. [28]

    versus maximizing the difference between causal and dense model empowerment (Eq. 10). Comparing ECL with empowerment (w/ Emp) against ECL with distance (w/ Dis) across three DMC tasks, our results in Figure 30 show that ECL w/ Dis achieves superior performance, and ECL w/ Emp ...

    2025

  14. [2003]

    Mastering diverse domains through world models

    Danijar Hafner, Jurgis Pasukonis, Jimmy Ba, and Timothy Lillicrap. Mastering diverse domains through world models. arXiv preprint arXiv:2301.04104,

    arXiv

  15. [2015]

    Ex- ploiting causal graph priors with posterior sampling for reinforcement learning

    Mirco Mutti, Riccardo De Santi, Marcello Restelli, Alexander Marx, and Giorgia Ramponi. Ex- ploiting causal graph priors with posterior sampling for reinforcement learning. arXiv preprint arXiv:2310.07518, 2023a. Mirco Mutti, Riccardo De Santi, Emanuele Rossi, Juan Felipe Cald...

    arXiv

  16. [2018]

    Octo: An open-source generalist robot policy

    Octo Model Team, Dibya Ghosh, Homer Walke, Karl Pertsch, Kevin Black, Oier Mees, Sudeep Dasari, Joey Hejna, Tobias Kreiman, Charles Xu, et al. Octo: An open-source generalist robot policy. arXiv preprint arXiv:2405.12213,

    arXiv

  17. [2020]

    Adarl: What, where, and how to adapt in transfer reinforcement learning

    Biwei Huang, Fan Feng, Chaochao Lu, Sara Magliacane, and Kun Zhang. Adarl: What, where, and how to adapt in transfer reinforcement learning. arXiv preprint arXiv:2107.02729,

    arXiv

  18. [2021]

    Information prioritization through empowerment in visual model-based rl

    Homanga Bharadhwaj, Mohammad Babaeizadeh, Dumitru Erhan, and Sergey Levine. Information prioritization through empowerment in visual model-based rl. arXiv preprint arXiv:2204.08585,

    arXiv

  19. [2022]

    Variational empowerment as representation learning for goal-based reinforcement learning

    Jongwook Choi, Archit Sharma, Honglak Lee, Sergey Levine, and Shixiang Shane Gu. Variational empowerment as representation learning for goal-based reinforcement learning. arXiv preprint arXiv:2106.01404,

    arXiv

  20. [2024]

    Diversity is all you need: Learning skills without a reward function

    Benjamin Eysenbach, Abhishek Gupta, Julian Ibarz, and Sergey Levine. Diversity is all you need: Learning skills without a reward function. arXiv preprint arXiv:1802.06070,

    arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.