Pith. sign in

REVIEW 3 major objections 5 minor 1 cited by

Null Counterfactual Factor Interactions for Goal-Conditioned Reinforcement Learning

T0 review · 3 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Filtering hindsight relabeled goals by null-counterfactual interactions improves sample efficiency up to 4x in object-centric goal-conditioned reinforcement learning.

desk verdict A genuinely new interaction definition and a sensible hindsight filter, but the inference accuracy claim is measured against contact, not null counterfactuals, and the simulated-nulling proxy is the real load-bearing risk. read the letter →

arxiv 2505.03172 v1 pith:O2ELQCW5 submitted 2025-05-06 cs.LG cs.AI

Caleb Chuck , Fan Feng , Carl Qi , Chang Shi , Siddhant Agarwal , Amy Zhang , Scott Niekum This is my paper · ORCID
classification cs.LGcs.AI
keywords goal-conditionedreinforcementlearninghindsightrelabelingnullcounterfactualfactorinteractionscausalinferenceobject-centriccontrolsampleefficiencymaskeddynamicsmodels
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Hindsight relabeling in goal-conditioned reinforcement learning gives high reward to trajectories that never touch the target object, which floods the replay buffer with useless behavior in object-centric tasks. The paper's core claim is that this failure is fixable by filtering relabeled trajectories with a causal interaction test: a cause factor interacts with a target when removing the cause to a null state changes the target's transition. To operationalize that, the paper introduces NCII, a masked forward model that learns to predict null-counterfactual outcomes and labels interactions, and HInt, which only relabels trajectories where a path in the unrolled interaction graph runs from the agent's action to the goal factor. Empirically, NCII lowers interaction misprediction rates relative to prior inference baselines, and HInt improves sample efficiency by up to 4x across Spriteworld, Robosuite, Robot Air Hockey, and Franka Kitchen. If the definition holds up, hindsight replay can be made safe for domains where the agent must learn to act on objects rather than learn to ignore them.

What carries the argument

The central object is the null-counterfactual interaction test: a comparison between a learned forward model's log-likelihood of the observed next state with all factors present versus with the candidate cause 'nulled' to its absent state. NCII instantiates this as a masked dynamics model $f(s,a,B(v);\theta)$ over factored MDP states, trained by maximum likelihood on trajectories with varying subsets of factors; the interaction model $h(s;\phi)$ distills the null test into fast, soft predictions and is retrained jointly with $f$. HInt then converts the learned interaction graph $B^{(t)}$ at each timestep into an unrolled temporal interaction graph and filters hindsight relabeling to trajectories containing a control-target path, since those are trajectories where the agent, rather than passive motion, is responsible for the goal factor's change.

What would settle it

Train NCII on a domain where the target object moves predictably under its own dynamics (for example constant gravity) while a second object also contacts it, then compare NCII's interaction labels against true contact timestamps: elevated misprediction exactly on contact-free but self-predictable spans would show that the simulated-nulling proxy confounds interaction with self-predictability.

Watch

Extended reading notes

Core claim

The paper's central claim is that interactions defined by null counterfactuals give a tractable, learnable signal for which trajectories hindsight relabeling should use. Formally, every state factor $S_i$ has a null state $s_{i,\circ}$, and $S_i$ interacts with target $S_j$ in transition $(s,a,s')$ exactly when $p(S_j=s'_j \mid S=s) \neq p(S_j=s'_j \mid S=s^{\circ}_{s_i})$. NCII trains a masked forward model on trajectories with varying subsets of factors, then uses the difference in log-likelihood between the observed outcome with and without the candidate cause nulled to label interactions. HInt then keeps a trajectory only if the unrolled interaction graph contains a path from the action node to the target factor, relabeling with the state at a time on that path. Across Spriteworld, Robosuite, Robot Air Hockey, and Franka Kitchen, HInt matches or exceeds vanilla hindsight, prioritized replay, and interaction-based exploration baselines, reaching the same performance up to 4x sooner.

Load-bearing premise

The load-bearing premise is that every state factor has a null state whose removal leaves all unrelated transitions unchanged, and that in physical domains a simulated nulling proxy—treating a target as non-interacting whenever a model using only that target's own history predicts its next state well—faithfully reproduces what would happen without the cause; if that proxy misfires, for example when gravity or friction makes a target's motion predictable while another object is also acting on it, the inferred interaction labels and thus HInt's filtering decisions inherit the error.

Editorial extensions

If this is right

  • Hindsight buffers in object-centric GCRL should contain only trajectories whose interaction graph has an action-to-target path; filtering out the rest removes the dominant 'target did nothing' relabeled goals.
  • Because NCII matches or beats prior interaction inference methods even without ground-truth contact labels, HInt can be deployed in physical domains where contacts are not directly observable.
  • HInt with learned NCII interactions reaches the same or better asymptotic performance than HInt with ground-truth contacts, so the benefit does not hinge on perfect inference.
  • In locomotion or navigation settings with no object to act on, the interaction filter adds little or can be detrimental, which bounds the method's applicability to interaction-centric tasks.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A consequence the paper leaves implicit is that the null-counterfactual test is a continuous responsibility signal, so a graded version could serve as reward shaping or an exploration bonus rather than only a binary hindsight gate.
  • The simulated-nulling proxy ties NCII's accuracy to the passive model's autoregressive quality; a testable extension is to train the masked forward model with null masks generated from the passive signal at train time and compare against fixed masks.
  • Because the filter rejects whole trajectories, it may discard episodes where a gradual, weak push eventually moves the target; scoring trajectories by interaction count or magnitude rather than binary membership would be a natural extension.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes two connected methods. NCII infers object interactions by comparing a learned forward model's prediction of a target factor's next state with its prediction under a nulled cause factor, formalized in Definition 3.1 and Eqs. (2)-(3). When explicit null trajectories are unavailable, Appendix D substitutes a passive-signal proxy: transitions in which the target is predictable from its own state are treated as non-interactions used to generate pseudo-null training data (Eqs. (6)-(7)). HInt then uses the inferred interaction graph to filter hindsight relabeling, keeping only trajectories containing a path from the action to the target factor, and commits the hindsight goal to an interaction time step. Experiments compare NCII's misprediction rate with JACI, gradient, attention, and NCD baselines in Random DAG, Spriteworld, Robosuite, Air Hockey, and Franka Kitchen (Table 1), and compare HInt's goal-conditioned RL sample efficiency with HER, prioritized replay, f-pg, ELDEN, and CAI (Figure 4 and Appendix M), reporting up to a 4x improvement.

Significance. If the claims are fully supported, the paper makes a useful contribution: it gives a tractable, formal definition of interaction based on null counterfactuals, applies it to interaction inference, and demonstrates that interaction filtering can improve hindsight relabeling in object-centric GCRL domains. Strengths include the explicit formalization in Definition 3.1, the iterative joint training of the forward and interaction models, the breadth of benchmark domains, evaluation against external task reward for the RL result, and a reproducibility statement with code in the appendix. The main weakness is that the interaction-inference claim in physical domains rests on an unvalidated proxy for the null counterfactual and an evaluation label (contact) that the paper itself acknowledges can disagree with the proposed definition. This does not invalidate the external-reward RL results, but it does mean the NCII accuracy claim, as stated, is not currently established.

major comments (3)
  1. [Appendix D, Eqs. (6)-(7); Section 5.1; Definition 3.1] The simulated nulling proxy is unvalidated against the paper's own null-counterfactual criterion. Appendix D replaces the genuine null state of Definition 3.1 with a passive autoregressive signal: Eq. (6) declares a target non-interacting when f(s_j) predicts s'_j well, and Eq. (7) uses such states to generate pseudo-null training data. This substitution is never checked against actual null counterfactuals. The failure mode is concrete: in Air Hockey, gravity can make the puck's motion predictable from its own state even while the paddle is striking it, and in Robosuite, quasistatic friction can make a pushed block's motion predictable from its own state. In such cases the passive signal is high despite an interaction, so Eq. (3) will under-detect interactions. Table 1 cannot arbitrate this because its ground truth in physical domains is contact, not the Definition 3.1 null-counterfactual criterion, and Section 5.2.1 explicitly notes that contact and null-counterfactual interaction can disagree ("rejecting contact that only produces minute changes"). The authors should either validate the passive-signal proxy in a domain with explicit null trajectories or report evaluation against true null-counterfactual labels; otherwise the NCII accuracy claim in Table 1 is not supported.
  2. [Section 4.1, Eq. (4); Algorithm 1] There is a load-bearing inconsistency in the definition of the interaction model h. The text states h:S×A→B, but Eq. (4) trains h(s;φ) on state alone, and Algorithm 1 lists h(s;ϕ) as an input. Since the training targets Null(s,a,θ) in Eq. (3) depend on the action, a state-only function cannot represent the action column of B and cannot produce the action-to-target paths that HInt uses for filtering in Section 4.2. The authors should clarify whether h actually receives (s,a); if it does, Eq. (4), Algorithm 1, and the discussion in Appendix E (where control factors are instead identified by training 2n separate models) need to be corrected consistently. If h is intentionally state-only, the paper must explain how action edges are obtained for the "action graph" and "control-target" filtering criteria.
  3. [Table 1 caption; Appendix D; Figure 4] The interaction inference accuracy is reported on test sets reweighted to 50% interactions (Table 1 caption), whereas Appendix D states that natural interaction frequencies in the physical domains range from 0.5% to 0.01%. HInt's hindsight filter operates on natural-frequency trajectories, so a 50%-interaction test set does not measure the operating regime of the downstream algorithm. The manuscript should report confusion-matrix metrics (precision, recall, and misprediction rate) at natural interaction frequencies, or otherwise show that the reweighted accuracy transfers to the rare-interaction regime that motivates the method.
minor comments (5)
  1. [Eq. (3)] The expression in Eq. (3) appears to take the logarithm of a difference of probabilities; the intended null test is a difference of log-likelihoods of the observed next state, and the argument should be the observed next state s'_j rather than s_j. Please correct the notation and the closing parenthesis.
  2. [Appendix D, Eq. (7)] There is a typo: "Bernolli" should be "Bernoulli".
  3. [Section 5.2 and Figure 4] The abstract claims "up to 4x" sample-efficiency improvement, but the paper does not report a numeric fixed-budget comparison or a statistical significance test; with only 5 trials per condition, the visual separation in Figure 4 should be supplemented with confidence intervals or a significance test for the claimed improvement magnitude.
  4. [Section 5.1, Table 1] The text states that NCII achieves a "statistically significant reduction in misprediction rate" but no significance tests are reported; either include pairwise significance tests or temper the wording to "lower misprediction rate".
  5. [Section 4.1, Eq. (4)] Please clarify whether the binary cross-entropy loss in Eq. (4) is computed over all entries of the interaction matrix B including the action column, and specify how the action-column targets are obtained when h is trained on state only.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: NCII and HInt are evaluated against external contact labels and task rewards; simulated nulling is an unvalidated proxy, not a self-referential reduction.

full rationale

NCII's interaction inference is scored against external ground-truth contact labels (Table 1) and HInt is scored against task reward (Figure 4); neither score is a function of the quantities being fitted. The masked forward-model objective (Eq. 2), null test (Eq. 3), and interaction-model objective (Eq. 4) form an iterative self-bootstrapping loop, but the loop is anchored to observed transitions and its output is validated externally, so it does not reduce to its own inputs. Appendix D's simulated nulling replaces true null states with a passive-predictability proxy (Eqs. 6-7); this is a substantive modeling assumption and a correctness risk, but it is not a circular reduction because the reported inference accuracy is measured against contacts, not against the passive signal. Citations to the authors' prior work (Chuck et al. 2023, 2024b) supply heuristics such as passive error, reweighting, and the JACI baseline, but the paper states and tests the null-counterfactual definition itself rather than relying on those citations to establish the central result.

Assumptions & free parameters 4 free parameters · 5 assumptions · 1 invented entities

The central method rests on the null counterfactual assumption and on simulated nulling through a passive signal. These are the main imported premises, and they are not independently verified outside the paper. The RL and inference results depend on several hand-chosen thresholds and reweighting choices, which are listed above.

free parameters (4)
  • epsilon_null = 1.0 (log-likelihood space)
    Threshold for the null counterfactual interaction test in Equation 3. Chosen by hand; Appendix F shows an ablation indicating robustness, but it remains a free parameter of the inference procedure.
  • epsilon_sim-null = not reported (used in Appendix D, Equation 7)
    Probability of randomly nulling a factor in simulated nulling when the passive signal is high. This is a hand-chosen hyperparameter that controls how null training data are synthesized.
  • epsilon_passive = not reported
    Log-likelihood threshold for deciding that a target object's passive prediction is accurate, used to identify non-interactions in Appendix D. This is a free threshold for the passive signal proxy.
  • interaction reweighting ratio = 20% sampling weight for low-passive-likelihood states
    Appendix D states that low passive likelihood states are upweighted so they are sampled on average 20% of the time. This artificially changes the training distribution and is a hand-set parameter.
assumptions (5)
  • ad hoc to paper Every state factor Si has a null state si,o such that nulling a non-interacting factor leaves the target transition distribution unchanged.
    This is Definition 3.1, the central inductive bias that makes interaction inference tractable. It is not a standard assumption in physics or RL; it is introduced for the method and is not independently verified.
  • domain assumption The learned masked forward model f(s,a,B;theta) accurately approximates the true transition distribution, including under null masks, so that log-likelihood differences in Equation 3 reflect real interactions.
    NCII relies on the forward model's confidence calibration to detect interactions. If the model is poorly calibrated, especially out of distribution under nulling, the inference is unreliable.
  • ad hoc to paper A target object is non-interacting when its own autoregressive prediction f(s_j) predicts s'_j well (the passive signal).
    This proxy, introduced in Appendix D, is used to simulate nulling in domains without explicit null trajectories. It assumes predictable motion implies no interaction, which can fail under gravity or friction.
  • domain assumption Interactions are rare events: 0.5% to 0.01% of states in the physical domains.
    Appendix D uses this rarity to justify reweighting and to argue that the passive signal is a useful prior. This is empirically asserted, not derived.
  • domain assumption Ground truth interactions in physical domains are given by physical contacts between objects.
    Section 5.1 defines ground truth as contacts. This assumption is used only for evaluation, not for training, but it is misaligned with the paper's own null-counterfactual definition, as acknowledged in Section 5.2.1.
invented entities (1)
  • Null state s_i,o for each factor
    purpose: Provides a counterfactual world where a cause object does not exist, enabling the null counterfactual interaction test.
    The null state is a postulated modeling construct. The paper assumes it exists for every factor but provides no external falsifiable prediction tied to it; its validity is only evidenced indirectly by downstream RL performance.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Null Counterfactual Factor Interactions for Goal-Conditioned Reinforcement Learning." pith.science (2026). https://pith.science/paper/O2ELQCW5

@misc{pith2026250503172,
  author       = {Pith},
  title        = {Pith review of: Null Counterfactual Factor Interactions for Goal-Conditioned Reinforcement Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/O2ELQCW5}},
  note         = {Machine review of arXiv:2505.03172}
}
read the original abstract

Hindsight relabeling is a powerful tool for overcoming sparsity in goal-conditioned reinforcement learning (GCRL), especially in certain domains such as navigation and locomotion. However, hindsight relabeling can struggle in object-centric domains. For example, suppose that the goal space consists of a robotic arm pushing a particular target block to a goal location. In this case, hindsight relabeling will give high rewards to any trajectory that does not interact with the block. However, these behaviors are only useful when the object is already at the goal -- an extremely rare case in practice. A dataset dominated by these kinds of trajectories can complicate learning and lead to failures. In object-centric domains, one key intuition is that meaningful trajectories are often characterized by object-object interactions such as pushing the block with the gripper. To leverage this intuition, we introduce Hindsight Relabeling using Interactions (HInt), which combines interactions with hindsight relabeling to improve the sample efficiency of downstream RL. However because interactions do not have a consensus statistical definition tractable for downstream GCRL, we propose a definition of interactions based on the concept of null counterfactual: a cause object is interacting with a target object if, in a world where the cause object did not exist, the target object would have different transition dynamics. We leverage this definition to infer interactions in Null Counterfactual Interaction Inference (NCII), which uses a "nulling'' operation with a learned model to infer interactions. NCII is able to achieve significantly improved interaction inference accuracy in both simple linear dynamics domains and dynamic robotic domains in Robosuite, Robot Air Hockey, and Franka Kitchen and HInt improves sample efficiency by up to 4x.

Figures

Figures reproduced from arXiv: 2505.03172 by the authors.

Figure 1
Figure 1. Figure (a) shows a case when a null counterfactual interaction occurs between the cause [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. An example of the unrolled dynamic interaction graph, where an edge indicates an [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Visualizations of domains used for evaluation. Goals are in green and target objects are in [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (12 more)
Figure 4
Figure 4. Figure 4: Comparison of HInt and HInt with NCII against baselines, 5 trials for each. HInt with NCII [PITH_FULL_IMAGE:figures/full_fig_p009_4.png]
Figure 5
Figure 5. Figure 5: Relative position heatmap between initial state and a) sampled or “desired” goal, b) hindsight goals, c) goals after HInt filtering, d) goals removed by HInt, over 3000 goals in Spriteworld default. 3 modified air hockey domains, and 3 modified Robosuite domains and Fr…
Figure 6
Figure 6. Figure 6: The data flow for the HInt method. An interaction matrix [PITH_FULL_IMAGE:figures/full_fig_p016_6.png]
Figure 7
Figure 7. Figure 7: Comparison of HInt and HInt with NCII 5 trials for each. The shading indicates standard [PITH_FULL_IMAGE:figures/full_fig_p016_7.png]
Figure 8
Figure 8. Figure 8: The Pointnet-based architecture used for interactions. Multilayer methods repeatedly [PITH_FULL_IMAGE:figures/full_fig_p019_8.png]
Figure 9
Figure 9. Figure 9: The GNN architecture used for interactions. Similarly, shared colors (green, yellow) denote [PITH_FULL_IMAGE:figures/full_fig_p020_9.png]
Figure 10
Figure 10. Figure 10: Comparison of HInt and Hindsight using different HER sampling schemes “final”, “future” [PITH_FULL_IMAGE:figures/full_fig_p024_10.png]
Figure 11
Figure 11. Figure 11: Comparison of HInt, Hindsight and Vanilla RL, 3 trials for each, on Spriteworld Default [PITH_FULL_IMAGE:figures/full_fig_p025_11.png]
Figure 12
Figure 12. Figure 12: Addition of Causal Action Influence (CAI) [PITH_FULL_IMAGE:figures/full_fig_p025_12.png]
Figure 13
Figure 13. Figure 13: Selected frames from a successful policy rollout for Spriteworld Obstacles [PITH_FULL_IMAGE:figures/full_fig_p026_13.png]
Figure 15
Figure 15. Figure 15: Selected frames from a successful policy rollout for Air Hockey Default [PITH_FULL_IMAGE:figures/full_fig_p026_15.png]
Figure 16
Figure 16. Figure 16: Selected frames from a successful policy rollout for Franka Kitchen [PITH_FULL_IMAGE:figures/full_fig_p026_16.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Learning Task-Sufficient World Models by Synergizing Agentic Exploration and Structured Modeling

    cs.LG 2026-07 conditional novelty 6.0 of 10

    Closed-loop agentic probing plus minimality/sufficiency masking recovers compact task-sufficient world-model latents that improve sample-efficient policy learning and cross-task generalization.

Reference graph

Works this paper leans on

56 extracted references · 22 canonical work pages · cited by 1 Pith paper

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [2]

    f-policy gradients: A general framework for goal-conditioned rl using f-divergences

    Siddhant Agarwal, Ishan Durugkar, Peter Stone, and Amy Zhang. f-policy gradients: A general framework for goal-conditioned rl using f-divergences. Advances in Neural Information Processing Systems, 36, 2023

    work page 2023

  3. [3]

    Hindsight experience replay

    Marcin Andrychowicz, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, OpenAI Pieter Abbeel, and Wojciech Zaremba. Hindsight experience replay. Advances in neural information processing systems, 30, 2017

    2017

  4. [4]

    Addressing hindsight bias in multigoal reinforcement learning

    Chenjia Bai, Lingxiao Wang, Yixin Wang, Zhaoran Wang, Rui Zhao, Chenyao Bai, and Peng Liu. Addressing hindsight bias in multigoal reinforcement learning. IEEE Transactions on Cybernetics, 53 0 (1): 0 392--405, 2021

    work page 2021

  5. [6]

    Causal sufficiency and actual causation

    Sander Beckers. Causal sufficiency and actual causation. Journal of Philosophical Logic, 50 0 (6): 0 1341--1374, June 2021. doi:10.1007/s10992-021-09601-z. URL https://doi.org/10.1007/s10992-021-09601-z

    work page doi:10.1007/s10992-021-09601-z 2021

  6. [7]

    Context-Specific Independence in Bayesian Networks

    Craig Boutilier, Nir Friedman, Moises Goldszmidt, and Daphne Koller. Context-specific independence in bayesian networks. arXiv preprint arXiv:1302.3562, 2013

    work page Pith review arXiv 2013

  7. [8]

    Woulda, coulda, shoulda: Counterfactually-guided policy search

    Lars Buesing, Theophane Weber, Yori Zwols, Sebastien Racaniere, Arthur Guez, Jean-Baptiste Lespiau, and Nicolas Heess. Woulda, coulda, shoulda: Counterfactually-guided policy search. arXiv preprint arXiv:1811.06272, 2018

    arXiv 2018

  8. [9]

    Goal-conditioned reinforcement learning with imagined subgoals

    Elliot Chane-Sane, Cordelia Schmid, and Ivan Laptev. Goal-conditioned reinforcement learning with imagined subgoals. In International conference on machine learning, pp.\ 1430--1440. PMLR, 2021

    2021

Show all 56 references
  1. [10]

    Hypothesis-driven skill discovery for hierarchical deep reinforcement learning

    Caleb Chuck, Supawit Chockchowwat, and Scott Niekum. Hypothesis-driven skill discovery for hierarchical deep reinforcement learning. In 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp.\ 5572--5579. IEEE, 2020

    2020

  2. [11]

    Granger-causal hierarchical skill discovery

    Caleb Chuck, Kevin Black, Aditya Arjun, Yuke Zhu, and Scott Niekum. Granger-causal hierarchical skill discovery. arXiv preprint arXiv:2306.09509, 2023

    2023 arXiv

  3. [12]

    Robot air hockey: A manipulation testbed for robot learning with reinforcement learning

    Caleb Chuck, Carl Qi, Michael J Munje, Shuozhe Li, Max Rudolph, Chang Shi, Siddhant Agarwal, Harshit Sikchi, Abhinav Peri, Sarthak Dayal, et al. Robot air hockey: A manipulation testbed for robot learning with reinforcement learning. arXiv preprint arXiv:2405.03113, 2024 a

    2024 arXiv

  4. [13]

    Automated discovery of functional actual causes in complex environments

    Caleb Chuck, Sankaran Vaidyanathan, Stephen Giguere, Amy Zhang, David Jensen, and Scott Niekum. Automated discovery of functional actual causes in complex environments. arXiv preprint arXiv:2404.10883, 2024 b

    2024 arXiv

  5. [14]

    Curriculum-guided hindsight experience replay

    Meng Fang, Tianyi Zhou, Yali Du, Lei Han, and Zhengyou Zhang. Curriculum-guided hindsight experience replay. Advances in neural information processing systems, 32, 2019

    2019

  6. [15]

    Learning dynamic attribute-factored world models for efficient multi-object reinforcement learning

    Fan Feng and Sara Magliacane. Learning dynamic attribute-factored world models for efficient multi-object reinforcement learning. Advances in Neural Information Processing Systems, 36, 2024

    2024

  7. [16]

    Relay policy learning: Solving long-horizon tasks via imitation and reinforcement learning

    Abhishek Gupta, Vikash Kumar, Corey Lynch, Sergey Levine, and Karol Hausman. Relay policy learning: Solving long-horizon tasks via imitation and reinforcement learning. Conference on Robot Learning (2019), 2019

    2019

  8. [17]

    Actual causality

    Joseph Y Halpern. Actual causality. MIT Press, 2016

    2016

  9. [18]

    On discovery of local independence over continuous variables via neural contextual decomposition

    Inwoo Hwang, Yunhyeok Kwak, Yeon-Ji Song, Byoung-Tak Zhang, and Sanghack Lee. On discovery of local independence over continuous variables via neural contextual decomposition. In Conference on Causal Learning and Reasoning, pp.\ 448--472. PMLR, 2023

    2023

  10. [19]

    Learning to achieve goals

    Leslie Pack Kaelbling. Learning to achieve goals. In IJCAI, volume 2, pp.\ 1094--8. Citeseer, 1993

    1993

  11. [20]

    Efficient reinforcement learning in factored mdps

    Michael Kearns and Daphne Koller. Efficient reinforcement learning in factored mdps. In IJCAI, volume 16, pp.\ 740--747, 1999

    1999

  12. [21]

    What can i do here? learning new skills by imagining visual affordances

    Alexander Khazatsky, Ashvin Nair, Daniel Jing, and Sergey Levine. What can i do here? learning new skills by imagining visual affordances. In 2021 IEEE International Conference on Robotics and Automation (ICRA), pp.\ 14291--14297. IEEE, 2021

    2021

  13. [22]

    Auto-encoding variational bayes

    Diederik P Kingma. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114, 2013

    2013 arXiv

  14. [23]

    Kipf and Max Welling

    Thomas N. Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. In International Conference on Learning Representations, 2017. URL https://openreview.net/forum?id=SJU4ayYgl

    2017

  15. [24]

    Archer: Aggressive rewards to counter bias in hindsight experience replay

    Sameera Lanka and Tianfu Wu. Archer: Aggressive rewards to counter bias in hindsight experience replay. arXiv preprint arXiv:1809.02070, 2018

    2018 arXiv

  16. [25]

    Generalized hindsight for reinforcement learning

    Alexander Li, Lerrel Pinto, and Pieter Abbeel. Generalized hindsight for reinforcement learning. Advances in neural information processing systems, 33: 0 7754--7767, 2020

    2020

  17. [26]

    Continuous control with deep reinforcement learning

    TP Lillicrap. Continuous control with deep reinforcement learning. arXiv preprint arXiv:1509.02971, 2015

    2015 arXiv

  18. [27]

    Biscuit: Causal representation learning from binary interactions

    Phillip Lippe, Sara Magliacane, Sindy L \"o we, Yuki M Asano, Taco Cohen, and Efstratios Gavves. Biscuit: Causal representation learning from binary interactions. In Uncertainty in Artificial Intelligence, pp.\ 1263--1273. PMLR, 2023

    2023

  19. [28]

    Goal-conditioned reinforcement learning: Problems and solutions

    Minghuan Liu, Menghui Zhu, and Weinan Zhang. Goal-conditioned reinforcement learning: Problems and solutions. International Joint Conference on Artificial Intelligence (IJCAI), 2022

    2022

  20. [29]

    Physgen: Rigid-body physics-grounded image-to-video generation

    Shaowei Liu, Zhongzheng Ren, Saurabh Gupta, and Shenlong Wang. Physgen: Rigid-body physics-grounded image-to-video generation. In European Conference on Computer Vision, pp.\ 360--378. Springer, 2024

    2024

  21. [30]

    Offline goal-conditioned reinforcement learning via f -advantage regression

    Jason Yecheng Ma, Jason Yan, Dinesh Jayaraman, and Osbert Bastani. Offline goal-conditioned reinforcement learning via f -advantage regression. Advances in neural information processing systems, 35: 0 310--323, 2022

    2022

  22. [31]

    Localizing external contact using proprioceptive sensors: The contact particle filter

    Lucas Manuelli and Russ Tedrake. Localizing external contact using proprioceptive sensors: The contact particle filter. In 2016 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp.\ 5062--5069. IEEE, 2016

    2016

  23. [32]

    Visual reinforcement learning with imagined goals

    Ashvin V Nair, Vitchyr Pong, Murtaza Dalal, Shikhar Bahl, Steven Lin, and Sergey Levine. Visual reinforcement learning with imagined goals. Advances in neural information processing systems, 31, 2018

    2018

  24. [33]

    Causality

    Judea Pearl. Causality. Cambridge University Press, New York, 2000

    2000

  25. [34]

    Counterfactual data augmentation using locally factored dynamics

    Silviu Pitis, Elliot Creager, and Animesh Garg. Counterfactual data augmentation using locally factored dynamics. Advances in Neural Information Processing Systems, 33: 0 3976--3990, 2020

    2020

  26. [35]

    Mocoda: Model-based counterfactual data augmentation

    Silviu Pitis, Elliot Creager, Ajay Mandlekar, and Animesh Garg. Mocoda: Model-based counterfactual data augmentation. Advances in Neural Information Processing Systems, 35: 0 18143--18156, 2022

    2022

  27. [36]

    Markov decision processes

    Martin L Puterman. Markov decision processes. Handbooks in operations research and management science, 2: 0 331--434, 1990

    1990

  28. [37]

    Pointnet: Deep learning on point sets for 3d classification and segmentation

    Charles R Qi, Hao Su, Kaichun Mo, and Leonidas J Guibas. Pointnet: Deep learning on point sets for 3d classification and segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp.\ 652--660, 2017

    2017

  29. [38]

    The graph neural network model

    Franco Scarselli, Marco Gori, Ah Chung Tsoi, Markus Hagenbuchner, and Gabriele Monfardini. The graph neural network model. IEEE transactions on neural networks, 20 0 (1): 0 61--80, 2008

    2008

  30. [39]

    Prioritized experience replay

    Tom Schaul, John Quan, Ioannis Antonoglou, and David Silver. Prioritized experience replay. arXiv preprint arXiv:1511.05952, 2015

    2015 arXiv

  31. [40]

    Causal influence detection for improving efficiency in reinforcement learning

    Maximilian Seitzer, Bernhard Sch \"o lkopf, and Georg Martius. Causal influence detection for improving efficiency in reinforcement learning. Advances in Neural Information Processing Systems, 34: 0 22905--22918, 2021

    2021

  32. [41]

    Smore: Score models for offline goal-conditioned reinforcement learning

    Harshit Sikchi, Rohan Chitnis, Ahmed Touati, Alborz Geramifard, Amy Zhang, and Scott Niekum. Smore: Score models for offline goal-conditioned reinforcement learning. International Conference on Learning Representations, 2024 a

    2024

  33. [42]

    Score models for offline goal-conditioned reinforcement learning

    Harshit Sikchi, Rohan Chitnis, Ahmed Touati, Alborz Geramifard, Amy Zhang, and Scott Niekum. Score models for offline goal-conditioned reinforcement learning. International Conference on Learning Representations, 2024 b

    2024

  34. [43]

    A general reinforcement learning algorithm that masters chess, shogi, and go through self-play

    David Silver, Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Matthew Lai, Arthur Guez, Marc Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, et al. A general reinforcement learning algorithm that masters chess, shogi, and go through self-play. Science, 362 0 ...

    2018

  35. [44]

    Solving olympiad geometry without human demonstrations

    Trieu H Trinh, Yuhuai Wu, Quoc V Le, He He, and Thang Luong. Solving olympiad geometry without human demonstrations. Nature, 625 0 (7995): 0 476--482, 2024

    2024

  36. [45]

    Causal action influence aware counterfactual data augmentation

    N \'u ria Armengol Urp \' , Marco Bagatella, Marin Vlastelica, and Georg Martius. Causal action influence aware counterfactual data augmentation. arXiv preprint arXiv:2405.18917, 2024

    2024 arXiv

  37. [46]

    Attention is all you need

    A Vaswani. Attention is all you need. Advances in Neural Information Processing Systems, 2017

    2017

  38. [47]

    Causal dynamics learning for task-independent state abstraction

    Zizhao Wang, Xuesu Xiao, Zifan Xu, Yuke Zhu, and Peter Stone. Causal dynamics learning for task-independent state abstraction. arXiv preprint arXiv:2206.13452, 2022

    2022 arXiv

  39. [48]

    Elden: Exploration via local dependencies

    Zizhao Wang, Jiaheng Hu, Peter Stone, and Roberto Martin-Martin. Elden: Exploration via local dependencies. In Advances in Neural Information Processing Systems (NeurIPS), 2023

    2023

  40. [49]

    Cobra: Data-efficient model-based rl through unsupervised object discovery and curiosity-driven exploration

    Nicholas Watters, Loic Matthey, Matko Bosnjak, Christopher P Burgess, and Alexander Lerchner. Cobra: Data-efficient model-based rl through unsupervised object discovery and curiosity-driven exploration. arXiv preprint arXiv:1905.09275, 2019

    1905 arXiv

  41. [50]

    Outracing champion gran turismo drivers with deep reinforcement learning

    Peter R Wurman, Samuel Barrett, Kenta Kawamoto, James MacGlashan, Kaushik Subramanian, Thomas J Walsh, Roberto Capobianco, Alisa Devlic, Franziska Eckert, Florian Fuchs, et al. Outracing champion gran turismo drivers with deep reinforcement learning. Nature, 602 0 (7896): 0 22...

    2022

  42. [51]

    Curiosity-driven experience prioritization via density estimation

    Rui Zhao and Volker Tresp. Curiosity-driven experience prioritization via density estimation. arXiv preprint arXiv:1902.08039, 2019

    1902 arXiv

  43. [52]

    Maximum entropy-regularized multi-goal reinforcement learning

    Rui Zhao, Xudong Sun, and Volker Tresp. Maximum entropy-regularized multi-goal reinforcement learning. In International Conference on Machine Learning, pp.\ 7553--7562. PMLR, 2019

    2019

  44. [53]

    How does goal relabeling improve sample efficiency? In Forty-first International Conference on Machine Learning, 2024

    Sirui Zheng, Chenjia Bai, Zhuoran Yang, and Zhaoran Wang. How does goal relabeling improve sample efficiency? In Forty-first International Conference on Machine Learning, 2024

    2024

  45. [54]

    robosuite: A modular simulation framework and benchmark for robot learning

    Yuke Zhu, Josiah Wong, Ajay Mandlekar, Roberto Mart \' n-Mart \' n, Abhishek Joshi, Soroush Nasiriany, and Yifeng Zhu. robosuite: A modular simulation framework and benchmark for robot learning. arXiv preprint arXiv:2009.12293, 2020

    2009 arXiv

  46. [55]

    @esa (Ref

    \@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should ...

  47. [56]

    \@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@firs...

  48. [57]

    @open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibset...

    2019 arXiv

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.