{"id":"5843f192-a874-4025-90ac-15497ec14e92","arxiv_id":"2505.03172","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A new null-counterfactual interaction detector (NCII) and a hindsight-filtering method (HInt) improve goal-conditioned RL sample efficiency in object-centric domains.","lead":"Researchers introduce a new way to detect when one object genuinely affects another in robot simulations, by asking what would happen if the cause object simply did not exist. They use this detection to filter which past experiences get reused in goal-conditioned reinforcement learning, improving sample efficiency by up to 4x in object-centric robotics tasks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Simulated nulling (App. D) substitutes passive predictability for true null counterfactuals; unvalidated, this proxy can mislabel interactions when gravity/friction dominate target motion, undermining NCII accuracy and HInt's NCII-based filtering.","rationale":"The paper's two headline claims—NCII's interaction inference accuracy and HInt's 4x sample efficiency gain—both depend, in the physical domains, on Appendix D's simulated nulling being a faithful stand-in for Definition 3.1's null states. The reader's CONDITIONAL verdict already flags this assumption; my stress-test agrees and sharpens the failure mode. If gravity/friction makes a target's motion predictable from its own state, the passive signal is high for genuinely interacting transitions, so the pseudo-null training data (Eq. 7) teaches the forward model that nulling the cause is a no-op, biasing Eq. 3 toward false negatives. Because Table 1 uses contact as ground truth and the paper itself notes contact is not identical to null counterfactual, the reported accuracy cannot rule out this bias. A direct comparison against true null labels is cheap in the same simulators and would settle whether the central RL claim rests on a valid induction. If the test passes, the claims stand; if it fails, the NCII accuracy claim and the NCII-based HInt results would need to be re-evaluated. Verdict remains CONDITIONAL as the reader stated; no change is needed, but the condition should be made explicit in revision.","tokens_in":22245,"tokens_out":8138,"duration_ms":79147,"concrete_test":"Concretely: in a simulated domain such as Spriteworld or Air Hockey, construct a held-out set of transitions (s,a,s'). For each, produce the true null-counterfactual next-state distribution by re-running the environment with the cause factor removed (e.g., deleting the cause object or setting its state to the null marker used by the environment), and label the transition as interacting iff this changes the target's next-state distribution (Eq. 1). Then train NCII on the same rollouts using (a) genuine null trajectories and (b) the Appendix D simulated-nulling proxy (Eqs. 6-7). Compare the two models' misprediction rates on the subset of test transitions where the passive autoregressive model's likelihood is high but the true null label says interacting.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Definition 3.1 requires comparing the observed transition probability against the probability under a genuine null state where the cause factor is absent (Eq. 1). In physical domains, no explicit null trajectories exist, so Appendix D replaces them with 'simulated nulling': a passive signal e_passive_j(s_j,s'_j) = log f(s_j)[s'_j] (Eq. 6), and transitions with high passive likelihood are treated as non-interactions and used to generate pseudo-null training data by randomly zeroing factor indices (Eq. 7). This substitution is never validated against actual null counterfactuals. The failure mode is concrete: if the target's motion is dominated by environment forces that are predictable from its own state (gravity in Air Hockey, friction in Robosuite), the passive signal is high even while a cause object is acting on the target. The masked forward model f(s,a,B(v);theta) is then trained on transitions where nulling the cause is labeled as no change, so Eq. 3 under-detects these interactions. HInt's NCII-based filtering (used in Sprite default/small, Robo default, Air default, Kitchen default/obstacles) inherits these labels. Additionally, Table 1 evaluates against contact, not null counterfactuals, and Section 5.2.1 explicitly notes the two can disagree ('rejecting contact that only produces minute changes'), so contact-based numbers cannot validate the proxy. The central claims thus hinge on an unexamined equivalence: that simulated nulling produces the same conditional distributions as genuine null states.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes two connected methods. NCII infers object interactions by comparing a learned forward model's prediction of a target factor's next state with its prediction under a nulled cause factor, formalized in Definition 3.1 and Eqs. (2)-(3). When explicit null trajectories are unavailable, Appendix D substitutes a passive-signal proxy: transitions in which the target is predictable from its own state are treated as non-interactions used to generate pseudo-null training data (Eqs. (6)-(7)). HInt then uses the inferred interaction graph to filter hindsight relabeling, keeping only trajectories containing a path from the action to the target factor, and commits the hindsight goal to an interaction time step. Experiments compare NCII's misprediction rate with JACI, gradient, attention, and NCD baselines in Random DAG, Spriteworld, Robosuite, Air Hockey, and Franka Kitchen (Table 1), and compare HInt's goal-conditioned RL sample efficiency with HER, prioritized replay, f-pg, ELDEN, and CAI (Figure 4 and Appendix M), reporting up to a 4x improvement.","tokens_in":22525,"tokens_out":6975,"duration_ms":68983,"significance":"If the claims are fully supported, the paper makes a useful contribution: it gives a tractable, formal definition of interaction based on null counterfactuals, applies it to interaction inference, and demonstrates that interaction filtering can improve hindsight relabeling in object-centric GCRL domains. Strengths include the explicit formalization in Definition 3.1, the iterative joint training of the forward and interaction models, the breadth of benchmark domains, evaluation against external task reward for the RL result, and a reproducibility statement with code in the appendix. The main weakness is that the interaction-inference claim in physical domains rests on an unvalidated proxy for the null counterfactual and an evaluation label (contact) that the paper itself acknowledges can disagree with the proposed definition. This does not invalidate the external-reward RL results, but it does mean the NCII accuracy claim, as stated, is not currently established.","major_comments":[{"comment":"The simulated nulling proxy is unvalidated against the paper's own null-counterfactual criterion. Appendix D replaces the genuine null state of Definition 3.1 with a passive autoregressive signal: Eq. (6) declares a target non-interacting when f(s_j) predicts s'_j well, and Eq. (7) uses such states to generate pseudo-null training data. This substitution is never checked against actual null counterfactuals. The failure mode is concrete: in Air Hockey, gravity can make the puck's motion predictable from its own state even while the paddle is striking it, and in Robosuite, quasistatic friction can make a pushed block's motion predictable from its own state. In such cases the passive signal is high despite an interaction, so Eq. (3) will under-detect interactions. Table 1 cannot arbitrate this because its ground truth in physical domains is contact, not the Definition 3.1 null-counterfactual criterion, and Section 5.2.1 explicitly notes that contact and null-counterfactual interaction can disagree (\"rejecting contact that only produces minute changes\"). The authors should either validate the passive-signal proxy in a domain with explicit null trajectories or report evaluation against true null-counterfactual labels; otherwise the NCII accuracy claim in Table 1 is not supported.","section":"Appendix D, Eqs. (6)-(7); Section 5.1; Definition 3.1"},{"comment":"There is a load-bearing inconsistency in the definition of the interaction model h. The text states h:S×A→B, but Eq. (4) trains h(s;φ) on state alone, and Algorithm 1 lists h(s;ϕ) as an input. Since the training targets Null(s,a,θ) in Eq. (3) depend on the action, a state-only function cannot represent the action column of B and cannot produce the action-to-target paths that HInt uses for filtering in Section 4.2. The authors should clarify whether h actually receives (s,a); if it does, Eq. (4), Algorithm 1, and the discussion in Appendix E (where control factors are instead identified by training 2n separate models) need to be corrected consistently. If h is intentionally state-only, the paper must explain how action edges are obtained for the \"action graph\" and \"control-target\" filtering criteria.","section":"Section 4.1, Eq. (4); Algorithm 1"},{"comment":"The interaction inference accuracy is reported on test sets reweighted to 50% interactions (Table 1 caption), whereas Appendix D states that natural interaction frequencies in the physical domains range from 0.5% to 0.01%. HInt's hindsight filter operates on natural-frequency trajectories, so a 50%-interaction test set does not measure the operating regime of the downstream algorithm. The manuscript should report confusion-matrix metrics (precision, recall, and misprediction rate) at natural interaction frequencies, or otherwise show that the reweighted accuracy transfers to the rare-interaction regime that motivates the method.","section":"Table 1 caption; Appendix D; Figure 4"}],"minor_comments":[{"comment":"The expression in Eq. (3) appears to take the logarithm of a difference of probabilities; the intended null test is a difference of log-likelihoods of the observed next state, and the argument should be the observed next state s'_j rather than s_j. Please correct the notation and the closing parenthesis.","section":"Eq. (3)"},{"comment":"There is a typo: \"Bernolli\" should be \"Bernoulli\".","section":"Appendix D, Eq. (7)"},{"comment":"The abstract claims \"up to 4x\" sample-efficiency improvement, but the paper does not report a numeric fixed-budget comparison or a statistical significance test; with only 5 trials per condition, the visual separation in Figure 4 should be supplemented with confidence intervals or a significance test for the claimed improvement magnitude.","section":"Section 5.2 and Figure 4"},{"comment":"The text states that NCII achieves a \"statistically significant reduction in misprediction rate\" but no significance tests are reported; either include pairwise significance tests or temper the wording to \"lower misprediction rate\".","section":"Section 5.1, Table 1"},{"comment":"Please clarify whether the binary cross-entropy loss in Eq. (4) is computed over all entries of the interaction matrix B including the action column, and specify how the action-column targets are obtained when h is trained on state only.","section":"Section 4.1, Eq. (4)"}],"recommendation":"major_revision","confidential_remarks":"The central gating issue is the validation of the simulated nulling proxy in Appendix D. The external-reward RL results and the reproducible code are strengths, and I do not question the novelty or the fit with the journal's scope. If the authors can validate the proxy against true null counterfactuals in at least one physical or synthetic domain, or explicitly restrict the NCII claims to the operationalized passive-signal definition, I would be willing to accept the paper after the revisions."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"One thing to know: this is a solid ICLR paper that deserves a serious referee, not a desk reject. The new idea is Definition 3.1—call an interaction a change in transition probability when a cause factor is replaced by a null state. That's a clean simplification of actual causality, and it's new enough. The authors build NCII on it with a masked forward model, then use NCII to filter hindsight relabeling in HInt. The RL results in Figure 4 look plausible: HInt consistently beats vanilla hindsight, and the up-to-4x sample efficiency gain is measured against external task reward, so there's no circularity there. The Figure 5 heatmap showing the hindsight goal distribution after filtering is a nice piece of evidence.\n\nWhat's weak is the inference evaluation. Table 1 reports misprediction rate against physical contact, but Definition 3.1 is about null counterfactuals. The paper itself admits contact and null counterfactual can disagree—rejecting contact that only produces minute changes. So the 'inference accuracy' numbers don't actually validate the definition. That's a real mismatch, but not fatal for the RL story, because the RL filter can work even with a contact-based proxy.\n\nThe bigger risk is Appendix D. In physical domains there are no explicit null trajectories, so they simulate nulling with a passive predictability signal: if an autoregressive model predicts the target's next state well, they treat it as non-interacting and zero out cause indices. That's an unexamined equivalence. If gravity or friction dominate the target's motion, the passive signal is high while another object is also acting, and the training labels for the masked model will be wrong. The stress-test note is correct that this is the load-bearing assumption. There is some indirect evidence it isn't catastrophic—HInt with NCII matches or beats HInt with contact ground truth in most domains—but that's not a direct validation.\n\nMinor: the reproducibility statement says code is in the appendix, but the arXiv manuscript doesn't actually contain it. Easy fix. Also several filter thresholds and reweighting parameters are hand-chosen, though they do include some ablations.\n\nBottom line: This is for researchers in goal-conditioned RL and causal interaction inference. The conceptual definition and the hindsight filtering idea are worth engaging with. I'd send it to reviewers, and I'd push the authors to validate the passive-nulling proxy, either in a toy domain where true nulls are available or by comparing with explicit null trajectories in Robosuite.","headline":"A genuinely new interaction definition and a sensible hindsight filter, but the inference accuracy claim is measured against contact, not null counterfactuals, and the simulated-nulling proxy is the real load-bearing risk.","tokens_in":23104,"tokens_out":2680,"would_cite":true,"duration_ms":25201,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Filtering hindsight relabeled goals by null-counterfactual interactions improves sample efficiency up to 4x in object-centric goal-conditioned reinforcement learning.","keywords":["goal-conditioned reinforcement learning","hindsight relabeling","null counterfactual","factor interactions","causal inference","object-centric control","sample efficiency","masked dynamics models"],"falsifier":"Train NCII on a domain where the target object moves predictably under its own dynamics (for example constant gravity) while a second object also contacts it, then compare NCII's interaction labels against true contact timestamps: elevated misprediction exactly on contact-free but self-predictable spans would show that the simulated-nulling proxy confounds interaction with self-predictability.","tokens_in":21955,"feed_emoji":"🎯","tokens_out":8331,"duration_ms":76548,"temperature":0.7,"pith_summary":"Hindsight relabeling in goal-conditioned reinforcement learning gives high reward to trajectories that never touch the target object, which floods the replay buffer with useless behavior in object-centric tasks. The paper's core claim is that this failure is fixable by filtering relabeled trajectories with a causal interaction test: a cause factor interacts with a target when removing the cause to a null state changes the target's transition. To operationalize that, the paper introduces NCII, a masked forward model that learns to predict null-counterfactual outcomes and labels interactions, and HInt, which only relabels trajectories where a path in the unrolled interaction graph runs from the agent's action to the goal factor. Empirically, NCII lowers interaction misprediction rates relative to prior inference baselines, and HInt improves sample efficiency by up to 4x across Spriteworld, Robosuite, Robot Air Hockey, and Franka Kitchen. If the definition holds up, hindsight replay can be made safe for domains where the agent must learn to act on objects rather than learn to ignore them.","feed_headline":"Null-counterfactual filter speeds up goal-conditioned RL 4x","feed_subtitle":"Removing trivial hindsight goals stops agents from learning to ignore the very object they should control.","key_machinery":"The central object is the null-counterfactual interaction test: a comparison between a learned forward model's log-likelihood of the observed next state with all factors present versus with the candidate cause 'nulled' to its absent state. NCII instantiates this as a masked dynamics model $f(s,a,B(v);\\theta)$ over factored MDP states, trained by maximum likelihood on trajectories with varying subsets of factors; the interaction model $h(s;\\phi)$ distills the null test into fast, soft predictions and is retrained jointly with $f$. HInt then converts the learned interaction graph $B^{(t)}$ at each timestep into an unrolled temporal interaction graph and filters hindsight relabeling to trajectories containing a control-target path, since those are trajectories where the agent, rather than passive motion, is responsible for the goal factor's change.","core_discovery":"The paper's central claim is that interactions defined by null counterfactuals give a tractable, learnable signal for which trajectories hindsight relabeling should use. Formally, every state factor $S_i$ has a null state $s_{i,\\circ}$, and $S_i$ interacts with target $S_j$ in transition $(s,a,s')$ exactly when $p(S_j=s'_j \\mid S=s) \\neq p(S_j=s'_j \\mid S=s^{\\circ}_{s_i})$. NCII trains a masked forward model on trajectories with varying subsets of factors, then uses the difference in log-likelihood between the observed outcome with and without the candidate cause nulled to label interactions. HInt then keeps a trajectory only if the unrolled interaction graph contains a path from the action node to the target factor, relabeling with the state at a time on that path. Across Spriteworld, Robosuite, Robot Air Hockey, and Franka Kitchen, HInt matches or exceeds vanilla hindsight, prioritized replay, and interaction-based exploration baselines, reaching the same performance up to 4x sooner.","pith_inferences":["A consequence the paper leaves implicit is that the null-counterfactual test is a continuous responsibility signal, so a graded version could serve as reward shaping or an exploration bonus rather than only a binary hindsight gate.","The simulated-nulling proxy ties NCII's accuracy to the passive model's autoregressive quality; a testable extension is to train the masked forward model with null masks generated from the passive signal at train time and compare against fixed masks.","Because the filter rejects whole trajectories, it may discard episodes where a gradual, weak push eventually moves the target; scoring trajectories by interaction count or magnitude rather than binary membership would be a natural extension."],"forward_implications":["Hindsight buffers in object-centric GCRL should contain only trajectories whose interaction graph has an action-to-target path; filtering out the rest removes the dominant 'target did nothing' relabeled goals.","Because NCII matches or beats prior interaction inference methods even without ground-truth contact labels, HInt can be deployed in physical domains where contacts are not directly observable.","HInt with learned NCII interactions reaches the same or better asymptotic performance than HInt with ground-truth contacts, so the benefit does not hinge on perfect inference.","In locomotion or navigation settings with no object to act on, the interaction filter adds little or can be detrimental, which bounds the method's applicability to interaction-centric tasks."],"supporting_citations":[{"why":"Defines hindsight experience replay, the relabeling mechanism HInt filters and the primary baseline it must beat.","marker":"Andrychowicz et al., 2017"},{"why":"Supplies the actual-causality formalization that the null-counterfactual definition is designed to make tractable.","marker":"Halpern, 2016"},{"why":"Proposes the actual-cause interaction framework HInt builds on and supplies the JACI baseline for interaction inference.","marker":"Chuck et al., 2024b"},{"why":"Provides Neural Contextual Decomposition, a comparison method in the inference experiments.","marker":"Hwang et al., 2023"},{"why":"Provides the gradient-based interaction inference method and the ELDEN exploration baseline.","marker":"Wang et al., 2023"},{"why":"Provides the controllability-based CAI baseline that HInt is compared against.","marker":"Seitzer et al., 2021"},{"why":"Supplies the locally factored dynamics approach to counterfactual data augmentation, an alternative to null-counterfactual filtering.","marker":"Pitis et al., 2020"},{"why":"Introduces the passive-error reweighting technique that NCII uses to simulate nulling when true null trajectories are unavailable.","marker":"Chuck et al., 2023"}],"fun_headline_variants":["Null counterfactuals spot real object touches for faster GCRL","HInt: hindsight that keeps object interactions, 4x sample boost","NCII finds causal object links to cut spurious hindsight goals","Nulling absent causes filters useless goals, speeds RL 4x"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that every state factor has a null state whose removal leaves all unrelated transitions unchanged, and that in physical domains a simulated nulling proxy—treating a target as non-interacting whenever a model using only that target's own history predicts its next state well—faithfully reproduces what would happen without the cause; if that proxy misfires, for example when gravity or friction makes a target's motion predictable while another object is also acting on it, the inferred interaction labels and thus HInt's filtering decisions inherit the error.","fun_headline_variants_meta":{"raw":{"variants":["Null counterfactuals spot real object touches for faster GCRL","HInt: hindsight that keeps object interactions, 4x sample boost","NCII finds causal object links to cut spurious hindsight goals","Nulling absent causes filters useless goals, speeds RL 4x"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000313,"raw_usage":{"total_tokens":1846,"prompt_tokens":1083,"completion_tokens":763,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":699,"completion_tokens_details":{"reasoning_tokens":700}},"tokens_in":699,"tokens_out":763,"duration_ms":6537,"temperature":1.0,"reasoning_tokens":700,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T23:57:20.077552+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train NCII on a domain where the target object moves predictably under its own dynamics (for example constant gravity) while a second object also contacts it, then compare NCII's interaction labels against true contact timestamps: elevated misprediction exactly on contact-free but self-predictable spans would show that the simulated-nulling proxy confounds interaction with self-predictability.","supporting_citations":[{"cited_title":"On discovery of local independence over continuous variables via neural contextual decomposition","cited_arxiv_id":null,"evidence_quote":"Provides Neural Contextual Decomposition, a comparison method in the inference experiments."},{"cited_title":"Counterfactual data augmentation using locally factored dynamics","cited_arxiv_id":null,"evidence_quote":"Supplies the locally factored dynamics approach to counterfactual data augmentation, an alternative to null-counterfactual filtering."}],"review_version":1}