{"id":"0441fa30-38fa-4878-8b74-3f07b7cee7eb","arxiv_id":"2502.10097","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":6,"one_line_summary":"CIP combines DirectLiNGAM-style causal masks for state-reward and action-reward links with counterfactual data augmentation and an empowerment objective to improve RL sample efficiency.","lead":"This paper introduces Causal Information Prioritization (CIP), a reinforcement learning method that infers which state and action dimensions cause rewards, then uses that information to augment data and guide exploration. The authors report large sample-efficiency gains across 39 continuous-control tasks, but the central data-augmentation mechanism rests on an unproven and likely invalid exchangeability assumption.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Counterfactual augmentation swaps state variables that can still affect rewards indirectly, so generated transitions are not valid MDP transitions.","rationale":"After reading the paper in good faith, the empirical results are extensive, but the central causal mechanism is the load-bearing part of the claim. The counterfactual augmentation is not a minor detail; it is the primary explanation for sample-efficiency gains and is used to construct training data. The validity of the swap requires an independence condition that is neither stated, proven, nor tested. A variable that merely lacks a direct edge to reward can still be an indirect cause, and the swap then fabricates impossible transitions. This would corrupt training data and cannot be the reliable source of improvement claimed. The linearity mismatch of DirectLiNGAM with the nonlinear reward function further undermines the learned masks. I agree with the reader's assessment: the central claim as written is broken. The paper could potentially be salvaged by reframing as an empirical method whose gains may arise from regularization or data augmentation effects, but that would be a different claim. Hence the verdict should remain REJECT.","tokens_in":29618,"tokens_out":6159,"duration_ms":56866,"concrete_test":"Construct a two-variable factored MDP with ground-truth graph s1_t -> s2_{t+1}, s2_t -> r_t, and no s1 -> r direct edge. Generate trajectories, run CIP's causal discovery to obtain M^{s->r}, and check whether s1 falls in the uncontrollable set. Then generate counterfactual transitions using the Section 4.1 swap and evaluate whether each synthetic tuple (s_t', a_t, s_{t+1}', r_t') is realizable under the true transition function, e.g., by comparing the true conditional next-state distribution with the swapped next state. If a large fraction of swaps are inconsistent, the augmentation is invalid. Separately, generate data from a nonlinear reward, e.g., r = ||M⊙s||^2, and test whether DirectLiNGAM recovers the true parent set; if not, the learned masks are unreliable.","verdict_should_be":"REJECT","load_bearing_attack":"The central claim is that CIP improves sample efficiency by prioritizing causal information, with the key mechanism being the counterfactual data augmentation of Section 4.1. The method defines the uncontrollable set U_s as state variables with no direct edge to the reward (Eq. 4) and constructs synthetic transitions by swapping those variables across trajectories. This is valid only if swapped variables are causally isolated: they must have no influence on the reward or on any other state variable, at any time. The paper never establishes this. In a general factored MDP, a variable with no direct edge to reward can still affect reward through indirect paths, such as s_i -> s_j -> r, or through the transition dynamics, such as s_i_t -> s_j_{t+1}, which changes future rewards. Swapping such a variable produces a tuple (s_t', a_t, s_{t+1}', r_t') that is not a realizable transition of the true MDP, because the next-state components that depend on the swapped variable are left unchanged. The reward mask in Eq. 2 does not help, because the augmented object is a full transition, and the dynamics P(s'|s,a) still couple the swapped dimension to others. Assumption 4 only asserts self-loop edges, and Propositions 1-2 concern direct edges to reward; neither implies the required independence. Additionally, DirectLiNGAM is used to learn the causal matrices, but it assumes a linear non-Gaussian SEM, whereas the reward function in Eq. 2 is unrestricted and in practice nonlinear; no identifiability result is provided for this mismatch. The empirical gains could therefore come from something other than the stated causal mechanism, leaving the central claim unsupported.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper proposes Causal Information Prioritization (CIP), a reinforcement learning method that uses causal discovery (DirectLiNGAM) on collected transitions to estimate state-to-reward and action-to-reward causal matrices, then uses the state-to-reward matrix to perform counterfactual data augmentation by swapping state variables that have no direct edge to the reward, reweights actions by the action-to-reward matrix, and adds a causality-aware empowerment term to the policy objective. The method is evaluated on 36 (or 39, the text is inconsistent) continuous control tasks, including locomotion, manipulation, sparse-reward, and pixel-based settings, and is reported to outperform SAC, ACE, BAC, and IFactor in most tasks. The paper also includes ablations, hyperparameter sensitivity analyses, and a generalization study on MT1/MT10.","tokens_in":29937,"tokens_out":11217,"duration_ms":94959,"significance":"The paper addresses an important problem—sample efficiency in reinforcement learning—and the idea of using causal structure to guide data augmentation and exploration is timely. The experimental scope is unusually broad: 5 environments, pixel-based and sparse-reward variants, 4 random seeds, and statistical analyses (pairwise t-tests, IQM, optimality gap). The authors provide code in the supplementary material and report computation costs. If the causal mechanism were sound, the near-100% success rates on Meta-World and Adroit Hand would be a substantial contribution. However, the theoretical foundation is not established: the counterfactual augmentation relies on an exchangeability assumption that is neither proved nor tested and is generally false in factored MDPs with cross-variable dependencies; the identifiability theorem is asserted without proof and is incompatible with the nonlinear reward functions in Eq. (2); and the empowerment objective in Eqs. (5)-(9) is not internally consistent. These gaps undermine the causal interpretation of the empirical gains.","major_comments":[{"comment":"The counterfactual augmentation step is valid only if the swapped variables are causally isolated, i.e., they have no influence on the reward or on any other state variable at any time. The uncontrollable set U_s in Eq. (4) is defined by the absence of a direct edge s_i^t → r_t, but a variable without a direct edge can still influence the reward through indirect paths, e.g., s_i^t → s_j^t → r_t or s_i^t → s_j^{t+1} → r_{t+1}, and the transition dynamics couple the swapped components to the unswapped ones. Swapping (s_i^t, s_i^{t+1}) across trajectories while leaving the other coordinates unchanged therefore produces tuples (s̃_t, a_t, s̃_{t+1}, r_t) that are not realizable transitions of the MDP. Assumption 4 (self-loop edges) and Propositions 1-2 (direct edges to reward) do not rule out such couplings. The paper neither proves the required exchangeability nor tests whether the augmented transitions are dynamically consistent; this is the central mechanism of the claimed improvement.","section":"Section 4.1, Eq. (4)"},{"comment":"Theorem 1 asserts that the causal matrices M^{s→r} and M^{a→r} are identifiable from the factored-MDP reward model, but the proof is omitted; Proposition 2's proof is also deferred with 'a similar line of reasoning.' More importantly, the identifiability statement is incompatible with the unrestricted reward function R in Eq. (2): DirectLiNGAM assumes a linear non-Gaussian structural equation model, while continuous-control reward functions are generally nonlinear (e.g., quadratic penalties, contact terms). No identifiability result for the nonlinear case is provided, and Appendix D.3.7 compares DirectLiNGAM only with GES and PC, not against known ground-truth graphs. Because the causal matrices are the sole basis for both the augmentation and the action reweighting, this gap is load-bearing.","section":"Appendix B, Theorem 1"},{"comment":"The action-reward causal matrix M^{a→r} is learned from a local buffer that already contains the synthetic transitions generated by the state-reward model in Step 1 (Algorithm 1: augmented transitions are added to D_c, and Step 2 samples D_a from D_c). This creates a feedback loop in which errors in the state-reward mask and in the swap operation propagate into the action-reward weights, which then reweight actions, influence the policy, and affect future data collection. The paper provides no analysis of how this self-training loop affects the reliability of the causal estimates; in particular, invalid augmentations can reinforce initial errors rather than be corrected. This concern is material because the action reweighting and empowerment objective depend on M^{a→r}.","section":"Section 4.2 and Algorithm 1"},{"comment":"The derivation of the causality-aware empowerment objective is internally inconsistent. Eq. (5) defines the objective as a mutual information, but Eqs. (6)-(7) insert the causal weights M^{a_i→r} as multiplicative factors inside the entropy summands, and Eq. (8) drops the weights entirely when writing the log-ratio form. Eq. (9) adds αE^{φ_c}(s) inside the expectation over a_t, although the empowerment term as defined is a function of the policy and state, not of the sampled action. If the objective is a heuristic reward-shaping term, this should be stated explicitly; as written, the learning objective in Eq. (8) cannot be derived from Eqs. (6)-(7), and the causal action reweighting ('reweight them as (ω_1 a_1, ...)') is never specified in the algorithm or the implementation details.","section":"Section 4.2, Eqs. (5)-(9)"}],"minor_comments":[{"comment":"Inconsistent task counts: the Abstract and Conclusion state 39 tasks, while Section 5.1, Figure 3, and Figure 4 state 36 tasks; the pixel-based section mentions 3 DMControl tasks but Figure 6 includes 4 tasks (including Cartpole). Please reconcile these numbers.","section":"Abstract, Section 5.1, Figure 3, Figure 4"},{"comment":"In Algorithm 1, Step 1's comment says 'Learn causal mask matrix M^{a→r}' but the context requires M^{s→r}; this is a typo that obscures the pipeline.","section":"Algorithm 1"},{"comment":"The causal discovery comparison refers to 'DirectLiNAM' (twice) instead of DirectLiNGAM; also 'ReacherEeasy' appears in Figure 15. These typos should be fixed.","section":"Appendix D.3.7 and Figure 15"},{"comment":"The reliability evaluation excludes the Adroit Hand door task 'due to CIP's exceptional performance there'; excluding a task on this basis can bias the optimality-gap comparison and should be justified or documented as a sensitivity analysis.","section":"Section 5.3, Figure 8"},{"comment":"The claim that CIP 'consistently outperforms' is stronger than the evidence in Table 1: pairwise t-tests show significant superiority in only 5 of 8 locomotion tasks, and BAC has a higher mean return on Ant. The text should be qualified accordingly.","section":"Section 5.2, Table 1"}],"recommendation":"reject","confidential_remarks":"The manuscript is an arXiv version of a paper already published at ICLR 2025. The ICLR acceptance is not a substitute for journal-level scrutiny. In my view, the counterfactual augmentation is invalid under general factored MDPs and the identifiability theorem is unproven, so the central causal claim is not defensible without substantial additional theory and experiments. The broad empirical evaluation is a strength, but it does not compensate for the unsupported core mechanism."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know. First, the empirical story is strong: CIP beats SAC, ACE, and BAC across a wide set of continuous control tasks, with ablations showing both the augmentation and the empowerment term contribute. Second, the causal story that explains these gains does not hold up. The counterfactual augmentation in Section 4.1 swaps state variables that merely lack a direct edge to the reward. That is not enough to make them safe to swap. A variable with no direct edge can still influence the reward through other state variables or through the transition dynamics, and swapping it produces a tuple that the true MDP would never generate. The paper never establishes the needed isolation; Assumption 4 only asserts self-loops, and Propositions 1-2 are about direct reward edges. So the mechanism the paper credits for the improvements is unsupported. You can still believe the augmentation helps—the ablation says it does—but not for the stated causal reason. It may be acting as generic data augmentation.\n\nThe new piece is honest and useful: the combination of state-reward and action-reward causal masks with empowerment is new relative to IFactor, ACE, and the CoDA/MoCoDA line. The benchmark is broad: 36 tasks (the abstract says 39), pixel-based and sparse-reward settings, ablations, statistical tests, and code in the supplement. That is a real contribution, though the task-count inconsistency makes you check whether the numbers were assembled carefully.\n\nTwo more soft spots. DirectLiNGAM assumes linear non-Gaussian noise, but the reward function in Eq. 2 is not assumed linear; there is no identifiability result for that mismatch. Theorem 1 is asserted with the proof deferred, and the appendix does not actually deliver a full proof. Also, hyperparameters like the temperature α are tuned on the reported tasks, so the headline comparisons are not fully clean.\n\nWho should read this: someone comparing causal add-ons to SAC empirically will get a useful data point; someone building on the causal mechanism should be careful. For peer review, I would send it out—it is substantial and the flaws are about interpretation, not about sloppy execution—but I would expect the referees to reject in its current form, or demand major revision that either proves the swap validity or reframes the contribution as empirical augmentation.","headline":"Strong empirical package, but the causal mechanism at the core is not justified, so the paper as written overclaims.","tokens_in":30505,"tokens_out":3125,"would_cite":false,"duration_ms":29535,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that a reinforcement-learning agent becomes dramatically more sample-efficient when it first learns which state and action dimensions causally drive reward, then swaps unrelated state variables across trajectories to…","keywords":["causal reinforcement learning","sample efficiency","factored MDP","counterfactual data augmentation","empowerment","causal discovery","DirectLiNGAM","continuous control"],"falsifier":"Construct a factored MDP where $s_1 \\to s_2 \\to r$ and $s_1$ has no direct edge to $r$. Since DirectLiNGAM estimates linear direct effects, $s_1$ could fall into the uncontrollable set; if CIP then swaps $s_1$ across trajectories, the synthetic transitions mix inconsistent $s_1$–$s_2$ pairs and the learned policy should degrade relative to CIP with augmentation disabled, or relative to a variant that masks $s_1$ entirely. Running that comparison would settle whether the counterfactual swap assumption holds.","tokens_in":29394,"feed_emoji":"🎯","tokens_out":6914,"duration_ms":52914,"temperature":0.7,"pith_summary":"CIP is a reinforcement-learning method that infers which state and action dimensions causally affect the reward, and then concentrates learning on those dimensions to cut sample complexity. It does this in two ways: by swapping causally irrelevant state variables across collected transitions to synthesize extra training data, and by reweighting actions according to their learned causal influence while maximizing a causality-aware empowerment objective. The paper claims this combination consistently outperforms strong baselines across 39 continuous-control tasks, including sparse-reward and pixel-based settings, with near-100% success on the Meta-World and Adroit door manipulation benchmarks. If true, it offers a practical recipe for making RL sample-efficient in real-world robotics, where environment interactions are expensive.","feed_headline":"Causal prioritization lifts RL to near-perfect scores on 17 tasks","feed_subtitle":"Swapping reward-irrelevant variables and causally reweighting actions helps RL agents master hard manipulation faster.","key_machinery":"The central object is the reward-guided causal mask pair $(M^{s\\to r}, M^{a\\to r})$ learned from a factored MDP, together with the uncontrollable set $U_s = \\{s_i \\mid M^{s\\to r}\\cdot(s_i^t, r_t) < \\theta\\}$. The uncontrollable set drives counterfactual data augmentation: variables in $U_s$ are assumed to be exchangeable across transitions, and swapping them generates synthetic training data that emphasize causally relevant state dimensions. The action mask $M^{a\\to r}$ then reweights actions, and the causality-aware empowerment objective $E^{\\pi_c}(s) = \\max_a I(a_t; s_{t+1}\\mid s_t; M)$, computed as the entropy difference between the policy and an inverse dynamics model (each weighted by $M^{a\\to r}$), is added to the Bellman target $T^{\\pi_c}Q = r(s_t, a_t) + \\gamma E[Q(s_{t+1}, a_{t+1}) + \\alpha E^{\\pi_c}(s)]$. The mutual-information objective is what converts the passive causal masks into an exploration bonus that prefers actions with predictable, causally relevant consequences.","core_discovery":"CIP treats the environment as a factored MDP whose reward is $r_t = R(M^{s\\to r}\\odot s_t, M^{a\\to r}\\odot a_t, \\epsilon_{r,t})$, where the binary/weight matrices $M^{s\\to r}$ and $M^{a\\to r}$ encode which state and action dimensions causally influence reward. It learns these masks with DirectLiNGAM from collected trajectories. State dimensions with no edge to the reward form the 'uncontrollable set' $U_s$; pairs of transitions sharing uncontrollable variables are counterfactually augmented by swapping those variables, producing synthetic tuples $(\\tilde s_t, \\tilde a_t, \\tilde s_{t+1}, \\tilde r_t)$ without additional environment interaction. Separately, the learned action-reward weights $\\omega$ reweight actions, and a causality-aware empowerment objective $E^{\\pi_c}(s) = H(\\pi_c(a_t|s_t)) - H(\\pi_c(a_t|s_t, s_{t+1}))$ (with each entropy term masked by $M^{a\\to r}$) is added to the Bellman update. The paper's central claim is that this combination, counterfactual state-swapping plus causally reweighted empowerment, is what makes policies learn faster and reach near-perfect success on tasks like Adroit door opening where baseline SAC, ACE, and BAC barely progress.","pith_inferences":["A testable extension: on environments where a state variable has no direct edge to reward but acts indirectly through another state variable, CIP's swap creates invalid transitions; a variant that masks such variables instead of swapping them should reveal whether the augmentation's validity assumption is doing the work.","If the causal masks are accurate, the empowerment objective is essentially a controllability bonus over causally relevant action dimensions; one could drop the causal discovery step and use learned action importance to see how much of the gain is causal versus merely dimensional attention.","The paper's own future-work list joins CIP with object-centric world models; a concrete next step would reuse $M^{s\\to r}$ as a sparsification prior on object attributes in a 3D manipulation context."],"forward_implications":["On the 17 Meta-World robot-arm tasks, CIP reaches a near-perfect normalized score of 100, and it achieves a near-100% success rate on the 28-dimensional Adroit door task where SAC, BAC, and ACE make little progress.","Ablations show that removing the empowerment term hurts performance the most, and removing counterfactual data augmentation reduces sample efficiency, so both components carry the reported gains.","CIP's state-swapping augmentation works without extra environment interaction, which is what makes its sample-efficiency gains not an artifact of more data.","In pixel-based DMControl tasks, CIP combined with IFactor's latent states outperforms IFactor alone, indicating the causal prioritization transfers to high-dimensional observations."],"supporting_citations":[{"why":"Supplies DirectLiNGAM, the linear non-Gaussian causal discovery algorithm used to learn the state–reward and action–reward weight matrices.","marker":"(Shimizu et al., 2011)"},{"why":"Provides the factored MDP framework that lets CIP treat each state and action dimension as a node in a causal graph.","marker":"(Kearns & Koller, 1999)"},{"why":"Gives the SCM and d-separation machinery used to state the identifiability assumptions for the causal masks.","marker":"(Pearl, 2009)"},{"why":"Defines SAC, the maximum-entropy actor-critic baseline that CIP extends and compares against across all tasks.","marker":"(Haarnoja et al., 2018)"},{"why":"Provides IFactor, the pixel-based latent factorization baseline CIP is combined with in image-input experiments.","marker":"(Liu et al., 2024)"},{"why":"Defines ACE, the causality-aware entropy-regularization baseline CIP is compared with in continuous control.","marker":"(Ji et al., 2024a)"},{"why":"Introduces empowerment, the mutual-information control measure underlying CIP's causality-aware empowerment objective.","marker":"(Klyubin et al., 2005)"},{"why":"Supplies the faithfulness and Markov assumptions under which the paper's identifiability theorem for the masks is stated.","marker":"(Spirtes et al., 2001)"}],"fun_headline_variants":["Causal masks and counterfactual swaps speed RL learning","Reward-guided causal inference makes RL sample-efficient","CIP: Prioritizing causal info to master complex RL tasks","Counterfactual state swaps and causal empowerment boost RL"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that any state variable with no direct causal edge to the reward can be swapped across transitions without changing the dynamics or reward: the paper assumes such variables have no indirect influence through other state dimensions, and it neither proves nor tests this.","fun_headline_variants_meta":{"raw":{"variants":["Causal masks and counterfactual swaps speed RL learning","Reward-guided causal inference makes RL sample-efficient","CIP: Prioritizing causal info to master complex RL tasks","Counterfactual state swaps and causal empowerment boost RL"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000307,"raw_usage":{"total_tokens":1808,"prompt_tokens":1047,"completion_tokens":761,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":663,"completion_tokens_details":{"reasoning_tokens":705}},"tokens_in":663,"tokens_out":761,"duration_ms":7607,"temperature":1.0,"reasoning_tokens":705,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T19:27:26.183003+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Construct a factored MDP where $s_1 \\to s_2 \\to r$ and $s_1$ has no direct edge to $r$. Since DirectLiNGAM estimates linear direct effects, $s_1$ could fall into the uncontrollable set; if CIP then swaps $s_1$ across trajectories, the synthetic transitions mix inconsistent $s_1$–$s_2$ pairs and the learned policy should degrade relative to CIP with augmentation disabled, or relative to a variant that masks $s_1$ entirely. Running that comparison would settle whether the counterfactual swap assumption holds.","supporting_citations":[{"cited_title":"For pixel-based DMControl environments, we employ IFactor (Liu et al.,","cited_arxiv_id":null,"evidence_quote":"Defines SAC, the maximum-entropy actor-critic baseline that CIP extends and compares against across all tasks."},{"cited_title":"The experimental results in Figure 32 across three tasks demonstrate that our chosen DirectLiNAM method exhibits superior performance compared to both alternatives","cited_arxiv_id":null,"evidence_quote":"Supplies the faithfulness and Markov assumptions under which the paper's identifiability theorem for the masks is stated."}],"review_version":1}