{"id":"10e65723-8143-49ca-85a8-f5d08fece6fd","arxiv_id":"2502.02327","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":1,"one_line_summary":"PGCR trains a policy to generate modified user states and an encoder to ignore the changed parts, claiming this finds causally relevant features, and reports improved offline RL recommendation performance.","lead":"This paper proposes a two-stage method that learns which parts of a user state matter for rewards, then encodes only those parts, for offline recommendation algorithms. It reports higher cumulative rewards when the method is added to standard RL agents, but the causal mechanism it claims is not derived or validated.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The intervened state s^I is the next state under do(a_t), not a modified current state; the CRC-preservation premise behind Eq. (3) and the MSE encoder objective is unsupported.","rationale":"The reader's weakest_assumption correctly identifies the load-bearing flaw: s^I, generated by intervening on the action, is in fact the next state under the MDP dynamics, not a modified version of the current state that preserves CRCs. My analysis confirms and sharpens this: Eq. (2) makes this explicit, and Eq. (3) then applies an invalid do(s_t := s^I) substitution to the reward function. The entire causal feature-selection mechanism and the interpretation of the encoder's MSE objective depend on this one identification. Without it, the Wasserstein reward, the encoder training, and the claim that representations focus on causally relevant components are all unsupported. The paper's empirical results and ablation do provide evidence that the procedure improves cumulative reward relative to baselines, but they do not distinguish the causal explanation from alternative mechanisms such as representation augmentation or temporal smoothing; the ablation only compares against a randomly sampled state, which is even weaker than using the actual next state. Proposition 2 is a standard conditional sufficiency result and does not validate the causal interpretation, since its assumptions are not shown to be satisfied. Therefore I agree with the reader's REJECT verdict and see no reason to alter it. The concrete test I propose would settle the matter empirically: in a synthetic environment with a known CRC/CIRC decomposition, the intervention-generated s^I can be directly checked for CRC preservation and the learned representation for causal content. If the test were to unexpectedly show CRC preservation, the verdict might need reconsideration, but based on the paper's own equations, the concern stands.","tokens_in":17000,"tokens_out":5039,"duration_ms":51364,"concrete_test":"Construct a synthetic recommender MDP with a known CRC/CIRC split: let s_t = (c_t, i_t), transition s_{t+1} = (c_t + delta_c(c_t, a_t, epsilon), i_{t+1}) with a known nontrivial delta_c, and reward r_t = g(c_t, a_t). Implement PGCR exactly as described (Algorithm 1), generating s^I_t via do(a_t := a^I_t). Then (1) measure whether the CRC component of s^I_t equals c_t; under any generic transition it will not, directly falsifying the premise in Section 3.2. (2) Train the PGCR encoder and evaluate whether the latent representation z_t supports linear decoding of c_t substantially better than of i_t (e.g., compare R^2 of linear probes). If s^I_t's CRC differs from c_t or the encoder does not preferentially encode c_t, the central causal claim is unsupported. An alternative, complementary check: replace s^I_t with the actual next state s_{t+1} from the offline dataset in Eq.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that PGCR's causal feature selection policy produces modified states s^I that retain only the causally relevant components (CRCs) of the original state s_t, enabling the encoder (Eq. 6) to learn CRC-focused representations. This requires s^I to be s_t with CIRCs altered while CRCs are unchanged. However, Section 3.2 defines the intervention as do(a_t := a^I_t), and Eq. (2) derives the resulting distribution as P(s_{t+1} | s_t, a^I_t, epsilon_{t+1}) — i.e., the distribution of the next state after taking action a^I_t. A sample s^I from this distribution is a one-step-ahead state, not a modified version of the current state. Nothing in the SCM of Eq. (1) guarantees that the transition function f_P acts as identity on CRCs; in general, s_{t+1} shares neither the same CRC values nor even the same CRC support as s_t. The paper nonetheless asserts (after Eq. 2) that s^I 'is expected to preserve only the causally relevant components of the original state s_t' and, in Eq. (3), treats substituting s_t := s^I in the reward function as a valid intervention measuring the causal effect of s_t on r_t. This is a category error: r_t is defined on the current state, and using a future state as the substitute conflates temporal evolution with causal counterfactual modification. Consequently, the Wasserstein reward in Eq. (5), which compares rewards under s_t and s^I, does not measure the causal relevance of state components, and the MSE loss in Eq. (6) does not provably encourage invariance to CIRCs. The empirical gains may arise from data augmentation or from aligning representations with future states, neither of which validates the causal CRC story. Proposition 2 is a standard sufficient-condition result and does not repair this gap, since the conditions it assumes (e.g., r_t independent of s_t given z_t and a_t) are exactly what the unvalidated construction was supposed to ensure.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Policy-Guided Causal Representation (PGCR), a two-stage framework for offline reinforcement learning-based recommender systems (RLRS). In stage one, a causal feature selection policy is trained to generate a 'modified state' s^I by intervening on the action a_t, with the stated goal of retaining causally relevant components (CRCs) of the current state s_t while altering causally irrelevant components (CIRCs). In stage two, an encoder is trained with an MSE loss between latent representations of s_t and s^I, intending to make the representation focus on CRCs. The paper provides two theoretical propositions (identifiability of the causal effect of a_t on s_{t+1}, and an optimal policy depending only on the latent representation) and reports experiments on MovieLens, Coat, KuaiRec, KuaiRand, and the VirtualTaobao simulator, showing improved cumulative and average rewards over DDPG, SAC, and TD3 baselines.","tokens_in":2000,"tokens_out":2499,"duration_ms":56989,"significance":"If the causal feature selection mechanism were sound, PGCR would offer a principled approach to state representation learning in offline recommendation, potentially improving long-term reward by focusing on reward-relevant user features and augmenting limited offline data. The empirical results are consistent across several datasets and backbones, and the paper includes an ablation indicating that the learned intervention contributes beyond random state substitution. However, the significance of the work rests entirely on the causal interpretation, and, as detailed below, the causal mechanism is not supported. Without a valid causal grounding, the method reduces to an ad-hoc state alignment procedure whose mechanism is unexplained. The paper also ships no code and gives limited implementation detail, which weakens reproducibility claims.","major_comments":[{"comment":"The central premise of the paper is that the intervened state s^I, generated by do(a_t := a^I_t), is a modified version of the current state s_t that preserves only the causally relevant components. However, Eq. (2) explicitly defines the distribution of s^I as the distribution of s_{t+1} under the intervention, i.e., P_{M;do(a_t:=a^I_t)}(s_{t+1}) = E_{s_t, epsilon_{t+1}}[P(s_{t+1}|s_t, a^I_t, epsilon_{t+1})]. In the MDP of Eq. (1), the intervention on a_t produces the next state, not an altered version of the current state. The transition function f_P is arbitrary and need not act as identity on any components of s_t; in general, s_{t+1} can differ from s_t in every component, including those that are causally relevant to r_t. The assertion after Eq. (2) that 'This intervened state s^I is expected to preserve only the causally relevant components of the original state s_t' is therefore unsupported and is a category error: a future state is not a valid counterfactual surrogate for the current state.","section":"Section 3.2, Eq. (2)"},{"comment":"Equation (3) treats replacing s_t by s^I in the reward function r_t = f_R(s_t, a_t) as a legitimate intervention do(s_t := s^I), and uses the resulting reward distribution to measure the causal effect of components of s_t on r_t. This is invalid because r_t is defined on the current state; substituting a one-step-ahead state into the reward function conflates temporal evolution with a counterfactual state modification. Consequently, the Wasserstein distance in Eq. (4) and the reward function in Eq. (5) do not measure the causal relevance of the components of s_t. The entire stage-one training objective is built on this invalid substitution.","section":"Section 3.2, Eq. (3)"},{"comment":"The MSE loss in Eq. (6) aligns the latent representations of s_t and s^I_t. Even if s^I_t were a valid surrogate for a modified current state, the loss only enforces that the encoder be invariant to the difference between s_t and s^I_t. It does not guarantee that the encoder preserves information about the CRCs; a trivial encoder that maps all states to the same vector would minimize the loss exactly but would lose all information. Proposition 2 lists sufficient conditions (r_t is independent of s_t given z_t and a_t, and transition consistency) for the optimal policy to depend only on z_t, but the paper does not prove that the encoder trained with Eq. (6) satisfies these conditions. Thus the central claim that 'the encoder learns to isolate the CRCs' is not established.","section":"Section 3.3, Eq. (6)"},{"comment":"The reward function in Eq. (5) defines the notion of 'causally relevant components' through the Wasserstein distance between the original and intervened reward distributions, and the causal feature selection policy is trained to maximize that same reward. The claim that the selected components are causal is therefore true by construction of the training objective, not by an external causal validation. The ablation in Table 2 comparing PGCR against PGCR-C (with a randomly sampled state) shows that the learned intervention helps, but it does not demonstrate that the improvement arises from identifying causal components; it could simply reflect that the learned s^I is a more useful data augmentation than a random state. A comparison against a non-causal augmentation that uses actual next states from the dataset (without the intervention) would be needed to separate these effects.","section":"Section 3.2, Eq. (5) and Section 4.4"}],"minor_comments":[{"comment":"The text says 'VritualTB' in the evaluation metric description; this is a typo for 'VirtualTB'.","section":"Section 4.1.1"},{"comment":"Reference [21] contains 'Schlkopf' (missing 'ö'); it should be 'Schölkopf'.","section":"References"},{"comment":"The derivation in Eq. (2) writes P(s_t|do(a_t)) = P(s_t), but this equality holds only because s_t is not a descendant of a_t in the causal graph; stating this explicitly would improve clarity.","section":"Section 3.2, Eq. (2)"},{"comment":"The notation in Eq. (3) is confusing: it sums over a_t and eta_t but uses P(a_t|do(s_t := s^I)) as though the action distribution were recomputed under a counterfactual state, while the SCM in Eq. (1) defines a_t as a function of s_t (not of a modified s^I). The equation appears to mix interventional and observational distributions without a clear derivation.","section":"Section 3.2, Eq. (3)"},{"comment":"The proof of Proposition 2 has a gap: in Step 4, the statement that 'the latent state dynamics depend only on the latent representation' is exactly the second bullet of the proposition, not a consequence of it, and the proof does not show how the expectation over the next state in the Bellman equation can be replaced by the latent transition. The proof would benefit from a more explicit inductive argument.","section":"Appendix C"}],"recommendation":"reject","confidential_remarks":"The paper's central causal claim is unsupported by a category error in the definition of the intervened state. This is not a local fix: the method's training objectives (Eqs. (5) and (6)) and its theoretical justifications (Propositions 1 and 2) all depend on the premise that do(a_t := a^I_t) produces a modified version of the current state, which is not what the MDP dynamics in Eq. (1) specify. The empirical improvements may still be interesting as a representation learning or data augmentation technique, but the manuscript would require a major reframing, removal of the causal claims, and a new empirical validation of the non-causal mechanism. Given the paper's framing, I believe rejection is appropriate. The authors may consider resubmitting a revised version that either provides a formally correct intervention procedure that genuinely modifies the current state's components or clearly presents the method as a distribution-matching state alignment method without causal terminology."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here is my take. The method is a plausible empirical heuristic, but the causal argument is a category error. The intervention do(a_t := a^I_t) produces a sample from P(s_{t+1} | s_t, a^I_t), i.e., a one-step-ahead state, not a modified version of s_t with causally irrelevant components changed. The paper simply asserts after Eq. (2) that s^I 'is expected to preserve only the causally relevant components of the original state s_t' and then uses it in Eq. (3) as if do(s_t := s^I) were a valid intervention on the current state. Nothing in the SCM guarantees that the transition function acts as identity on some components; in general, s_{t+1} has different support. So the Wasserstein reward compares rewards under the current state and a future state; that conflates temporal evolution with causal counterfactual modification. The encoder loss in Eq. (6) may align representations with future states, which could be a useful data augmentation, but it does not provably focus on CRCs. Proposition 1 is a standard back-door identifiability result for the effect of a_t on s_{t+1}, and it does not validate the leap. Proposition 2 is a standard sufficient-condition result, but its conditions (e.g., r_t ⟂ s_t | z_t, a_t) are exactly what the unvalidated construction was supposed to ensure. There is also a circularity: the reward function in Eq. (5) defines causal relevance via a Wasserstein distance, and the policy is trained to minimize that same distance, so the claim that retained components are causal is true by construction, not by measurement.\n\nCredit where due: the two-stage design with a Wasserstein reward is a new combination, and the experiments are reasonably thorough: four offline datasets plus an online simulator, variance reported, and an ablation removing the causal agent. The improvements over DDPG/SAC/TD3 are consistent. But the ablation only replaces the causal agent with a random state; it does not compare to non-causal learned interventions or to bisimulation-based representation learning, which the paper itself cites as closely related.\n\nWho is this for? Practitioners in offline RLRS who want a data-augmentation trick might find it useful, but the causal interpretation should be treated with skepticism. A serious referee should engage with it because the empirical results are non-trivial, but the paper needs major revision: either the causal machinery must be respecified (e.g., by actually constructing counterfactual states via a learned transition model) or the causal claims should be dropped and the method presented as a heuristic regularizer. I would accept it for peer review but with the expectation of heavy revision.","headline":"The paper reports consistent empirical gains, but the central causal interpretation is a category error: the intervened state is the next state, not a modified current state, so the causal feature selection story collapses.","tokens_in":17937,"tokens_out":3654,"would_cite":false,"duration_ms":34138,"reading_group":"maybe","serious_thinker":"no","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper introduces PGCR, a two-stage framework that uses a Wasserstein-distance-guided policy to isolate causally relevant state components and trains an encoder to keep only those components, improving offline RL recommenders.","keywords":["offline reinforcement learning","recommender systems","causal state representation","causal feature selection","Wasserstein distance","state representation learning","counterfactual intervention"],"falsifier":"Run PGCR on a synthetic offline recommender environment whose state features are labeled reward-relevant and reward-irrelevant by construction; if the learned causal feature selection policy preserves known-irrelevant features or alters known-relevant ones—or if the encoder's latent representations remain sensitive to changes in known-irrelevant features—the central claim that PGCR isolates causally relevant components is refuted.","tokens_in":16798,"feed_emoji":"🎯","tokens_out":12859,"duration_ms":109190,"temperature":0.7,"pith_summary":"Offline reinforcement-learning recommenders must learn from fixed user histories, and their raw state vectors contain noisy, reward-irrelevant features that can mislead the policy. This paper sets out to establish that a two-stage method, PGCR, can solve that by first learning a policy that intervenes on the recommendation action to produce modified states retaining only the causally relevant components, and then training an encoder so that original and modified states map to the same latent point, forcing the representation to ignore irrelevant variation. The intervention policy is guided by a Wasserstein-distance reward that measures how much the reward distribution changes; small changes mean the causally relevant components were preserved. The paper reports that adding PGCR to standard off-policy algorithms improves cumulative and average reward across four offline recommendation benchmarks and an online simulator, and that replacing the causal policy with random states removes most of the gain.","feed_headline":"Causal state filtering lifts offline recommender systems","feed_subtitle":"Keeping only reward-relevant user features lifts offline RL recommenders on four datasets and a simulator.","key_machinery":"The load-bearing object is the causal feature selection policy and the modified states it generates. The policy acts by replacing the action with an intervened value, producing a modified state $s^I$ that is meant to retain the causally relevant components while altering the rest; its reward is $\\exp(-\\lambda W_1(P^{do(s_t := s^I)}(r_t), P(r_t)))$, so the agent is pushed to choose interventions that leave the reward distribution nearly unchanged. The second-stage encoder $\\phi$ is trained with the MSE objective $\\|\\phi(s_t) - \\phi(s^I_t)\\|_2^2$, which aligns the latent representations of states that differ only in irrelevant components. The framework is completed by a back-door identifiability argument for the intervention's effect on the next state and by a proposition showing the optimal policy depends only on the latent state when reward and transition dynamics factor through it.","core_discovery":"The paper's central claim is that the causal effect of state components on reward can be isolated through an action intervention, and that the resulting modified state, $s^I$, can act as a supervised signal for representation learning. Concretely, intervening as $do(a_t := a^I_t)$ yields a state that shares the original state's causally relevant components but differs in the irrelevant ones; comparing reward distributions before and after the intervention with the first-order Wasserstein distance gives a reward that drives the causal feature selection policy to preserve reward-relevant features. The encoder trained by minimizing the squared distance between encoded original and modified states therefore learns a latent space where only those features survive. Theoretical results in the paper establish identifiability of the effect of action on next state via the back-door criterion and show that, under the learned representation, the optimal policy can be written as a function of the latent state alone.","pith_inferences":["Editorial inference: the same two-stage recipe—an intervention policy rewarded by distributional shift, followed by an MSE-trained encoder—could transfer to offline control problems outside recommendation, such as robotics or healthcare, whenever the state has reward-relevant and reward-irrelevant features.","Editorial inference: the method's promised invariance could be tested directly on a synthetic environment with a known feature partition; the encoder should be insensitive to changes in known-irrelevant features and sensitive to changes in known-relevant ones.","Editorial inference: because the Wasserstein reward compares reward distributions under intervention, applying PGCR to purely logged data would require a learned reward model or an assumption that counterfactual rewards are observable, an extension the paper leaves implicit."],"forward_implications":["Adding PGCR to DDPG, SAC, and TD3 raises cumulative and average reward on four offline recommendation benchmarks relative to the unmodified algorithms.","On the online simulator, PGCR-enhanced algorithms achieve higher one-step click-through rate than their base versions.","The intervention policy generates additional modified states from the offline dataset, providing a form of data augmentation that mitigates missing transitions.","Under the representation conditions the paper proves, the optimal recommendation policy depends only on the latent state, so downstream decisions can ignore the full raw state.","Ablation with randomly sampled states instead of causal interventions lowers cumulative and average reward, showing the causal selection step drives the improvement."],"supporting_citations":[{"why":"Documents the missing-transition and fixed-data challenges of offline RL recommender systems that motivate PGCR's data augmentation.","marker":"[2]"},{"why":"Supplies the bisimulation metrics perspective that inspires the Wasserstein-distance reward for state abstraction.","marker":"[7]"},{"why":"Serves as a base RL algorithm (TD3) whose PGCR variant is evaluated in the experiments.","marker":"[8]"},{"why":"Serves as a base RL algorithm (SAC) whose PGCR variant is evaluated in the experiments.","marker":"[12]"},{"why":"Serves as a base RL algorithm (DDPG) and is used to generate the offline demonstrations for training the causal agent.","marker":"[18]"},{"why":"Provides do-calculus and the back-door criterion used in Proposition 1's identifiability proof.","marker":"[20]"},{"why":"Supplies the causal inference definitions used in the appendix for d-separation and adjustment.","marker":"[21]"},{"why":"Converts the benchmark recommendation datasets into RL environments used in the offline experiments.","marker":"[28]"},{"why":"Identifies pitfalls of bisimulation-based representations in offline RL, motivating PGCR's causal feature selection.","marker":"[29]"}],"fun_headline_variants":["Causal state filtering lifts offline RL recommenders","Policy-guided causal selection improves offline recommender systems","Wasserstein-guided filtering keeps reward-relevant user features","Intervention-derived representations boost offline RL recommendation"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the state reached after forcing a different action is the same current state with only its reward-irrelevant parts changed, rather than a genuinely different next state produced by the environment's dynamics.","fun_headline_variants_meta":{"raw":{"variants":["Causal state filtering lifts offline RL recommenders","Policy-guided causal selection improves offline recommender systems","Wasserstein-guided filtering keeps reward-relevant user features","Intervention-derived representations boost offline RL recommendation"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00078,"raw_usage":{"total_tokens":3460,"prompt_tokens":971,"completion_tokens":2489,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":587,"completion_tokens_details":{"reasoning_tokens":2429}},"tokens_in":587,"tokens_out":2489,"duration_ms":15099,"temperature":1.0,"reasoning_tokens":2429,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-09T12:31:42.826696+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run PGCR on a synthetic offline recommender environment whose state features are labeled reward-relevant and reward-irrelevant by construction; if the learned causal feature selection policy preserves known-irrelevant features or alters known-relevant ones—or if the encoder's latent representations remain sensitive to changes in known-irrelevant features—the central claim that PGCR isolates causally relevant components is refuted.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the bisimulation metrics perspective that inspires the Wasserstein-distance reward for state abstraction."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Serves as a base RL algorithm (SAC) whose PGCR variant is evaluated in the experiments."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the causal inference definitions used in the appendix for d-separation and adjustment."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Converts the benchmark recommendation datasets into RL environments used in the offline experiments."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Identifies pitfalls of bisimulation-based representations in offline RL, motivating PGCR's causal feature selection."}],"review_version":1}