Policy-Guided Causal State Representation for Offline Reinforcement Learning Recommendation

Lina Yao; Siyu Wang; Xiaocong Chen

REVIEW 4 major objections 5 minor 36 references

Policy-Guided Causal State Representation for Offline Reinforcement Learning Recommendation

T0 review · 4 major / 5 minor · reviewed 2026-08-09 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper introduces PGCR, a two-stage framework that uses a Wasserstein-distance-guided policy to isolate causally relevant state components and trains an encoder to keep only those components, improving offline RL recommenders.

desk verdict The paper reports consistent empirical gains, but the central causal interpretation is a category error: the intervened state is the next state, not a modified current state, so the causal feature selection story collapses. read the letter →

arxiv 2502.02327 v1 pith:PPMJXP4L submitted 2025-02-04 cs.IR cs.LG

Siyu Wang , Xiaocong Chen , Lina Yao This is my paper · ORCID

classification cs.IRcs.LG

keywords offlinereinforcementlearningrecommendersystemscausalstaterepresentationfeatureselectionWassersteindistancecounterfactualintervention

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Offline reinforcement-learning recommenders must learn from fixed user histories, and their raw state vectors contain noisy, reward-irrelevant features that can mislead the policy. This paper sets out to establish that a two-stage method, PGCR, can solve that by first learning a policy that intervenes on the recommendation action to produce modified states retaining only the causally relevant components, and then training an encoder so that original and modified states map to the same latent point, forcing the representation to ignore irrelevant variation. The intervention policy is guided by a Wasserstein-distance reward that measures how much the reward distribution changes; small changes mean the causally relevant components were preserved. The paper reports that adding PGCR to standard off-policy algorithms improves cumulative and average reward across four offline recommendation benchmarks and an online simulator, and that replacing the causal policy with random states removes most of the gain.

What carries the argument

The load-bearing object is the causal feature selection policy and the modified states it generates. The policy acts by replacing the action with an intervened value, producing a modified state $s^I$ that is meant to retain the causally relevant components while altering the rest; its reward is $\exp(-\lambda W_1(P^{do(s_t := s^I)}(r_t), P(r_t)))$, so the agent is pushed to choose interventions that leave the reward distribution nearly unchanged. The second-stage encoder $\phi$ is trained with the MSE objective $\|\phi(s_t) - \phi(s^I_t)\|_2^2$, which aligns the latent representations of states that differ only in irrelevant components. The framework is completed by a back-door identifiability argument for the intervention's effect on the next state and by a proposition showing the optimal policy depends only on the latent state when reward and transition dynamics factor through it.

What would settle it

Run PGCR on a synthetic offline recommender environment whose state features are labeled reward-relevant and reward-irrelevant by construction; if the learned causal feature selection policy preserves known-irrelevant features or alters known-relevant ones—or if the encoder's latent representations remain sensitive to changes in known-irrelevant features—the central claim that PGCR isolates causally relevant components is refuted.

Watch

Extended reading notes

Core claim

The paper's central claim is that the causal effect of state components on reward can be isolated through an action intervention, and that the resulting modified state, $s^I$, can act as a supervised signal for representation learning. Concretely, intervening as $do(a_t := a^I_t)$ yields a state that shares the original state's causally relevant components but differs in the irrelevant ones; comparing reward distributions before and after the intervention with the first-order Wasserstein distance gives a reward that drives the causal feature selection policy to preserve reward-relevant features. The encoder trained by minimizing the squared distance between encoded original and modified states therefore learns a latent space where only those features survive. Theoretical results in the paper establish identifiability of the effect of action on next state via the back-door criterion and show that, under the learned representation, the optimal policy can be written as a function of the latent state alone.

Load-bearing premise

The load-bearing premise is that the state reached after forcing a different action is the same current state with only its reward-irrelevant parts changed, rather than a genuinely different next state produced by the environment's dynamics.

Editorial extensions

If this is right

Adding PGCR to DDPG, SAC, and TD3 raises cumulative and average reward on four offline recommendation benchmarks relative to the unmodified algorithms.
On the online simulator, PGCR-enhanced algorithms achieve higher one-step click-through rate than their base versions.
The intervention policy generates additional modified states from the offline dataset, providing a form of data augmentation that mitigates missing transitions.
Under the representation conditions the paper proves, the optimal recommendation policy depends only on the latent state, so downstream decisions can ignore the full raw state.
Ablation with randomly sampled states instead of causal interventions lowers cumulative and average reward, showing the causal selection step drives the improvement.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Editorial inference: the same two-stage recipe—an intervention policy rewarded by distributional shift, followed by an MSE-trained encoder—could transfer to offline control problems outside recommendation, such as robotics or healthcare, whenever the state has reward-relevant and reward-irrelevant features.
Editorial inference: the method's promised invariance could be tested directly on a synthetic environment with a known feature partition; the encoder should be insensitive to changes in known-irrelevant features and sensitive to changes in known-relevant ones.
Editorial inference: because the Wasserstein reward compares reward distributions under intervention, applying PGCR to purely logged data would require a learned reward model or an assumption that counterfactual rewards are observable, an extension the paper leaves implicit.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Here is my take. The method is a plausible empirical heuristic, but the causal argument is a category error. The intervention do(a_t := a^I_t) produces a sample from P(s_{t+1} | s_t, a^I_t), i.e., a one-step-ahead state, not a modified version of s_t with causally irrelevant components changed. The paper simply asserts after Eq. (2) that s^I 'is expected to preserve only the causally relevant components of the original state s_t' and then uses it in Eq. (3) as if do(s_t := s^I) were a valid intervention on the current state. Nothing in the SCM guarantees that the transition function acts as identity on some components; in general, s_{t+1} has different support. So the Wasserstein reward compares rewards under the current state and a future state; that conflates temporal evolution with causal counterfactual modification. The encoder loss in Eq. (6) may align representations with future states, which could be a useful data augmentation, but it does not provably focus on CRCs. Proposition 1 is a standard back-door identifiability result for the effect of a_t on s_{t+1}, and it does not validate the leap. Proposition 2 is a standard sufficient-condition result, but its conditions (e.g., r_t ⟂ s_t | z_t, a_t) are exactly what the unvalidated construction was supposed to ensure. There is also a circularity: the reward function in Eq. (5) defines causal relevance via a Wasserstein distance, and the policy is trained to minimize that same distance, so the claim that retained components are causal is true by construction, not by measurement.

Credit where due: the two-stage design with a Wasserstein reward is a new combination, and the experiments are reasonably thorough: four offline datasets plus an online simulator, variance reported, and an ablation removing the causal agent. The improvements over DDPG/SAC/TD3 are consistent. But the ablation only replaces the causal agent with a random state; it does not compare to non-causal learned interventions or to bisimulation-based representation learning, which the paper itself cites as closely related.

Who is this for? Practitioners in offline RLRS who want a data-augmentation trick might find it useful, but the causal interpretation should be treated with skepticism. A serious referee should engage with it because the empirical results are non-trivial, but the paper needs major revision: either the causal machinery must be respecified (e.g., by actually constructing counterfactual states via a learned transition model) or the causal claims should be dropped and the method presented as a heuristic regularizer. I would accept it for peer review but with the expectation of heavy revision.

Referee Report

4 major / 5 minor

Summary. The paper proposes Policy-Guided Causal Representation (PGCR), a two-stage framework for offline reinforcement learning-based recommender systems (RLRS). In stage one, a causal feature selection policy is trained to generate a 'modified state' s^I by intervening on the action a_t, with the stated goal of retaining causally relevant components (CRCs) of the current state s_t while altering causally irrelevant components (CIRCs). In stage two, an encoder is trained with an MSE loss between latent representations of s_t and s^I, intending to make the representation focus on CRCs. The paper provides two theoretical propositions (identifiability of the causal effect of a_t on s_{t+1}, and an optimal policy depending only on the latent representation) and reports experiments on MovieLens, Coat, KuaiRec, KuaiRand, and the VirtualTaobao simulator, showing improved cumulative and average rewards over DDPG, SAC, and TD3 baselines.

Significance. If the causal feature selection mechanism were sound, PGCR would offer a principled approach to state representation learning in offline recommendation, potentially improving long-term reward by focusing on reward-relevant user features and augmenting limited offline data. The empirical results are consistent across several datasets and backbones, and the paper includes an ablation indicating that the learned intervention contributes beyond random state substitution. However, the significance of the work rests entirely on the causal interpretation, and, as detailed below, the causal mechanism is not supported. Without a valid causal grounding, the method reduces to an ad-hoc state alignment procedure whose mechanism is unexplained. The paper also ships no code and gives limited implementation detail, which weakens reproducibility claims.

major comments (4)

[Section 3.2, Eq. (2)] The central premise of the paper is that the intervened state s^I, generated by do(a_t := a^I_t), is a modified version of the current state s_t that preserves only the causally relevant components. However, Eq. (2) explicitly defines the distribution of s^I as the distribution of s_{t+1} under the intervention, i.e., P_{M;do(a_t:=a^I_t)}(s_{t+1}) = E_{s_t, epsilon_{t+1}}[P(s_{t+1}|s_t, a^I_t, epsilon_{t+1})]. In the MDP of Eq. (1), the intervention on a_t produces the next state, not an altered version of the current state. The transition function f_P is arbitrary and need not act as identity on any components of s_t; in general, s_{t+1} can differ from s_t in every component, including those that are causally relevant to r_t. The assertion after Eq. (2) that 'This intervened state s^I is expected to preserve only the causally relevant components of the original state s_t' is therefore unsupported and is a category error: a future state is not a valid counterfactual surrogate for the current state.
[Section 3.2, Eq. (3)] Equation (3) treats replacing s_t by s^I in the reward function r_t = f_R(s_t, a_t) as a legitimate intervention do(s_t := s^I), and uses the resulting reward distribution to measure the causal effect of components of s_t on r_t. This is invalid because r_t is defined on the current state; substituting a one-step-ahead state into the reward function conflates temporal evolution with a counterfactual state modification. Consequently, the Wasserstein distance in Eq. (4) and the reward function in Eq. (5) do not measure the causal relevance of the components of s_t. The entire stage-one training objective is built on this invalid substitution.
[Section 3.3, Eq. (6)] The MSE loss in Eq. (6) aligns the latent representations of s_t and s^I_t. Even if s^I_t were a valid surrogate for a modified current state, the loss only enforces that the encoder be invariant to the difference between s_t and s^I_t. It does not guarantee that the encoder preserves information about the CRCs; a trivial encoder that maps all states to the same vector would minimize the loss exactly but would lose all information. Proposition 2 lists sufficient conditions (r_t is independent of s_t given z_t and a_t, and transition consistency) for the optimal policy to depend only on z_t, but the paper does not prove that the encoder trained with Eq. (6) satisfies these conditions. Thus the central claim that 'the encoder learns to isolate the CRCs' is not established.
[Section 3.2, Eq. (5) and Section 4.4] The reward function in Eq. (5) defines the notion of 'causally relevant components' through the Wasserstein distance between the original and intervened reward distributions, and the causal feature selection policy is trained to maximize that same reward. The claim that the selected components are causal is therefore true by construction of the training objective, not by an external causal validation. The ablation in Table 2 comparing PGCR against PGCR-C (with a randomly sampled state) shows that the learned intervention helps, but it does not demonstrate that the improvement arises from identifying causal components; it could simply reflect that the learned s^I is a more useful data augmentation than a random state. A comparison against a non-causal augmentation that uses actual next states from the dataset (without the intervention) would be needed to separate these effects.

minor comments (5)

[Section 4.1.1] The text says 'VritualTB' in the evaluation metric description; this is a typo for 'VirtualTB'.
[References] Reference [21] contains 'Schlkopf' (missing 'ö'); it should be 'Schölkopf'.
[Section 3.2, Eq. (2)] The derivation in Eq. (2) writes P(s_t|do(a_t)) = P(s_t), but this equality holds only because s_t is not a descendant of a_t in the causal graph; stating this explicitly would improve clarity.
[Section 3.2, Eq. (3)] The notation in Eq. (3) is confusing: it sums over a_t and eta_t but uses P(a_t|do(s_t := s^I)) as though the action distribution were recomputed under a counterfactual state, while the SCM in Eq. (1) defines a_t as a function of s_t (not of a modified s^I). The equation appears to mix interventional and observational distributions without a clear derivation.
[Appendix C] The proof of Proposition 2 has a gap: in Step 4, the statement that 'the latent state dynamics depend only on the latent representation' is exactly the second bullet of the proposition, not a consequence of it, and the proof does not show how the expectation over the next state in the Bellman equation can be replaced by the latent transition. The proof would benefit from a more explicit inductive argument.

Circularity Check

2 steps flagged · score 7.0 of 10

PGCR's central claim that the learned policy retains causally relevant components is a restatement of its own reward function: CRCs are defined by their effect on reward, and the policy is trained to minimize the reward-distribution change, making the encoder's 'causal representation' claim true by construction rather than by independent causal identification.

self definitional [Section 3.1 (CRC/CIRC definitions) and Section 3.2, Eq. (5) and following text]
"By maximizing this reward, the agent is incentivized to select actions that minimize the Wasserstein distance between the intervened and original reward distributions. This encourages the agent to choose actions that retain the causally relevant components of the state, effectively filtering out causally irrelevant features."

Section 3.1 defines CRCs as components whose modification leads to significant changes in rewards and CIRCs as components whose alteration has only a weak causal effect on rewards. Equation (5) defines the policy reward as exp(-lambda * W1(P^{do(s_t:=s^I)}(r_t), P(r_t))), so maximizing it is exactly minimizing the reward-distribution change caused by the intervention. The sentence following Eq. (5) equates this minimization with retaining the causally relevant components of the state. Since the two criteria are the same, the claim that the learned policy isolates CRCs is a restatement of the training objective, not a derived or independently tested property. The encoder in Eq.
other [Section 3.2, Eqs. (2)-(3)]
"Solving this model for the distribution of s_{t+1} yields the causal effect of a_t on s_{t+1}, denoted as P^{do(a_t:=a^I_t)}(s_{t+1})... Since the CRCs are the parts of s_t that have a significant causal impact on rewards, we regard the new state s^I, induced by the intervention on a_t, as an effective intervention on s_t in the original tuple {s_t,a_t,s_{t+1},r_t}."

Equation (2) gives the interventional distribution of s_{t+1} under do(a_t := a^I_t); a sample s^I from it is a one-step-ahead state, not a modified copy of the current state. Equation (3) nevertheless substitutes s^I into the current reward function P(r_t | s_t, a_t, eta_t) as if it were a counterfactual value of s_t. The SCM in Eq. (1) has r_t = f_R(s_t,a_t), so nothing in the model implies that the next state shares the CRCs of s_t. The paper's assertion that s^I 'is expected to preserve only the causally relevant components of the original state s_t' is therefore an added premise, and it is exactly the conclusion needed for the MSE encoder of Eq. (6) to be interpretable as causal representation learning.

full rationale

The paper's main derivation chain is: (i) define CRCs as state components whose modification changes the reward distribution; (ii) train a 'causal feature selection policy' with the reward in Eq. (5), which is high precisely when the intervened reward distribution is close to the original; (iii) assert that the resulting intervened state s^I therefore retains only CRCs; (iv) train an encoder with Eq. (6) to map s_t and s^I_t to the same latent code; (v) conclude the encoder focuses on CRCs. Step (iii) is the load-bearing move and it is definitional: the training objective and the definition of CRC are the same criterion (effect on reward). Thus the 'prediction' that s^I preserves CRCs is the objective being optimized, not an independent finding. The additional step from Eq. (2) to Eq. (3) is a category error: Eq. (2) derives the distribution of the next state under do(a_t := a^I_t), while Eq. (3) substitutes that next-state sample into r_t = f_R(s_t,a_t) as though it were a counterfactual current state. The paper gives no SCM-based argument that s^I equals s_t with only CIRCs changed; this is the conclusion assumed. The empirical results (Tables 1-2, Figures 2-4) are independent evidence that the procedure helps downstream RL, but they do not test the causal claim; the ablation only replaces the causal agent with a random state generator, and improved reward could come from data augmentation or regularization rather than from isolating CRCs. No load-bearing self-citation was found; citations to the authors' prior work are contextual. Overall, the central causal-representation claim reduces to the training objective by definition, so the circularity score is 7.

Assumptions & free parameters 1 free parameters · 5 assumptions · 1 invented entities

The method depends on several assumptions that are not derived from the MDP. The most important is that intervening on the action produces a state s^I that is a modified version of the current state with only causally irrelevant components changed; this is asserted, not proven. The Wasserstein-distance reward defines the notion of causal relevance in a way that is consistent with the policy's objective, making the causal selection claim tautological. Standard do-calculus and back-door results are invoked but do not justify the state-preservation step.

free parameters (1)

lambda (Wasserstein reward scaling) = optimal range 0.1 to 0.2 on VirtualTB
Equation (5) introduces lambda to control reward sensitivity; the hyperparameter study shows performance is highly sensitive to it, and it is tuned only on VirtualTB without justification on other datasets.

assumptions (5)

standard math Back-door criterion and do-calculus rules hold for the MDP SCM.
Used in Proposition 1 and Eqs (2)-(3) to derive intervention distributions.
domain assumption The reward function f_R in Eq (1) is deterministic given s_t and a_t; the only noise enters through state transitions and action selection.
This simplifies the do-calculus derivations but may not hold for noisy user feedback.
ad hoc to paper The next state s^I after intervening on the action is a valid surrogate for the current state s_t with only CIRCs altered.
This is asserted in Section 3.2 without justification and is the load-bearing assumption of the method.
ad hoc to paper Training the encoder with MSE on pairs (s_t, s^I_t) yields an encoder satisfying the sufficiency conditions of Proposition 2.
No proof connects the MSE objective to the conditional independence and transition consistency assumptions of Proposition 2.
domain assumption The Wasserstein distance between reward distributions adequately quantifies the causal effect of state components on reward.
Equations (4)-(5) treat this distance as a proxy for causal relevance, but no formal equivalence is shown.

invented entities (1)

Intervened state s^I_t (treated as modified current state)
purpose: Provides a training signal for the encoder to ignore features changed by the action intervention
s^I_t is generated by the learned causal policy from the offline dataset; it is not externally validated as a state that differs from s_t only in causally irrelevant components. No falsifiable prediction is attached to this construct.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Policy-Guided Causal State Representation for Offline Reinforcement Learning Recommendation." pith.science (2026). https://pith.science/paper/PPMJXP4L

@misc{pith2026250202327,
  author       = {Pith},
  title        = {Pith review of: Policy-Guided Causal State Representation for Offline Reinforcement Learning Recommendation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/PPMJXP4L}},
  note         = {Machine review of arXiv:2502.02327}
}

read the original abstract

In offline reinforcement learning-based recommender systems (RLRS), learning effective state representations is crucial for capturing user preferences that directly impact long-term rewards. However, raw state representations often contain high-dimensional, noisy information and components that are not causally relevant to the reward. Additionally, missing transitions in offline data make it challenging to accurately identify features that are most relevant to user satisfaction. To address these challenges, we propose Policy-Guided Causal Representation (PGCR), a novel two-stage framework for causal feature selection and state representation learning in offline RLRS. In the first stage, we learn a causal feature selection policy that generates modified states by isolating and retaining only the causally relevant components (CRCs) while altering irrelevant components. This policy is guided by a reward function based on the Wasserstein distance, which measures the causal effect of state components on the reward and encourages the preservation of CRCs that directly influence user interests. In the second stage, we train an encoder to learn compact state representations by minimizing the mean squared error (MSE) loss between the latent representations of the original and modified states, ensuring that the representations focus on CRCs. We provide a theoretical analysis proving the identifiability of causal effects from interventions, validating the ability of PGCR to isolate critical state components for decision-making. Extensive experiments demonstrate that PGCR significantly improves recommendation performance, confirming its effectiveness for offline RL-based recommender systems.

Figures

Figures reproduced from arXiv: 2502.02327 by the authors.

**Figure 2.** The 1-step CTR performance in the VirtualTaobao [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗

**Figure 3.** Performance comparisons in VirtualTB: (a) DDPG as the backbone, (b) SAC as the backbone, and (c) TD3 as the [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗

Figures from the paper (1 more)

**Figure 4.** Figure 4: Hyper Parameter Study in VirtualTB exploration in RLRS by focusing on causally relevant aspects of user interactions. Wang et al. [25] introduced a method to generate counterfactual user interactions based on a causal view of MDP for data augmentation. In a related vei…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

36 extracted references · 25 canonical work pages

[1]

M Mehdi Afsar, Trafford Crump, and Behrouz Far. 2022. Reinforcement learning based recommender systems: A survey. Comput. Surveys 55, 7 (2022), 1–38

work page 2022
[2]

Xiaocong Chen, Siyu Wang, Julian McAuley, Dietmar Jannach, and Lina Yao
[3]

Xiaocong Chen, Siyu Wang, Lianyong Qi, Yong Li, and Lina Yao. 2023. Intrinsically motivated reinforcement learning based recommendation with counterfactual data augmentation. World Wide Web 26, 5 (2023), 3253–3274

work page 2023
[4]

Xiaocong Chen, Siyu Wang, and Lina Yao. 2024. Maximum-Entropy Regularized Decision Transformer with Reward Relabelling for Dynamic Recommendation. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining (Barcelona, Spain) (KDD ’24). Association for Computing Machinery, New York, NY, USA, 376–384. https://doi.org/10.1145/363...

arXiv 2024
[5]

Xiaocong Chen, Lina Yao, Julian McAuley, Guanglin Zhou, and Xianzhi Wang
[6]

Xiaocong Chen, Lina Yao, Aixin Sun, Xianzhi Wang, Xiwei Xu, and Liming Zhu

work page
[7]

Norm Ferns, Prakash Panangaden, and Doina Precup. 2011. Bisimulation metrics for continuous Markov decision processes. SIAM J. Comput. 40, 6 (2011), 1662– 1714

work page 2011
[8]

Scott Fujimoto, Herke Hoof, and David Meger. 2018. Addressing function ap- proximation error in actor-critic methods. In International conference on machine learning. PMLR, 1587–1596

2018

Show all 36 references

[9]

Chongming Gao, Kexin Huang, Jiawei Chen, Yuan Zhang, Biao Li, Peng Jiang, Shiqi Wang, Zhong Zhang, and Xiangnan He. 2023. Alleviating matthew effect of offline reinforcement learning in interactive recommendation. In Proceedings of the 46th International ACM SIGIR Conference o...

2023
[10]

Chongming Gao, Shijun Li, Wenqiang Lei, Jiawei Chen, Biao Li, Peng Jiang, Xiangnan He, Jiaxin Mao, and Tat-Seng Chua. 2022. KuaiRec: A fully-observed dataset and insights for evaluating recommender systems. In Proceedings of the 31st ACM International Conference on Information...

2022
[11]

Chongming Gao, Shijun Li, Yuan Zhang, Jiawei Chen, Biao Li, Wenqiang Lei, Peng Jiang, and Xiangnan He. 2022. KuaiRand: An Unbiased Sequential Recommen- dation Dataset with Randomly Exposed Videos. In Proceedings of the 31st ACM International Conference on Information and Knowl...

2022
[12]

Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. 2018. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International conference on machine learning . PMLR, 1861– 1870

2018
[13]

Xiangnan He, Yang Zhang, Fuli Feng, Chonggang Song, Lingling Yi, Guohui Ling, and Yongdong Zhang. 2023. Addressing confounding feature issue for causal recommendation. ACM Transactions on Information Systems 41, 3 (2023), 1–23

2023
[14]

Biwei Huang, Chaochao Lu, Liu Leqi, José Miguel Hernández-Lobato, Clark Glymour, Bernhard Schölkopf, and Kun Zhang. 2022. Action-sufficient state representation learning for control with structural constraints. In International Conference on Machine Learning . PMLR, 9260–9279

2022
[15]

Timothée Lesort, Natalia Díaz-Rodríguez, Jean-Franois Goudou, and David Filliat
[16]

Haoxuan Li, Kunhan Wu, Chunyuan Zheng, Yanghao Xiao, Hao Wang, Zhi Geng, Fuli Feng, Xiangnan He, and Peng Wu. 2024. Removing hidden confounding in recommendation: a unified multi-task learning approach. Advances in Neural Information Processing Systems 36 (2024)

2024
[17]

Haoxuan Li, Chunyuan Zheng, Peng Wu, Kun Kuang, Yue Liu, and Peng Cui
[18]

Timothy P Lillicrap, Jonathan J Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. 2015. Continuous control with deep reinforcement learning. arXiv preprint arXiv:1509.02971 (2015)

2015 arXiv
[19]

Tariq Mahmood and Francesco Ricci. 2007. Learning and adaptivity in interactive recommender systems. In Proceedings of the ninth international conference on Electronic commerce. 75–84

2007
[20]

Judea Pearl. 2009. Causality. Cambridge university press

2009
[21]

Jonas Peters, Dominik Janzing, and Bernhard Schlkopf. 2017. Elements of Causal Inference: Foundations and Learning Algorithms . The MIT Press

2017
[22]

In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining

Who should be given incentives? counterfactual optimal treatment regimes learning for recommendation. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining . 1235–1247
[23]

Jing-Cheng Shi, Yang Yu, Qing Da, Shi-Yong Chen, and An-Xiang Zeng. 2019. Virtual-taobao: Virtualizing real-world online retail environment for reinforce- ment learning. In Proceedings of the AAAI Conference on Artificial Intelligence , Vol. 33. 4902–4909

2019
[24]

Siyu Wang, Xiaocong Chen, Dietmar Jannach, and Lina Yao. 2023. Causal decision transformer for recommender systems via offline reinforcement learning. In Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval . 1599–1608

2023
[25]

Siyu Wang, Xiaocong Chen, Julian McAuley, Sally Cripps, and Lina Yao. 2023. Plug-and-Play Model-Agnostic Counterfactual Policy Synthesis for Deep Re- inforcement Learning-Based Recommendation. IEEE Transactions on Neural Networks and Learning Systems (2023)

2023
[26]

Siyu Wang, Xiaocong Chen, and Lina Yao. 2024. On Causally Disentangled State Representation Learning for Reinforcement Learning based Recommender Systems. arXiv preprint arXiv:2407.13091 (2024)

2024 arXiv
[27]

Tobias Schnabel, Adith Swaminathan, Ashudeep Singh, Navin Chandak, and Thorsten Joachims. 2016. Recommendations as treatments: Debiasing learning and evaluation. In international conference on machine learning . PMLR, 1670– 1679

2016
[28]

Yuanqing Yu, Chongming Gao, Jiawei Chen, Heng Tang, Yuefeng Sun, Qian Chen, Weizhi Ma, and Min Zhang. 2024. EasyRL4Rec: An Easy-to-use Library for Reinforcement Learning Based Recommender Systems. In Proceedings of the 47th International ACM SIGIR Conference on Research and De...

2024
[29]

Hongyu Zang, Xin Li, Leiji Zhang, Yang Liu, Baigui Sun, Riashat Islam, Remi Tachet des Combes, and Romain Laroche. 2024. Understanding and addressing the pitfalls of bisimulation-based representations in offline reinforcement learning. Advances in Neural Information Processing...

2024
[30]

Amy Zhang, Rowan Thomas McAllister, Roberto Calandra, Yarin Gal, and Sergey Levine. 2021. Learning Invariant Representations for Reinforcement Learning without Reconstruction. In International Conference on Learning Representations . https://openreview.net/forum?id=-2FCwDKRREu

2021
[31]

back-door

Yang Zhang, Fuli Feng, Xiangnan He, Tianxin Wei, Chonggang Song, Guohui Ling, and Yongdong Zhang. 2021. Causal intervention for leveraging popularity bias in recommendation. In Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Informatio...

2021
[32]

Yichao Wang, Huifeng Guo, Bo Chen, Weiwen Liu, Zhirong Liu, Qi Zhang, Zhicheng He, Hongkun Zheng, Weiwei Yao, Muyu Zhang, et al. 2022. Causalint: Causal inspired intervention for multi-scenario recommendation. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Disco...

2022
[2018]

Neural Networks 108 (2018), 379–392

State representation learning for control: An overview. Neural Networks 108 (2018), 379–392

2018
[2021]

In Proceedings of the 30th ACM International Conference on Information & Knowledge Management

Generative inverse deep reinforcement learning for online recommenda- tion. In Proceedings of the 30th ACM International Conference on Information & Knowledge Management. 201–210
[2023]

Knowledge-Based Systems 264 (2023), 110335

Deep reinforcement learning in recommender systems: A survey and new perspectives. Knowledge-Based Systems 264 (2023), 110335. https://doi.org/10. 1016/j.knosys.2023.110335

2023
[2024]

ACM Transactions on Information Systems 42, 6 (2024), 1–26

On the opportunities and challenges of offline reinforcement learning for recommender systems. ACM Transactions on Information Systems 42, 6 (2024), 1–26

2024

Pith tools

Reviewed August 9, 2026 · model on record in the stance chip above.

[1] [1]

M Mehdi Afsar, Trafford Crump, and Behrouz Far. 2022. Reinforcement learning based recommender systems: A survey. Comput. Surveys 55, 7 (2022), 1–38

work page 2022

[2] [2]

Xiaocong Chen, Siyu Wang, Julian McAuley, Dietmar Jannach, and Lina Yao

[3] [3]

Xiaocong Chen, Siyu Wang, Lianyong Qi, Yong Li, and Lina Yao. 2023. Intrinsically motivated reinforcement learning based recommendation with counterfactual data augmentation. World Wide Web 26, 5 (2023), 3253–3274

work page 2023

[4] [4]

Xiaocong Chen, Siyu Wang, and Lina Yao. 2024. Maximum-Entropy Regularized Decision Transformer with Reward Relabelling for Dynamic Recommendation. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining (Barcelona, Spain) (KDD ’24). Association for Computing Machinery, New York, NY, USA, 376–384. https://doi.org/10.1145/363...

arXiv 2024

[5] [5]

Xiaocong Chen, Lina Yao, Julian McAuley, Guanglin Zhou, and Xianzhi Wang

[6] [6]

Xiaocong Chen, Lina Yao, Aixin Sun, Xianzhi Wang, Xiwei Xu, and Liming Zhu

work page

[7] [7]

Norm Ferns, Prakash Panangaden, and Doina Precup. 2011. Bisimulation metrics for continuous Markov decision processes. SIAM J. Comput. 40, 6 (2011), 1662– 1714

work page 2011

[8] [8]

Scott Fujimoto, Herke Hoof, and David Meger. 2018. Addressing function ap- proximation error in actor-critic methods. In International conference on machine learning. PMLR, 1587–1596

2018