Pith. sign in

REVIEW 5 major objections 5 minor 83 references

DARLR: Dual-Agent Offline Reinforcement Learning for Recommender Systems with Dynamic Reward

T0 review · 5 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Two-agent reward shaping tops offline RL recommenders on four datasets

desk verdict Plausible incremental advance in model-based offline RL for RecSys, but the dynamic reward mechanism is oversold and two of four benchmark gains are within noise. read the letter →

arxiv 2505.07257 v1 pith:FWL2VFZI submitted 2025-05-12 cs.IR

Yi Zhang , Ruihong Qiu , Xuwei Xu , Jiajun Liu , Sen Wang This is my paper · ORCID
classification cs.IR
keywords offlinereinforcementlearningrecommendersystemsdynamicrewardshapingdual-agentframeworkuncertaintypenaltyreferenceuserselectionmodel-basedRL
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that model-based offline reinforcement learning for recommender systems fails not because the world model is inaccurate, but because existing methods freeze the world model's reward function and uncertainty penalty during policy learning, propagating those inaccuracies. The authors propose DARLR, a dual-agent framework in which a selector agent picks reference users by balancing similarity and diversity, and a recommender agent averages the world model's predicted rewards over those users to dynamically reshape the reward signal. They also derive an uncertainty penalty from the selector's own statistics, so conservatism adapts as training proceeds. If correct, this means offline recommenders can approach the performance of policies trained on ground-truth rewards without any online interaction, just by treating the world model as an evolving object rather than a fixed look-up table.

What carries the argument

The central mechanism is the dual-agent loop: a selector policy $\pi_{\text{sel}}$ chooses reference users by maximizing an intrinsic reward $r^{\text{sel}} = \hat{r} + \lambda_s r_s^{\text{sel}} + \lambda_d r_d^{\text{sel}}$, where $r_s^{\text{sel}}$ is cosine similarity to the target user and $r_d^{\text{sel}}$ is average dissimilarity from already selected users. The recommender then uses the averaged predicted reward $\hat{r}(u, i_t) = \frac{1}{|\mathcal{U}_{S,t}|}\sum_{u' \in \mathcal{U}_{S,t}} \hat{r}(u', i_t)$ as its reward signal, and the uncertainty penalty $P'_U = |\hat{r} - \hat{r}_{-1}| / (r_s^{\text{sel}} + r_d^{\text{sel}})$ adapts to how representative the selected set is. This machinery converts reward refinement and risk estimation from one-shot static computations into actions taken inside the same reinforcement learning loop that trains the recommender.

What would settle it

On any of the four datasets, compare DARLR's reward estimates against ground-truth rewards for a held-out set of users and items, then run a variant where the selector picks reference users uniformly at random. If the random-selection variant achieves the same or lower reward error, or if DARLR's dynamic reward curves do not stay below the static baselines' error curves, the central claim that learned selection drives the improvement is falsified.

Watch

Extended reading notes

Core claim

DARLR claims that continuously refining reward estimations during policy learning—rather than treating rewards as static look-up tables—mitigates the impact of inaccuracies in the world model's reward predictions for offline recommender systems. A selector agent sequentially gathers a small set of reference users whose predicted feedback vectors are similar to the target user's yet diverse among themselves; the recommender then replaces the target user's predicted reward with the average reward over that set at each interaction step. The same selection statistics feed a dynamically adjusted uncertainty penalty, so risky actions are penalized more when the selected reference set is not representative. Empirically, DARLR achieves the highest single-step reward and cumulative reward among all compared baselines on KuaiRand, KuaiRec, Coat, and Yahoo, with curves consistently closer to the ground-truth-reward upper bound than static-reward methods.

Load-bearing premise

The averaged reward over users selected for similarity and diversity is more accurate than the target user's own predicted reward, even though all predictions come from the same error-prone frozen world model; if the world model is systematically biased, the selector may pick similarly biased users and the averaged reward will inherit rather than correct that bias.

Editorial extensions

If this is right

  • Offline recommender policies can be trained to near-ground-truth performance without online interaction, as long as rewards are refined during learning rather than frozen.
  • The dual-agent design suggests a general recipe: any model-based offline RL system can allocate a second policy to actively select which data points to trust when estimating rewards.
  • Adaptive uncertainty penalties tied to the selection process should be preferred over static ensemble-variance penalties when offline logs are sparse or biased.
  • On dense datasets like KuaiRec, fewer reference users suffice; on sparse datasets like KuaiRand, more reference users help, indicating the method self-tunes to data sparsity.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The selector's averaging trick may work even with a systematically biased world model if the bias is roughly additive across users, but if the bias is user-specific and correlated with similarity, the averaging could reinforce rather than cancel the error; this is testable with synthetic bias injection.
  • The same dual-agent reward-refinement idea could transfer to offline RL outside recommender systems, such as robotics or healthcare, wherever a learned reward model is used as a frozen proxy for a sparse environment.
  • Constraining the selector to a small user subset (the paper notes clustering can speed it up) suggests a practical path to scaling dynamic reward shaping to millions of users.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. This paper proposes DARLR, a dual-agent offline model-based RL framework for recommender systems. A selector agent chooses a set of reference users for each recommendation step using an intrinsic reward that combines the world-model reward prediction, user similarity, and diversity (Eqs. 10-12). A recommender agent then uses the average world-model reward over the selected users as the reward estimate (Eq. 15) and an adaptive uncertainty penalty (Eq. 16) when training the recommendation policy. The authors claim that this dynamic reward estimation mitigates the impact of world-model reward inaccuracies, and they report improvements over baselines on KuaiRand, KuaiRec, Coat, and Yahoo, together with ablations and hyperparameter sensitivity studies.

Significance. If the central mechanism were established, the work would offer a practical way to use peer-user information in model-based offline RL for recommendations, a direction with limited prior work. The manuscript has clear strengths: all model-based baselines share the same pretrained world model; the ablation study isolates the similarity/diversity components and static-versus-dynamic designs; the GT-reward upper bound provides a useful reference; and the authors release code. However, the load-bearing claims, namely that Eq. (15) yields more accurate rewards for the target user and that the empirical gains over ROLeR are statistically meaningful, are not yet supported by the evidence presented, so the significance is conditional on substantial revision.

major comments (5)
  1. [§3.3.1, Eq. (15)] Eq. (15) replaces the target user's predicted reward with the average of predictions from a reference-user set drawn from the same frozen world model, but the paper gives no argument or empirical demonstration that this average is closer to the true reward for the target user than the direct prediction \hat{r}(u,i_t). Because the selector's intrinsic reward in Eq. (12) includes \hat{r} itself, a selector trained to maximize that reward tends to choose users for whom the world model predicts high rewards; if those predictions share a systematic bias (e.g., popularity- or density-dependent overestimation), the averaged reward in Eq. (15) inherits and may amplify that bias. Please provide a direct correctness check: on a held-out set, compare the mean absolute error of \hat{r}(u,i), of the Eq. (15) average, and of a random-user average against ground truth, stratified by interaction density, and add an ablation in which the selector's reward excludes \hat{r}.
  2. [§4.4, Figure 4] The 'Error' plotted in Figure 4 is not defined tightly enough to support the claim that DARLR produces more accurate reward estimates. The text says only that it is the deviation between ground truth and estimated rewards; it does not state which state-action pairs are included, whose ground truth is used (the target user's or the reference users'), how trajectories are sampled, or whether the DARLR curve uses Eq. (15) or the direct prediction. Without this protocol, the lower DARLR curve could reflect selection of reference users whose true rewards are easier to predict, rather than improved estimates for the target user. Please specify the exact metric, the aggregation, and the evaluation protocol, and include error bars or multiple seeds.
  3. [Tables 2 and 3; §4.2] The headline empirical claim is not supported by the reported statistics. On KuaiRand, DARLR's cumulative reward (13.8152±1.9351) differs from ROLeR (13.4553±1.5086) by about one third of the pooled standard deviation; on Yahoo the difference (68.5418 vs. 68.3637) is within one standard deviation; on Coat the difference (78.0429 vs. 76.1603) is less than one pooled standard deviation; and on KuaiRec (35.2203 vs. 33.2457) the gap is also smaller than the reported variability. No significance tests are reported, so statements such as 'superior performance' and 'outperforms ROLeR significantly' are not justified. Please add paired bootstrap or equivalent tests across the common seeds, or present effect sizes with confidence intervals and soften the claims accordingly.
  4. [Abstract; §3.4; §5] The manuscript repeatedly describes DARLR as dynamically 'updating world models' and as having an 'evolving world model', but Algorithm 1, Eq. (15), and Eq. (17) update only the selector, the recommender, and the shaped reward/uncertainty; the pretrained world-model parameters and the per-user feedback vectors p_u and \hat{r} in Eq. (3) are not updated during policy learning. This distinction matters because the claimed contribution over ROLeR/DORL is the dynamic correction of reward inaccuracy, and the implemented mechanism is best described as dynamic reward estimation via reference-user averaging. Please either revise the terminology throughout (including the abstract and related-work comparison) or actually update the world model and then evaluate whether that parameter update is responsible for the gains.
  5. [§3.3.2, Eq. (16)] The adaptive uncertainty penalty in Eq. (16) has two undeclared technical issues. First, the denominator r^{sel}_s + r^{sel}_d can be negative, zero, or very small because r^{sel}_s is a cosine similarity in its raw range [-1,1], so the penalty can flip sign or blow up; no clipping or additive constant is specified. Second, the numerator uses the difference between consecutive reward estimates \hat{r} - \hat{r}_{-1}, but after user selection \hat{r} is the average over a potentially different reference-user set at each step, so the difference may reflect the changing composition of the selected set rather than the evolving accuracy of the world model. Please clarify what exactly is tracked in \hat{r}_{-1}, add a stability mechanism for the denominator, and test sensitivity to this choice.
minor comments (5)
  1. [Eqs. (3) and (15)] Notation: Eq. (3) defines \hat{r} as an average over K world models, while Eq. (15) redefines \hat{r} as an average over reference users; using the same symbol for two different estimators is confusing. Use a distinct symbol such as \bar{r} for the reference-user average.
  2. [§3.2.1, Eqs. (7) and (10)] The selector state in Eq. (7) concatenates the recommender state with p_u, described as a row of the predicted user-item feedback matrix, but the exact dimensionality and how p_u is constructed from the world-model predictions or embeddings are not stated. Clarify this, since it affects the selector state size and the cosine similarity in Eq. (10).
  3. [§4.1.4] The hyperparameters K_sel, lambda_s, lambda_d, lambda_U, and lambda_E are tuned over dataset-specific ranges, but no validation-split or model-selection procedure is described. If the ranges were chosen on the test set, the hyperparameter sensitivity analysis does not demonstrate robustness; state how the final values were selected.
  4. [§4.3, Table 4] For the variants DARLR w. r_static and DARLR w. P_U,static, the text says these change the reward function and uncertainty penalty to those of ROLeR, but it is not explicit whether the selector is still used. State that explicitly so the ablation isolates the intended components.
  5. [Throughout] There are several typos and formatting artifacts: 'Acknowledegments' in Section 7, 'coeffients' in Section 4.5, and the broken word 'Reco- mmender' in the introduction should be corrected.

Circularity Check

1 steps flagged · score 4.0 of 10

The dynamic reward in Eq. 15 averages the same frozen world-model predictions that train the selector in Eq. 12, so the claimed 'refinement' is a self-referential statistic rather than an independent correction.

  1. self definitional [Sec. 3.2.1, Eq. (12); Sec. 3.3.1, Eq. (15)]
    "Combining the three parts, the intrinsic reward model is: r^{sel} = \hat{r} + \lambda_s r^{sel}_s + \lambda_d r^{sel}_d , (12) ... Then, the dynamic reward shaping can be implemented by averaging over the set: \hat{r}(u, i_t) = \frac{\sum_{u'\in u_{S,t}} \hat{r}(u', i_t)}{|u_{S,t}|} , (15)"

    The selector policy is trained with Eq. (12), whose first term is the world model's own prediction \hat{r}; therefore the selected reference set u_{S,t} is itself a function of \hat{r}. Eq. (15) then defines the recommender's 'refined' reward as the average of the same \hat{r} over that \hat{r}-dependent set. No independent or corrective signal enters between these two equations: the similarity gain (Eq. 10) and diversity gain (Eq. 11) are also computed from the world model's predicted feedback vectors p_u. Thus the dynamic reward is a self-referential statistic of the frozen model's predictions, not a correction of its inaccuracies.

full rationale

DARLR's central mechanism is the dynamic reward in Eq. (15), which averages the frozen world model's predicted rewards over reference users chosen by the selector. Because the selector's own intrinsic reward in Eq. (12) includes that same \hat{r}, the selection process and the resulting dynamic reward are coupled through the identical biased prediction source. This is a genuine self-referential loop: the method claims to mitigate reward inaccuracy, but the 'refined' estimate is a function of the very predictions whose inaccuracy it is meant to reduce. The paper does not invoke a load-bearing self-citation or a uniqueness theorem, and the final performance is benchmarked against ground-truth rewards and external baselines, which provides some independent evidence. However, Figure 4's 'Error' definition is not tightly specified, so it does not fully rule out the possibility that the selector simply finds reference users whose world-model predictions are easier rather than more accurate. Overall, the construction exhibits partial circularity in the reward-shaping claim, but the empirical comparison keeps the paper from being entirely circular. Score 4 reflects this moderate, localized self-referentiality rather than a fully forced derivation.

Assumptions & free parameters 9 free parameters · 5 assumptions · 0 invented entities

The central claim relies on several domain assumptions about user similarity, the informativeness of frozen reward predictions, and the correctness of an ad hoc intrinsic reward for the selector. The main free parameters are the five method hyperparameters plus architecture choices, all tuned per dataset. No new physical or conceptual entities are introduced beyond the algorithmic components.

free parameters (9)
  • lambda_s (similarity gain coefficient) = tuned per dataset from {0.5, 1, 2, 5}
    Controls the weight of user similarity in the selector intrinsic reward (Eq 12); no fixed value across datasets.
  • lambda_d (diversity gain coefficient) = tuned per dataset from {0.01, 0.05, 0.1, 0.5}
    Controls the weight of diversity among selected users in the selector intrinsic reward (Eq 12).
  • lambda_U (uncertainty penalty coefficient) = tuned per dataset from {0.01, 0.05, 0.1, 0.5, 1}
    Scales the dynamic uncertainty penalty in the final reward (Eq 17).
  • lambda_E (entropy penalty coefficient) = tuned per dataset from {0.01, 0.05, 0.1, 0.5, 1}
    Scales the entropy penalty retained from DORL (Eq 17).
  • K_sel (number of reference users) = varies by dataset: [10,20,30,40] for KuaiRec/Coat, [50,100,150,200] for KuaiRand, [25,50,75,100] for Yahoo
    Termination condition for the selection process; tuned per dataset and reported to significantly affect performance.
  • Transformer encoder layers = tuned from {1,2,3}
    Number of encoder layers in the state transition functions for selector and recommender (Eq 8 and Eq 14).
  • Attention heads = tuned from {1,2,3}
    Number of attention heads in the selector and recommender state trackers.
  • Selector window size w_sel = tuned from {3,5,10}
    Window size for the selector's Transformer transition function (Eq 8).
  • Recommender window size w_rec = tuned from {3,5,10}
    Window size for the recommender's Transformer transition function (Eq 14).
assumptions (5)
  • domain assumption Similar users share similar preferences, so cosine similarity between predicted feedback vectors is a valid signal for selecting reference users.
    Invoked in Eq 10 and Sec 3.2.1 ('based on that similar users may share similar preferences'), relying on prior work rather than proof.
  • domain assumption Averaging reward predictions over selected reference users improves the reward estimate for the current user.
    Core mechanism in Eq 15; no theoretical guarantee, especially when the same frozen world model generates all predictions.
  • ad hoc to paper The intrinsic reward in Eq 12 correctly aligns the selector's objective with reward-estimation accuracy.
    This reward model is designed for this paper; the reported ablations support it, but there is no independent verification that maximizing it yields representative user sets.
  • domain assumption World model predictions are informative enough that selecting users based on them does not compound errors.
    The method bootstraps entirely from the frozen world model; if predictions are systematically biased, the selector may select similarly biased users and amplify the bias.
  • domain assumption A2C is sufficient to train both the selector and recommender policies in this setting.
    A2C is chosen for simplicity and consistency with baselines, but no convergence guarantees are provided for the dual-agent interaction.

how reviews work

0 comments
Cite this review

Pith. "Pith review of DARLR: Dual-Agent Offline Reinforcement Learning for Recommender Systems with Dynamic Reward." pith.science (2026). https://pith.science/paper/FWL2VFZI

@misc{pith2026250507257,
  author       = {Pith},
  title        = {Pith review of: DARLR: Dual-Agent Offline Reinforcement Learning for Recommender Systems with Dynamic Reward},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/FWL2VFZI}},
  note         = {Machine review of arXiv:2505.07257}
}
read the original abstract

Model-based offline reinforcement learning (RL) has emerged as a promising approach for recommender systems, enabling effective policy learning by interacting with frozen world models. However, the reward functions in these world models, trained on sparse offline logs, often suffer from inaccuracies. Specifically, existing methods face two major limitations in addressing this challenge: (1) deterministic use of reward functions as static look-up tables, which propagates inaccuracies during policy learning, and (2) static uncertainty designs that fail to effectively capture decision risks and mitigate the impact of these inaccuracies. In this work, a dual-agent framework, DARLR, is proposed to dynamically update world models to enhance recommendation policies. To achieve this, a \textbf{\textit{selector}} is introduced to identify reference users by balancing similarity and diversity so that the \textbf{\textit{recommender}} can aggregate information from these users and iteratively refine reward estimations for dynamic reward shaping. Further, the statistical features of the selected users guide the dynamic adaptation of an uncertainty penalty to better align with evolving recommendation requirements. Extensive experiments on four benchmark datasets demonstrate the superior performance of DARLR, validating its effectiveness. The code is available at https://github.com/ArronDZhang/DARLR.

Figures

Figures reproduced from arXiv: 2505.07257 by the authors.

Figure 1
Figure 1. Recommendation policies trained from scratch with [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Overall Framework of DARLR with three main [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. The overall performance on KuaiRand and KuaiRec. [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: The mean reward differences (Error) during train [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: Fig.5. The cumulative reward is used as the evaluation metric and [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 5
Figure 5. Figure 5: Hyperparameter sensitivity with different [PITH_FULL_IMAGE:figures/full_fig_p009_5.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

83 extracted references · 69 canonical work pages

  1. [1]

    M Mehdi Afsar, Trafford Crump, and Behrouz Far. 2022. Reinforcement learning based recommender systems: A survey.Comput. Surveys(2022)

    2022

  2. [2]

    Rishabh Agarwal, Dale Schuurmans, and Mohammad Norouzi. 2020. An opti- mistic perspective on offline reinforcement learning. InICML

    2020

  3. [3]

    Jesús Bobadilla, Fernando Ortega, Antonio Hernando, and Abraham Gutiérrez

  4. [4]

    Robin Burke. 2002. Hybrid recommender systems: Survey and experiments.User Modeling and User-Adapted Interaction(2002)

    work page 2002

  5. [5]

    Lucian Busoniu, Robert Babuska, and Bart De Schutter. 2008. A comprehensive survey of multiagent reinforcement learning.TSMC(2008)

    work page 2008

  6. [6]

    Jianxin Chang, Chenbin Zhang, Zhiyi Fu, Xiaoxue Zang, Lin Guan, Jing Lu, Yiqun Hui, Dewei Leng, Yanan Niu, Yang Song, et al. 2023. TWIN: TWo-stage interest network for lifelong user behavior modeling in CTR prediction at kuaishou. In SIGKDD

    work page 2023

  7. [7]

    Haokun Chen, Xinyi Dai, Han Cai, Weinan Zhang, Xuejian Wang, Ruiming Tang, Yuzhou Zhang, and Yong Yu. 2019. Large-scale interactive recommendation with tree-structured policy gradient. InAAAI

    work page 2019

  8. [8]

    Minmin Chen, Alex Beutel, Paul Covington, Sagar Jain, Francois Belletti, and Ed H Chi. 2019. Top-k off-policy correction for a REINFORCE recommender system. InWSDM

    work page 2019

Show all 83 references
  1. [9]

    Minmin Chen, Bo Chang, Can Xu, and Ed H Chi. 2021. User response models to improve a reinforce recommender system. InWSDM

    2021

  2. [10]

    Xiaocong Chen, Siyu Wang, Julian McAuley, Dietmar Jannach, and Lina Yao

  3. [11]

    Xiaocong Chen, Lina Yao, Julian McAuley, Guanglin Zhou, and Xianzhi Wang

  4. [12]

    Christopher Diehl, Timo Sebastian Sievernich, Martin Krüger, Frank Hoffmann, and Torsten Bertram. 2023. Uncertainty-aware model-based offline reinforcement learning for automated driving.IEEE RA-L(2023)

    2023

  5. [13]

    Xing Fang, Qichao Zhang, Yinfeng Gao, and Dongbin Zhao. 2022. Offline re- inforcement learning for autonomous driving with real world driving data. In ITSC

    2022

  6. [14]

    Deep reinforcement learning in recommender systems: A survey and new perspectives.KBS(2023)

    2023

  7. [15]

    Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine

  8. [16]

    Scott Fujimoto, David Meger, and Doina Precup. 2019. Off-policy deep reinforce- ment learning without exploration. InICML

    2019

  9. [17]

    Jun Feng, Heng Li, Minlie Huang, Shichen Liu, Wenwu Ou, Zhirong Wang, and Xiaoyan Zhu. 2018. Learning to collaborate: Multi-scenario ranking via multi- agent reinforcement learning. InWWW

    2018

  10. [18]

    Chongming Gao, Shijun Li, Wenqiang Lei, Jiawei Chen, Biao Li, Peng Jiang, Xiangnan He, Jiaxin Mao, and Tat-Seng Chua. 2022. KuaiRec: A fully-observed dataset and insights for evaluating recommender systems. InCIKM

    2022

  11. [19]

    Chongming Gao, Shijun Li, Yuan Zhang, Jiawei Chen, Biao Li, Wenqiang Lei, Peng Jiang, and Xiangnan He. 2022. KuaiRand: An Unbiased Sequential Recom- mendation Dataset with Randomly Exposed Videos. InCIKM

    2022

  12. [20]

    Chongming Gao, Shiqi Wang, Shijun Li, Jiawei Chen, Xiangnan He, Wenqiang Lei, Biao Li, Yuan Zhang, and Peng Jiang. 2023. CIRS: Bursting filter bubbles by counterfactual interactive recommender system.TOIS(2023)

    2023

  13. [21]

    Chongming Gao, Kexin Huang, Jiawei Chen, Yuan Zhang, Biao Li, Peng Jiang, Shiqi Wang, Zhong Zhang, and Xiangnan He. 2023. Alleviating matthew effect of offline reinforcement learning in interactive recommendation. InSIGIR

    2023

  14. [22]

    Jin Huang, Harrie Oosterhuis, Bunyamin Cetinkaya, Thijs Rood, and Maarten de Rijke. 2022. State encoders in reinforcement learning for recommendation: A reproducibility study. InSIGIR

    2022

  15. [23]

    Eugene Ie, Vihan Jain, Jing Wang, Sanmit Narvekar, Ritesh Agarwal, Rui Wu, Heng-Tze Cheng, Tushar Chandra, and Craig Boutilier. 2019. SLATEQ: a tractable decomposition for reinforcement learning with recommendation sets. InIJCAI

    2019

  16. [24]

    Michael Janner, Justin Fu, Marvin Zhang, and Sergey Levine. 2019. When to trust your model: Model-based policy optimization. InNeurIPS

    2019

  17. [25]

    Huifeng Guo, Ruiming Tang, Yunming Ye, Zhenguo Li, and Xiuqiang He. 2017. DeepFM: a factorization-machine based neural network for CTR prediction. In IJCAI

    2017

  18. [26]

    Wang-Cheng Kang and Julian McAuley. 2018. Self-attentive sequential recom- mendation. InICDM

    2018

  19. [27]

    Rahul Kidambi, Aravind Rajeswaran, Praneeth Netrapalli, and Thorsten Joachims

  20. [28]

    Aviral Kumar, Justin Fu, Matthew Soh, George Tucker, and Sergey Levine. 2019. Stabilizing off-policy q-learning via bootstrapping error reduction. InNIPS

    2019

  21. [29]

    Wei Jiang, Xinyi Gao, Guandong Xu, Tong Chen, and Hongzhi Yin. 2024. Chal- lenging Low Homophily in Social Recommendation. InWWW

    2024

  22. [30]

    Tze Leung Lai and Herbert Robbins. 1985. Asymptotically efficient adaptive allocation rules.Advances in Applied Mathematics(1985)

    1985

  23. [31]

    Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. 2020. Offline Rein- forcement Learning: Tutorial, Review, and Perspectives on Open Problems.CoRR abs/2005.01643 (2020)

    2020 arXiv

  24. [32]

    InNeurIPS

    Morel: Model-based offline reinforcement learning. InNeurIPS

  25. [33]

    Xiao-Yin Liu, Xiao-Hu Zhou, Guo-Tao Li, Hao Li, Mei-Jiang Gui, Tian-Yu Xiang, De-Xing Huang, and Zeng-Guang Hou. 2024. MICRO: Model-Based Offline Reinforcement Learning with a Conservative Bellman Operator. InIJCAI

    2024

  26. [34]

    Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. 2020. Conserva- tive q-learning for offline reinforcement learning. InNeurIPS

    2020

  27. [35]

    Jianlan Luo, Perry Dong, Jeffrey Wu, Aviral Kumar, Xinyang Geng, and Sergey Levine. 2023. Action-quantized offline reinforcement learning for robotic skill learning. InCoRL. 1348–1361

    2023

  28. [36]

    Yadan Luo, Zi Huang, Zheng Zhang, Ziwei Wang, Jingjing Li, and Yang Yang. 2019. Curiosity-driven reinforcement learning for diverse visual paragraph generation. InMM

    2019

  29. [37]

    Yang Li, Tong Chen, Yadan Luo, Hongzhi Yin, and Zi Huang. 2021. Discovering Collaborative Signals for Next POI Recommendation with Iterative Seq2Graph Augmentation. InIJCAI

    2021

  30. [38]

    Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timo- thy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. 2016. Asynchro- nous methods for deep reinforcement learning. InICML

    2016

  31. [39]

    Ryan Lowe, Yi I Wu, Aviv Tamar, Jean Harb, OpenAI Pieter Abbeel, and Igor Mordatch. 2017. Multi-agent actor-critic for mixed cooperative-competitive environments. InNeurIPS

    2017

  32. [40]

    Rafael Figueiredo Prudencio, Marcos ROA Maximo, and Esther Luna Colombini

  33. [41]

    Ruihong Qiu, Zi Huang, and Hongzhi Yin. 2021. Memory augmented multi- instance contrastive predictive coding for sequential recommendation. InICDM

    2021

  34. [42]

    Benjamin M Marlin and Richard S Zemel. 2009. Collaborative prediction and ranking with non-random missing data. InRecSys

    2009

  35. [43]

    Ruihong Qiu, Jingjing Li, Zi Huang, and Hongzhi Yin. 2019. Rethinking the item order in session-based recommendation with graph neural networks. InCIKM

    2019

  36. [44]

    Bei Peng, Tabish Rashid, Christian Schroeder de Witt, Pierre-Alexandre Kamienny, Philip Torr, Wendelin Böhmer, and Shimon Whiteson. 2021. Facmac: Factored multi-agent centralised policy gradients. InNeurIPS

    2021

  37. [45]

    Yuhang Ran, Yi-Chen Li, Fuxiang Zhang, Zongzhang Zhang, and Yang Yu. 2023. Policy regularization with dataset constraint for offline reinforcement learning. InICML

    2023

  38. [46]

    A survey on offline reinforcement learning: Taxonomy, review, and open problems.TNNLS(2023)

    2023

  39. [47]

    Marc Rigter, Bruno Lacerda, and Nick Hawes. 2022. RAMBO-RL: Robust Adver- sarial Model-Based Offline Reinforcement Learning. InNeurIPS

    2022

  40. [48]

    Ruihong Qiu, Zi Huang, Hongzhi Yin, and Zijian Wang. 2022. Contrastive learning for representation degeneration problem in sequential recommendation. InWSDM

    2022

  41. [49]

    Samarth Sinha, Ajay Mandlekar, and Animesh Garg. 2022. S4rl: Surprisingly simple self-supervision for offline reinforcement learning in robotics. InCoRL. 907–917

    2022

  42. [50]

    Ruihong Qiu, Hongzhi Yin, Zi Huang, and Tong Chen. 2020. Gag: Global attributed graph neural network for streaming session-based recommendation. InSIGIR

    2020

  43. [51]

    2018.Reinforcement learning: An intro- duction

    Richard S Sutton and Andrew G Barto. 2018.Reinforcement learning: An intro- duction. MIT press

    2018

  44. [52]

    Tabish Rashid, Mikayel Samvelyan, Christian Schroeder De Witt, Gregory Far- quhar, Jakob Foerster, and Shimon Whiteson. 2020. Monotonic value function factorisation for deep multi-agent reinforcement learning.JMLR(2020)

    2020

  45. [53]

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. InNeurIPS

    2017

  46. [54]

    Tobias Schnabel, Adith Swaminathan, Ashudeep Singh, Navin Chandak, and Thorsten Joachims. 2016. Recommendations as treatments: Debiasing learning and evaluation. InICML

    2016

  47. [55]

    Hengliang Wang and Kedian Mu. 2020. Aspect-Level Attributed Network Em- bedding via Variational Graph Neural Networks. InDASFAA

    2020

  48. [56]

    Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. 2020. Learning to summarize with human feedback.NeurIPS(2020)

    2020

  49. [57]

    Xu Wang, Jiangxia Cao, Zhiyi Fu, Kun Gai, and Guorui Zhou. 2025. HoME: Hier- archy of Multi-Gate Experts for Multi-Task Learning at Kuaishou. InSIGKDD

    2025

  50. [58]

    Adith Swaminathan and Thorsten Joachims. 2015. Counterfactual risk minimiza- tion: Learning from logged bandit feedback. InICML

    2015

  51. [59]

    Shaowei Wei, Zhengwei Wu, Xin Li, Qintong Wu, Zhiqiang Zhang, Jun Zhou, Lihong Gu, and Jinjie Gu. 2024. Leave No One Behind: Online Self-Supervised DARLR: Dual-Agent Offline Reinforcement Learning for Recommender Systems with Dynamic Reward SIGIR ’25, July 13–18, 2025, Padua, ...

    2024

  52. [60]

    Anna Vettoruzzo, Mohamed-Rafik Bouguelia, Joaquin Vanschoren, Thorsteinn Rögnvaldsson, and KC Santosh. 2024. Advances and challenges in meta-learning: A technical review.TPAMI(2024)

    2024

  53. [61]

    Xin Xin, Alexandros Karatzoglou, Ioannis Arapakis, and Joemon M Jose. 2020. Self-supervised reinforcement learning for recommender systems. InSIGIR

    2020

  54. [62]

    Foster, and Sham M

    Ruosong Wang, Dean P. Foster, and Sham M. Kakade. 2021. What are the Statis- tical Limits of Offline RL with Linear Function Approximation?. InICLR

    2021

  55. [63]

    Wanqi Xue, Qingpeng Cai, Ruohan Zhan, Dong Zheng, Peng Jiang, Kun Gai, and Bo An. 2023. ResAct: Reinforcing Long-term Engagement in Sequential Recommendation with Residual Actor. InICLR

    2023

  56. [64]

    Ziyu Wang, Alexander Novikov, Konrad Zolna, Josh S Merel, Jost Tobias Sprin- genberg, Scott E Reed, Bobak Shahriari, Noah Siegel, Caglar Gulcehre, Nicolas Heess, et al. 2020. Critic regularized regression. InNeurIPS

    2020

  57. [65]

    Tianhe Yu, Garrett Thomas, Lantao Yu, Stefano Ermon, James Y Zou, Sergey Levine, Chelsea Finn, and Tengyu Ma. 2020. Mopo: Model-based offline policy optimization. InNeurIPS

    2020

  58. [66]

    Ronald J Williams. 1992. Simple statistical gradient-following algorithms for connectionist reinforcement learning.Machine learning(1992)

    1992

  59. [67]

    Huichu Zhang, Siyuan Feng, Chang Liu, Yaoyao Ding, Yichen Zhu, Zihan Zhou, Weinan Zhang, Yong Yu, Haiming Jin, and Zhenhui Li. 2019. Cityflow: A multi- agent reinforcement learning environment for large scale city traffic scenario. In WWW

    2019

  60. [68]

    Wanqi Xue, Qingpeng Cai, Zhenghai Xue, Shuo Sun, Shuchang Liu, Dong Zheng, Peng Jiang, Kun Gai, and Bo An. 2023. PrefRec: Recommender Systems with Human Preferences for Reinforcing Long-term User Engagement. InSIGKDD

    2023

  61. [69]

    Shuai Zhang, Lina Yao, Aixin Sun, and Yi Tay. 2019. Deep learning based rec- ommender system: A survey and new perspectives.ACM computing surveys (2019)

    2019

  62. [70]

    Tianhe Yu, Aviral Kumar, Rafael Rafailov, Aravind Rajeswaran, Sergey Levine, and Chelsea Finn. 2021. Combo: Conservative offline model-based policy optimization. InNeurIPS

    2021

  63. [71]

    Yang Zhang, Chenwei Zhang, and Xiaozhong Liu. 2017. Dynamic scholarly col- laborator recommendation via competitive multi-agent reinforcement learning. InRecSys

    2017

  64. [72]

    Yuanqing Yu, Chongming Gao, Jiawei Chen, Heng Tang, Yuefeng Sun, Qian Chen, Weizhi Ma, and Min Zhang. 2024. EasyRL4Rec: An Easy-to-use Library for Reinforcement Learning Based Recommender Systems. InSIGIR

    2024

  65. [73]

    Xiangyu Zhao, Long Xia, Lixin Zou, Hui Liu, Dawei Yin, and Jiliang Tang. 2020. Whole-chain recommendations. InCIKM

    2020

  66. [74]

    Michael R Zhang, Nishkrit Desai, Juhan Bae, Jonathan Lorraine, and Jimmy Ba

  67. [75]

    Using large language models for hyperparameter optimization.arXiv preprint arXiv:2312.04528(2023)

    2023 arXiv

  68. [77]

    Yi Zhang, Ruihong Qiu, Jiajun Liu, and Sen Wang. 2024. ROLeR: Effective Reward Shaping in Offline Reinforcement Learning for Recommender Systems. InCIKM

    2024

  69. [79]

    Dongyang Zhao, Liang Zhang, Bo Zhang, Lizhou Zheng, Yongjun Bao, and Weipeng Yan. 2020. Mahrl: Multi-goals abstraction based deep hierarchical reinforcement learning for recommendations. InSIGIR

    2020

  70. [81]

    Xiangyu Zhao, Xudong Zheng, Xiwang Yang, Xiaobing Liu, and Jiliang Tang

  71. [82]

    InSIGKDD

    Jointly learning to recommend and advertise. InSIGKDD

  72. [83]

    Peilin Zhou, You-Liang Huang, Yueqi Xie, Jingqi Gao, Shoujin Wang, Jae Boum Kim, and Sunghun Kim. 2024. Is Contrastive Learning Necessary? A Study of Data Augmentation vs Contrastive Learning in Sequential Recommendation. In WWW

    2024

  73. [2013]

    Recommender systems survey.KBS(2013)

    2013

  74. [2020]

    D4RL: Datasets for Deep Data-Driven Reinforcement Learning.CoRR abs/2004.07219 (2020)

    2020 arXiv

  75. [2023]

    On the opportunities and challenges of offline reinforcement learning for recommender systems.TOIS(2023)

    2023

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.