Reward Balancing Revisited: Enhancing Offline Reinforcement Learning for Recommender Systems

Fan Zhou; Ning Luo; Peng Jiang; Teng Sha; Wenzheng Shu; Xialong Liu; Yanhua Cheng; Yanxiang Zeng; Yongxiang Tang

REVIEW 4 major objections 5 minor 19 references

Reward Balancing Revisited: Enhancing Offline Reinforcement Learning for Recommender Systems

T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read R3S claims that subtracting diffusion-sampling variance from the reward and adding decayed diversity penalties makes offline RL recommenders both more accurate and more diverse, beating eleven baselines on three datasets.

desk verdict Competent incremental extension of ROLeR with a plausible but unvalidated uncertainty term; worth a referee, not a must-read. read the letter →

arxiv 2506.22112 v2 pith:HJVTEPES submitted 2025-06-27 cs.IR

Wenzheng Shu , Yanxiang Zeng , Yongxiang Tang , Teng Sha , Ning Luo , Yanhua Cheng , Xialong Liu , Fan Zhou

show 1 more author

Peng Jiang

This is my paper

classification cs.IR

keywords offlinereinforcementlearningrecommendersystemsdiffusionmodelrewardshapinguncertaintyestimationdiversitypenaltyinteractiverecommendationworld

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper proposes R3S, an offline reinforcement learning method for recommender systems whose central claim is that reward quality improves when the reward signal is reallocated in two ways: penalizing the uncertainty of a diffusion-based reward world model, and adding decayed diversity penalties that encourage varied recommendations. The intended effect is to correct the world model's intrinsic fluctuations while keeping the policy from collapsing onto popular items, which the paper frames as harmonizing heterogeneous user preferences. If the claim holds, offline recommender policies can be trained from logged interaction data with less reward noise and more interactive diversity, without needing online exploration. The paper reports that R3S outperforms eleven baselines on Coat, Yahoo, and KuaiRand and that each component of the reward scheme contributes to the result.

What carries the argument

The load-bearing object is the composite reward of Eq. (3), built on three components. The diffusion world model, using the DiffRec backbone, conditions a denoising network on user and item embeddings and produces $\hat r_D$ by averaging $M$ reverse samples; its spread defines $P_D$, the intrinsic uncertainty penalty. The diversity penalties are $P_E=-D_{KL}(\pi_\beta(\cdot|s)\|\pi_u(\cdot|s))$, which measures how far the behavior policy is from uniform and penalizes over-concentration on popular items, and $P_I$, which computes entropy over $k$ randomly sampled earlier positions in the trajectory to capture long-range interaction variety. The decay function $\omega(l)=\alpha(e^{-\xi l}+1)$, with $\alpha=5\times10^{-1}$ and $\xi=1.0$, interpolates between these two penalties as the interaction proceeds.

What would settle it

On held-out interaction data, compute $P_D$ for each state-action pair and compare it with the actual squared error of the world model's mean reward prediction: if $P_D$ does not rank-order those errors, or if an ablation that removes the $-\lambda_1 P_D$ term does not degrade any measured accuracy of the world model, then the central claim fails.

Watch

Extended reading notes

Core claim

R3S's central discovery is the reallocated reward rule in Eq. (3): $\hat r = \hat r_D - \lambda_1 P_D + \lambda_2[(1-\omega(l))P_I+\omega(l)P_E]$, trained on a diffusion-based world model. $\hat r_D$ is the mean of $M$ reverse diffusion samples; $P_D=\frac{1}{M}\sum_{m=1}^{M}(x_m-r_D)^2$ is the variance across those samples, used as an intrinsic uncertainty penalty; $P_E$ is a KL-divergence entropy penalty that keeps the policy close to uniform; and $P_I$ is a new interactive penalty computed over $k$ randomly sampled past positions rather than a fixed recent window. An exponential decay $\omega(l)=\alpha(e^{-\xi l}+1)$ shifts weight from $P_E$ early in a trajectory to $P_I$ later. The paper's claim is that this combination makes the world model's reward predictions more accurate and lets the policy balance immediate diversity with long-range interaction dependencies, which is supported by the reported gains over eleven baselines.

Load-bearing premise

The paper's result depends on the variance among the diffusion model's repeated reward samples being a true measure of reward-prediction uncertainty, so that subtracting it improves the world model rather than merely adding sampling noise.

Editorial extensions

If this is right

Policies trained with Eq. (3) should earn higher cumulative reward per interaction on offline recommender benchmarks, because reward noise is reduced before value learning.
The decayed penalty schedule should make a recommender explore broadly early in a session and capture long-range state dependencies later, which the paper reports as longer effective interactions on KuaiRand.
Diffusion world models can supply the uncertainty signal for reward shaping without training an ensemble, lowering the cost of uncertainty-aware offline RL.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

One testable extension is to replace $P_D$ with an independently calibrated uncertainty estimate, such as ensemble disagreement or conformal prediction intervals; if $P_D$ is mostly sampling noise, the calibrated estimate should match or beat R3S at comparable cost.
The ablations suggest the decay schedule could be tuned to data density: the entropy penalty $P_E$ helps on small sparse datasets like Coat, while the interactive penalty $P_I$ helps on complex logs like KuaiRand, so an automatic schedule would likely improve robustness.
The same uncertainty-minus-variance reward correction could be applied to any offline RL setting with a diffusion-based dynamics model, for example continuous control from logged trajectories, where reward noise and state diversity are also central concerns.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

You'll want to know: this paper extends ROLeR by adding a diffusion-based uncertainty penalty and a decay-weighted mix of two diversity penalties, and reports consistent gains over ROLeR on three datasets. It's a modest, honest increment, not a field reorganization. The strongest part is the breadth of baselines and the real-world datasets; the softest part is that the uncertainty mechanism is never actually validated.

What's new: R3S takes the diffusion backbone from DiffRec, the entropy penalty from DORL, and the overall offline RL setup from ROLeR, then adds two twists. First, it uses the sample variance of M reverse-sampled rewards from the diffusion world model as an intrinsic uncertainty penalizer P_D. Second, it combines the existing entropy penalty P_E with a new interactive penalty P_I that samples k random positions in the history, weighted by an exponential decay. That specific combination is not in the cited works, and the empirical comparison against ROLeR and DORL is legitimate and reasonably thorough.

The paper does some things well. The experimental table includes eleven baselines, three real-world datasets, and three metrics. The ablations at least attempt to separate uncertainty, diversity, and the two penalty types. The claims in the text are mostly consistent with the tables, and the authors clearly acknowledge the lineage of their method.

The soft spots are real but not disqualifying. The definition of P_D uses 'x_m' in Eq. (1)'s vicinity, which is not defined; presumably it should be the reward sample. More importantly, the paper never shows that this variance correlates with predictive error. It could mostly be sampling noise from the diffusion reverse process. The ablation w/o U replaces the diffusion world model with a DeepFM predictor, so it changes both the architecture and the uncertainty penalty at once; you cannot attribute the gain to the uncertainty term. No significance tests are reported, and the margins over ROLeR on Yahoo and KuaiRand are within standard errors. Table 2 has no error bars at all. The hyperparameters alpha and xi are stated but not investigated. No code is released.

Despite these gaps, the central comparative claim is plausible: R3S does not look worse than ROLeR on any dataset, and the reward-shaping recipe could be useful to practitioners. The paper is short and likely a companion-format paper, which explains some of the missing detail.

Who should read it: people working on offline RL for recommender systems, especially those building on DORL or ROLeR. It deserves a serious referee if submitted to a full venue, but it needs a stronger mechanism validation and significance testing before acceptance.

My recommendation: engage with it as a modest, potentially useful extension, but do not treat the uncertainty claim as established.

Referee Report

4 major / 5 minor

Summary. The paper proposes R3S, an offline reinforcement learning framework for recommender systems built on an A2C policy and a diffusion-based world model. The world model predicts rewards by averaging M reverse samples (Eq. 1) and derives an uncertainty penalty P_D from their spread. The policy reward in Eq. (3) subtracts λ1 P_D and adds a weighted combination of an entropy penalty P_E and a new interactive penalty P_I, with an exponential decay ω(l) that favors P_E early and P_I later. Experiments on Coat, Yahoo, and KuaiRand compare R3S against eleven baselines, and ablations are reported on Coat and KuaiRand. The paper claims improved world-model accuracy, harmonization of heterogeneous user preferences, and state-of-the-art cumulative reward.

Significance. If the central claims held, the diffusion-based uncertainty penalty and the decayed diversity penalties would be a useful recipe for offline RL recommenders, extending ROLeR and DiffRec. The paper uses three real-world datasets and eleven baselines, and the method is described in enough detail to be reimplemented. However, the evidence is not yet sufficient: the uncertainty quantity is uncalibrated, the key ablation changes two factors at once, and the improvements over the strongest baseline overlap with the reported variance on two of three datasets. The idea is worth pursuing, but the empirical support needs substantial strengthening before the claims are acceptable.

major comments (4)

[3.1, Eq. (1) and P_D definition] The uncertainty penalty P_D is load-bearing for the claim that R3S improves the accuracy of world models, but its definition is not consistent. Eq. (1) defines r_D as the average of W_phi[r_T^m] over M samples from N(0,I), while the text immediately after defines P_D as (1/M) Σ (x_m − r_D)^2, introducing x_m without definition. If x_m denotes the input feature vector, the expression is dimensionally mismatched with a reward; if it is a typo for the reward sample from the reverse process, P_D is simply the sample variance of M stochastic reverse samples. In either reading, there is no evidence that this quantity captures predictive (epistemic) uncertainty rather than reverse-process sampling noise: M is never reported, no comparison is made to other uncertainty estimators such as ensemble variance or MC dropout, and no calibration against reward prediction errors is provided. The authors should fix the notation, report M, and include a direct validation of P_D as an uncertainty estimate, together with an isolated ablation that removes only the P_D term while keeping the diffusion world model fixed.
[4.3, Table 2 (w/o U)] The ablation labeled w/o U replaces the diffusion world model with a DeepFM predictor and removes the uncertainty penalty simultaneously. This changes two factors at once, so any performance difference cannot be attributed to the uncertainty subtraction; it could equally be due to the generative capacity of the diffusion model. An ablation that keeps the diffusion world model and simply sets λ1 = 0 is needed. In addition, Table 2 reports no variances or significance tests, and the ablation directions differ between Coat and KuaiRand (e.g., w/oD is better than w/oU on Coat but worse on KuaiRand), so the current results do not support the mechanistic interpretation given in Section 4.3.
[4.2, Table 1] The claimed superiority over the strongest baseline ROLeR is not statistically supported. On Yahoo, R3S (69.223 ± 2.212) and ROLeR (68.364 ± 1.855) overlap by more than a standard error; on KuaiRand, R3S (14.087 ± 1.828) and ROLeR (13.455 ± 1.509) also overlap, and R3S has a lower LENGTH (28.841 vs. 29.270) despite the claim of harmonizing heterogeneous preferences. No significance tests, effect sizes, or per-seed results are reported. The authors should report paired tests (e.g., bootstrap or paired t-test over seeds) and discuss the practical significance of the differences; as presented, the table supports at most a modest improvement on Coat.
[3.2-3.3, Eq. (3) and hyperparameters] The decay function ω(l) = α(e^{−ξl} + 1) sets α = 0.5 and ξ = 1.0 without derivation or sensitivity analysis, and the window size k for P_I and the number of reverse samples M are not reported. Because the balance between P_E and P_I in Eq. (3) is the paper's second contribution, these values are load-bearing. A sensitivity sweep over α, ξ, k, and M is needed to show that the method does not hinge on a single tuned setting. The assertion that randomly sampling k positions captures long-range dependencies better than a contiguous subsequence is also not supported by any ablation; a comparison between the two sampling schemes would be informative.

minor comments (5)

[1, 3.1, 2.1] There are several typos and formatting issues: 'Challanges' in Section 1, 'Foward Training' in Section 3.1, the garbled phrase 'the 𝑗-th cumulative reward gain' in Section 2.1, and the line break 'harmo-nizes' in the abstract.
[3.1, Eq. (1)] The notation W_phi[·] is not defined as an operator; please clarify whether it denotes the full diffusion model mapping from an initial Gaussian sample to a reward prediction.
[Figure 2 and Eq. (2)] The caption uses π_β for the behavior policy while Eq. (2) writes π_β and π_u; please clarify the relation between the behavior policy and the uniform policy, and between π_β and the target policy π_θ.
[4.1] The paper refers to [17] for dataset details but gives no summary statistics or splits for Coat, Yahoo, or KuaiRand; adding one sentence with dataset sizes and preprocessing choices would aid reproducibility.
[Table 2 caption] The caption 'Note that: (w/o) without (U) diffusion uncertainty, and using DeepFM [4] instead; (D) diversity mechanism; (P_E) entropy penalizer; (P_I) interactive penalizer' is grammatically awkward and should be rewritten for clarity.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: the central claim rests on external benchmarks and is not equivalent to the paper's inputs by construction.

full rationale

The derivation chain in R3S is not circular in the sense defined by the review criteria. The reward formulation in Eq. (3) combines a diffusion-derived mean reward r_D, an uncertainty penalty P_D computed from multiple reverse samples, and diversity penalties P_E and P_I; none of these are defined in terms of the final evaluation metrics or the final policy objective. The empirical claim that R3S improves world-model accuracy and user diversity is tested against external datasets (Coat, Yahoo, KuaiRand) and external baselines such as ROLeR, CQL, BCQ, and MOPO, so the central comparison is not fitted-then-predicted on the same quantity. The only self-citations are to prior work coauthored by one of the present authors: [2] (DORL) for the entropy penalty P_E and [3] (KuaiRand) as a dataset; these are adopted as components or benchmarks rather than used to justify the core contribution, so they are not load-bearing in a circular way. The concerns raised about P_D, such as the notational inconsistency between x_m and r_D and the lack of calibration against alternative uncertainty estimates, are validity and reproducibility issues rather than demonstrations that the paper's result reduces to its own inputs. Therefore the appropriate finding is no significant circularity, with the score reflecting only a minor non-load-bearing self-citation.

Assumptions & free parameters 6 free parameters · 6 assumptions · 0 invented entities

The method depends on several hand-set hyperparameters and on assumptions about diffusion uncertainty and entropy-based diversity that are asserted rather than derived. The benchmark comparisons are external, so the central empirical claim is not circular, but the design choices are justified post hoc by the same experiments.

free parameters (6)

alpha (decay scale) = 0.5
Set by hand in Section 3.2; controls the magnitude of the exponential decay term in the diversity reward. No sensitivity analysis is reported.
xi (decay rate) = 1.0
Set by hand in Section 3.2; controls how quickly P_E is replaced by P_I over trajectory steps. No sensitivity analysis is reported.
lambda1 (uncertainty weight) = not reported
Weight on the diffusion uncertainty penalty in Eq. (3). Values are not stated in the paper, and no tuning grid is given.
lambda2 (diversity weight) = not reported
Weight on the combined entropy and interactive penalty in Eq. (3). Values and tuning procedure are omitted.
k (sampling window size for P_I) = not reported
The number of randomly sampled past positions used by the interactive penalty P_I in Section 3.2; not specified, yet it defines P_I.
M (number of reverse samples) = not reported
Number of diffusion reverse samples used to estimate mean reward and uncertainty in Eq. (1); no value is given.

assumptions (6)

domain assumption Offline recommendation can be modeled as an MDP with transition T and reward r learned from logged data.
Section 2.1 defines the MDP without discussing whether user feedback in offline logs is unbiased or satisfies the Markov property; this is a standard assumption inherited from [18].
domain assumption The variance of multiple diffusion reverse samples is a valid measure of intrinsic reward uncertainty.
Section 3.1 defines P_D as the sample variance of M reverse samples and treats it as inherent predictive uncertainty without calibration or ablation against other uncertainty estimates.
domain assumption An entropy penalty against the uniform distribution increases diversity and mitigates the Matthew effect.
Section 3.2 imports P_E from DORL [2] and builds P_I on the same premise; no independent justification is given.
ad hoc to paper The interactive penalty P_I over k randomly sampled positions captures long-range dependencies better than a contiguous subsequence.
Section 3.2 asserts this property but provides no theorem, no controlled test, and no visualization; it is the key design choice distinguishing R3S from ROLeR.
ad hoc to paper Exponential decay with alpha=0.5 and xi=1.0 balances early entropy and late interactive diversity.
Section 3.2 states the values without derivation or sensitivity analysis; the shape of the weighting function is assumed, not learned.
domain assumption Hyperparameters and evaluation protocols from [13], [17], and [18] transfer unchanged to the new components.
Section 4.1 says configurations are adopted from prior work, but the new terms introduce lambda1, lambda2, M, and k that are not covered by those configurations.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Reward Balancing Revisited: Enhancing Offline Reinforcement Learning for Recommender Systems." pith.science (2026). https://pith.science/paper/HJVTEPES

@misc{pith2026250622112,
  author       = {Pith},
  title        = {Pith review of: Reward Balancing Revisited: Enhancing Offline Reinforcement Learning for Recommender Systems},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/HJVTEPES}},
  note         = {Machine review of arXiv:2506.22112}
}

read the original abstract

Offline reinforcement learning (RL) has emerged as a prevalent and effective methodology for real-world recommender systems, enabling learning policies from historical data and capturing user preferences. In offline RL, reward shaping encounters significant challenges, with past efforts to incorporate prior strategies for uncertainty to improve world models or penalize underexplored state-action pairs. Despite these efforts, a critical gap remains: the simultaneous balancing of intrinsic biases in world models and the diversity of policy recommendations. To address this limitation, we present an innovative offline RL framework termed Reallocated Reward for Recommender Systems (R3S). By integrating inherent model uncertainty to tackle the intrinsic fluctuations in reward predictions, we boost diversity for decision-making to align with a more interactive paradigm, incorporating extra penalizers with decay that deter actions leading to diminished state variety at both local and global scales. The experimental results demonstrate that R3S improves the accuracy of world models and efficiently harmonizes the heterogeneous preferences of the users.

Figures

Figures reproduced from arXiv: 2506.22112 by the authors.

**Figure 2.** Outline of the R3S . It comprises a diffusion-based world model along with an A2C paradigm. Note that 𝜋𝛽 denotes the inherent behavior within offline data, whereas 𝜋𝜃 represents the target policy to be optimized. rather than statistically evaluating the differences among distributions within additional ensembles of the world models. Considering each interaction 𝒙𝑚, we can refine the uncertainty of reward 𝑃𝐷 direct… view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

19 extracted references · 15 canonical work pages

[1]

Scott Fujimoto, David Meger, and Doina Precup. 2019. Off-Policy Deep Rein- forcement Learning without Exploration. In ICML, Vol. 97. PMLR, 2052–2062

work page 2019
[2]

Chongming Gao, Kexin Huang, Jiawei Chen, Yuan Zhang, Biao Li, Peng Jiang, Shiqi Wang, Zhong Zhang, and Xiangnan He. 2023. Alleviating Matthew Effect of Offline Reinforcement Learning in Interactive Recommendation. In SIGIR. ACM, 238–248

work page 2023
[3]

Chongming Gao, Shijun Li, Yuan Zhang, Jiawei Chen, Biao Li, Wenqiang Lei, Peng Jiang, and Xiangnan He. 2022. KuaiRand: An Unbiased Sequential Recom- mendation Dataset with Randomly Exposed Videos. In CIKM. ACM, 3953–3957

work page 2022
[4]

Huifeng Guo, Ruiming Tang, Yunming Ye, Zhenguo Li, and Xiuqiang He. 2017. DeepFM: A Factorization-Machine based Neural Network for CTR Prediction. In IJCAI. ijcai.org, 1725–1731

2017
[5]

Michael Janner, Justin Fu, Marvin Zhang, and Sergey Levine. 2019. When to Trust Your Model: Model-Based Policy Optimization. InAdvances in Neural Information Processing Systems, Vol. 32. 12498–12509

work page 2019
[6]

Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. 2020. Conser- vative Q-Learning for Offline Reinforcement Learning. In Advances in Neural Information Processing Systems, Vol. 33. 1179–1191

work page 2020
[7]

Tze Leung Lai and Herbert Robbins. 1985. Asymptotically Efficient Adaptive Allocation Rules. Advances in Applied Mathematics 6, 1 (1985), 4–22

work page 1985
[8]

Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. 2020. Offline rein- forcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint arXiv:2005.01643 (2020)

arXiv 2020

Show all 19 references

[9]

Benjamin M Marlin and Richard S Zemel. 2009. Collaborative Prediction and Ranking with Non-Random Missing Data. In Proceedings of the 2009 ACM Con- ference on Recommender Systems . ACM, 5–12

2009
[10]

Volodymyr Mnih. 2016. Asynchronous Methods for Deep Reinforcement Learn- ing. arXiv preprint arXiv:1602.01783 (2016)

2016 arXiv
[11]

Tobias Schnabel, Adith Swaminathan, Ashudeep Singh, Navin Chandak, and Thorsten Joachims. 2016. Recommendations as Treatments: Debiasing Learning and Evaluation. In ICML, Vol. 48. PMLR, 1670–1679

2016
[12]

Adith Swaminathan and Thorsten Joachims. 2015. Counterfactual Risk Minimiza- tion: Learning from Logged Bandit Feedback. In ICML, Vol. 37. PMLR, 814–823

2015
[13]

Wenjie Wang, Yiyan Xu, Fuli Feng, Xinyu Lin, Xiangnan He, and Tat-Seng Chua
[14]

Ziyu Wang, Alexander Novikov, Konrad Zolna, Josh S Merel, Jost Tobias Springen- berg, Scott E Reed, Bobak Shahriari, Noah Siegel, Caglar Gulcehre, Nicolas Heess, et al. 2020. Critic Regularized Regression. In Advances in Neural Information Processing Systems, Vol. 33. 7768–7778

2020
[15]

Xin Xin, Alexandros Karatzoglou, Ioannis Arapakis, and Joemon M Jose. 2020. Self-Supervised Reinforcement Learning for Recommender Systems. In SIGIR. ACM, 931–940

2020
[16]

Tianhe Yu, Garrett Thomas, Lantao Yu, Stefano Ermon, James Y Zou, Sergey Levine, Chelsea Finn, and Tengyu Ma. 2020. MOPO: Model-based Offline Policy Optimization. In Advances in Neural Information Processing Systems , Vol. 33. 14129–14142

2020
[17]

Yuanqing Yu, Chongming Gao, Jiawei Chen, Heng Tang, Yuefeng Sun, Qian Chen, Weizhi Ma, and Min Zhang. 2024. EasyRL4Rec: An Easy-to-use Library for Reinforcement Learning Based Recommender Systems. In SIGIR. ACM, 977–987

2024
[18]

Yi Zhang, Ruihong Qiu, Jiajun Liu, and Sen Wang. 2024. ROLeR: Effective Reward Shaping in Offline Reinforcement Learning for Recommender Systems. In CIKM. ACM, 3269–3278

2024
[2023]

In SIGIR

Diffusion Recommender Model. In SIGIR. ACM, 832–841

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.

[1] [1]

Scott Fujimoto, David Meger, and Doina Precup. 2019. Off-Policy Deep Rein- forcement Learning without Exploration. In ICML, Vol. 97. PMLR, 2052–2062

work page 2019

[2] [2]

Chongming Gao, Kexin Huang, Jiawei Chen, Yuan Zhang, Biao Li, Peng Jiang, Shiqi Wang, Zhong Zhang, and Xiangnan He. 2023. Alleviating Matthew Effect of Offline Reinforcement Learning in Interactive Recommendation. In SIGIR. ACM, 238–248

work page 2023

[3] [3]

Chongming Gao, Shijun Li, Yuan Zhang, Jiawei Chen, Biao Li, Wenqiang Lei, Peng Jiang, and Xiangnan He. 2022. KuaiRand: An Unbiased Sequential Recom- mendation Dataset with Randomly Exposed Videos. In CIKM. ACM, 3953–3957

work page 2022

[4] [4]

Huifeng Guo, Ruiming Tang, Yunming Ye, Zhenguo Li, and Xiuqiang He. 2017. DeepFM: A Factorization-Machine based Neural Network for CTR Prediction. In IJCAI. ijcai.org, 1725–1731

2017

[5] [5]

Michael Janner, Justin Fu, Marvin Zhang, and Sergey Levine. 2019. When to Trust Your Model: Model-Based Policy Optimization. InAdvances in Neural Information Processing Systems, Vol. 32. 12498–12509

work page 2019

[6] [6]

Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. 2020. Conser- vative Q-Learning for Offline Reinforcement Learning. In Advances in Neural Information Processing Systems, Vol. 33. 1179–1191

work page 2020

[7] [7]

Tze Leung Lai and Herbert Robbins. 1985. Asymptotically Efficient Adaptive Allocation Rules. Advances in Applied Mathematics 6, 1 (1985), 4–22

work page 1985

[8] [8]

Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. 2020. Offline rein- forcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint arXiv:2005.01643 (2020)

arXiv 2020