Pith. sign in

REVIEW 3 major objections 6 minor 54 references

Supervised Learning-enhanced Multi-Group Actor Critic for Live Stream Allocation in Feed

T0 review · 3 major / 6 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read SL-MGAC stabilizes reinforcement learning for live-stream allocation in a large mixed feed, outperforming all baselines in offline evaluation and an SAC-based baseline in a production A/B test.

desk verdict A genuine industrial RL deployment whose headline mechanism is undermined by label leakage in Eq. 10, so the empirical gains don't yet support the central claim. read the letter →

arxiv 2412.10381 v6 pith:CZ7EBTVY submitted 2024-11-28 cs.IR cs.AI

Jingxin Liu , Xiang Gao , Yisha Li , Xin Li , Haiyang Lu , Ben Wang This is my paper · ORCID
classification cs.IRcs.AI
keywords ReinforcementLearningRecommendationSystemVarianceReductionLiveStreamAllocationActor-CriticConstrainedMarkovDecisionProcessOfflinePolicyEvaluationOnlineA/BTesting
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

At the final stage of a mixed short-video and live-stream feed, the system must decide for each user request whether to inject at most one live stream into the video feed. The paper argues that this decision cannot be made greedily: over-injecting live streams raises live-stream metrics in the short run but reduces total app usage and retention. To make reinforcement learning usable for this high-risk decision, the authors propose SL-MGAC, an actor-critic algorithm whose critic is augmented with multi-task supervised reward prediction, whose state is decomposed across user activity groups, and whose Q-values and rewards are normalized. The reported evidence is an offline policy evaluation where SL-MGAC reaches a cumulative reward of 458.49 versus 443.01 for the best baseline, and a five-day online A/B test against a SAC-based allocation baseline showing +7.431% live-stream watch time, +0.197% video watch time, +0.121% app usage duration, and +0.086% user retention, with a visibly more stable live-stream injection ratio.

What carries the argument

The load-bearing identity is the additive critic decomposition $Q_{\phi_i}(s_t,a_t)=R_{\theta_i}(s_t,a_t)+\gamma T_{\xi_i}(s_t,a_t)$ (Eq. 8), in which the Reward Prediction Network $R$ is trained both by a supervised loss on the reconstructed watch time and by the TD target of the critic. The supervised branch uses posterior one-hot bin vectors $o(t)_l, o(t)_v$ and linear reconstruction of the watch time from a predicted within-bin proportion $\delta$ (Eq. 10), which the paper argues reduces variance relative to predicting raw watch time. Independent Multi-Group State Decomposition modules in actor and critic partition users into $K$ activity-level groups, each with its own MLP, so that sparse live-stream feedback is shared within groups rather than learned per sample. Layer normalization on inputs and softmax normalization of Q-values in the actor loss complete the stabilization machinery.

What would settle it

Retrain SL-MGAC on the same offline dataset with the posterior one-hot bin vectors in Eq. 10 replaced by a feature-only estimate (or dropped) while keeping multi-group decomposition, distribution discretization, and normalization fixed, and compare cumulative reward and Q-value variance: if performance stays near 458.49 the mechanism survives, and if it drops toward 449.13 the label conditioning is the load-bearing part.

Watch

Extended reading notes

Core claim

On the paper's own terms, SL-MGAC establishes that the instability that blocks RL deployment in live-stream allocation can be traced to high-variance reward and Q-value learning, and that this variance can be contained by three coordinated changes. First, the critic is split into a Reward Prediction Network and a Q Residual Network; the reward branch is trained by supervised multi-task learning to predict live-stream and video watch-time proportions inside discretized time bins, which restricts bootstrapping error accumulation. Second, a multi-group state decomposition partitions users into K groups by historical live-stream activity and runs separate group-wise MLPs for actor and critic, reducing prediction variance. Third, Layer Normalization, reward normalization, and a softmax-normalized Q-value in the actor loss keep gradients stable. With these changes, the paper reports both higher cumulative reward than all baselines in offline evaluation and positive gains on every online engagement metric while keeping the live-stream injection ratio steadier.

Load-bearing premise

The reward-prediction branch of the critic is fed the watch-time bin that each training sample actually fell into, which is the very quantity it is meant to predict; if this label conditioning distorts the critic rather than merely reducing variance, the reported stability and gains are not attributable to the proposed mechanism.

Editorial extensions

If this is right

  • If the reported offline and online results hold, RL can be deployed at the final allocation stage of a large feed, where a policy collapse would directly harm ad and e-commerce exposure.
  • The policy network alone is small enough for sub-20ms inference on a high-request-rate system, because the complex critic is used only as a training-time teacher.
  • The platform-level constraint becomes a tunable knob: the penalty coefficient $\lambda$ in the reward $\tilde r = y_l - \frac{\lambda}{B} y_v$ makes the allocator more conservative as $\lambda$ grows and more greedy as it turns negative.
  • The multi-group state decomposition is transferable to other binary injection scenarios such as advertising and e-commerce allocations, as the authors state.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A testable next step would ablate only the posterior one-hot inputs in Eq. 10 while keeping multi-group decomposition and distribution discretization fixed; the current ablations remove several components at once, so the individual contribution of the label conditioning is not yet isolated.
  • If this mechanism is the true source of stability, the same split of a critic into a supervised reward predictor plus a residual Q-value could apply to any recommender-side RL task with noisy, heavy-tailed feedback, not just live-stream injection.
  • The ablation data suggest the softmax-normalized actor loss (Q-norm) is the single largest contributor, since removing it drops offline reward from 458.49 to 392.12; exploring a temperature-scaled variant or a learned baseline might push performance further.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper proposes SL-MGAC, an actor-critic algorithm for deciding whether to insert a live stream into a short-video feed at Kuaishou. The algorithm combines a multi-group user decomposition, layer normalization, a critic decomposed into a reward prediction network and a Q residual network, distribution-discretized reward learning, and a softmax-normalized actor loss. Offline NCIS evaluation and a five-day online A/B test report improvements over SAC and other baselines in live-stream watch time while maintaining app usage duration and retention. The central claimed contribution is that multi-task supervised reward learning plus multi-group decomposition reduces variance and improves stability in this final-stage live-stream allocation problem.

Significance. If the claims held, this would be a practically valuable industrial RL deployment with released code and a clear risk story. The variance-reduction components (multi-group decomposition, normalization, Q-normalization) are plausible and worth studying. However, the main validation is undermined by a label-leakage flaw in the reward prediction network and by tuning of the constraint weight on the evaluation metrics, so the experiments do not currently support the central attribution.

major comments (3)
  1. [Section 3.3.2, Eq. (10) and Eq. (7)] The current-term reward prediction network is conditioned on the realized reward. In Eq. (10), R_theta_i(s_t,a_t) is computed using o_l(t) and o_v(t), the one-hot vectors of the actual live-stream and short-video watch-time bins of the logged sample, so these inputs are derived from r_t, the reward the RPN is supposed to predict. Consequently Q_phi_i(s_t,a_t) in Eq. (8) is a posterior quantity conditioned on the observed outcome of a_t, and this leaked Q is used both in the actor loss (Eq. (13)) and in the critic regression (Eq. (7)). The same inconsistency appears in the target term: Eq. (7) takes a max over a_{t+1}, but o(t+1) encodes the realized reward of the logged next action, so for the non-logged next action the bin is unobserved and the target cannot be evaluated as written unless the max is silently replaced by evaluation at the logged action. The paper's remark after Eq. (10) only justifies the availability of o(t+1); it does not address the current-term leakage or the max-over-action problem. A leakage-free ablation, in which o_l/o_v are removed from the current RPN input, is needed to support the central claim that supervised reward learning stabilizes critic learning; without it, the offline and online gains in Tables 1 and 3 cannot be attributed to the proposed mechanism.
  2. [Section 4.3 and Appendix C, Table 5] The constraint weight lambda is selected on the same online metrics used to claim constraint satisfaction. Table 5 reports live-stream DAU, watch times, app usage duration, and user retention for lambda in {0.2, 0.1, -0.1, -0.2}, and lambda = 0.1 is then presented as the setting in which SL-MGAC improves app usage duration and retention. Because this is selection on the evaluation metric, the constraint-satisfying behavior reported in Table 3 is at least partly a tuning artifact and is not evidence that the algorithm is inherently better at satisfying platform-level constraints. Please pre-specify a validation period or report the selection rule, provide the same lambda sweep for the baselines, and treat the final lambda as a selected hyperparameter with associated uncertainty.
  3. [Section 4.4 and Section 4.7, Tables 1-3] All offline and online comparisons are point estimates without confidence intervals, standard errors, number of independent runs, or significance tests. The offline advantage over the best baseline is 458.49 versus 443.01 for RLUR, and the online gains are +0.197% video watch time, +0.121% app usage duration, and +0.086% user retention; these magnitudes are small enough that they could easily be within day-to-day or user-segment noise. The stability comparisons in Fig. 4 and Fig. 8 also lack a statistical test and rest on visual inspection of shaded areas and amplitude densities. In addition, the NCIS offline evaluation does not state the logging policy, the propensity model, or the clipping constant, which are essential for interpreting the cumulative-reward numbers. Without this statistical and methodological support, the abstract's claims of outperforming baselines and exhibiting enhanced stability are not established.
minor comments (6)
  1. [Section 3.3.2 and Table 4] The relationship between the N_l+1 and N_v+1 one-hot vectors in Eq. (10) and the eight-dimensional output of the RPN should be spelled out; the extra bin for the a_t = 0 case is mentioned in the text but is not reflected in the bin-boundary table.
  2. [Eq. (7)] The notation Q_label(s_{t+1}) is misleading because the target depends on (s_t, a_t) through r(s_t, a_t); please rename it to Q_label(s_t, a_t) or simply 'target'.
  3. [Figure 4 caption] The phrase 'over 10 rounds of training' should state whether the curves are averages over 10 random seeds, 10 dataset shuffles, or 10 independent training runs.
  4. [Table 3 caption] The caption should state explicitly that all values are relative improvements over the SAC baseline; otherwise the rows for L2R, Dummy, and SL-MGAC-0 are hard to interpret.
  5. [Appendix C] The section titled 'More Ablation Studies' is a hyperparameter sensitivity analysis for lambda, not an ablation of the model components; the title should be adjusted to reflect its content.
  6. [Eq. (5)] The reward y_l - (lambda/B)y_v is not exactly the penalized reward of Eq. (4), which contains the factor (1+lambda)y_l; the text should explain that lambda has been rescaled by 1/(1+lambda), otherwise the connection to the Lagrange multiplier in Eq. (3) is confusing.

Circularity Check

1 steps flagged · score 6.0 of 10

Eq. 10 feeds the realized reward-bin one-hot into the Reward Prediction Network, so the critic Q used in the actor loss is partly conditioned on the true reward; the claimed variance reduction is in part an oracle effect.

  1. fitted input called prediction [Section 3.3.2, Eq. (10); used in Eq. (8) critic decomposition, Eq. (11) supervised loss, and Eq. (13) actor loss.]
    "Let 𝑜(𝑡)𝑙∈ R𝑁𝑙+1,𝑜(𝑡)𝑣∈ R𝑁𝑣+1 be one-hot vectors representing the time bins in which the real live stream (short video) watch time of sample(𝑠𝑡,𝑎𝑡,𝑟𝑡) falls. Note that a separate bin is set for the case𝑎𝑡 = 0. Then, the RPN𝑅𝜃𝑖 can be modeled by multi-task neural networks: 𝑅𝜃𝑖(𝑠𝑡,𝑎𝑡) =𝑠𝑖𝑔𝑚𝑜𝑖𝑑(𝐹Γ𝑖(𝑠𝑡,𝑎𝑡)− 𝐺Θ𝑖(𝑠𝑡,𝑎𝑡)). Note that we introduce the posterior one-hot vectors𝑜(𝑡)𝑙 and𝑜(𝑡)𝑣 in Eq. 10 will not affect the calculation of𝑄′ 𝜙𝑖(𝑠𝑡+1,·), because we have already reserved𝑟𝑡+1 in the datasetD."

    o(t)_l and o(t)_v are computed from the realized watch times y_l and y_v of the same sample, and those watch times define the reward: r(s_t,a_t)=y_l in Eq. 1, with the penalized reward in Eq. 5 equal to y_l - (lambda/B)y_v. Hence R_theta_i(s_t,a_t) in Eq. 10 is not a prediction from (s_t,a_t); it is a posterior reconstruction that already knows the reward bin. Since Q_phi_i(s_t,a_t)=R_theta_i(s_t,a_t)+gamma*T_xi_i(s_t,a_t) in Eq. 8, the Q value used in the actor loss (Eq. 13) and regressed in the critic loss (Eq. 7) is partly an oracle. The supervised loss L_SL in Eq.

full rationale

The central claimed mechanism is that supervised reward learning stabilizes critic learning and reduces Q-value variance (Section 3.3.2 and the contributions list). Eq. 10 makes the Reward Prediction Network a function of the realized watch-time bins o(t)_l and o(t)_v, which are exactly the information defining the reward r_t. Therefore the Q function used in the actor loss is not a counterfactual prediction from (s_t,a_t); it is partly a posterior value conditioned on the true reward. This is a fitted input called prediction rather than an independent derivation. The paper's own defense ('we have already reserved r_{t+1} in the dataset') addresses only the target-network term and does not address that R_theta_i(s_t,a_t) is computed with the current sample's reward bin. Consequently, the ablation gains attributed to MG-SD and multi-task SL in Section 4.5 are confounded: a critic with oracle bin features would be expected to show lower variance and higher Q accuracy even without the proposed decomposition. Separately, the choice lambda=0.1 appears tuned on the online A/B metrics in Appendix C Table 5, so the constraint-satisfaction numbers are not fully independent, although this is hyperparameter selection rather than by-construction circularity. No load-bearing self-citation chain was found: the Layer Normalization argument cites external work (Yue et al., [45]) and the RCPO/AWAC references are external. Overall, the derivation is not fully self-contained because the main reward-learning component has label access, so a partial circularity score of 6 is appropriate.

Assumptions & free parameters 5 free parameters · 7 assumptions · 0 invented entities

The free parameters are all engineering choices tuned on the reported objectives, and the axiomatic assumptions are standard RL/OPE modeling assumptions. The key additional burden is Eq. 10, where the reward prediction uses posterior true reward bins, effectively treating the label as an input.

free parameters (5)
  • lambda = 0.1
    Cost weight in the simplified reward of Eq. 5; tuned via online A/B experiments (Appendix C Table 5) and selected partly because it yields positive app usage duration and retention.
  • K = 6
    Number of user groups in the Multi-Group State Decomposition module; chosen from sensitivity analysis in Fig. 5.
  • gamma = 0.9
    Discount factor for all methods; standard choice but not derived, and it affects cumulative reward comparisons.
  • exploration epsilon = 0.2
    Maximal exploration probability in epsilon-greedy online exploration (Eq. 16); set by hand.
  • Watch-time bin boundaries = Live [0,6,15,30,60,100,600,1200]s; Video [0,3,10,25,50,100,600,1200]s
    Distribution discretization bins in Table 4; chosen by hand and directly determine the supervised reward learning targets.
assumptions (7)
  • domain assumption The request-level infinite-horizon CMDP with binary action and instantaneous constraint c(s,a)=y_v/B - y_l adequately models user engagement and retention.
    Section 2 defines the problem this way; if the environment is not Markov or the constraint is not the right proxy, the whole optimization target is misspecified.
  • standard math The simplification from Eq. 4 to Eq. 5 preserves the optimal policy because the dropped factor (1+lambda) is a positive constant scaling of the reward.
    Eq. 4 gives (1+lambda)y_l - (lambda/B)y_v; Eq. 5 uses y_l - (lambda/B)y_v. Positive affine scaling of the reward does not change the argmax policy, but this equivalence is not stated in the paper.
  • domain assumption The Normalized Capped Importance Sampling estimator provides an unbiased or low-bias evaluation of the target policies from the logged data.
    Section 4.4 cites NCIS but gives no clipping thresholds, propensity model details, or coverage analysis; the offline cumulative rewards in Table 1 rest on this assumption.
  • domain assumption The layer normalization NTK bound from [45] transfers to the proposed actor and critic networks.
    Section 3.3.1 applies a theoretical result about layer normalization to their architecture without verifying the conditions for their specific networks.
  • standard math The AWAC-style actor loss in Eq. 13 is equivalent to the KL-constrained RL problem in Eq. 14, as proved in [33].
    Section 3.3.3 relies on the cited AWAC theoretical result to justify the modified actor loss.
  • domain assumption Off-policy training on real-time logged data with epsilon-greedy exploration provides adequate coverage and a valid behavior policy for the actor-critic update.
    Sections 3.4 and 4.3 use streaming logged data; no off-policy correction or behavior policy estimation is described for the critic updates.
  • domain assumption Splitting users into K groups by 3-week cumulative live-stream watch time yields a stable and useful decomposition of user state.
    Section 3.2 introduces the user-group partition as a natural approach; its effectiveness is only shown empirically through ablation, with no theoretical grounding.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Supervised Learning-enhanced Multi-Group Actor Critic for Live Stream Allocation in Feed." pith.science (2026). https://pith.science/paper/CZ7EBTVY

@misc{pith2026241210381,
  author       = {Pith},
  title        = {Pith review of: Supervised Learning-enhanced Multi-Group Actor Critic for Live Stream Allocation in Feed},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/CZ7EBTVY}},
  note         = {Machine review of arXiv:2412.10381}
}
read the original abstract

In the context of a short video & live stream mixed recommendation scenario, the live stream recommendation system (RS) decides whether to allocate at most one live stream into the video feed for each user request. To maximize long-term user engagement, it is crucial to determine an optimal live stream policy for accurate live stream allocation. The inappropriate live stream allocation policy can significantly affect the duration of the usage app and user retention, which ignores the long-term negative impact of live stream allocation. Recently, reinforcement learning (RL) has been widely applied in recommendation systems to capture long-term user engagement. However, traditional RL algorithms often face divergence and instability problems, which restricts the application and deployment in the large-scale industrial recommendation systems, especially in the aforementioned challenging scenario. To address these challenges, we propose a novel Supervised Learning-enhanced Multi-Group Actor Critic algorithm (SL-MGAC). Specifically, we introduce a supervised learning-enhanced actor-critic framework that incorporates variance reduction techniques, where multi-task reward learning helps restrict bootstrapping error accumulation during critic learning. Additionally, we design a multi-group state decomposition module for both actor and critic networks to reduce prediction variance and improve model stability. We also propose a novel reward function to prevent overly greedy live stream allocation. Empirically, we evaluate the SL-MGAC algorithm using offline policy evaluation (OPE) and online A/B testing. Experimental results demonstrate that the proposed method not only outperforms baseline methods under the platform-level constraints but also exhibits enhanced stability in online recommendation scenarios.

Figures

Figures reproduced from arXiv: 2412.10381 by the authors.

Figure 1
Figure 1. Structure of a short video & live stream mixed rec [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Overall framework of the SL-MGAC algorithm. The SL (RL) MG-SD Module is short for the Multi-Group State [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. System Architecture of the SL-MGAC algorithm. [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: The Q value curves between SL-MGAC and SL [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: Performance of different numbers of user group [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 7
Figure 7. Figure 7: Data distributions of online live stream room (or viewer) count, live stream watch time and short video watch time. [PITH_FULL_IMAGE:figures/full_fig_p011_7.png]
Figure 8
Figure 8. Figure 8: The amplitude distribution of live stream injection ratio. [PITH_FULL_IMAGE:figures/full_fig_p012_8.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

54 extracted references · 21 canonical work pages

  1. [1]

    M Mehdi Afsar, Trafford Crump, and Behrouz Far. 2022. Reinforcement learning based recommender systems: A survey. Comput. Surveys 55, 7 (2022), 1–38

    2022

  2. [2]

    Eitan Altman. 2021. Constrained Markov decision processes . Routledge

    work page 2021

  3. [3]

    Oron Anschel, Nir Baram, and Nahum Shimkin. 2017. Averaged-dqn: Variance reduction and stabilization for deep reinforcement learning. In International conference on machine learning . PMLR, 176–185

    work page 2017

  4. [4]

    Jimmy Lei Ba. 2016. Layer normalization. arXiv preprint arXiv:1607.06450 (2016)

    arXiv 2016

  5. [5]

    Qingpeng Cai, Shuchang Liu, Xueliang Wang, Tianyou Zuo, Wentao Xie, Bin Yang, Dong Zheng, Peng Jiang, and Kun Gai. 2023. Reinforcing user retention in a billion scale short video recommender system. In Companion Proceedings of the ACM Web Conference 2023. 421–426

    work page 2023

  6. [6]

    Ming Chen and Xiuze Zhou. 2020. DeepRank: Learning to rank with neural networks for recommendation. Knowledge-Based Systems 209 (2020), 106478

    work page 2020

  7. [7]

    Petros Christodoulou. [n. d.]. Soft actor-critic for discrete action settings. arXiv

    work page

  8. [8]

    Vibhavari Dasagi, Jake Bruce, Thierry Peynot, and Jürgen Leitner. 2019. Ctrl- z: Recovering from instability in reinforcement learning. arXiv preprint arXiv:1910.03732 (2019)

    work page Pith review arXiv 2019

Show all 54 references
  1. [9]

    Romain Deffayet, Thibaut Thonet, Jean-Michel Renders, and Maarten De Rijke

  2. [10]

    Shibhansh Dohare, Qingfeng Lan, and A Rupam Mahmood. 2023. Overcoming policy collapse in deep reinforcement learning. In Sixteenth European Workshop on Reinforcement Learning

    2023

  3. [11]

    Vincent François-Lavet, Raphael Fonteneau, and Damien Ernst. 2015. How to discount deep reinforcement learning: Towards new dynamic strategies. arXiv preprint arXiv:1512.02011 (2015)

    2015 arXiv

  4. [12]

    Scott Fujimoto and Shixiang Shane Gu. 2021. A minimalist approach to offline reinforcement learning. Advances in neural information processing systems 34 (2021), 20132–20145

    2021

  5. [13]

    Scott Fujimoto, Herke Hoof, and David Meger. 2018. Addressing function ap- proximation error in actor-critic methods. In International conference on machine learning. PMLR, 1587–1596

    2018

  6. [14]

    Scott Fujimoto, David Meger, and Doina Precup. 2019. Off-policy deep rein- forcement learning without exploration. In International conference on machine learning. PMLR, 2052–2062

    2019

  7. [15]

    Evan Greensmith, Peter L Bartlett, and Jonathan Baxter. 2004. Variance Reduction Techniques for Gradient Estimates in Reinforcement Learning.Journal of Machine Learning Research 5, 9 (2004)

    2004

  8. [16]

    Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. 2018. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International conference on machine learning . PMLR, 1861– 1870

    2018

  9. [17]

    Trevor Hastie, Robert Tibshirani, Jerome H Friedman, and Jerome H Friedman

  10. [18]

    Geoffrey Hinton. 2015. Distilling the Knowledge in a Neural Network. arXiv preprint arXiv:1503.02531 (2015)

    2015 arXiv

  11. [19]

    Eugene Ie, Vihan Jain, Jing Wang, Sanmit Narvekar, Ritesh Agarwal, Rui Wu, Heng-Tze Cheng, Tushar Chandra, and Craig Boutilier. 2019. SlateQ: A tractable decomposition for reinforcement learning with recommendation sets. (2019)

    2019

  12. [20]

    Arthur Jacot, Franck Gabriel, and Clément Hongler. 2018. Neural tangent ker- nel: Convergence and generalization in neural networks. Advances in neural information processing systems 31 (2018)

    2018

  13. [21]

    Eric Jang, Shixiang Gu, and Ben Poole. 2016. Categorical reparameterization with gumbel-softmax. arXiv preprint arXiv:1611.01144 (2016)

    2016 arXiv

  14. [22]

    Diederik P Kingma. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014)

    2014 arXiv

  15. [23]

    Ilya Kostrikov, Ashvin Nair, and Sergey Levine. 2021. Offline reinforcement learning with implicit q-learning. arXiv preprint arXiv:2110.06169 (2021). KDD ’25, August 3–7, 2025, Toronto, ON, Canada Jingxin Liu et al

    2021 arXiv

  16. [24]

    Aviral Kumar, Justin Fu, Matthew Soh, George Tucker, and Sergey Levine. 2019. Stabilizing off-policy q-learning via bootstrapping error reduction. Advances in neural information processing systems 32 (2019)

    2019

  17. [25]

    Yuxi Li. 2017. Deep reinforcement learning: An overview. arXiv preprint arXiv:1701.07274 (2017)

    2017 arXiv

  18. [26]

    Guogang Liao, Ze Wang, Xiaoxu Wu, Xiaowen Shi, Chuheng Zhang, Yongkang Wang, Xingxing Wang, and Dong Wang. 2022. Cross dqn: Cross deep q network for ads allocation in feed. In Proceedings of the ACM Web Conference 2022 . 401– 409

    2022

  19. [27]

    TP Lillicrap. 2015. Continuous control with deep reinforcement learning. arXiv preprint arXiv:1509.02971 (2015)

    2015 arXiv

  20. [28]

    Shuchang Liu, Qingpeng Cai, Bowen Sun, Yuhao Wang, Ji Jiang, Dong Zheng, Peng Jiang, Kun Gai, Xiangyu Zhao, and Yongfeng Zhang. 2023. Exploration and regularization of the latent action space in recommendation. In Proceedings of the ACM Web Conference 2023. 833–844

    2023

  21. [29]

    Hongzi Mao, Shaileshh Bojja Venkatakrishnan, Malte Schwarzkopf, and Moham- mad Alizadeh. 2018. Variance reduction for reinforcement learning in input- driven environments. arXiv preprint arXiv:1807.02264 (2018)

    2018 arXiv

  22. [30]

    Dadong Miao, Yanan Wang, Guoyu Tang, Lin Liu, Sulong Xu, Bo Long, Yun Xiao, Lingfei Wu, and Yunjiang Jiang. 2021. Sequential Search with Off-Policy Reinforcement Learning. In Proceedings of the 30th ACM International Conference on Information & Knowledge Management . 4006–4015

    2021

  23. [31]

    Volodymyr Mnih. 2013. Playing atari with deep reinforcement learning. arXiv preprint arXiv:1312.5602 (2013)

    2013 arXiv

  24. [32]

    Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. 2015. Human-level control through deep reinforcement learning. nature 518, 7540 (2015), 529–533

    2015

  25. [33]

    Ashvin Nair, Abhishek Gupta, Murtaza Dalal, and Sergey Levine. 2020. Awac: Accelerating online reinforcement learning with offline datasets. arXiv preprint arXiv:2006.09359 (2020)

    2020 arXiv

  26. [34]

    Joshua Romoff, Peter Henderson, Alexandre Piché, Vincent Francois-Lavet, and Joelle Pineau. 2018. Reward estimation for variance reduction in deep reinforce- ment learning. arXiv preprint arXiv:1805.03359 (2018)

    2018 arXiv

  27. [35]

    Hao Sun, Lei Han, Rui Yang, Xiaoteng Ma, Jian Guo, and Bolei Zhou. 2022. Exploit reward shifting in value-based deep-rl: Optimistic curiosity-based exploration and conservative exploitation via linear reward shaping. Advances in neural information processing systems 35 (2022),...

    2022

  28. [36]

    Richard S Sutton, David McAllester, Satinder Singh, and Yishay Mansour. 1999. Policy gradient methods for reinforcement learning with function approximation. Advances in neural information processing systems 12 (1999)

    1999

  29. [37]

    Adith Swaminathan and Thorsten Joachims. 2015. The self-normalized estimator for counterfactual learning. advances in neural information processing systems 28 (2015)

    2015

  30. [38]

    Chen Tessler, Daniel J Mankowitz, and Shie Mannor. 2018. Reward constrained policy optimization. arXiv preprint arXiv:1805.11074 (2018)

    2018 arXiv

  31. [39]

    Masatoshi Uehara, Chengchun Shi, and Nathan Kallus. 2022. A review of off- policy evaluation in reinforcement learning. arXiv preprint arXiv:2212.06355 (2022)

    2022 arXiv

  32. [40]

    Hado Van Hasselt, Arthur Guez, and David Silver. 2016. Deep reinforcement learning with double q-learning. InProceedings of the AAAI conference on artificial intelligence, Vol. 30

    2016

  33. [41]

    A Vaswani. 2017. Attention is all you need. Advances in Neural Information Processing Systems (2017)

    2017

  34. [42]

    Christopher John Cornish Hellaby Watkins. 1989. Learning from delayed rewards. (1989)

    1989

  35. [43]

    Xin Xin, Alexandros Karatzoglou, Ioannis Arapakis, and Joemon M Jose. 2020. Self-supervised reinforcement learning for recommender systems. In Proceedings of the 43rd International ACM SIGIR conference on research and development in Information Retrieval. 931–940

    2020

  36. [44]

    Wanqi Xue, Qingpeng Cai, Zhenghai Xue, Shuo Sun, Shuchang Liu, Dong Zheng, Peng Jiang, Kun Gai, and Bo An. 2023. PrefRec: recommender systems with human preferences for reinforcing long-term user engagement. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discove...

    2023

  37. [45]

    Yang Yue, Rui Lu, Bingyi Kang, Shiji Song, and Gao Huang. 2024. Understanding, predicting and better resolving Q-value divergence in offline-RL. Advances in Neural Information Processing Systems 36 (2024)

    2024

  38. [46]

    Gengrui Zhang, Yao Wang, Xiaoshuang Chen, Hongyi Qian, Kaiqiao Zhan, and Ben Wang. 2024. UNEX-RL: Reinforcing Long-Term Rewards in Multi-Stage Recommender Systems with UNidirectional EXecution. In Proceedings of the AAAI Conference on Artificial Intelligence , Vol. 38. 9305–9313

    2024

  39. [47]

    Qihua Zhang, Junning Liu, Yuzhuo Dai, Yiyan Qi, Yifan Yuan, Kunlun Zheng, Fan Huang, and Xianfeng Tan. 2022. Multi-task fusion via reinforcement learning for long-term user satisfaction in recommender systems. In Proceedings of the 28th ACM SIGKDD conference on knowledge disco...

    2022

  40. [48]

    Yu Zhang and Qiang Yang. 2021. A survey on multi-task learning. IEEE transac- tions on knowledge and data engineering 34, 12 (2021), 5586–5609

    2021

  41. [49]

    Guorui Zhou, Xiaoqiang Zhu, Chenru Song, Ying Fan, Han Zhu, Xiao Ma, Yanghui Yan, Junqi Jin, Han Li, and Kun Gai. 2018. Deep interest network for click-through rate prediction. In Proceedings of the 24th ACM SIGKDD international conference on knowledge discovery & data mining ...

    2018

  42. [50]

    Lixin Zou, Long Xia, Zhuoye Ding, Jiaxing Song, Weidong Liu, and Dawei Yin

  43. [54]

    In Proceedings of the 25th ACM SIGKDD international conference on knowledge discovery & data mining

    Reinforcement learning to optimize long-term user engagement in recom- mender systems. In Proceedings of the 25th ACM SIGKDD international conference on knowledge discovery & data mining . 2810–2818. Supervised Learning-enhanced Multi-Group Actor Critic for Live Stream Allocat...

    2025

  44. [2009]

    The elements of statistical learning: data mining, inference, and prediction . Vol. 2. Springer

  45. [2019]

    arXiv preprint arXiv:1910.07207 ([n. d.])

    1910 arXiv

  46. [2023]

    In Pro- ceedings of the Sixteenth ACM International Conference on Web Search and Data Mining

    Generative slate recommendation with reinforcement learning. In Pro- ceedings of the Sixteenth ACM International Conference on Web Search and Data Mining. 580–588

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.