Pith. sign in

REVIEW 3 major objections 5 minor 70 references

By turning item-level outcomes into residual token credits, HRPO improves session-level utility and online target cost.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

HRPO decomposes item-level rewards into token-level 'residual credits' along semantic identifier hierarchies and optimizes the generator with a PPO-style objective, improving session utility in KuaiSim and production A/B tests.

T0 review reviewed 2026-08-05 challenge →

load-bearing objection HRPO is a genuine, reproducible contribution to token-level post-training for SID decoders, and the online A/B test gives it real grounding; the cohort-level reward proxy is a legitimate weakness but not a fatal one. the 3 major comments →

arxiv 2608.00750 v1 pith:GGDGG2D3 submitted 2026-08-01 cs.IR

Hierarchical Residual Policy Optimization for Generative Recommendations

Kaifeng Guo , Yiming Yang , Jingtong Gao , Guolei Zeng , Fukang Yang , Yukang Liang , Peng Jiang , Qingpeng Cai
show 1 more author
Xiangyu Zhao
This is my paper
classification cs.IR
keywords generative recommendationsemantic identifierstoken-level credit assignmenthierarchical autoregressive modelingpreference alignmentrecommender system simulationpolicy optimization
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Generative recommenders choose items by decoding hierarchical semantic identifiers (SIDs) one token at a time, but their training usually imitates logged behavior instead of optimizing downstream utility. HRPO is a post-training procedure that turns the single terminal reward of an exposed item into a dense learning signal for every SID token. It smooths prefix-level utilities over feature-based user cohorts, takes differences along the SID tree as residual token credits, and accumulates them into credit-to-go returns, then optimizes with a clipped, KL-regularized objective. The paper reports that this improves session-level utility over prior post-training baselines in a public-data simulator and lifts a target cost metric in an online A/B test without shifting exposure volume.

Core claim

The paper's central claim is that the bottleneck in post-trained SID recommenders is token-level credit assignment: broadcasting a terminal item reward uniformly over the whole SID path gives weak, conflicting supervision on shared prefixes. HRPO instead estimates a smoothed expected utility for every SID prefix within feature-defined user cohorts, defines the credit of each token as the marginal utility of moving from its parent prefix to its own prefix, and accumulates these residual credits into a credit-to-go return. Because the residuals telescope, credit-to-go equals the terminal prefix utility minus the current prefix baseline, making token returns comparable across candidates at the

What carries the argument

The load-bearing object is the residual credit-to-go, G_t(c,y) = sum_{k=t}^L r_k = mu(c,y_{1:L}) - mu(c,y_{1:t-1}), where mu is a cohort-smoothed utility estimate for a SID prefix and r_t = mu(c,y_{1:t}) - mu(c,y_{1:t-1}) is the marginal credit of choosing token y_t. The residualization strips away the prefix baseline so early coarse decisions and later fine decisions receive signals on a common scale, and the telescoping sum lets the entire terminal outcome be distributed along the decoding path. RRPO consumes these returns through per-position normalization inside a small proposal set, PPO-style clipping, and a KL trust-region penalty toward the frozen reference policy, which together keep

Load-bearing premise

The load-bearing premise is that the smoothed cohort average for a SID prefix estimates what that prefix is worth for the particular user; if users in the same cohort diverge too much, token credits become biased toward cohort-level popularity and the policy may optimize the wrong utility.

What would settle it

Permute users at random into cohorts, retrain HRPO, and compare session-level total reward to the feature-based grouping result; if random cohorts retain the full gain instead of falling toward the no-cohorting baseline, the credit signal is not actually personalizing. A complementary check is to stratify cohorts by within-cohort variance in click behavior and test whether HRPO's gain over the best item-level baseline shrinks as variance grows.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • SID decoders can be post-trained from logged item-level outcomes alone, without needing logged serving probabilities or online exploration.
  • Early coarse tokens and later fine tokens get separate learning signals, so an error at one layer of the SID hierarchy is not broadcast to all layers.
  • Session-level return improves in the simulator while exposure coverage stays comparable to other offline baselines, indicating the gain is not from concentrating on a few popular items.
  • In production, the target cost metric improves across traffic segments while exposure and cost remain near the control policy, consistent with conservative updates in the serving distribution.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper: the same residual-credit recipe could apply to any tree-structured action space, such as hierarchical product taxonomies or multi-step decision pipelines, where feedback arrives only at leaves.
  • Beyond the paper: cohort smoothing is effectively an imputation step for non-exposed candidates; a natural extension is learned cohort embeddings or continuous user representations instead of deterministic buckets.
  • Beyond the paper: within-cohort preference variance is the unquantified risk; if users in a cohort diverge too much, token credits may drift toward popularity rather than personal utility.
  • Beyond the paper: the reported transfer evidence includes one auxiliary simulator plus a production test sharing the same logging distribution, so transfer to a different item taxonomy or behavior distribution remains untested.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes HRPO, a post-training framework for SID-based generative recommenders. Logged feedback is available only for the final exposed item, so HRPO first groups users into deterministic cohorts and computes smoothed prefix-level utility estimates mu_hat(c, y_{1:t}) via shrinkage (Eq. 8). It then defines residual token credits as successive differences of these prefix utilities (Eq. 9) and accumulates them into a credit-to-go signal (Eqs. 10-11). These signals are optimized by RRPO, a PPO-style objective with clipped token-level likelihood ratios, within-proposal position-wise normalization, and KL regularization (Eqs. 12-15). The paper reports offline experiments on KuaiRand/KuaiSim, an additional Agent4Rec-MovieLens study, and an online A/B test in a production advertising system, claiming consistent gains in session-level utility and business metrics.

Significance. If the claims hold, the paper makes a useful and timely contribution: it addresses a real limitation of SID-based generative recommendation, namely that terminal item-level rewards are weak and misaligned supervision for hierarchical token decisions. The residual decomposition is elegant and the telescoping identity in Eq. 11 correctly shows that token credits form a consistent decomposition of the terminal prefix score. The paper also provides public code and an archived artifact, and the online A/B test is a genuine external grounding that is held out from offline training. These are real strengths. However, the central quantitative claims rest on an approximation that is not verified: the cohort-smoothed prefix utility in Eq. 8 is used as if it were a proxy for user-specific utility, and the offline and online evaluations lack statistical reporting. The contribution is defensible in principle, but the evidence as presented is not yet at the level claimed in the abstract.

major comments (3)
  1. [§4.7, Table 6] The entire token-level credit signal is derived from mu_hat(c, y_{1:t}), a cohort-level shrinkage estimate of prefix utility. Eq. (11) shows that the credit-to-go equals the cohort mean terminal utility minus the cohort mean baseline for the prefix. This gives a valid policy-gradient objective for the cohort-average reward surrogate, but it is unbiased for the generating user's utility only if within-cohort utility is sufficiently homogeneous. Table 8 indicates that default cohorts are broad aggregates (median 62, mean 216 logged interactions per cohort), and the cohort mapping in Eq. (6) is based on a few stable features. No diagnostic is reported for the approximation error mu_hat(c,y) - E[R(u,y) | u in c], nor is there a personalization check. Consequently, the reported offline gains and even the online target-cost lifts could be driven by optimizing a cohort-average popularity proxy
  2. [§4.7, Table 6] The offline evaluation reports single runs without standard errors, confidence intervals, or significance tests. Tables 2-5 show large differences (e.g., HRPO Total reward 10.528 vs 8.137 for SPRec), but the reader cannot tell whether these differences are stable across seeds or within run-to-run noise. The hyperparameters in Sec. 4.2 and Fig. 3 were apparently tuned on the same evaluation setup, which further increases the risk of overfitting the reported numbers. Please report mean and standard deviation over at least 3-5 seeds, and, where appropriate, paired significance tests or effect-size statements. If the authors intentionally report single runs for cost reasons, that limitation should be stated explicitly and the strength of the comparative claims should be softened.
  3. [§4.7, Table 6] The online A/B result is the strongest external evidence, but Table 6 reports only relative percentage changes with no confidence intervals, p-values, or guardrail thresholds. The Target Cost lifts are 0.168%, 0.186%, and 3.490%, while Exposure and Cost also move non-trivially in some segments (e.g., Fiction IAA exposure +0.750%, cost +0.823%). Without uncertainty quantification and a statement of the experiment duration and traffic volume, the characterization of the result as 'validated online gains' (RQ4 and Sec. 4.7) is stronger than the table supports. Please provide confidence intervals and significance levels, and clarify whether Exposure and Cost movements are within pre-specified guardrails or are simply not statistically significant.
minor comments (5)
  1. [§2.3, Eq. (5)] Notation is inconsistent: Eq. (5) uses y_{<t} while most of the paper uses y_{1:t-1}. Use one convention throughout.
  2. [§3.4, Eq. (15)] The KL trust-region term KL_TR is defined as an expectation under pi_{theta_old}; please clarify how this expectation is estimated in the minibatch update and whether the same proposal set Y(x) is used for the KL estimate.
  3. [Fig. 3] The caption says bars indicate EpisodeLen and the line indicates Total reward, but the two quantities have different scales; the figure should show dual axes with clear labels so the sensitivity claims are readable.
  4. [§4.1] 'Puresubset' should be 'Pure subset'. Also, the description of how the simulator's click-based reward relates to the scalarized utility r(b) in Eq. (1) would benefit from one sentence making the mapping explicit.
  5. [Appendix C] The scaling-law diagnostic is interesting but disconnected from the HRPO conclusions; please state explicitly what this appendix is intended to show and why the reader should interpret it as supporting the main claims.

Circularity Check

0 steps flagged

No significant circularity: HRPO's token-credit derivation is an algebraic rewriting of logged-reward smoothing, and the central claims are validated on held-out simulator rollouts and an online A/B test independent of training.

full rationale

The paper's derivation chain is self-contained and does not reduce a claimed prediction to a fitted input. Eq. (8) estimates smoothed prefix utilities from logged outcomes; Eqs. (9)-(11) are telescoping identities that rewrite these estimates as residual token credits and credit-to-go signals; Eq. (12) normalizes them within a proposal set; Eq. (15) is a clipped policy-gradient objective with a KL penalty. No step fits a parameter and then reports the same quantity as a predicted result: the smoothing pseudo-count, group size, KL coefficient, and clip range are hyperparameters, and the evaluated outcomes (KuaiSim session return, Agent4Rec AvgR, online Target Cost) are not equal to the training signals. The online A/B test is held out from training, and the offline evaluations use public simulator benchmarks rather than quantities derived from the method's own fitted values. Self-citations to KuaiRand [13] and KuaiSim [55] are references to public data and simulator artifacts, not to a uniqueness theorem or an unverified premise that supplies the method's validity. The cohort-smoothed prefix utility in Eq. (8) raises a potential bias concern about within-cohort heterogeneity, but that is a correctness/robustness question, not circularity. The score of 2 reflects only the presence of minor, non-load-bearing self-citations in the evaluation stack; there is no construction-equivalent step or fitted-input-called-prediction in the derivation.

Axiom & Free-Parameter Ledger

6 free parameters · 4 axioms · 0 invented entities

The central method depends on cohort-level value estimation as the reward model and on a reference policy for proposals. The four listed axioms are the main unverified premises; several hyperparameters are tuned on the target simulator, which is a mild circularity concern.

free parameters (6)
  • shrinkage pseudo-count alpha = 100
    Controls variance-bias trade-off of prefix utility estimates; tuned in Section 4.5 (Figure 3d).
  • KL regularization coefficient beta_KL = 0.1
    Trust-region penalty in RRPO objective; tuned in Section 4.5 (Figure 3b).
  • PPO clip epsilon_clip = 0.2
    Clips likelihood ratios; tuned in Section 4.5 (Figure 3c).
  • proposal group size W = 18
    Number of candidates in proposal set; tuned in Section 4.5 (Figure 3a).
  • cohort mapping g = deterministic tuple bucketing, 120 cohorts
    Groups users for prefix utility pooling; sensitivity in Appendix A.2 (Tables 7-8).
  • reference policy sync period M_sync = 20 steps
    Frequency of updating frozen reference; listed in Section 4.2, not swept.
axioms (4)
  • domain assumption The linear scalarization r(b) = sum_m w_m b_m defines the true utility, with pre-specified weights w_m
    Eq. (1); if these weights are mis-specified, the optimized objective differs from business metrics.
  • ad hoc to paper Cohort-level smoothed prefix utility mu_hat(c, y_{1:t}) approximates the expected utility of a prefix for any user in cohort c
    Eq. (8), Section 3.2; load-bearing for credit assignment, not validated against per-user reward models.
  • domain assumption The reference policy pi_theta_old is an adequate proposal distribution for conservative optimization without importance weighting from the logging policy
    Alg. 1 step 11 and Eq. (13); the paper assumes no off-policy correction is needed.
  • domain assumption KuaiSim simulator rollouts are a faithful proxy for real user behavior for the offline gains
    Section 4.1 explicitly says simulator is a proxy; online A/B is the ground truth.

reviewed 2026-08-05 · how reviews work

0 comments
Cite this review

Pith. "Pith review of Hierarchical Residual Policy Optimization for Generative Recommendations." pith.science (2026). https://pith.science/paper/GGDGG2D3

@misc{pith2026260800750,
  author       = {Pith},
  title        = {Pith review of: Hierarchical Residual Policy Optimization for Generative Recommendations},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/GGDGG2D3}},
  note         = {Machine review of arXiv:2608.00750}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Generative recommenders select items by autoregressively decoding semantic identifiers (SIDs), whose token positions induce a coarse-to-fine hierarchy over the item space. In practice, SID decoders are trained via supervised next-token prediction, which imitates logged trajectories rather than directly optimizing downstream utility. This motivates post-training with outcome feedback to guide decoding toward higher utility. However, logged feedback is only observed for the final exposed item, causing most post-training methods to operate at the item level and broadcast the same terminal signal across all SID tokens. As a result, token-level credit assignment becomes sparse, high-variance, and layer-dependent. To this end, we propose Hierarchical Residual Policy Optimization (HRPO), a post-training framework that converts item-level outcomes into dense, token-aligned learning signals for conservative token-wise improvement. Specifically, HRPO first estimates SID prefix-level utilities via group-wise reward smoothing over feature-based user clusters. It then decomposes these utilities into residual token credits and accumulates them into credit-to-go signals. Finally, Residual-Return Policy Optimization (RRPO) optimizes the residual credits using clipped updates, group-normalized advantages, and KL regularization to preserve stability. Experiments on a public dataset and an online A/B test in a large-scale commercial system show consistent gains in session-level utility and key business metrics. Source code and the archived artifact are available for reproduction.

Figures

Figures reproduced from arXiv: 2608.00750 by Fukang Yang, Guolei Zeng, Jingtong Gao, Kaifeng Guo, Peng Jiang, Qingpeng Cai, Xiangyu Zhao, Yiming Yang, Yukang Liang.

Figure 1
Figure 1. Figure 1: Token-level credit assignment for hierarchical Se [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Overview of HRPO. where 𝑔(·) is a fixed mapping implemented in the system; Appen￾dix A.2 reports the concrete KuaiRand features, cohort support, and grouping sensitivity. For each record with cohort 𝑐 = 𝑐(𝑢) and path y, we update cohort-conditional prefix statistics along the trie path: 𝑛(𝑐, 𝑦1:𝑡 ) ← 𝑛(𝑐, 𝑦1:𝑡 )+1, 𝑠(𝑐, 𝑦1:𝑡 ) ← 𝑠(𝑐, 𝑦1:𝑡 )+𝑟, 𝑡 ∈ [1, 𝐿], (7) Empirical means 𝑠(𝑐, 𝑦1:𝑡 )/𝑛(𝑐, 𝑦1:𝑡 ) can be … view at source ↗
Figure 3
Figure 3. Figure 3: Hyperparameter sensitivity under offline training [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Production-backbone architecture used for A/B serving and the scaling-law diagnostic. [PITH_FULL_IMAGE:figures/full_fig_p008_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Online/simulator serving and offline RRPO training [PITH_FULL_IMAGE:figures/full_fig_p008_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Scaling-law diagnostic for A/B model sizes. [PITH_FULL_IMAGE:figures/full_fig_p012_6.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

70 extracted references · 22 canonical work pages · 1 internal anchor

  1. [1]

    Himan Abdollahpouri. 2019. Popularity Bias in Ranking and Recommendation. InProceedings of the AAAI/ACM Conference on AI, Ethics, and Society (AIES). 529–530. https://doi.org/10.1145/3306618.3314309

    arXiv 2019

  2. [2]

    Courville, and Yoshua Bengio

    Dzmitry Bahdanau, Philemon Brakel, Kelvin Xu, Anirudh Goyal, Ryan Lowe, Joelle Pineau, Aaron C. Courville, and Yoshua Bengio. 2017. An Actor-Critic Algorithm for Sequence Prediction. InInternational Conference on Learning Rep- resentations (ICLR). https://arxiv.org/abs/1607.07086

    Pith/arXiv arXiv 2017

  3. [3]

    Charles, D

    Léon Bottou, Jonas Peters, Joaquin Quiñonero-Candela, Denis X. Charles, D. Max Chickering, Elon Portugaly, Dipankar Ray, Patrice Simard, and Ed Snelson. 2013. Counterfactual Reasoning and Learning Systems: The Example of Computational Advertising.Journal of Machine Learning Research14, Nov (2013), 3207–3260. http://leon.bottou.org/papers/bottou-jmlr-2013

    work page 2013

  4. [4]

    Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Michael Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. 2021. Decision Transformer: Reinforcement Learning via Sequence Modeling. InAdvances in Neural Information Processing Systems (NeurIPS). arXiv:2106.01345

    Pith/arXiv arXiv 2021

  5. [5]

    Yuxin Chen, Junfei Tan, An Zhang, Zhengyi Yang, Leheng Sheng, Enzhi Zhang, Xiang Wang, and Tat-Seng Chua. 2024. On Softmax Direct Preference Optimiza- tion for Recommendation. InAdvances in Neural Information Processing Systems (NeurIPS). https://doi.org/10.48550/arXiv.2406.09215 arXiv:2406.09215 [cs.IR]

    Pith/arXiv arXiv doi:10.48550/arxiv.2406.09215 2024

  6. [6]

    Christiano, Jan Leike, Tom B

    Paul F. Christiano, Jan Leike, Tom B. Brown, Miljan Martic, Shane Legg, and Dario Amodei. 2017. Deep Reinforcement Learning from Human Preferences. In Advances in Neural Information Processing Systems (NeurIPS). https://arxiv.org/ abs/1706.03741

    Pith/arXiv arXiv 2017

  7. [7]

    Paul Covington, Jay Adams, and Emre Sargin. 2016. Deep Neural Networks for YouTube Recommendations. InProceedings of the 10th ACM Conference on Recommender Systems (RecSys). 191–198. https://doi.org/10.1145/2959100.2959190

    arXiv 2016

  8. [8]

    Jiaxin Deng, Shiyao Wang, Kuo Cai, Lejian Ren, Qigen Hu, Weifeng Ding, Qiang Luo, and Guorui Zhou. 2025. OneRec: Unifying Retrieve and Rank with Generative Recommender and Iterative Preference Alignment. https://doi.org/10.48550/ arXiv.2502.18965 arXiv:2502.18965 [cs.IR]

    Pith/arXiv arXiv doi:10.48550/arxiv.2502.18965 2025

  9. [9]

    Miroslav Dudík, John Langford, and Lihong Li. 2011. Doubly Robust Policy Evaluation and Learning. InProceedings of the 28th International Conference on Machine Learning (ICML). 1097–1104

    work page 2011

  10. [10]

    Bradley Efron and Carl Morris. 1975. Data Analysis Using Stein’s Estimator and Its Generalizations.J. Amer. Statist. Assoc.70, 350 (1975), 311–319. https: //doi.org/10.1080/01621459.1975.10479864

    arXiv 1975

  11. [11]

    Scott Fujimoto, Herke van Hoof, and David Meger. 2018. Addressing Function Approximation Error in Actor-Critic Methods. InProceedings of the 35th Interna- tional Conference on Machine Learning (ICML). https://doi.org/10.48550/arXiv. 1802.09477 arXiv:1802.09477 [cs.LG]

    Pith/arXiv arXiv doi:10.48550/arxiv.1802.09477 2018

  12. [12]

    Chongming Gao, Ruijun Chen, Shuai Yuan, Kexin Huang, Yuanqing Yu, and Xiangnan He. 2025. SPRec: Self-Play to Debias LLM-based Recommendation. In Proceedings of the ACM Web Conference 2025. 5075–5084. https://doi.org/10.1145/ 3696410.3714524 arXiv:2412.09243

    work page arXiv 2025

  13. [13]

    Chongming Gao, Shijun Li, Yuan Zhang, Jiawei Chen, Biao Li, Wenqiang Lei, Peng Jiang, and Xiangnan He. 2022. KuaiRand: An Unbiased Sequential Recom- mendation Dataset with Randomly Exposed Videos. InProceedings of the 31st ACM International Conference on Information and Knowledge Management (CIKM). https://doi.org/10.1145/3511808.3557624 arXiv:2208.08696 [cs.IR]

    arXiv 2022

  14. [14]

    Jingtong Gao, Yewen Li, Shuai Mao, Peng Jiang, Nan Jiang, Yejing Wang, Qing- peng Cai, Fei Pan, Peng Jiang, Kun Gai, Bo An, and Xiangyu Zhao. 2025. Genera- tive Auto-Bidding with Value-Guided Explorations. https://doi.org/10.48550/ arXiv.2504.14587 arXiv:2504.14587 [cs.LG]

    work page doi:10.48550/arxiv.2504.14587 2025

  15. [15]

    Shijie Geng, Shuchang Liu, Zuohui Fu, Yingqiang Ge, and Yongfeng Zhang. 2022. Recommendation as Language Processing (RLP): A Unified Pretrain, Personalized Prompt & Predict Paradigm (P5). InRecSys. 299–315. https://doi.org/10.1145/ 3523227.3546767

    work page arXiv 2022

  16. [16]

    Balázs Hidasi, Alexandros Karatzoglou, Linas Baltrunas, and Domonkos Tikk

  17. [17]

    Chi, Julian McAuley, and Derek Zhiyuan Cheng

    Yupeng Hou, Jianmo Ni, Zhankui He, Noveen Sachdeva, Wang-Cheng Kang, Ed H. Chi, Julian McAuley, and Derek Zhiyuan Cheng. 2025. ActionPiece: Contextually Tokenizing Action Sequences for Generative Recommendation. InProceedings of the 42nd International Conference on Machine Learning (ICML). arXiv:2502.13581 [cs.IR]

    Pith/arXiv arXiv 2025

  18. [18]

    Eugene Ie, Chih-wei Hsu, Martin Mladenov, Vihan Jain, Sanmit Narvekar, Jing Wang, Rui Wu, and Craig Boutilier. 2019. RecSim: A Configurable Simulation Platform for Recommender Systems. https://doi.org/10.48550/arXiv.1909.04847 arXiv:1909.04847 [cs.IR]

    Pith/arXiv arXiv doi:10.48550/arxiv.1909.04847 2019

  19. [19]

    Eugene Ie, Vihan Jain, Jing Wang, Sanmit Narvekar, Ritesh Agarwal, Rui Wu, Heng-Tze Cheng, Tushar Chandra, and Craig Boutilier. 2019. SlateQ: A Tractable Decomposition for Reinforcement Learning with Recommendation Sets. InPro- ceedings of the Twenty-Eighth International Joint Conference on Artificial Intelli- gence, IJCAI-19. 2592–2599. https://doi.org/1...

    doi:10.24963/ijcai.2019/360 2019

  20. [20]

    Dietmar Jannach and Himan Abdollahpouri. 2023. A survey on multi-objective recommender systems.Frontiers in Big Data6 (2023). https://doi.org/10.3389/ fdata.2023.1157899

    work page arXiv 2023

  21. [21]

    Yangqin Jiang, Yuhao Yang, Lianghao Xia, Da Luo, Kangyi Lin, and Chao Huang

  22. [22]

    Wang-Cheng Kang and Julian McAuley. 2018. Self-Attentive Sequential Rec- ommendation. In2018 IEEE International Conference on Data Mining (ICDM). https://doi.org/10.1109/ICDM.2018.00035

    arXiv 2018

  23. [23]

    Ron Kohavi, Alex Deng, Brian Frasca, Roger Longbotham, Toby Walker, and Ya Xu. 2012. Trustworthy Online Controlled Experiments: Five Puzzling Outcomes Explained. InProceedings of the 18th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD). 786–794. https://doi.org/10.1145/ 2339530.2339653

    work page arXiv 2012

  24. [24]

    Xiaoyu Kong, Leheng Sheng, Junfei Tan, Yuxin Chen, Jiancan Wu, An Zhang, Xiang Wang, and Xiangnan He. 2025. MiniOneRec: An Open-Source Framework for Scaling Generative Recommendation. https://doi.org/10.48550/arXiv.2510. 24431 arXiv:2510.24431 [cs.IR]

    doi:10.48550/arxiv.2510 2025

  25. [25]

    Andrew Levy, George Konidaris, Robert Platt, and Kate Saenko. 2019. Learning Multi-Level Hierarchies with Hindsight. InInternational Conference on Learning Representations. arXiv:1712.00948

    Pith/arXiv arXiv 2019

  26. [26]

    Jing Li, Pengjie Ren, Zhumin Chen, Zhaochun Ren, Tian Lian, and Jun Ma. 2017. Neural Attentive Session-based Recommendation. InProceedings of the 26th ACM International Conference on Information and Knowledge Management (CIKM). https://doi.org/10.1145/3132847.3132926

    arXiv 2017

  27. [27]

    Lihong Li, Wei Chu, John Langford, and Xuanhui Wang. 2011. Unbiased Offline Evaluation of Contextual-bandit-based News Article Recommendation Algo- rithms. InProceedings of the 4th ACM International Conference on Web Search and Data Mining (WSDM). 297–306. https://doi.org/10.1145/1935826.1935878 arXiv:1003.5956

    work page arXiv 2011

  28. [28]

    Lillicrap, Jonathan J

    Timothy P. Lillicrap, Jonathan J. Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. 2015. Continuous Control with Deep Reinforcement Learning. https://doi.org/10.48550/arXiv.1509.02971 arXiv:1509.02971 [cs.LG]

    Pith/arXiv arXiv doi:10.48550/arxiv.1509.02971 2015

  29. [29]

    Shuchang Liu, Qingpeng Cai, Bowen Sun, Yuhao Wang, Ji Jiang, Dong Zheng, Peng Jiang, Kun Gai, Xiangyu Zhao, and Yongfeng Zhang. 2023. Exploration and Regularization of the Latent Action Space in Recommendation. InProceedings of the ACM Web Conference 2023. 833–844. https://doi.org/10.1145/3543507.3583244

    work page arXiv 2023

  30. [30]

    H. Brendan McMahan, Gary Holt, David Sculley, Michael Young, Dietmar Ebner, Julian Grady, Lan Nie, Todd Phillips, Eugene Davydov, Daniel Golovin, Sharat Chikkerur, Dan Liu, Martin Wattenberg, Arnar Mar Hrafnkelsson, Tom Boulos, and Jeremy Kubica. 2013. Ad click prediction: a view from the trenches. InKDD

    work page 2013

  31. [31]

    Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu

    Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Tim- othy P. Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. 2016. Asynchronous Methods for Deep Reinforcement Learning.arXiv preprint arXiv:1602.01783(2016). https://doi.org/10.48550/arXiv.1602.01783

    Pith/arXiv arXiv doi:10.48550/arxiv.1602.01783 2016

  32. [32]

    Ng, Daishi Harada, and Stuart Russell

    Andrew Y. Ng, Daishi Harada, and Stuart Russell. 1999. Policy Invariance un- der Reward Transformations: Theory and Application to Reward Shaping. In Proceedings of the 16th International Conference on Machine Learning (ICML). https://doi.org/10.5555/645528.657613

    arXiv 1999

  33. [33]

    Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schul- man, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. 2022. Training Lan- guage Models to Follow Instructions with Human...

    Pith/arXiv arXiv doi:10.48550/arxiv.2203.02155 2022

  34. [34]

    R. L. Plackett. 1975. The Analysis of Permutations.Journal of the Royal Statistical Society: Series C (Applied Statistics)24, 2 (1975), 193–202. https://doi.org/10.2307/ 2346567

    work page 1975

  35. [35]

    Manning, and Chelsea Finn

    Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, and Chelsea Finn. 2023. Direct Preference Optimization: Your Language Model is Secretly a Reward Model. https://doi.org/10.48550/arXiv.2305.18290 arXiv:2305.18290 [cs.LG]

    Pith/arXiv arXiv doi:10.48550/arxiv.2305.18290 2023

  36. [36]

    Tran, Jonah Samost, Maciej Kula, Ed H

    Shashank Rajput, Nikhil Mehta, Anima Singh, Raghunandan Keshavan, Trung Vu, Lukasz Heldt, Lichan Hong, Yi Tay, Vinh Q. Tran, Jonah Samost, Maciej Kula, Ed H. Chi, and Maheswaran Sathiamoorthy. 2023. Recommender Systems with Generative Retrieval. InAdvances in Neural Information Processing Systems (NeurIPS). https://doi.org/10.48550/arXiv.2305.05065 arXiv:...

    Pith/arXiv arXiv doi:10.48550/arxiv.2305.05065 2023

  37. [37]

    Marc’Aurelio Ranzato, Sumit Chopra, Michael Auli, and Wojciech Zaremba. 2016. Sequence Level Training with Recurrent Neural Networks. InInternational Con- ference on Learning Representations (ICLR). https://arxiv.org/abs/1511.06732

    Pith/arXiv arXiv 2016

  38. [38]

    John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz

  39. [39]

    John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel

  40. [40]

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov

  41. [41]

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. 2024. DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models. https: //doi.org/10.48550/arXiv.2402.03300 arXiv:2402.03300 [cs.CL]

    Pith/arXiv arXiv doi:10.48550/arxiv.2402.03300 2024

  42. [42]

    Harald Steck. 2011. Item Popularity and Recommendation Accuracy. InPro- ceedings of the 5th ACM Conference on Recommender Systems (RecSys). 125–132. https://doi.org/10.1145/2043932.2043957

    arXiv 2011

  43. [43]

    Fei Sun, Jun Liu, Jian Wu, Changhua Pei, Xiao Lin, Wenwu Ou, and Peng Jiang. 2019. BERT4Rec: Sequential Recommendation with Bidirectional En- coder Representations from Transformer. InProceedings of the 28th ACM Inter- national Conference on Information and Knowledge Management (CIKM). https: //doi.org/10.1145/3357384.3357895

    arXiv 2019

  44. [44]

    arXiv:1506.02438 Hierarchical Residual Policy Optimization for Generative Recommendations KDD ’26, August 09–13, 2026, Jeju Island, Republic of Korea

    High-Dimensional Continuous Control Using Generalized Advantage Estimation. arXiv:1506.02438 Hierarchical Residual Policy Optimization for Generative Recommendations KDD ’26, August 09–13, 2026, Jeju Island, Republic of Korea

    Pith/arXiv arXiv 2026

  45. [45]

    Adith Swaminathan and Thorsten Joachims. 2015. Counterfactual Risk Min- imization: Learning from Logged Bandit Feedback. InProceedings of the 32nd International Conference on Machine Learning (ICML). 814–823. arXiv:1502.02362

    Pith/arXiv arXiv 2015

  46. [46]

    Junfei Tan, Yuxin Chen, An Zhang, Junguang Jiang, Bin Liu, Ziru Xu, Han Zhu, Jian Xu, Bo Zheng, and Xiang Wang. 2025. Reinforced Preference Op- timization for Recommendation. https://doi.org/10.48550/arXiv.2510.12211 arXiv:2510.12211 [cs.IR]

    work page doi:10.48550/arxiv.2510.12211 2025

  47. [47]

    Tran, Mostafa Dehghani, Jianmo Ni, Dara Bahri, Harsh Mehta, Zhen Qin, Kai Hui, Zhe Zhao, Jai Gupta, Tal Schuster, William W

    Yi Tay, Vinh Q. Tran, Mostafa Dehghani, Jianmo Ni, Dara Bahri, Harsh Mehta, Zhen Qin, Kai Hui, Zhe Zhao, Jai Gupta, Tal Schuster, William W. Cohen, and Donald Metzler. 2022. Transformer Memory as a Differentiable Search Index. InAdvances in Neural Information Processing Systems. https://doi.org/10.5555/ 3600270.3601857 arXiv:2202.06991 [cs.IR]

    work page arXiv 2022

  48. [48]

    Hanbing Wang, Xiaorui Liu, Wenqi Fan, Xiangyu Zhao, Venkataramana Kini, Devendra Pratap Yadav, Fei Wang, Zhen Wen, and Hui Liu. 2025. Rethinking Large Language Model Architectures for Sequential Recommendations. InProceedings of the 14th International Joint Conference on Natural Language Processing and the 4th Conference of the Asia-Pacific Chapter of the...

    work page doi:10.18653/v1/2025.ijcnlp-long.180 2025

  49. [49]

    Maolin Wang, Xinjian Zhao, Wanyu Wang, Sheng Zhang, Jiansheng Li, Bowen Yu, Binhao Wang, Shucheng Zhou, Dawei Yin, Qing Li, Ruocheng Guo, and Xiangyu Zhao. 2026. Embedding in Recommender Systems: A Survey.ACM Transactions on Information Systems(2026). https://doi.org/10.1145/3812652

    work page doi:10.1145/3812652 2026

  50. [50]

    Sutton and Andrew G

    Richard S. Sutton and Andrew G. Barto. 2018.Reinforcement Learning: An Intro- duction(2 ed.). MIT Press. https://doi.org/10.5555/3312046

    work page doi:10.5555/3312046 2018

  51. [51]

    Yejing Wang, Shengyu Zhou, Jinyu Lu, Qidong Liu, Xinhang Li, Wenlin Zhang, Feng Li, Pengjie Wang, Jian Xu, Bo Zheng, and Xiangyu Zhao. 2025. GFlowGR: Fine-tuning Generative Recommendation Frameworks with Generative Flow Networks. https://doi.org/10.48550/arXiv.2506.16114 arXiv:2506.16114 [cs.IR]

    Pith/arXiv arXiv doi:10.48550/arxiv.2506.16114 2025

  52. [52]

    Yejing Wang, Shengyu Zhou, Jinyu Lu, Ziwei Liu, Langming Liu, Maolin Wang, Wenlin Zhang, Feng Li, Wenbo Su, Pengjie Wang, Jian Xu, and Xiangyu Zhao

    work page

  53. [53]

    Williams

    Ronald J. Williams. 1992. Simple Statistical Gradient-Following Algorithms for Connectionist Reinforcement Learning.Machine Learning8, 3-4 (1992), 229–256. https://doi.org/10.1007/BF00992696

    doi:10.1007/bf00992696 1992

  54. [54]

    An Zhang, Yuxin Chen, Leheng Sheng, Xiang Wang, and Tat-Seng Chua. 2024. On Generative Agents in Recommendation. InProceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval. 1807–1817. https://doi.org/10.1145/3626772.3657844

    arXiv 2024

  55. [55]

    Kesen Zhao, Shuchang Liu, Qingpeng Cai, Xiangyu Zhao, Ziru Liu, Dong Zheng, Peng Jiang, and Kun Gai. 2023. KuaiSim: A Comprehensive Simu- lator for Recommender Systems. https://doi.org/10.48550/arXiv.2309.12645 arXiv:2309.12645 [cs.IR]

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2309.12645 2023

  56. [56]

    Wenjie Wang, Honghui Bao, Xinyu Lin, Jizhi Zhang, Yongqi Li, Fuli Feng, See- Kiong Ng, and Tat-Seng Chua. 2024. Learnable Item Tokenization for Generative Recommendation. InProceedings of the 33rd ACM International Conference on Information and Knowledge Management (CIKM). 2400–2409. https://doi.org/10. 1145/3627673.3679569

    arXiv 2024

  57. [57]

    Xiangyu Zhao, Liang Zhang, Zhuoye Ding, Long Xia, Jiliang Tang, and Dawei Yin

  58. [58]

    Bowen Zheng, Enze Liu, Zhongfu Chen, Zhongrui Ma, Yue Wang, Wayne Xin Zhao, and Ji-Rong Wen. 2025. Pre-training Generative Recommender with Multi-Identifier Item Tokenization. https://doi.org/10.48550/arXiv.2504.04400 arXiv:2504.04400 [cs.IR]

    Pith/arXiv arXiv doi:10.48550/arxiv.2504.04400 2025

  59. [59]

    Qiyong Zhong, Jiajie Su, Yunshan Ma, Julian McAuley, and Yupeng Hou. 2025. Pctx: Tokenizing Personalized Context for Generative Recommendation. https: //doi.org/10.48550/arXiv.2510.21276 arXiv:2510.21276 [cs.IR]

    work page doi:10.48550/arxiv.2510.21276 2025

  60. [60]

    Han Zhu, Xiang Li, Pengye Zhang, Guozheng Li, Jie He, Han Li, and Kun Gai

  61. [61]

    Lixin Zou, Long Xia, Zhuoye Ding, Jiaxing Song, Weidong Liu, and Dawei Yin

    work page

  62. [63]

    Xiangyu Zhao, Long Xia, Liang Zhang, Zhuoye Ding, Dawei Yin, and Jiliang Tang. 2018. Deep Reinforcement Learning for Page-wise Recommendations. In Proceedings of the 12th ACM Conference on Recommender Systems (RecSys). 95–103. https://doi.org/10.1145/3240323.3240374

    arXiv 2018

  63. [69]

    InProceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining

    Learning Tree-based Deep Model for Recommender Systems. InProceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. 1079–1088. https://doi.org/10.1145/3219819.3219826

    arXiv

  64. [2013]

    https://doi.org/10.1145/2487575.2488200

    1222–1230. https://doi.org/10.1145/2487575.2488200

    work page arXiv

  65. [2015]

    arXiv:1502.05477

    Trust Region Policy Optimization. arXiv:1502.05477

    Pith/arXiv arXiv

  66. [2016]

    In International Conference on Learning Representations (ICLR)

    Session-based Recommendations with Recurrent Neural Networks. In International Conference on Learning Representations (ICLR). https://doi.org/10. 48550/arXiv.1511.06939 arXiv:1511.06939 [cs.IR]

    Pith/arXiv arXiv doi:10.48550/arxiv.1511.06939

  67. [2017]

    https://doi.org/10.48550/arXiv

    Proximal Policy Optimization Algorithms. https://doi.org/10.48550/arXiv. 1707.06347 arXiv:1707.06347 [cs.LG]

    Pith/arXiv arXiv doi:10.48550/arxiv.1707.06347

  68. [2019]

    Reinforcement Learning to Optimize Long-term User Engagement in Recommender Systems. InKDD. 2810–2818. https://doi.org/10.1145/3292500. 3330668 A HRPO Algorithms This appendix summarizes the offline post-training loop used in the paper. The procedure optimizes the RRPO objective (Eq.(15)) using cohort-conditioned smoothed prefix means (Eq. (8)) and residu...

    doi:10.1145/3292500 2026

  69. [2025]

    InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)

    RecLM: Recommendation Instruction Tuning. InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 15443–15459. https://doi.org/10.18653/v1/2025.acl-long.751

    doi:10.18653/v1/2025.acl-long.751 2025

  70. [2026]

    https://doi.org/10.48550/arXiv.2511.18793 arXiv:2511.18793 [cs.AI]

    NEZHA: A Zero-sacrifice and Hyperspeed Decoding Architecture for Generative Recommendations. https://doi.org/10.48550/arXiv.2511.18793 arXiv:2511.18793 [cs.AI]

    work page doi:10.48550/arxiv.2511.18793

This paper was first reviewed by deepseek-v4-flash on August 5, 2026.