Pith. sign in

REVIEW 4 major objections 4 minor 37 references

A two-tower recommender retriever trained with off-policy REINFORCE on multi-step importance weights can raise estimated session-level reward over next-item baselines while keeping standard recall largely intact.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 08:55 UTC pith:AO7GWH7X

load-bearing objection Plausible method, weak OPE evidence: the multi-step off-policy REINFORCE extension is worth building on, but the headline claim of higher session reward is not established by the reported numbers. the 4 major comments →

arxiv 2607.02818 v2 pith:AO7GWH7X submitted 2026-07-02 cs.IR

Session-Level Optimization for Large-Scale Retrieval using REINFORCE with Multi-Step Off-Policy Correction

Artem Matveev , Sergei Makeev , Aleksei Krasilnikov , Vladimir Baikalov , Sergei Liamaev , Kirill Khrylchenko This is my paper
classification cs.IR
keywords off-policy reinforcement learningREINFORCEtwo-tower retrievalrecommender systemsimportance samplingoff-policy evaluationtest-time scalingsequential decision-making
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper tries to show that a two-tower candidate retriever can be trained to maximize a whole session of user satisfaction instead of just predicting the next click or like. The method is off-policy REINFORCE: the model learns from pre-collected sessions, reweighting each logged action by a product of target-to-behavior policy ratios so the gradient points toward higher discounted session reward. On a public five-billion-interaction dataset, the authors report that this session-level agent scores higher on off-policy estimates of cumulative reward than next-item and next-positive baselines, while recall@100 stays within about a point of the baselines. They also report that spending more inference-time compute to simulate future feedback and pick the best rollout raises predicted long-term returns without updating the policy.

Core claim

The central claim is that autoregressive off-policy REINFORCE with multi-step importance weights is a viable training objective for large-scale two-tower retrieval when the goal is long-term session reward. The authors formulate a music recommendation session as a sequential decision problem, define immediate reward as a like indicator plus one-tenth of the listened ratio, and train a transformer-based retriever on logged sessions. Instead of the one-step importance correction used in earlier off-policy recommenders, they reweight each step by up to ten capped products of current and future policy ratios, made possible by processing an entire session in one forward pass. Using a learned user

What carries the argument

The load-bearing object is the off-policy REINFORCE gradient with multi-step importance weights: each action's log-probability is weighted by the discounted future session return and by a capped product of up to ten ratios of target-policy probability over estimated behavior-policy probability. This lets the model optimize session-level return from pre-collected logs without online interaction. The second component is a user feedback model, a causal transformer that predicts like probability and listened-ratio for any candidate action; it powers both the doubly robust off-policy evaluation and the inference-time rollout selection procedure.

Load-bearing premise

The whole comparison rests on the learned user feedback model and the estimated behavior-policy probabilities being accurate enough that the off-policy estimates preserve the true ordering of policies; if the feedback model is no better than a constant predictor, the reported session-reward gains are not established.

What would settle it

Train a trivial feedback model that always predicts the base like rate and the mean listened ratio; if its reward MAE matches the reported 0.0340, then the DM and DR comparisons in the paper cannot distinguish REINFORCE from the baselines.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Session-level off-policy REINFORCE can improve estimated cumulative session reward over next-item and next-positive objectives without additional data, training time, or compute.
  • Multi-step importance correction with about ten factors and a cap of 0.5 outperforms both one-step correction and no correction in the reported experiments.
  • Recall on actions associated with low future session return drops relative to the next-item baseline, suggesting the RL objective steers away from low-return reproductions.
  • Test-time scaling with a feedback model raises model-based session-return estimates for all models, so inference compute can substitute for a policy update.
  • A learned user feedback model enables doubly robust off-policy evaluation for sequential recommendation, with DR estimates landing closer to the observed behavior return than step-IS alone.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the offline policy ordering survives contact with real users, the approach is a drop-in upgrade: initialize from a next-item model, add the session-level loss, and keep the same two-tower serving stack.
  • The figure showing Positive-CE matching the REINFORCE model once rollouts are added suggests part of the measured gain may come from the inference-time selection mechanism rather than the training objective; isolating the two would require a live experiment.
  • The same multi-step importance weighting should transfer to token-based generative retrieval, where an autoregressive structure already exists and behavior-policy probabilities can be estimated from the same logs.
  • The shared-backbone variant sketched in the discussion, if it holds up, would collapse the three-model stack into one model and turn test-time scaling into a standard reranking step; that remains an untested prediction.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper proposes an autoregressive two-tower retriever trained with off-policy REINFORCE for session-level recommendation, using a multi-step approximation of importance weights enabled by the autoregressive formulation. It also trains a user feedback model to support doubly robust off-policy evaluation and introduces a feedback-model-based test-time scaling procedure. Experiments on the Yambda-5B dataset report that the REINFORCE agent achieves higher off-policy estimates of cumulative session reward than next-item and next-positive baselines while remaining competitive on recall@100, and that increasing the number of inference-time rollouts increases predicted model-based returns. The methodological contribution is framed as an extension of prior one-step off-policy correction and as an adaptation of sequential OPE and test-time scaling to large-scale retrieval.

Significance. If the central claim holds, the paper would make a useful contribution: it shows that a session-level REINFORCE objective with multi-step importance weighting can be trained on a public large-scale dataset, that sequential doubly robust OPE can be applied to recommendation, and that model-based rollouts can serve as a form of test-time scaling. The use of a public dataset, temporal splitting, matched training budgets, and a clear separation of behavior/target/feedback models are strengths. However, the evidence for the central claim rests entirely on OPE estimates whose reliability is not established: the user feedback model is not benchmarked against trivial baselines, no uncertainty quantification is provided, the Step-IS estimates are on a different scale from the observed behavior return, and the RQ2 result is circular by construction. The estimator mathematics is standard, but the empirical validation is currently too fragile to support the abstract's claim.

major comments (4)
  1. [Section 4.2, Table 2] The user feedback model is the foundation of the DM and DR estimates in Table 3, yet its evaluation is incomplete. Reward MAE of 0.0340 is reported without comparison to a constant predictor. Given that 1.13% of actions are liked and the listened-ratio component is dense, a predictor that always outputs the marginal means could plausibly achieve a similar MAE. Like PR-AUC is 0.2432, above the random baseline of 0.0113, but remains low in absolute terms. Without a trivial baseline and per-component calibration metrics, the small DM differences in Table 3 (0.5395 vs. 0.5384 vs. 0.5279) cannot be interpreted as evidence of a policy ordering.
  2. [Section 4.2, Table 3 and Tables 4-6] No error bars, confidence intervals, or significance tests are reported for any OPE estimate. Step-IS values (1.31-1.47) are roughly 2.2 times the directly observed behavior return (0.5855), indicating substantial bias from the estimated behavior policy or from importance-weight clipping. Since the hyperparameters in Tables 4-6 are selected using the same OPE metrics, the reported gains may be overfit to estimator noise. The authors should provide bootstrap CIs, significance tests, and ideally validate the estimators by checking whether they recover the known behavior return when applied to the behavior policy.
  3. [Section 4.3, Figure 4] The RQ2 result is circular: rollouts are selected by maximizing the learned feedback model's predicted return and then evaluated by the same feedback model. The expected maximum of N sampled scores increases with N even if the feedback model is uninformative, so the monotone DM curve is nearly forced by construction. The text's claim that an ineffective model would produce an almost constant curve is not correct. A control experiment (e.g., a random reward model, or comparing against selecting the first rollout) and independent evaluation are needed before this can be presented as evidence of improved long-term return.
  4. [Section 3.1, Eq. (2)] The behavior policy probability in Eq. (2) requires normalization over the entire item catalog A, but the implementation description only mentions sampled-softmax training with logQ correction. Sampled softmax training does not directly provide normalized probabilities for importance weights. If the denominator is approximated or restricted to a subset, the importance weights in Step-IS and DR are biased beyond the usual clipping bias. The paper should state exactly how the full denominator is computed or approximated, and quantify the effect of this approximation on the reported OPE estimates.
minor comments (4)
  1. [Figure 4] The figure shows point estimates only. Adding bootstrap confidence intervals or multiple-seed variation would clarify whether the monotone trend is meaningful.
  2. [Notation] The symbol T is used both for trajectory length T_xi and for rollout length in Section 3.2, which is confusing. Consider using L or H for the rollout horizon.
  3. [Section 4.2, text after Table 2] The statement that Reward MAE 'remains low' is not informative without a baseline. Reporting the MAE of a constant predictor (mean reward) and the Reward MAE of the model would directly address this.
  4. [Tables 5-6] Several differences are very small (e.g., DM 0.5390 vs. 0.5395). Without variance estimates, it is hard to assess whether the claimed optimum at K=10 and cap=0.5 is supported by the data.

Circularity Check

1 steps flagged

RQ2 test-time scaling result is circular: rollouts are selected and evaluated by the same learned feedback model, so the monotone DM curve follows from max-of-N by construction; the paper's own Section 4.3 caveat concedes this.

specific steps
  1. fitted input called prediction [Section 4.3 (Test-Time Scaling), Figure 4; selection step in Section 3.2, step (3); abstract]
    "Since both roll-out selection and subsequent evaluation are performed using the same user feedback model, these results do not constitute independent evidence that the real policy quality improves. They only show that the proposed procedure can improve the model-based reward estimate. ... Increasing the number of parallel roll-outs yields higher DM estimates for all models. This shows that test-time scaling can provide an additional mechanism for improving model-based session-level utility without changing the parameters of the policy itself."

    The DM estimate used as the evaluation metric is computed with the same learned user feedback model (Section 3.1) that is used to score roll-outs and select the argmax trajectory (Section 3.2, step 3: 'each roll-out is then scored by its predicted discounted return, and the best roll-out, j★, is selected'). Selecting the maximum of N model-predicted returns and then evaluating the selected trajectory with the same model forces the expected DM to increase with N for any non-degenerate score distribution, including pure noise. The paper's defense that an uninformative feedback model would give an almost constant curve is therefore incorrect: the maximum of random scores grows with N. The abstract's 'higher model-based long-term returns' is thus a restatement of the selection rule, not an ind

full rationale

The core RQ1 result—off-policy REINFORCE training against observed rewards improves Step-IS/DM/DR estimates relative to supervised baselines—is not circular in the formal sense: the policy is trained on logged reward data, baselines share architecture and initialization, and the OPE estimators, while potentially biased, are not defined in terms of the trained policy. The main defect there is external validity (e.g., the user feedback model is not benchmarked against a constant predictor, and Step-IS values are ~2.2x the observed behavior return), which is a correctness risk, not a by-construction circularity. The serious circular step is RQ2: test-time scaling is evaluated with the same feedback model used to select roll-outs, so the monotone increase in DM with the number of roll-outs is mathematically forced by argmax selection. The paper explicitly acknowledges this non-independence in Section 4.3, but still presents the result as a contribution and does not carry the caveat into the abstract. No load-bearing self-citation chain is present; self-citations are background/scale references. Overall score 7 reflects one central claim that reduces by construction while the other central claim retains independent, albeit imperfectly validated, content.

Axiom & Free-Parameter Ledger

7 free parameters · 5 axioms · 0 invented entities

The ledger shows where the paper's own choices enter. gamma, the 0.1 reward weight, and the feedback-loss weighting define what is being optimized; K and the IW cap are tuned on the reported ablations; the behavior-policy softmax and the feedback model are domain assumptions whose accuracy is never validated against external benchmarks. No invented entities are introduced.

free parameters (7)
  • discount factor gamma = 0.9
    Chosen by hand (Section 3); controls how much future reward counts in both the RL objective and all OPE estimates.
  • reward weight for listened ratio = 0.1
    r_t = 1{like} + 0.1 * ratio (Section 3); ad hoc calibration that defines the quantity being optimized and evaluated.
  • importance-weight cap c = 0.5
    Selected from ablation (Table 6); heavy capping makes multi-step weights saturate at 0.5 and largely determines the effective correction.
  • number of IS factors K = 10
    Selected from ablation (Table 5); with c=0.5, K=10 mostly shapes which products reach the cap.
  • rollout length T (test-time scaling) = 8
    Hand-chosen (Section 3.4); all test-time-scaling figures use T=8.
  • top-k sampling k for rollouts = 4
    Hand-chosen (Section 3.4, Figure 4); controls candidate diversity in test-time scaling.
  • feedback loss weights = 0.2 like / 0.8 ratio
    Hand-chosen weighting (Section 3.1); affects the quality of the feedback model used in DM/DR evaluation.
axioms (5)
  • domain assumption MDP transition: user feedback depends only on current state and action
    Section 2.1 formulates recommendation as an MDP; if user feedback depends on longer history or hidden context beyond the encoded state, the rollout model and IS correction are misspecified.
  • domain assumption Behavior policy softmax probabilities are computable and accurate
    Eq. 2 defines the behavior policy as a full softmax over the catalog, but only sampled-softmax training is described; the exact normalization used for importance weights is not stated.
  • ad hoc to paper Truncated and capped importance weights yield sufficiently unbiased gradients
    Section 3.1 drops leading factors and caps weights at 0.5; the induced bias is not quantified or compared with full-trajectory IS.
  • domain assumption User feedback model generalizes from behavior-policy data to the support of the target policies
    The feedback model is trained on behavior-data state-action pairs (Section 3.1) but used for DM/DR rollouts of policies that may place probability on actions underrepresented in the training data.
  • standard math Standard policy-gradient / importance-sampling / DR theory
    REINFORCE gradient (Section 2.2) and IPS/DR bias formulas (Section 2.3) are taken as given from Williams 1992, Dudik et al. 2011, Jiang & Li 2016, and Farajtabar et al. 2018.

pith-pipeline@v1.3.0-alltime-deepseek · 15228 in / 21136 out tokens · 205012 ms · 2026-08-02T08:55:01.572864+00:00 · methodology

0 comments
read the original abstract

Two-tower models are a widely used paradigm for large-scale retrieval in recommendation. However, they are typically trained with myopic supervised objectives, such as next-item prediction, that do not directly optimize long-term user satisfaction. In this work, we formulate recommendation as a session-level sequential decision-making problem and train a two-tower retriever autoregressively with off-policy REINFORCE on pre-collected data. Unlike the one-step off-policy correction used in prior work, we propose a multi-step approximation of importance weights enabled by the autoregressive formulation. To support offline evaluation, we train a user feedback model that simulates user responses to generated recommendations. This lets us adapt doubly robust off-policy evaluation for sequential decision-making to recommendation, a setting that has received limited attention. We further introduce a feedback-model-based test-time scaling procedure that simulates future responses and selects the recommendation with the highest predicted long-term return. Experiments on the public large-scale Yambda-5B dataset show that our RL agent achieves higher off-policy estimates of cumulative session reward than next-item and next-positive prediction baselines, while remaining competitive on conventional retrieval metrics. Moreover, allocating more inference-time compute to simulating future responses yields higher model-based long-term returns without updating the policy.

Figures

Figures reproduced from arXiv: 2607.02818 by Aleksei Krasilnikov, Artem Matveev, Kirill Khrylchenko, Sergei Liamaev, Sergei Makeev, Vladimir Baikalov.

Figure 1
Figure 1. Figure 1: Autoregressive off-policy REINFORCE training. To [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: User feedback model. Given the user history and [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 4
Figure 4. Figure 4: DM estimate versus the number of inference-time [PITH_FULL_IMAGE:figures/full_fig_p008_4.png] view at source ↗

Review history (2 revisions) →

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

37 extracted references · 10 linked inside Pith

  1. [1]

    Anirudhan Badrinath, Prabhat Agarwal, Laksh Bhasin, Jaewon Yang, Jiajing Xu, and Charles Rosenberg. 2025. PinRec: Outcome-Conditioned, Multi-Token Generative Retrieval for Industry-Scale Recommendation Systems.arXiv preprint arXiv:2504.10507(2025). arXiv:2504.10507 [cs.IR]

    Pith/arXiv arXiv 2025

  2. [2]

    Claes M Cassel, Carl E Särndal, and Jan H Wretman. 1976. Some results on generalized difference estimation and generalized regression estimation for finite populations.Biometrika63, 3 (1976), 615–620

    1976

  3. [3]

    Minmin Chen, Alex Beutel, Paul Covington, Sagar Jain, Francois Belletti, and Ed H. Chi. 2018. Top-K Off-Policy Correction for a REINFORCE Recommender System.arXiv preprint arXiv:1812.02353(2018). arXiv:1812.02353 [cs.LG]

    Pith/arXiv arXiv 2018

  4. [4]

    Minmin Chen, Can Xu, Vince Gatto, Devanshu Jain, Aviral Kumar, and Ed H. Chi. 2022. Off-Policy Actor-Critic for Recommender Systems. InProceedings of the 16th ACM Conference on Recommender Systems. doi:10.1145/3523227.3546758

    arXiv 2022

  5. [5]

    Maurizio Ferrari Dacrema, Simone Boglio, Paolo Cremonesi, and Dietmar Jannach

  6. [6]

    Miroslav Dudik, John Langford, and Lihong Li. 2011. Doubly Robust Policy Evaluation and Learning. InProceedings of the 28th International Conference on Machine Learning

    2011

  7. [7]

    Mehrdad Farajtabar, Yinlam Chow, and Mohammad Ghavamzadeh. 2018. More robust doubly robust off-policy evaluation. InInternational Conference on Machine Learning. PMLR, 1447–1456

    2018

  8. [8]

    Chengcheng Guo, Kuo Cai, Yu Zhou, Qiang Luo, Ruiming Tang, Han Li, Kun Gai, and Guorui Zhou. 2026. PROMISE: Process Reward Models Unlock Test-Time Scaling Laws in Generative Recommendations.arXiv preprint arXiv:2601.04674 (2026). arXiv:2601.04674 [cs.IR]

    arXiv 2026

  9. [9]

    Xuegang Hao, Ming Zhang, Alex Li, Xiangyu Qian, Zhi Ma, Yanlong Zang, Shi- jie Yang, Zhongxuan Han, Xiaolong Ma, Jinguang Liu, Zhen Li, Zhida Jiang, Shusheng Wang, Ning Tang, Yanchen Qiao, Chenxiang Yang, Chen Sun, Jincheng Yuan, Chunhua Peng, Heng Hu, Peijun Yang, Baopeng Yuan, Caiyun Qiu, Zhao- long Xing, Haofei Yuan, Haipeng Zhang, Yuzhang Guo, Weijie ...

    arXiv 2025

  10. [10]

    Yitong Ji, Aixin Sun, Jie Zhang, and Chenliang Li. 2023. A critical study on data leakage in recommender system offline evaluation.ACM Transactions on Information Systems41, 3 (2023), 1–27

    2023

  11. [11]

    Nan Jiang and Lihong Li. 2016. Doubly Robust Off-policy Value Evaluation for Reinforcement Learning. InProceedings of the 33rd International Conference on Machine Learning. 652–661

    2016

  12. [12]

    Kirill Khrylchenko and Alexander Fritzler. 2023. Personalized Transformer-based Ranking for e-Commerce at Yandex.arXiv preprint arXiv:2310.03481(2023)

    Pith/arXiv arXiv 2023

  13. [13]

    Kirill Khrylchenko, Artem Matveev, Sergei Makeev, and Vladimir Baikalov. 2025. Scaling Recommender Transformers to One Billion Parameters.arXiv preprint arXiv:2507.15994(2025). arXiv:2507.15994 [cs.IR]

    arXiv 2025

  14. [14]

    B Ravi Kiran, Ibrahim Sobh, Victor Talpaert, Patrick Mannion, Ahmad A Al Sallab, Senthil Yogamani, and Patrick Pérez. 2021. Deep reinforcement learning for autonomous driving: A survey.IEEE transactions on intelligent transportation systems23, 6 (2021), 4909–4926

    2021

  15. [15]

    Jens Kober, J Andrew Bagnell, and Jan Peters. 2013. Reinforcement learning in robotics: A survey.The International Journal of Robotics Research32, 11 (2013), 1238–1274

    2013

  16. [16]

    Sergey Levine and Vladlen Koltun. 2013. Guided policy search. InInternational conference on machine learning. PMLR, 1–9

    2013

  17. [17]

    Zhanyu Liu, Shiyao Wang, Xingmei Wang, Rongzhou Zhang, Jiaxin Deng, Honghui Bao, Jinghao Zhang, Wuchao Li, Pengfei Zheng, Xiangyu Wu, Yifei Hu, Qigen Hu, Xinchen Luo, Lejian Ren, Zixing Zhang, Qianqian Wang, Kuo Cai, Yunfan Wu, Hongtao Cheng, Zexuan Cheng, Lu Ren, Huanjie Wang, Yi Su, Ruiming Tang, Kun Gai, and Guorui Zhou. 2025. OneRec-Think: In-Text Rea...

    arXiv 2025

  18. [18]

    Yu A Malkov and Dmitry A Yashunin. 2018. Efficient and robust approximate nearest neighbor search using hierarchical navigable small world graphs.IEEE transactions on pattern analysis and machine intelligence42, 4 (2018), 824–836

    2018

  19. [19]

    Zaiqiao Meng, Richard McCreadie, Craig Macdonald, and Iadh Ounis. 2020. Ex- ploring data splitting strategies for the evaluation of recommendation models. In Proceedings of the 14th acm conference on recommender systems. 681–686

    2020

  20. [20]

    Nikil Pancha, Andrew Zhai, Jure Leskovec, and Charles Rosenberg. 2022. Pinner- Former: Sequence Modeling for User Representation at Pinterest. InProceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining. ACM, 3702–3712. doi:10.1145/3534678.3539156

    arXiv 2022

  21. [21]

    Alexander Ploshkin, Vladislav Tytskiy, Alexey Pismenny, Vladimir Baikalov, Evgeny Taychinov, Artem Permiakov, Daniil Burlakov, Eugene Krofto, and Niko- lay Savushkin. 2025. Yambda-5B: A Large-Scale Multi-modal Dataset for Ranking and Retrieval.arXiv preprint arXiv:2505.22238(2025). arXiv:2505.22238 [cs.IR]

    Pith/arXiv arXiv 2025

  22. [22]

    Keshavan, Trung Vu, Lukasz Heldt, Lichan Hong, Yi Tay, Vinh Q

    Shashank Rajput, Nikhil Mehta, Anima Singh, Raghunandan H. Keshavan, Trung Vu, Lukasz Heldt, Lichan Hong, Yi Tay, Vinh Q. Tran, Jonah Samost, Ma- ciej Kula, Ed H. Chi, and Maheswaran Sathiamoorthy. 2023. Recommender Systems with Generative Retrieval.arXiv preprint arXiv:2305.05065(2023). arXiv:2305.05065 [cs.IR]

    Pith/arXiv arXiv 2023

  23. [23]

    David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershel- vam, Marc Lanctot, et al . 2016. Mastering the game of Go with deep neural networks and tree search.nature529, 7587 (2016), 484–489

    2016

  24. [24]

    Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. 2024. Scaling LLM Test- Time Compute Optimally can be More Effective than Scaling Model Parameters. arXiv preprint arXiv:2408.03314(2024). arXiv:2408.03314 [cs.LG]

    Pith/arXiv arXiv 2024

  25. [25]

    Dan Tito Svenstrup, Jonas Hansen, and Ole Winther. 2017. Hash Embeddings for Efficient Word Representations. InAdvances in Neural Information Processing Systems, Vol. 30

    2017

  26. [26]

    Peiyi Wang, Lei Li, Zhihong Shao, Runxin Xu, Damai Dai, Yifei Li, Deli Chen, Yu Wu, and Zhifang Sui. 2024. Math-Shepherd: Verify and Reinforce LLMs Step-by- step without Human Annotations. InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics, 9426–9439....

    doi:10.18653/v1/2024.acl-long.510 2024

  27. [27]

    Peiyao Wang, Zhan Shi, Amina Shabbeer, and Ben London. 2025. Off-policy eval- uation of candidate generators in two-stage recommender systems. InProceedings of the Nineteenth ACM Conference on Recommender Systems. 350–359

    2025

  28. [28]

    Ruoxi Wang, Rakesh Shivanna, Derek Zhiyuan Cheng, Sagar Jain, Dong Lin, Lichan Hong, and Ed H. Chi. 2020. DCN V2: Improved Deep & Cross Network and Practical Lessons for Web-scale Learning to Rank Systems.arXiv preprint arXiv:2008.13535(2020). arXiv:2008.13535 [cs.IR]

    Pith/arXiv arXiv 2020

  29. [29]

    Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2022. Self-consistency improves chain of thought reasoning in language models.arXiv preprint arXiv:2203.11171(2022)

    Pith/arXiv arXiv 2022

  30. [30]

    Williams

    Ronald J. Williams. 1992. Simple Statistical Gradient-Following Algorithms for Connectionist Reinforcement Learning.Machine Learning8 (1992), 229–256. doi:10.1007/BF00992696

    doi:10.1007/bf00992696 1992

  31. [31]

    Xue Xia, Pong Eksombatchai, Nikil Pancha, Dhruvil Deven Badani, Po-Wei Wang, Neng Gu, Saurabh Vishwas Joshi, Nazanin Farahpour, Zhiyuan Zhang, and Andrew Zhai. 2023. TransAct: Transformer-based Realtime User Action Model for Recommendation at Pinterest.arXiv preprint arXiv:2306.00248(2023). arXiv:2306.00248 [cs.IR]

    Pith/arXiv arXiv 2023

  32. [32]

    Doris Xin, Nicolas Mayoraz, Hubert Pham, Karthik Lakshmanan, and John R Anderson. 2017. Folding: Why good models sometimes make spurious recommen- dations. InProceedings of the eleventh ACM conference on recommender systems. 201–209

    2017

  33. [33]

    Ji Yang, Xinyang Yi, Derek Zhiyuan Cheng, Lichan Hong, Yang Li, Simon Xiaom- ing Wang, Taibai Xu, and Ed H Chi. 2020. Mixed negative sampling for learning two-tower neural networks in recommendations. InCompanion proceedings of the web conference 2020. 441–447

    2020

  34. [34]

    Xinyang Yi, Ji Yang, Lichan Hong, Derek Zhiyuan Cheng, Lukasz Heldt, Aditee Kumthekar, Zhe Zhao, Li Wei, and Ed H. Chi. 2019. Sampling-Bias-Corrected Neural Modeling for Large Corpus Item Recommendations. InProceedings of the 13th ACM Conference on Recommender Systems. ACM, 269–277. doi:10.1145/ 3298689.3346996

    arXiv 2019

  35. [35]

    Guorui Zhou, Jiaxin Deng, Jinghao Zhang, Kuo Cai, Lejian Ren, Qiang Luo, Qianqian Wang, Qigen Hu, et al. 2025. OneRec Technical Report.arXiv preprint arXiv:2506.13695(2025). arXiv:2506.13695 [cs.IR]

    arXiv 2025

  36. [36]

    Guorui Zhou, Hengrui Hu, Hongtao Cheng, Huanjie Wang, Jiaxin Deng, Jinghao Zhang, Kuo Cai, Lejian Ren, et al . 2025. OneRec-V2 Technical Report.arXiv preprint arXiv:2508.20900(2025). arXiv:2508.20900 [cs.IR]

    Pith/arXiv arXiv 2025

  37. [2021]

    doi:10.1145/3434185

    A Troubling Analysis of Reproducibility and Progress in Recommender Systems Research.ACM Transactions on Information Systems39, 2, Article 20 (2021), 49 pages. doi:10.1145/3434185

    doi:10.1145/3434185 2021