Pith. sign in

REVIEW 3 major objections 5 minor 42 references

The paper claims that offline generative recommenders can be fine-tuned by weighting logged interactions with exp(r/λ), and that a coverage-versus-noise trade-off controlled by λ explains when this beats reward-model-based alignment.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-05 00:11 UTC pith:HY7URAXV

load-bearing objection Honest analysis of a known algorithm with a clean coverage-noise tradeoff, but the theory applies to the exact tilt while the experiments train a parametric proxy. the 3 major comments →

arxiv 2608.00816 v1 pith:HY7URAXV submitted 2026-08-01 cs.IR

Exponential Reward Weighting for Fine-Tuning Generative Recommenders under Sparse and Noisy Feedback

Keertana Chidambaram , Sanath Kumar Krishnamurthy , Qiuling Xu , Ko-Jen Hsiao , Moumita Bhattacharya This is my paper
classification cs.IR
keywords generative recommendationoffline fine-tuningexponential reward weightingreward hackingnoisy feedbackcoverage costtemperature trade-offsequential recommendation
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper argues that when a generative recommender is post-trained offline on logged user feedback, the reliable move is not to train a reward model and optimize it, but to weight each logged interaction by exp(r/λ) and continue supervised training. It proves that the suboptimality of the resulting policy against the optimal policy splits into two costs: a coverage cost for how rarely the logging policy produced the optimal trajectory, and a noise cost from imperfect feedback. The temperature λ trades these two costs off, and the theory gives a closed-form optimal λ* = 2√(R_max ε / log(1/p*)). On three public benchmarks and a large proprietary dataset, performance follows the predicted inverted-U as λ varies, with Exp-RSFT beating behavior cloning, linear reward weighting, PPO, and DPO, while the reward-model methods degrade by over-optimizing an unreliable reward head.

Core claim

The central claim is Theorem 4.4: for a fixed context, with sub-Gaussian reward noise and true rewards bounded in [0, R_max], the value gap between the exponential-tilt policy and the optimal trajectory is at most λ log(1/p*(s)) + R_max(e^{2ε/λ} − 1), where p*(s) is the probability the logging policy assigns to the optimal trajectory and ε grows like σ√(2 log(2|T|/δ)). For λ ≥ 2ε this is at most λ log(1/p*) + 4R_max ε/λ, minimized at λ* = 2√(R_max ε / log(1/p*)). Low temperatures exploit high-reward trajectories harder but amplify reward noise; high temperatures are more noise-robust but stay close to the logging policy. The paper verifies the predicted inverted-U temperature sweep empirical

What carries the argument

The exponential tilt π*_λ(τ|s) ∝ π_β(τ|s) exp(R(τ)/λ), the explicit solution of KL-constrained reward maximization, projected onto an autoregressive policy by temperature-weighted maximum likelihood (Eq. 5). The proof uses a variational identity F(π) = Σ π R̂ − λ KL(π∥π_β), shows the tilted policy uniquely maximizes F, and then converts the perturbation of the partition function caused by reward noise into the e^{2ε/λ} − 1 noise term. The temperature λ is the knob that balances the coverage term λ log(1/p*) against this noise term.

Load-bearing premise

The formal bounds describe an idealized tilted policy, not the actual neural network trained on a finite logged dataset, and the paper does not bound how much the finite-sample weighted maximum-likelihood model deviates from that ideal; the predicted temperature trade-off transfers to practice only if the model can represent the tilt and the logged data cover it.

What would settle it

On a synthetic bandit with known logging probabilities, known sub-Gaussian noise σ, and bounded rewards, fit Exp-RSFT at many λ values. If the best λ does not move toward 2√(R_max ε/log(1/p*)) as noise or coverage changes, or if the ranking-versus-λ curve is monotone rather than inverted-U, the decomposition's temperature trade-off is wrong. More directly, compare the value of the trained parametric policy with the exact tilted policy at the same λ: a large gap that does not shrink with dataset size would show the parametric projection, not the trade-off, drives the results.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Share X Bluesky LinkedIn Reddit HN

If this is right

  • In sparse, noisy, offline recommendation, a reward-model-free objective can suffice for post-training: Exp-RSFT improves ranking over behavior cloning and linear reward weighting without a reward model, propensity scores, or online exploration.
  • Temperature λ is not a nuisance hyperparameter but the control variable of a predicted trade-off: the theory gives an interior optimum, and the paper's sweeps confirm an inverted-U, so practitioners should tune λ rather than assume a fixed reward weight.
  • The noise cost grows only logarithmically with catalog size (O(σ√(T log|A|))), so exponential weighting should remain stable as catalogs scale.
  • Reward-model-based methods such as PPO and DPO can inflate predicted reward while degrading ranking, which is direct evidence that optimizing an unreliable learned reward is unsafe in exactly this regime.
  • The coverage cost decomposes as a sum of per-step costs along the optimal trajectory, so the penalty for recovering a target trajectory grows linearly rather than exponentially with the recommendation horizon.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The paper does not bound the gap between the exact tilted policy π*_λ and the finite-sample parametric model π_θ that Algorithm 1 actually trains; a natural extension is to test whether that approximation gap, not the temperature trade-off, dominates on very sparse logs.
  • Because the noise bound uses a union bound over trajectories, it assumes no strong dependence among reward errors; with session-level correlated feedback the effective noise may exceed σ√(2 log(2|T|/δ)), pushing the optimal λ above the formula's prediction.
  • A synthetic bandit with known logging probabilities, known noise level, and bounded rewards could isolate the theory from model approximation: the best λ should shift toward 2√(R_max ε/log(1/p*)) as noise or coverage changes, and the minimized bound should predict achievable regret.
  • The variational argument would likely carry over to preference pairs or multi-objective rewards by exponentiating reward differences or using per-objective temperatures, giving testable variants of Exp-RSFT beyond scalar logged rewards.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. Exp-RSFT fine-tunes a generative recommender by weighting logged trajectories by exp(R/λ) and maximizing the weighted log-likelihood. The paper claims that, under sub-Gaussian reward noise, the suboptimality of the resulting policy relative to any comparator is bounded by a coverage cost λ D_KL(π̃∥π_β) plus a noise cost, and that these competing costs produce an optimal temperature λ* and an inverted-U performance curve. Experiments on ML-1M, ML-20M, Amazon Books, and a proprietary streaming dataset show Exp-RSFT matching or slightly improving over linear reward-weighted SFT and outperforming PPO/DPO, with temperature sweeps exhibiting inverted-U curves.

Significance. If the central claim were established, the paper would provide a simple theoretical rationale for using exponential reward weighting in offline recommendation with sparse, noisy feedback, avoiding reward models and propensity scores. The technical appendix is largely sound for the exact tilted policy π_λ: the variational identity, the sub-Gaussian union bound, and the total-variation argument in Theorems 4.2–4.3 are valid, and Theorem 4.4's algebra is correct. The paper's strengths include trajectory-level analysis recovering the contextual bandit, a closed-form bound-minimizing temperature, and a broad empirical evaluation of four datasets with a reproducible setup. The weaknesses are equally central: the theorems concern an idealized population tilt rather than the finite-sample parametric policy actually trained, and the inverted-U prediction is derived from minimizing an upper bound, so it is not a logical consequence of the theory. These gaps undermine the paper's main explanatory claim as currently written.

major comments (3)
  1. [Section 4 / Algorithm 1, Eq. (5)] Theorems 4.2–4.4 bound the value of the exact tilted population policy π_λ(τ|s) ∝ π_β(τ|s) exp(R̂(τ)/λ), defined using noisy rewards for every trajectory in the support T. Algorithm 1, however, minimizes the weighted MLE objective (5) over a parametric family (HSTU) using a finite logged dataset D, and the paper never bounds D_KL(π_λ ∥ π_θ) or |V^{π_λ}(s)−V^{π_θ}(s)|. The empirical objective also omits the per-context partition function Z(s) and sees only sampled trajectories, so small λ makes exp(R/λ) high-variance. Consequently, Theorem 4.4's trade-off is established for π_λ, not for the deployed π_θ; without a transfer bound or a direct measurement of the approximation error (e.g., a synthetic experiment where π_λ is known), the paper's central explanation of the observed inverted-U is not secure.
  2. [Section 4, Theorem 4.4; Section 5] The inverted-U 'prediction' is derived by minimizing the upper bound g(λ)=λ log(1/p*)+4R_max ε/λ on the suboptimality. An upper bound having a minimum at λ* implies nothing about the location of the minimum of the actual suboptimality V^*(s)−V^{π_λ}(s). Similarly, Theorem 4.2 is a lower bound on V^{π_λ} (or an upper bound on the gap), not an exact decomposition. Therefore the experimental temperature sweeps in Figure 1 cannot be cited as confirmation of Theorem 4.4; at most they are consistent with a heuristic reading. To support the prediction, the authors would need either a lower bound on the suboptimality that is also U-shaped, or an upper bound shown to be tight at the minimizer, or a separate argument that the slack is monotone in λ.
  3. [Section 3.2, Eqs. (4)–(5)] The derivation of the weighted-MLE objective drops the partition function Z(s) from Eq. (4). Unlike the single-context case, Z(s) depends on the context, so minimizing E_{τ∼D}[exp(R/λ) log π_θ] is not equivalent to minimizing Σ_s D_KL(π*(·|s)∥π_θ(·|s)); contexts with larger Z(s) are overweighted. Additionally, the statement that 'the tilted distribution π* remains within the same autoregressive policy class' is generally false for trajectory-level rewards: the tilted conditional at step t is an expectation over future actions and is not a function of the current state alone. The KL projection in (5) is a sensible heuristic, but it is an uncontrolled approximation on both counts.
minor comments (5)
  1. [Section 3.1, Eq. (3)] The notation 'Es∼d0' should be 'E_{s∼d0}' in Eq. (3) and surrounding text; the subscript is malformed.
  2. [Table 2] The header 'RM' is not defined in the caption or text; spell out 'Reward Model' and clarify how 'Reward-Model' predictions are produced (e.g., trained reward head on HSTU representations).
  3. [Tables 3 and 5 / Figures 1, 3, 4] All reported results come from a single training run per configuration, and the margins over RSFT are small (e.g., ML-1M NDCG@10 0.1465 vs. 0.1437). Report confidence intervals or multiple seeds, especially since the paper claims consistent improvement.
  4. [Section 4 and Appendix A] Theorems 4.2–4.4 are stated for a fixed context s. The experimental protocol averages over many test contexts; clarify how the high-probability statements extend to the test distribution, or at least note that a union bound over contexts would be needed.
  5. [Section 5 / Appendix B.1] The statement that 'the temperature λ and training epochs are specified in Section 5' is incomplete: the temperature sweep values appear only in figures. List the exact λ values used for each dataset.

Circularity Check

0 steps flagged

No significant circularity: the theoretical results are derived from explicit variational identities and stated noise assumptions, the algorithm is attributed to prior external work, and no fitted parameter is relabeled as a prediction.

full rationale

The derivation chain is self-contained rather than circular. Exp-RSFT is explicitly identified as a known algorithm family (RWR, MARWIL), so there is no renaming of prior work as new. The weighted-MLE objective (Eq. 5) is derived from the KL-constrained problem (Eq. 3) via Lagrangian and projection; dropping the partition function is valid because it is independent of the optimized parameters θ. Theorems 4.2–4.4 are proved from the variational identity (A3) and a sub-Gaussian union bound, not assumed: the coverage term is exactly the comparator KL divergence, and the noise term is a distribution-shift bound between the noisy and clean tilts. No fitted input is called a prediction: the closed-form λ* (Eq. 9) is explicitly said to require unknown quantities and is not used for hyperparameter selection; λ is tuned empirically (Section 4, end: "Although Equation (9) characterizes the optimal temperature λ*, evaluating it requires unknown quantities such as the reward noise level and the coverage of the optimal trajectory. Therefore, it cannot be directly used for hyperparameter selection in practice, and we tune λ empirically."). This stated limitation weakens the predictive force of the inverted-U claim but is not a circularity. The main substantive weakness is that Theorems 4.2–4.4 bound the exact population tilt πλ, while Algorithm 1 trains a finite-sample parametric πθ by weighted MLE; no D_KL(πλ∥πθ) or value-transfer bound is provided. That is a missing proof / correctness gap, not a circularity. Citations to Nair et al. (2020) and Wang et al. (2018) are external prior work and are not load-bearing self-citations.

Axiom & Free-Parameter Ledger

2 free parameters · 5 axioms · 0 invented entities

The central theoretical contribution is a population-level analysis of the exponential tilt; the only introduced parameter is the temperature λ. No new physical or model entities are postulated. The main implicit assumptions are sub-Gaussian noise, bounded rewards, finite trajectory space, and an unstated representational and finite-sample gap between the exact tilt and the trained parametric policy.

free parameters (2)
  • temperature λ = ≈0.5-1.0 on public benchmarks
    Central regularization knob in exp(R/λ); swept empirically and selected by best NDCG@10. The closed-form λ* from Theorem 4.4 is not computable in practice because R_max, ε, and p* are unknown.
  • training epochs = 30 (ML-1M), 12 (ML-20M), 10 (Amazon Books)
    Hyperparameter choices fixed per dataset for all algorithms; not part of the theory but affect the empirical comparisons.
axioms (5)
  • domain assumption Reward noise is zero-mean σ-sub-Gaussian (Assumption 4.1)
    Theorems 4.2-4.4 union-bound |ξ(τ)| over all trajectories; heavy-tailed or strongly dependent noise invalidates the guarantees. The paper lists this as a limitation.
  • domain assumption True rewards are bounded, R*(τ) ∈ [0, R_max]
    Theorems 4.3-4.4 bound the noisy-tilt value difference by R_max times total variation. The experiments use bounded feedback, but Theorem 4.2 does not require this.
  • domain assumption Finite trajectory set with |T| ≤ |A|^T for each context
    The probability event (A1) requires a union bound over finite T; large or continuous action spaces would need a different covering argument.
  • ad hoc to paper The parametric weighted-MLE solution approximately realizes the exact tilt πλ
    Section 3.2 defines π* in closed form and projects to πθ; Section 4 bounds V for πλ. No representational or finite-sample error is bounded, so the guarantees transfer to the deployed πθ only under this unstated approximation.
  • standard math Comparator policies have finite KL divergence to πβ
    Appendix A states that if D_KL(π̃ || πβ) = ∞, the bounds hold trivially, so all nontrivial comparators must be supported on the support of the logging policy.

pith-pipeline@v1.3.0-alltime-deepseek · 13861 in / 14588 out tokens · 170489 ms · 2026-08-05T00:11:45.868374+00:00 · methodology

0 comments
Cite this review

Pith. "Pith review of Exponential Reward Weighting for Fine-Tuning Generative Recommenders under Sparse and Noisy Feedback." pith.science (2026). https://pith.science/paper/HY7URAXV

@misc{pith2026260800816,
  author       = {Pith},
  title        = {Pith review of: Exponential Reward Weighting for Fine-Tuning Generative Recommenders under Sparse and Noisy Feedback},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/HY7URAXV}},
  note         = {Machine review of arXiv:2608.00816}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

In recommendation systems, users interact with only a small fraction of a vast item catalog, producing feedback that is both sparse and noisy. This challenges post-training generative recommenders: reward models trained from logged interactions often fail to generalize, while directly optimizing imperfect rewards can lead to reward over-optimization. We propose Exponential reward-weighted fine-tuning (Exp-RSFT), where each logged interaction is weighted by $\exp(r/\lambda)$, avoids this failure by optimizing directly on the logged rewards, with the temperature $\lambda$ regularizing against their noise. We theoretically show that Exp-RSFT's suboptimality decomposes into two costs: a coverage cost arising from limitations of the logging policy and a noise cost from imperfect feedback. The temperature $\lambda$ balances these competing effects, yielding an optimal tradeoff between exploiting high-reward behavior and robustness to noise. Across three public benchmarks and a large-scale industrial dataset, we verify this theoretical prediction: performance follows an inverted-U trend as a function of $\lambda$, while PPO and DPO often over-optimize unreliable reward models and degrade recommendation quality. Exp-RSFT consistently improves ranking performance without requiring online exploration or preference data.

Figures

Figures reproduced from arXiv: 2608.00816 by Keertana Chidambaram, Ko-Jen Hsiao, Moumita Bhattacharya, Qiuling Xu, Sanath Kumar Krishnamurthy.

Figure 1
Figure 1. Figure 1: NDCG@10 for different values of λ across datasets. “Avg Reward” in [PITH_FULL_IMAGE:figures/full_fig_p006_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Distribution of ratings in the three public test datasets. [PITH_FULL_IMAGE:figures/full_fig_p011_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Best-epoch values of all five metrics for different [PITH_FULL_IMAGE:figures/full_fig_p012_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Final-epoch values of all five metrics for different [PITH_FULL_IMAGE:figures/full_fig_p013_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: All metrics across training epochs for all algorithms on ML-1M. [PITH_FULL_IMAGE:figures/full_fig_p014_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: All metrics across training epochs for all algorithms on ML-20M. [PITH_FULL_IMAGE:figures/full_fig_p014_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: All metrics across training epochs for all algorithms on Amazon Books. [PITH_FULL_IMAGE:figures/full_fig_p015_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: RSFT versus Exp-RSFT across training epochs on ML-1M. [PITH_FULL_IMAGE:figures/full_fig_p015_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: RSFT versus Exp-RSFT across training epochs on ML-20M. [PITH_FULL_IMAGE:figures/full_fig_p016_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: RSFT versus Exp-RSFT across training epochs on Amazon Books. [PITH_FULL_IMAGE:figures/full_fig_p016_10.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

42 extracted references · 13 canonical work pages · 3 internal anchors

  1. [1]

    GR-LLMs: Recent Advances in Generative Recommendation Based on Large Language Models

    Gr-llms: Recent advances in generative recommendation based on large language models , author=. arXiv preprint arXiv:2507.06507 , year=

    work page internal anchor Pith review Pith/arXiv arXiv

  2. [2]

    Proceedings of the 30th ACM SIGKDD conference on Knowledge Discovery and Data Mining , pages=

    A review of modern recommender systems using generative models (gen-recsys) , author=. Proceedings of the 30th ACM SIGKDD conference on Knowledge Discovery and Data Mining , pages=

    work page

  3. [3]

    World Wide Web , volume=

    A survey on large language models for recommendation , author=. World Wide Web , volume=. 2024 , publisher=

    work page 2024

  4. [4]

    Advances in Neural Information Processing Systems , volume=

    Recommender systems with generative retrieval , author=. Advances in Neural Information Processing Systems , volume=

  5. [5]

    International Conference on Machine Learning (ICML) , year=

    Actions speak louder than words: Trillion-parameter sequential transducers for generative recommendations , author=. International Conference on Machine Learning (ICML) , year=

    work page

  6. [6]

    Proceedings of the 41st International Conference on Machine Learning , pages =

    Actions Speak Louder than Words: Trillion-Parameter Sequential Transducers for Generative Recommendations , author =. Proceedings of the 41st International Conference on Machine Learning , pages =. 2024 , volume =

    work page 2024

  7. [7]

    2018 IEEE international conference on data mining (ICDM) , pages=

    Self-attentive sequential recommendation , author=. 2018 IEEE international conference on data mining (ICDM) , pages=. 2018 , organization=

    2018

  8. [8]

    arXiv preprint arXiv:2502.18965 , year=

    Onerec: Unifying retrieve and rank with generative recommender and iterative preference alignment , author=. arXiv preprint arXiv:2502.18965 , year=

    Pith/arXiv arXiv

  9. [9]

    Advances in neural information processing systems , volume=

    Training language models to follow instructions with human feedback , author=. Advances in neural information processing systems , volume=

  10. [10]

    Advances in neural information processing systems , volume=

    Learning to summarize with human feedback , author=. Advances in neural information processing systems , volume=

  11. [11]

    Advances in neural information processing systems , volume=

    Conservative q-learning for offline reinforcement learning , author=. Advances in neural information processing systems , volume=

  12. [12]

    arXiv preprint arXiv:2005.01643 , year=

    Offline reinforcement learning: Tutorial, review, and perspectives on open problems , author=. arXiv preprint arXiv:2005.01643 , year=

    Pith/arXiv arXiv 2005

  13. [13]

    arXiv preprint arXiv:1707.06347 , year=

    Proximal policy optimization algorithms , author=. arXiv preprint arXiv:1707.06347 , year=

    Pith/arXiv arXiv

  14. [14]

    arXiv preprint arXiv:2402.03300 , year=

    Deepseekmath: Pushing the limits of mathematical reasoning in open language models , author=. arXiv preprint arXiv:2402.03300 , year=

    Pith/arXiv arXiv

  15. [15]

    Local Policy Improvement for Recommender Systems

    Local policy improvement for recommender systems , author=. arXiv preprint arXiv:2212.11431 , year=

    work page internal anchor Pith review Pith/arXiv arXiv

  16. [16]

    Proceedings of the 10th ACM conference on recommender systems , pages=

    Deep neural networks for youtube recommendations , author=. Proceedings of the 10th ACM conference on recommender systems , pages=

  17. [17]

    ResAct: Reinforcing Long-term Engagement in Sequential Recommendation with Residual Actor

    ResAct: Reinforcing long-term engagement in sequential recommendation with residual actor , author=. arXiv preprint arXiv:2206.02620 , year=

    work page internal anchor Pith review Pith/arXiv arXiv

  18. [18]

    The Journal of Machine Learning Research , volume=

    Batch learning from logged bandit feedback through counterfactual risk minimization , author=. The Journal of Machine Learning Research , volume=. 2015 , publisher=

    work page 2015

  19. [19]

    arXiv preprint arXiv:1103.4601 , year=

    Doubly robust policy evaluation and learning , author=. arXiv preprint arXiv:1103.4601 , year=

    Pith/arXiv arXiv

  20. [20]

    1998 , publisher=

    Reinforcement learning: An introduction , author=. 1998 , publisher=

    1998

  21. [21]

    arXiv preprint arXiv:1512.07679 , year=

    Deep reinforcement learning in large discrete action spaces , author=. arXiv preprint arXiv:1512.07679 , year=

    Pith/arXiv arXiv

  22. [22]

    Advances in Neural Information Processing Systems , volume=

    Exponentially weighted imitation learning for batched historical data , author=. Advances in Neural Information Processing Systems , volume=

    work page

  23. [23]

    arXiv preprint arXiv:1910.00177 , year=

    Advantage-weighted regression: Simple and scalable off-policy reinforcement learning , author=. arXiv preprint arXiv:1910.00177 , year=

    Pith/arXiv arXiv 1910

  24. [24]

    arXiv preprint arXiv:2006.09359 , year=

    Awac: Accelerating online reinforcement learning with offline datasets , author=. arXiv preprint arXiv:2006.09359 , year=

    Pith/arXiv arXiv 2006

  25. [25]

    Advances in neural information processing systems , volume=

    Direct preference optimization: Your language model is secretly a reward model , author=. Advances in neural information processing systems , volume=

  26. [26]

    International Conference on Artificial Intelligence and Statistics , pages=

    A general theoretical paradigm to understand learning from human preferences , author=. International Conference on Artificial Intelligence and Statistics , pages=. 2024 , organization=

    2024

  27. [27]

    arXiv preprint arXiv:2402.14740 , year=

    Back to basics: Revisiting reinforce style optimization for learning from human feedback in llms , author=. arXiv preprint arXiv:2402.14740 , year=

    Pith/arXiv arXiv

  28. [28]

    arXiv preprint arXiv:2305.14718 , year=

    Leftover lunch: Advantage-based offline reinforcement learning for language models , author=. arXiv preprint arXiv:2305.14718 , year=

    Pith/arXiv arXiv

  29. [29]

    The Thirty-ninth Annual Conference on Neural Information Processing Systems , year=

    Offline RL by Reward-Weighted Fine-Tuning for Conversation Optimization , author=. The Thirty-ninth Annual Conference on Neural Information Processing Systems , year=

  30. [30]

    Acm transactions on interactive intelligent systems (tiis) , volume=

    The movielens datasets: History and context , author=. Acm transactions on interactive intelligent systems (tiis) , volume=. 2015 , publisher=

    2015

  31. [31]

    Proceedings of the 38th international ACM SIGIR conference on research and development in information retrieval , pages=

    Image-based recommendations on styles and substitutes , author=. Proceedings of the 38th international ACM SIGIR conference on research and development in information retrieval , pages=

    work page

  32. [32]

    Proceedings of the 16th ACM SIGKDD international conference on Knowledge discovery and data mining , pages=

    Training and testing of recommender systems on data missing not at random , author=. Proceedings of the 16th ACM SIGKDD international conference on Knowledge discovery and data mining , pages=

    work page

  33. [33]

    international conference on machine learning , pages=

    Recommendations as treatments: Debiasing learning and evaluation , author=. international conference on machine learning , pages=. 2016 , organization=

    2016

  34. [34]

    arXiv preprint arXiv:2110.06169 , year=

    Offline reinforcement learning with implicit q-learning , author=. arXiv preprint arXiv:2110.06169 , year=

    Pith/arXiv arXiv

  35. [35]

    2007 IEEE International Symposium on Approximate Dynamic Programming and Reinforcement Learning , pages=

    Using reward-weighted regression for reinforcement learning of task space control , author=. 2007 IEEE International Symposium on Approximate Dynamic Programming and Reinforcement Learning , pages=. 2007 , organization=

    work page 2007

  36. [36]

    Advances in Neural Information Processing Systems , volume=

    Critic regularized regression , author=. Advances in Neural Information Processing Systems , volume=

  37. [37]

    Advances in Neural Information Processing Systems , volume=

    Scaling laws for reward model overoptimization in direct alignment algorithms , author=. Advances in Neural Information Processing Systems , volume=

    work page

  38. [38]

    arXiv preprint arXiv:2311.00168 , year=

    The alignment ceiling: Objective mismatch in reinforcement learning from human feedback , author=. arXiv preprint arXiv:2311.00168 , year=

    Pith/arXiv arXiv

  39. [39]

    arXiv preprint arXiv:2201.03544 , year=

    The effects of reward misspecification: Mapping and mitigating misaligned models , author=. arXiv preprint arXiv:2201.03544 , year=

    Pith/arXiv arXiv

  40. [40]

    Advances in Neural Information Processing Systems , volume=

    Defining and characterizing reward gaming , author=. Advances in Neural Information Processing Systems , volume=

  41. [41]

    Machine learning , volume=

    Finite-time analysis of the multiarmed bandit problem , author=. Machine learning , volume=. 2002 , publisher=

    2002

  42. [42]

    arXiv preprint arXiv:1606.06565 , year=

    Concrete problems in AI safety , author=. arXiv preprint arXiv:1606.06565 , year=

    Pith/arXiv arXiv