Pith. sign in

REVIEW 4 major objections 6 minor 2 cited by

Bias Fitting to Mitigate Length Bias of Reward Model in RLHF

T0 review · 4 major / 6 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Length bias in reward models is a learnable non-linear curve, and removing its correlation with the reward score improves alignment outcomes.

desk verdict A useful incremental extension of ODIN that fits a nonlinear length-reward curve; the main gap is that the debiasing loss targets correlation with the fitted curve, not with length itself, and the paper never reports the direct residual correlation. read the letter →

arxiv 2505.12843 v2 pith:VY6CAMKN submitted 2025-05-19 cs.LG cs.AI

Kangwen Zhao , Jianfeng Cai , Jinhua Zhu , Ruopei Sun , Dongyun Xue , Wengang Zhou , Li Li , Houqiang Li This is my paper · ORCID
classification cs.LGcs.AI
keywords lengthbiasrewardhackingmodelRLHFdebiasingnon-linearfittingDirectPreferenceOptimizationBest-of-N
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper proposes FiMi-RM, a three-stage procedure that treats length bias in reward models as a learnable non-linear function rather than a constant or linear penalty. The authors train a standard reward model, fit a lightweight model that maps only response length to the reward score, and then retrain the reward model while penalizing the Pearson correlation between its score and that fitted curve. When the debiased reward model is used in Direct Preference Optimization and Best-of-N, they report higher length-controlled win rate and reduced verbosity while keeping preference-modeling ability, at two model scales. They also characterize the fitted bias as strongly linear for short responses, decelerating for medium-length responses, and statistically indistinguishable from noise for very long responses.

What carries the argument

The load-bearing object is the fitting model $f(\mathrm{len}(y)) = W_{\mathrm{reg}}\cdot\mathrm{ResNet}(\mathrm{LE}(\mathrm{len}(y))) + b_{\mathrm{reg}}$, where $\mathrm{LE}$ is a sinusoidal length encoding modeled on positional encoding, and the ResNet has two residual layers. The model is trained only on length-reward pairs from the warm-up reward model, with a loss that maximizes Pearson correlation and minimizes mean squared error between the fitted scalar and the detached reward. That fitted scalar defines what the paper counts as the length-bias component; the debiasing stage then trains the reward model to have zero Pearson correlation with this component while preserving Bradley-Terry preference accuracy.

What would settle it

Compare the residual reward after subtracting the fitted curve against length on a held-out set using Spearman rank correlation or an estimate of mutual information; if substantial length dependence remains, zero Pearson correlation with one fitted curve has not removed length bias.

Watch

Extended reading notes

Core claim

The paper's central claim is that length bias in a reward model can be isolated by a lightweight fitting model $f$ that takes only the response length $\mathrm{len}(y)$ as input, and that removing the correlation with $f$'s prediction debiases the reward without destroying preference learning. Concretely, after a warm-up reward model is trained with the Bradley-Terry loss $\mathcal{L}_{BT}$, the fitting model is trained with $\mathcal{L}_{\mathrm{fit}} = -|\rho(r_{\mathrm{detach}}, \hat{r})| + \mathcal{L}_{\mathrm{mse}}$, where $\rho$ is the Pearson correlation coefficient. Then the reward model is retrained with $\mathcal{L}_{\mathrm{debiased}} = |\rho(r, \hat{r}_{\mathrm{detach}})| + \mathcal{L}_{BT}$. The authors report that this produces near-balanced accuracy between preference pairs where the chosen response is longer and pairs where the rejected response is longer, flattens the length-reward scatter, and raises length-controlled win rate under Direct Preference Optimization and Best-of-N at two model scales. The fitted curve reveals three phases: strong linearity below about 100 tokens, a decelerating upward trend from 100 to 200 tokens, and behavior indistinguishable from noise beyond 200 tokens.

Load-bearing premise

The load-bearing premise is that the fitting model's output, computed from length alone, captures exactly the length-bias component of the reward model, so that forcing the reward model to be uncorrelated with that single fitted curve removes the bias without touching genuine quality signals.

Editorial extensions

If this is right

  • Debiasing a reward model with a learned non-linear bias curve yields higher length-controlled win rate than both the vanilla reward model and a linear-assumption baseline under Direct Preference Optimization and Best-of-N.
  • The three-phase bias curve implies that a single linear length penalty is miscalibrated, under-penalizing short responses and over-penalizing long ones.
  • Downstream policies trained with the debiased reward model produce shorter and more balanced outputs while retaining or improving win rate, reducing verbosity-driven reward hacking.
  • Because the method works at two model scales with only mild overall accuracy loss on preference pairs, the fitted-curve correction can be added without reworking the rest of the alignment pipeline.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Inference: the fitted three-phase curve suggests a cheap piecewise-linear penalty, steep below 100 tokens, shallow from 100 to 200, and flat beyond 200, could approximate the full fitting model in deployment.
  • Inference: because zero Pearson correlation with one fitted curve does not imply statistical independence from length, residual higher-order length dependence may remain; a mutual-information version of the penalty would test whether this mechanism is complete.
  • Inference: the method may remove genuine length preference when users explicitly ask for detailed answers; conditioning the fitting model on the prompt or on user instructions would turn the bias component into a context-dependent quantity.
  • Inference: the same fit-then-decorrelate recipe could be applied to other reward-hacking signals such as formatting artifacts, by replacing the length encoder with an encoder of those surface features.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes FiMi-RM, a three-stage method to mitigate length bias in RLHF reward models. Stage 1 trains a standard Bradley-Terry reward model, deliberately retaining length bias. Stage 2 fits a lightweight model f(len(y)) — a sinusoidal length encoding followed by a two-layer ResNet and linear head — to the reward model's output using a combination of Pearson correlation and MSE losses, producing a predicted bias score. Stage 3 debiases the reward model by training with L'_pearson = |rho(r, hat_r_detach)| plus the original BT loss, alternating between fitting-model and reward-model updates. The authors evaluate on Anthropic HH data with Qwen2.5-1.5B and Qwen2.5-7B, reporting accuracy on C-longer/R-longer subsets, length-reward scatter plots, length-controlled Alpaca-Eval win rates under BoN and DPO, and length distributions of selected responses. They also report a fitted three-phase length-reward pattern: strongly linear for short responses, sublinear for medium lengths, and noise-like for long lengths.

Significance. If the central claim holds, the paper offers a practical, lightweight alternative to linear length-penalty methods and to ODIN's two-head architecture, with the potential advantage of capturing nonlinear length-reward relations. The use of length-controlled Alpaca-Eval as an external evaluation is a genuine strength, as it provides a check that is not defined by the method's own fitted curve. The balanced accuracy on C-longer and R-longer subsets is also a useful diagnostic. However, the significance is tempered by two unresolved issues: the debiasing objective in Eq. (9) only enforces zero Pearson correlation with one fitted scalar function of length, not independence from length, and the fitted curve is learned from the same reward model that is later debiased, so the reported three-phase pattern is not independently validated. The absence of confidence intervals and of an ablation against a linear version of the same fitting framework further limits the strength of the empirical claims.

major comments (4)
  1. [§3.3, Eq. (9)] The debiasing loss L'_pearson = |rho(r, hat_r_detach)| only forces the reward model output to be uncorrelated with the specific fitted scalar hat_r = f(len(y)). Zero Pearson correlation with one nonlinear function of length does not imply zero or even reduced correlation with length itself; for example, a reward that is linear in length can be nearly uncorrelated with a saturating or sinusoidal fitted curve over some ranges. The paper never reports rho(r, len) after debiasing, and Figure 2 shows only binned scatter plots. Since the central claim is that FiMi-RM decouples reward from length, the authors should directly report the post-debiasing Pearson correlation between r and len, or an equivalent independence test, to verify the mechanism.
  2. [§3.2, Fig. 5; Limitations] The fitted curve is trained on the output of the same warm-up reward model that is later debiased, and it approximates the marginal relation E[r | len]. That marginal relation mixes genuine length preference with content preferences that correlate with length; the Limitations section itself concedes that some length preference may be genuine. Consequently, removing the projection onto hat_r can remove true signal rather than only bias. The downstream LC-WR evaluation is external and partially addresses this, but the paper should provide additional evidence that the fitted curve corresponds to bias rather than to legitimate preference, for example by evaluating on a dataset with explicit length annotations or by comparing against a content-controlled baseline.
  3. [Tables 2 and 3; §4.2] The reported LC-WR and WR numbers are point estimates without confidence intervals, significance tests, or multiple seeds. Several differences are small (e.g., DPO 1.5B LC-WR 73.84 vs. 73.22; BoN 1.5B LC-WR 74.83 vs. 73.72; DPO 7B LC-WR 70.19 vs. 68.17), so it is unclear whether the improvements are statistically reliable. Additionally, the paper's key claim that nonlinear fitting is superior to linear debiasing is not directly supported: there is no ablation that replaces the ResNet fitting model with a linear function of length while keeping all other components unchanged. Such an ablation is necessary to establish that the improved LC-WR comes from the nonlinear structure rather than from the extra training signal or the decorrelation objective alone.
  4. [§3.3, Eq. (8)-(11)] The alternating training schedule in Eq. (10)-(11) is a central design choice but is not analyzed. The period a = 8 is fixed, and the paper does not report sensitivity to this hyperparameter, nor whether convergence of the reward model or the fitting model is affected by the alternation. At minimum, the authors should show that the results are stable for reasonable values of a, because the decorrelation effect depends on the fitting model remaining aligned with the current reward model during the debiasing stage.
minor comments (6)
  1. [Abstract and §1] There are several grammatical and typographical errors, including 'these approaches either mitigate bias without characterizing the bias form' (missing comma and subject-verb agreement), 'A part of existing approaches alleviate length bias' (number agreement), and 'does not characterizing the bias form' in §2.
  2. [Figure 2] The caption states that the average curve is 'more parallel to the x-axis' for the proposed method, but parallelism is not quantified. Reporting the slope or correlation of the binned averages would make the visual claim precise.
  3. [Table 1] The table would benefit from explicit counts or percentages for the C-longer and R-longer subsets in the caption or text; the text mentions 58% and 40% but the table itself does not show these numbers, making the accuracy comparison harder to interpret.
  4. [§4.1, Eq. (13)] The DPO objective is written with beta in the logarithm, but the authors do not state the value of beta used in experiments or whether it differs between the two model sizes; this is needed for reproducibility.
  5. [Figure 5] The step numbers in Figure 5 are described in the text but the training step at which the final debiased reward model is evaluated is not specified; the paper should clarify how the fitted curve shown at step 500 relates to the curve used in the debiasing stage.
  6. [§5 and Limitations] The conclusion and limitations sections are somewhat brief and do not discuss the lack of a direct test of the decorrelation mechanism, which is the main theoretical assumption of the method; a short paragraph acknowledging this and pointing to future work would improve the paper.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the debiasing objective is a self-referential projection, but the paper's central claims are tested on external benchmarks and do not reduce to the fitted curve.

full rationale

The derivation chain is self-contained. Stage 2 fits \hat r = modelf(len(y)) to the warm-up reward model's outputs using L_fit = -|rho(r_detach, \hat r)| + L_mse (Eqs. 4-6), and Stage 3 penalizes L'_pearson = |rho(r, \hat r_detach)| (Eq. 9). This is an iterative orthogonalization: the debiased model is trained to be uncorrelated with the fitted curve, but the paper does not define 'decoupled from length' as equivalent to 'uncorrelated with \hat r', nor does it claim a theorem that the two are equivalent. The main empirical results (LC-WR on Alpaca-Eval, WR, and length distributions under DPO and BoN) are measured on held-out data against external baselines, so they are not forced by the fitting procedure. Figure 5 illustrates the training fit of the bias model; the three-phase description is a descriptive reading of that fitted curve, not an independent prediction, and the Limitations section candidly concedes that some length-preference may be genuine—a construct-validity caveat rather than a circular step. There are no load-bearing self-citations or imported uniqueness theorems.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The paper introduces a learned bias function as the central new object. That function is fitted to the same reward model it later debiases, so several domain assumptions about what length bias is and how to remove it do the real work. No new physical or model-level entities beyond the fitting network are introduced.

free parameters (4)
  • Fitting model f(len(y)) (ResNet weights and regression head) = learned from reward model outputs
    The fitted curve is the operational definition of length bias and fully determines the debiasing target in Eq. (9).
  • Length encoding dimension d = 32
    Set in Sec. 3.2; controls the capacity of the length encoding.
  • Model alternation period a = 8
    Schedule in Eq. (11) for alternating fitting and debiasing training; chosen by hand.
  • Loss weighting for L_fit and L_debiased = implicitly 1.0 each
    No weighting scheme is reported; unit weighting is assumed and is a manual choice.
assumptions (5)
  • domain assumption Human preference labels contain a length-related spurious component that should be removed.
    Motivates the whole method; the authors note in the Limitations section that length may partially correlate with genuine preference.
  • ad hoc to paper The reward model's length bias is well-approximated by a function of length alone, learnable by a two-layer ResNet on sinusoidal length encoding.
    Sec. 3.2 defines model_f(len(y)); no expressivity or identifiability analysis is given.
  • ad hoc to paper Zero Pearson correlation between r and \hat r is sufficient to decouple length from reward.
    Eq. (9) uses |rho(r, \hat r_detach)| as the debiasing objective; Pearson zero does not imply statistical independence.
  • domain assumption The fitting model trained on the warm-up reward model's outputs remains valid during the alternating debiasing training.
    Alternating training in Eq. (10) assumes the fitted bias curve does not drift as the reward model changes.
  • standard math Bradley-Terry preference model as the reward training objective.
    Eq. (1), standard in RLHF.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Bias Fitting to Mitigate Length Bias of Reward Model in RLHF." pith.science (2026). https://pith.science/paper/VY6CAMKN

@misc{pith2026250512843,
  author       = {Pith},
  title        = {Pith review of: Bias Fitting to Mitigate Length Bias of Reward Model in RLHF},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/VY6CAMKN}},
  note         = {Machine review of arXiv:2505.12843}
}
read the original abstract

Reinforcement Learning from Human Feedback (RLHF) relies on reward models to align large language models with human preferences. However, RLHF often suffers from reward hacking, wherein policy learning exploits flaws in the trained reward model to maximize reward scores without genuinely aligning with human preferences. A significant example of such reward hacking is length bias, where reward models usually favor longer responses irrespective of actual response quality. Previous works on tackling length bias have notable limitations, these approaches either mitigate bias without characterizing the bias form, or simply assume a linear length-reward relation. To accurately model the intricate nature of length bias and facilitate more effective bias mitigation, we propose FiMi-RM (Bias Fitting to Mitigate Length Bias of Reward Model), a framework that autonomously learns and corrects underlying bias patterns. Our approach consists of three stages: First, we warm up by training a standard reward model which inherently contains length bias. Next, we deploy a lightweight fitting model to capture the non-linear relation between length and reward. Finally, we incorporate this learned relation into the reward model, effectively decoupling length from reward while preserving preference modeling capabilities. Experimental results demonstrate that FiMi-RM achieves a more balanced length-reward distribution. Furthermore, when applied to alignment algorithms such as Direct Preference Optimization (DPO) and Best-of-N (BoN), our debiased reward model improves length-controlled win rate and reduces verbosity without compromising its performance.

Figures

Figures reproduced from arXiv: 2505.12843 by the authors.

Figure 1
Figure 1. The overview of our method (part a). First, we use traditional reward model training to [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Scatter plot of reward versus length, with binned averages (black lines). Our method [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. The pairwise comparison of the distribution of responses selected by BoN. The figure [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Length distribution differences of chosen and rejected responses in the labeling stage of [PITH_FULL_IMAGE:figures/full_fig_p009_4.png]
Figure 5
Figure 5. Figure 5: The fitted curve of modelf at different steps in training. In general, the curve gradually aligns with the trend of the scatter points and eventually matches their pattern closely. From the last subfigure, the relation exhibits strong linearity at shorter length, with …

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Reward Bias Substitution: Single-Axis Bias Mitigations Redirect Optimization Pressure

    cs.AI 2026-05 accept novelty 7.0 of 10

    Single-axis reward bias mitigations redirect optimization pressure to correlated proxies, and audit-distribution scoring produces identical observables for successful mitigation, bias substitution, and overcorrection.

  2. Test-Time Scaling via Error Localization

    cs.LG 2026-07 conditional novelty 6.0 of 10

    TTEL uses feedback-induced token probability drops to localize the first error in a failed reasoning trace and branch a new generation from that prefix, improving pass@k per token on coding and math benchmarks.

Reference graph

Works this paper leans on

51 extracted references · 27 canonical work pages · cited by 2 Pith papers

  1. [1]

    A general language assistant as a laboratory for alignment.arXiv preprint arXiv:2112.00861, 2021

    Amanda Askell, Yuntao Bai, Anna Chen, Dawn Drain, Deep Ganguli, Tom Henighan, Andy Jones, Nicholas Joseph, Ben Mann, Nova DasSarma, et al. A general language assistant as a laboratory for alignment.arXiv preprint arXiv:2112.00861, 2021

    arXiv 2021

  2. [2]

    Training a helpful and harmless assistant with reinforcement learning from human feedback, 2022

    Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback, 2022

    work page 2022

  3. [3]

    Rank analysis of incomplete block designs: I

    Ralph Allan Bradley and Milton E Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons.Biometrika, 39(3/4):324–345, 1952

    1952

  4. [4]

    Noise contrastive alignment of language models with explicit rewards, 2024

    Huayu Chen, Guande He, Lifan Yuan, Ganqu Cui, Hang Su, and Jun Zhu. Noise contrastive alignment of language models with explicit rewards, 2024

    work page 2024

  5. [5]

    ODIN: Disentangled reward mitigates hacking in RLHF

    Lichang Chen, Chen Zhu, Jiuhai Chen, Davit Soselia, Tianyi Zhou, Tom Goldstein, Heng Huang, Mohammad Shoeybi, and Bryan Catanzaro. ODIN: Disentangled reward mitigates hacking in RLHF. InForty-first International Conference on Machine Learning, 2024

    work page 2024

  6. [6]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning, 2025

    DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning, 2025

    2025

  7. [7]

    Deepseek-v3 technical report, 2025

    DeepSeek-AI, Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, et al. Deepseek-v3 technical report, 2025

    work page 2025

  8. [8]

    Raft: Reward ranked finetuning for generative foundation model alignment.arXiv preprint arXiv:2304.06767, 2023

    Hanze Dong, Wei Xiong, Deepanshu Goyal, Yihan Zhang, Winnie Chow, Rui Pan, Shizhe Diao, Jipeng Zhang, Kashun Shum, and Tong Zhang. Raft: Reward ranked finetuning for generative foundation model alignment.arXiv preprint arXiv:2304.06767, 2023

    arXiv 2023

Show all 51 references
  1. [9]

    Rlhf workflow: From reward modeling to online rlhf, 2024

    Hanze Dong, Wei Xiong, Bo Pang, Haoxiang Wang, Han Zhao, Yingbo Zhou, Nan Jiang, Doyen Sahoo, Caiming Xiong, and Tong Zhang. Rlhf workflow: From reward modeling to online rlhf, 2024

    2024

  2. [10]

    Length-controlled alpacaeval: A simple way to debias automatic evaluators.arXiv preprint arXiv:2404.04475, 2024

    Yann Dubois, Balázs Galambosi, Percy Liang, and Tatsunori B Hashimoto. Length-controlled alpacaeval: A simple way to debias automatic evaluators.arXiv preprint arXiv:2404.04475, 2024

    2024 arXiv

  3. [11]

    Helping or herding? re- ward model ensembles mitigate but do not eliminate reward hacking

    Jacob Eisenstein, Chirag Nagpal, Alekh Agarwal, Ahmad Beirami, Alexander Nicholas D’Amour, Krishnamurthy Dj Dvijotham, Adam Fisch, Katherine A Heller, Stephen Robert Pfohl, Deepak Ramachandran, Peter Shaw, and Jonathan Berant. Helping or herding? re- ward model ensembles mitig...

    2024

  4. [12]

    Kto: Model alignment as prospect theoretic optimization.arXiv preprint arXiv:2402.01306, 2024

    Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky, and Douwe Kiela. Kto: Model alignment as prospect theoretic optimization.arXiv preprint arXiv:2402.01306, 2024

    2024 arXiv

  5. [13]

    Scaling laws for reward model overoptimization

    Leo Gao, John Schulman, and Jacob Hilton. Scaling laws for reward model overoptimization. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett, editors,Proceedings of the 40th International Conference on Machine Learning, vo...

    2023

  6. [14]

    A general theoretical paradigm to understand learning from human preferences

    Mohammad Gheshlaghi Azar, Zhaohan Daniel Guo, Bilal Piot, Remi Munos, Mark Rowland, Michal Valko, and Daniele Calandriello. A general theoretical paradigm to understand learning from human preferences. In Sanjoy Dasgupta, Stephan Mandt, and Yingzhen Li, editors, Proceedings of...

    2024

  7. [15]

    The llama 3 herd of models, 2024

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, et al. The llama 3 herd of models, 2024

    2024

  8. [16]

    Bonbon alignment for large language models and the sweetness of best-of-n sampling.arXiv preprint arXiv:2406.00832, 2024

    Lin Gui, Cristina Gârbacea, and Victor Veitch. Bonbon alignment for large language models and the sweetness of best-of-n sampling.arXiv preprint arXiv:2406.00832, 2024

    2024 arXiv

  9. [17]

    Deep residual learning for im- age recognition

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for im- age recognition. InProceedings of the IEEE Conference on Computer Vision and Pattern Recognition, June 2016

    2016

  10. [18]

    ORPO: Monolithic preference optimization without reference model

    Jiwoo Hong, Noah Lee, and James Thorne. ORPO: Monolithic preference optimization without reference model. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen, editors,Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 11170– 11189, M...

    2024

  11. [19]

    Post-hoc reward calibra- tion: A case study on length bias

    Zeyu Huang, Zihan Qiu, Zili Wang, Edoardo Ponti, and Ivan Titov. Post-hoc reward calibra- tion: A case study on length bias. InThe Thirteenth International Conference on Learning Representations, 2025

    2025

  12. [20]

    Gonzalez, Hao Zhang, and Ion Stoica

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large lan- guage model serving with pagedattention. InProceedings of the ACM SIGOPS 29th Symposium on Operating System...

    2023

  13. [21]

    Openassistant conversations -democratizing large language model alignment

    Andreas Köpf, DimitriV on Rütte, Sotiris Anagnostidis, Zhi-Rui Tam, Keith Stevens, Abdul- lahBarhoum Nguyen, et al. Openassistant conversations -democratizing large language model alignment

  14. [22]

    The alignment ceiling: Objective mismatch in rein- forcement learning from human feedback.arXiv preprint arXiv:2311.00168, 2023

    Nathan Lambert and Roberto Calandra. The alignment ceiling: Objective mismatch in rein- forcement learning from human feedback.arXiv preprint arXiv:2311.00168, 2023

    2023 arXiv

  15. [23]

    RRM: Robust reward model training mitigates reward hacking

    Tianqi Liu, Wei Xiong, Jie Ren, Lichang Chen, Junru Wu, Rishabh Joshi, Yang Gao, Jiaming Shen, Zhen Qin, Tianhe Yu, Daniel Sohn, Anastasia Makarova, Jeremiah Zhe Liu, Yuan Liu, Bilal Piot, Abe Ittycheriah, Aviral Kumar, and Mohammad Saleh. RRM: Robust reward model training mit...

    2025

  16. [24]

    Simpo: Simple preference optimization with a reference-free reward

    Yu Meng, Mengzhou Xia, and Danqi Chen. Simpo: Simple preference optimization with a reference-free reward. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang, editors,Advances in Neural Information Processing Systems, volume 37, pages 124198–1...

    2024

  17. [25]

    Expanding on what we missed with sycophancy, 2025

    OpenAI. Expanding on what we missed with sycophancy, 2025

    2025

  18. [26]

    Gpt-4 technical report, 2024

    OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, et al. Gpt-4 technical report, 2024

    2024

  19. [27]

    Training language models to follow instructions with human feedback.Advances in neural information processing systems, 35:27730–27744, 2022

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback.Advances in neural information processing systems, 35:27730...

    2022

  20. [28]

    Reward gaming in conditional text generation

    Richard Yuanzhe Pang, Vishakh Padmakumar, Thibault Sellam, Ankur Parikh, and He He. Reward gaming in conditional text generation. InProceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 4746–4763, 2023. 11

    2023

  21. [29]

    Pytorch: An imperative style, high-performance deep learning library.CoRR, abs/1912.01703, 2019

    Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, et al. Pytorch: An imperative style, high-performance deep learning library.CoRR, abs/1912.01703, 2019

    1912 arXiv

  22. [30]

    Notes on regression and inheritance in the case of two parents.Proceedings of the Royal Society of London, 58:240–242, 1895

    Karl Pearson. Notes on regression and inheritance in the case of two parents.Proceedings of the Royal Society of London, 58:240–242, 1895

  23. [31]

    Qwen2.5 technical report, 2025

    Qwen, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, et al. Qwen2.5 technical report, 2025

    2025

  24. [32]

    Direct preference optimization: Your language model is secretly a reward model

    Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors,Advances in Ne...

    2023

  25. [33]

    Warp: On the benefits of weight averaged rewarded policies, 2024

    Alexandre Ramé, Johan Ferret, Nino Vieillard, Robert Dadashi, Léonard Hussenot, Pierre-Louis Cedoz, Pier Giuseppe Sessa, Sertan Girgin, Arthur Douillard, and Olivier Bachem. Warp: On the benefits of weight averaged rewarded policies, 2024

    2024

  26. [34]

    Warm: On the benefits of weight averaged reward models, 2024

    Alexandre Ramé, Nino Vieillard, Léonard Hussenot, Robert Dadashi, Geoffrey Cideron, Olivier Bachem, and Johan Ferret. Warm: On the benefits of weight averaged reward models, 2024

    2024

  27. [35]

    Deepspeed: System optimizations enable training deep learning models with over 100 billion parameters

    Jeff Rasley, Samyam Rajbhandari, Olatunji Ruwase, and Yuxiong He. Deepspeed: System optimizations enable training deep learning models with over 100 billion parameters. In Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 3505–3...

    2020

  28. [36]

    Offline regularised reinforcement learning for large language models alignment, 2024

    Pierre Harvey Richemond, Yunhao Tang, Daniel Guo, Daniele Calandriello, Mohammad Ghesh- laghi Azar, Rafael Rafailov, Bernardo Avila Pires, Eugene Tarassov, Lucas Spangher, Will Ellsworth, Aliaksei Severyn, Jonathan Mallinson, Lior Shani, Gil Shamir, Rishabh Joshi, Tianqi Liu, ...

    2024

  29. [37]

    Proximal policy optimization algorithms.arXiv preprint arXiv:1707.06347, 2017

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms.arXiv preprint arXiv:1707.06347, 2017

    2017 arXiv

  30. [38]

    Bond: Aligning llms with best-of-n distillation.arXiv preprint arXiv:2407.14622, 2024

    Pier Giuseppe Sessa, Robert Dadashi, Léonard Hussenot, Johan Ferret, Nino Vieillard, Alexan- dre Ramé, Bobak Shariari, Sarah Perrin, Abe Friesen, Geoffrey Cideron, et al. Bond: Aligning llms with best-of-n distillation.arXiv preprint arXiv:2407.14622, 2024

    2024 arXiv

  31. [39]

    Deepseekmath: Pushing the limits of mathematical reasoning in open language models.arXiv preprint arXiv:2402.03300, 2024

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models.arXiv preprint arXiv:2402.03300, 2024

    2024 arXiv

  32. [40]

    Loose lips sink ships: Mitigating length bias in reinforcement learning from human feedback

    Wei Shen, Rui Zheng, Wenyu Zhan, Jun Zhao, Shihan Dou, Tao Gui, Qi Zhang, and Xuanjing Huang. Loose lips sink ships: Mitigating length bias in reinforcement learning from human feedback. InThe 2023 Conference on Empirical Methods in Natural Language Processing, 2023

    2023

  33. [41]

    A long way to go: Investigating length correlations in rlhf.arXiv preprint arXiv:2310.03716, 2023

    Prasann Singhal, Tanya Goyal, Jiacheng Xu, and Greg Durrett. A long way to go: Investigating length correlations in rlhf.arXiv preprint arXiv:2310.03716, 2023

    2023 arXiv

  34. [42]

    Ziegler, Ryan Lowe, Chelsea V oss, Alec Radford, Dario Amodei, and Paul Christiano

    Nisan Stiennon, Long Ouyang, Jeff Wu, Daniel M. Ziegler, Ryan Lowe, Chelsea V oss, Alec Radford, Dario Amodei, and Paul Christiano. Learning to summarize from human feedback, 2022

    2022

  35. [43]

    Learning to summarize with human feedback

    Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea V oss, Alec Radford, Dario Amodei, and Paul F Christiano. Learning to summarize with human feedback. Advances in neural information processing systems, 33:3008–3021, 2020

    2020

  36. [44]

    Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context, 2024

    Gemini Team, Petko Georgiev, Ving Ian Lei, Ryan Burnell, Libin Bai, Anmol Gulati, et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context, 2024. 12

    2024

  37. [45]

    Llama 2: Open foundation and fine-tuned chat models, 2023

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, et al. Llama 2: Open foundation and fine-tuned chat models, 2023

    2023

  38. [46]

    Attention is all you need

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In I. Guyon, U. V on Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors,Advances in Neural ...

    2017

  39. [47]

    Reward hacking in reinforcement learning, Nov 2024

    Lilian Weng. Reward hacking in reinforcement learning, Nov 2024

    2024

  40. [48]

    Qwen2 technical report, 2024

    An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, et al. Qwen2 technical report, 2024

    2024

  41. [49]

    Dapo: An open-source llm reinforcement learning system at scale, 2025

    Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, et al. Dapo: An open-source llm reinforcement learning system at scale, 2025

    2025

  42. [50]

    From lists to emojis: How format bias affects model alignment, 2024

    Xuanchang Zhang, Wei Xiong, Lichang Chen, Tianyi Zhou, Heng Huang, and Tong Zhang. From lists to emojis: How format bias affects model alignment, 2024

    2024

  43. [51]

    Fine-tuning language models from human preferences, 2020.URL https://arxiv

    Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences, 2020.URL https://arxiv. org/abs, page 14, 1909. 13

    2020

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.