Pith. sign in

REVIEW 4 major objections 6 minor 43 references

Off-Policy Corrected Reward Modeling for Reinforcement Learning from Human Feedback

T0 review · 4 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Off-policy reweighting fixes RLHF reward drift without new labels

desk verdict A genuinely useful off-policy correction for reward-model training that mostly delivers empirically, though its central theoretical claim about non-convergence outruns the proof. read the letter →

arxiv 2507.15507 v1 pith:TORMKZ2L submitted 2025-07-21 cs.LG cs.AIcs.CL

Johannes Ackermann , Takashi Ishida , Masashi Sugiyama This is my paper
classification cs.LGcs.AIcs.CL
keywords ReinforcementLearningfromHumanFeedbackrewardmodeldistributionshiftimportanceweightingoveroptimizationoff-policycorrectionpolicygradientBradley-Terry
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that overoptimization in RLHF is a distribution shift problem with a fix that costs no new human labels. In standard PPO-RLHF, the reward model is trained once on responses from the supervised fine-tuning policy $\pi_1$, but during RL it scores responses from a policy $\pi_i$ that keeps moving away from $\pi_1$; the authors argue this makes the reward model an inconsistent estimator of the true reward parameters, and hence the policy gradient is inconsistent, so standard RLHF generally does not converge to the optimal policy even with unlimited data. Their proposed Off-Policy Corrected Reward Modeling (OCRM) retrains the reward model with importance weights $\pi_i/\pi_1$ so that the loss is evaluated under the current policy, and they prove an estimation error bound showing this weighted estimator is consistent. Empirically, on summarization and chatbot tasks with a gold reward model, OCRM improves final alignment over PPO, DPO, WPO, and RLP-SPG using the same preference dataset.

What carries the argument

The load-bearing object is the importance weight $w(s, a_w, a_l) = \pi_i(a_w|s)\pi_i(a_l|s) / \pi_1(a_w|s)\pi_1(a_l|s)$, the ratio of current-policy to SFT-policy probabilities for the preferred and dispreferred completions. It arises because the human preference probability $P(a_w > a_l|s)$ is assumed independent of the policy that generated the responses, so the joint distribution ratio collapses to the policy ratio, turning the reward-model loss under $\pi_i$ into a reweighted loss under $\pi_1$. The consistency argument is carried by Theorem 1, a Rademacher-complexity bound of the form $L^{\pi_i}_{RM}(\tilde\theta) - L^{\pi_i}_{RM}(\theta^*) \le 4W R_{N_{RM}}(\mathcal{F}) + W C_l \sqrt{2/N_{RM} \ln(2/\delta)}$, where $W$ bounds the importance weight; since the complexity term vanishes as $N_{RM} \to \infty$, the importance-weighted estimator is consistent, whereas the unweighted estimator's excess loss contains distribution-shift terms that do not vanish.

What would settle it

Run the LM alignment experiment on TL;DR with unflattened importance weights; if the resulting reward model scores worse on current-policy pairs than the unweighted reward model while the flattened version keeps its advantage, then the consistency theorem is not the mechanism behind the empirical gains. Alternatively, collect fresh human labels on responses sampled from the trained policy and test whether the preference probabilities match the importance-weighted predictions of the original reward model; a systematic mismatch would violate the preference-invariance assumption used in Equation (5).

Watch

Extended reading notes

Core claim

The central claim is that the standard RLHF reward-modeling step is statistically inconsistent once the policy starts moving. The reward model $\hat\theta$ is the empirical minimizer of the Bradley-Terry loss on data from the SFT policy $\pi_1$, so with unlimited data it converges to $\theta^\circ$, the best approximation of the human reward on $\pi_1$'s distribution. But the policy gradient at step $i$ needs the best approximation $\theta^*$ on the current policy $\pi_i$'s distribution, and $\theta^\circ \neq \theta^*$ whenever $\pi_i \neq \pi_1$; therefore $\hat\theta$ is an inconsistent estimator of $\theta^*$, the gradient estimate built on it is inconsistent, and the whole pipeline can fail to reach the optimal policy no matter how much data it is fed. The paper's remedy is to importance-weight the reward-model loss by the ratio $\pi_i(a_w|s)\pi_i(a_l|s)/\pi_1(a_w|s)\pi_1(a_l|s)$, which re-targets the loss at $\pi_i$ without sampling new completions or labels, restoring consistency.

Load-bearing premise

The practical version of the method flattens the importance weights with $\eta = 0.001$ and $\alpha = 0.9$, which introduces a bias that the consistency theorem does not cover, so the real-world gains hinge on these flattened weights still being more accurate than unweighted training; the theory also assumes that human preference depends only on the two responses and not on which policy produced them.

Editorial extensions

If this is right

  • If the inconsistency argument is correct, any fixed-reward-model RLHF run carries an irreducible bias that grows with policy drift, and OCRM removes that bias without collecting new preference labels.
  • Iterating the correction (m = 2, 3, 5) keeps improving the gold win rate, with gains beginning to saturate by m = 5, so a few retraining rounds capture most of the available benefit.
  • The correction applies to any policy-gradient RLHF variant, including GRPO and RLOO, not only PPO, since it only changes how the reward model is trained.
  • Larger values of k (more policy updates per reward-model retrain) yield better KL-reward tradeoffs, suggesting the method works best as a coarse, staged correction rather than an every-step update.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the consistency claim holds, then online data collection is not logically required to keep a reward model honest: offline reweighting of existing preference data should be able to emulate the distribution shift, which could lower the cost of alignment pipelines that currently resample and relabel.
  • The same importance-weight correction could be applied to any downstream consumer of the reward model, such as rejection sampling or best-of-n selection, where distribution shift also distorts scores; the paper only demonstrates the PPO loop.
  • The flattening hyperparameters $\eta$ and $\alpha$ trade theoretical consistency for variance reduction; a data-dependent schedule that anneals toward $\eta = 1$ as the dataset grows could combine the consistency guarantee with the stability the experiments require.
  • The preference-invariance assumption is the part most likely to break in deployment; an immediate audit would be to compare human judgments on current-policy outputs with the old reward model's importance-weighted predictions.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper studies overoptimization in RLHF from a distribution-shift perspective. It argues that standard RLHF, which trains a reward model once on data from the SFT policy and then uses it to train later policies, produces an inconsistent estimate of the RM parameters under the shifted policy distribution and therefore an inconsistent policy gradient estimate, so that standard RLHF may fail to converge to the optimal policy even with unlimited data. The authors propose Off-Policy Corrected Reward Modeling (OCRM), which retrains the RM with importance weights that correct for the shift from the SFT policy to the current policy, without collecting new labels. They provide a Rademacher-complexity estimation error bound for the importance-weighted RM loss, present a didactic 2D example, and report experiments on TL;DR summarization and Alpaca-Farm using a gold reward model and GPT-4.1 nano feedback, showing that OCRM improves over PPO, DPO, WPO, and RLP-SPG baselines.

Significance. If the paper's theoretical claims are properly scoped, OCRM is a useful and practical contribution. The core idea of using importance weighting to correct the reward model for policy shift is clean, and the empirical evaluation is strong: the paper includes multiple base models (Pythia 1B/2.8B/6.9B, Qwen 2.5 1.5B), multiple seeds on the main summarization task, ablations of the two design choices (KL-reference change and value reset), sensitivity analyses for the new hyperparameters η, α, and k, a smaller-dataset study, and an additional evaluation with GPT-4.1 nano feedback. The Rademacher bound in Theorem 1 and its proof in Appendix A are standard and correct under the stated assumptions. The paper also provides code and detailed implementation notes, which strengthens reproducibility. However, the theoretical narrative in Section 4 overclaims: the paper's central non-convergence statement is not actually proven, and Theorem 1 bounds reward-model loss, not policy return or convergence. The practical method also introduces bias through flattened and relative importance weights, for which no theoretical guarantee is given.

major comments (4)
  1. [Section 4] The claim that standard RLHF 'generally do[es] not converge to the optimal policy' (Introduction and Abstract) is not established by the argument in Section 4. The section shows that the fixed RM minimizer θ_hat on π1 data is not the minimizer of the RM loss on the later policy πi, i.e., L^{πi}(θ_hat) ≥ L^{πi}(θ*). This is a statement about a surrogate loss. It does not imply that the policy gradient estimate is inconsistent, because (a) the Bradley-Terry reward is only identified up to action-independent, state-dependent additive shifts, which cancel in the advantage function; (b) since θ_H ∉ Θ, both θ_hat and θ* are misspecified surrogates, and a biased reward can still provide a gradient direction that improves the true objective; and (c) the paper provides no theorem or counterexample showing that gradient ascent with θ_hat fails to reach an optimal policy. The didactic experiment in Section 6.1 is suggestive but not a proof of the general claim. Please either prove the non-convergence claim under explicit assumptions or soften the abstract and introduction to say that the RM parameter estimate is inconsistent and the paper provides evidence that this can lead to stagnation.
  2. [Section 5.1 / Theorem 1] Theorem 1 bounds L^{πi}_RM(tilde_θ) - L^{πi}_RM(θ*), i.e., the reward-model loss gap, not a policy-return gap or a convergence statement. The paragraph after Eq. (a)-(b) says that the gap 'may be nonzero as NRM → ∞, demonstrating the benefit of our method,' but the benefit shown is only in the RM loss. The paper later concedes in Section 5.1 that 'this does not mean that RLHF with off-policy corrected RMs will necessarily converge to an optimal policy.' This concession is in tension with the stronger statements in the Introduction and Abstract. The theoretical contribution should be scoped consistently: OCRM provides a consistent estimate of θ*, the minimizer of the RM loss under πi, and this can improve the reward signal used by RLHF; it is not shown to provide consistent policy gradients or convergence to the optimal policy.
  3. [Section 5.2 and Table 3] The consistency theorem applies to the unflattened importance weight w(s, aw, al) = πi(aw|s)πi(al|s) / π1(aw|s)π1(al|s), but all language-model experiments use flattened, relative weights with η = 0.001 and α = 0.9. As the paper notes, flattening 'reintroduces some bias.' Therefore the deployed estimator is not the consistent estimator analyzed in Theorem 1, and no theoretical guarantee is provided for the biased version. The empirical results show that all tested η and α values improve over PPO, which supports the practical method, but the claim that OCRM 'provides a consistent estimate' should be restricted to the unflattened case, and the paper should state explicitly that the practical variant trades consistency for variance reduction.
  4. [Section 5, Eq. (5)] The derivation of the importance weight in Eq. (5) cancels the preference probability P(aw > al | s) from the numerator and denominator, which requires the human preference distribution over pairs to be invariant to the sampling policy. This is a substantive assumption: human annotators' preferences can depend on the reference distribution, and the offline dataset DRM contains pairs sampled under π1, not under πi. If the preference probability is policy-dependent, the cancellation fails. The paper should state this assumption explicitly and discuss its plausibility. The support condition πi(a|s) > 0 for all a is also not sufficient if the true data-generating process puts nonzero probability on pairs that are not in the support of π1's pair distribution.
minor comments (6)
  1. [Section 4] The phrase 'we know1 that L^{π1}_RM(θ_hat) → L^{π1}_RM(θ◦)' uses footnote 1 to refer to Theorem 2 in Appendix A, but the statement is pointwise consistency of the empirical minimizer, not the uniform bound; the citation is acceptable, but the wording could be clearer that convergence is in probability under standard regularity conditions.
  2. [Algorithm 1 (Appendix D)] In Algorithm 1, the notation 'R_{θ,i+1}(s, a)' and the variables 'a1' and 'a2' in the importance weight are not defined in the main text; please align this with the main-text notation R_{θ_i} and a_w, a_l.
  3. [Appendix A.1] In the proof of Theorem 2, the line 'ED[sup_θ (bR(θ) − L(θ))]' appears to be a typo for 'bL(θ)'; please correct this and check the surrounding display for consistency.
  4. [Figure 2] The legend in Figure 2 places 'Ours (m=3)' in the same list as the β values for PPO; since Ours uses β = 0.05, the plot would be clearer if the label indicated '(β = 0.05)' explicitly.
  5. [Section 5.2] The sentence 'In practice, we found that choosing a high k such that the new policy is trained until convergence on the current RM performs well' is supported only by single-seed experiments; please note this limitation where the statement is made.
  6. [Table 3 / Table 5] The hyperparameter table reports that α = 0.9 and α = 1.0 give nearly identical results, which Appendix E.2 explains by noting that relative IW is redundant with flattening; this explanation is useful and could be moved closer to the main-text table.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the proposed correction is a standard importance-weighting identity, and the main empirical claims are evaluated against a fixed external gold reward model.

full rationale

The paper's derivation chain is self-contained. The core identity in Eqs. (4)-(5) rewrites the pi_i reward-model loss as an importance-weighted expectation under pi_1; this is the standard change-of-measure identity from Shimodaira (2000), not an assumption that presupposes the conclusion. Theorem 1 is a conventional Rademacher generalization bound for the weighted empirical minimizer, and its proof in Appendix A builds on Theorem 2 with a bounded loss; it does not assume the consistency it claims. The Section 4 argument defines theta_o and theta* as the respective minimizers of L^{pi_1} and L^{pi_i}; the inequality L^{pi_i}(theta_o) >= L^{pi_i}(theta*) is true by definition of theta*, but this is a definitional comparison between two well-defined estimands, not a fitted quantity renamed as a prediction. The paper explicitly concedes in Section 5.1 that OCRM is not proven to converge to an optimal policy, so the strongest convergence claim is not being smuggled in through the theorem. Empirically, OCRM is evaluated against a fixed external gold reward model (Skywork-Reward-Llama-3.1-8B) and GPT-4.1-nano feedback, neither of which is fitted by the method. The only self-citation is Yamada et al. (2011) for relative importance weighting used with alpha=0.9 in the practical implementation; this variant is not part of the consistency theorem, and Table 3 shows results are insensitive to alpha, so it is not load-bearing. The skeptical concern that RM-loss inconsistency does not by itself imply policy-gradient inconsistency or non-convergence is better classified as a proof-gap or correctness risk than as circularity, since the paper does not derive the policy claim from an equation that already contains it.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

The central claim rests on standard importance-weighting assumptions, the Bradley-Terry model, and a set of practical hyperparameters. No new physical or conceptual entities are introduced.

free parameters (5)
  • η (importance weight flattening) = 0.001
    Chosen by hand to reduce variance of importance weights; reintroduces bias. Used in all LM experiments. Sensitivity shown in Table 3.
  • α (relative IW mixture) = 0.9
    Chosen by hand for relative importance weighting; the paper notes that using α=0.9 gives similar results to α=1.0 in a sensitivity check.
  • k (PPO updates between RM retrainings) = 100000
    Chosen as a practical trade-off; larger k improves final reward (Figure 4).
  • β (KL regularization strength) = 0.05 for Pythia 1B/2.8B, 0.1 for Qwen 2.5 1.5B and Pythia 6.9B GRPO
    Default from Huang et al. for OCRM; not optimized for OCRM. An ablation in Figure 2 shows the method still wins at the default β.
  • m (number of iterations) = 2 or 3
    Number of RM retraining rounds; more iterations improve results, with saturation at m=5 (Table 6).
assumptions (5)
  • domain assumption Bradley-Terry preference model (Eq. 1)
    Human preferences are modeled as logistic function of reward difference; this underlies the RM loss in Eq. (2).
  • domain assumption Preference distribution P(aw>al|s) is invariant to the policy generating responses
    Used in Eq. (5) to cancel the preference term from the importance weight, reducing it to the policy probability ratio.
  • domain assumption True human reward is not in the hypothesis class (θH ∉ Θ)
    Needed so that the best RM under π1 differs from the best RM under πi, giving non-vanishing loss gaps in Section 4.
  • standard math Bounded loss and bounded importance weights (lRM ≤ Cl, w ∈ [0,W])
    Required for the Rademacher complexity bound in Theorem 1; the paper notes bounded rewards suffice.
  • domain assumption Support condition: π_i(a|s) > 0 whenever π_1(a|s) > 0
    Required for importance weighting; the paper argues this holds for softmax-output LMs.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Off-Policy Corrected Reward Modeling for Reinforcement Learning from Human Feedback." pith.science (2026). https://pith.science/paper/TORMKZ2L

@misc{pith2026250715507,
  author       = {Pith},
  title        = {Pith review of: Off-Policy Corrected Reward Modeling for Reinforcement Learning from Human Feedback},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/TORMKZ2L}},
  note         = {Machine review of arXiv:2507.15507}
}
read the original abstract

Reinforcement Learning from Human Feedback (RLHF) allows us to train models, such as language models (LMs), to follow complex human preferences. In RLHF for LMs, we first train an LM using supervised fine-tuning, sample pairs of responses, obtain human feedback, and use the resulting data to train a reward model (RM). RL methods are then used to train the LM to maximize the reward given by the RM. As training progresses, the responses generated by the LM no longer resemble the responses seen by the RM during training, leading to the RM becoming inaccurate. The score given by the RM keeps increasing, but the learned behavior no longer matches the human preferences. This issue is known as overoptimization. We investigate overoptimization from the point of view of distribution shift and show that the shift results in an inconsistent estimate of the RM parameters, leading to an inconsistent estimate of the policy gradient. We propose Off-Policy Corrected Reward Modeling (OCRM), which iteratively off-policy corrects the RM using importance weighting, without requiring new labels or samples. This results in a more accurate RM, which empirically leads to an improved final policy. We validate our approach in experiments with summarization and chatbot datasets and show that it performs significantly better than standard RLHF methods and baselines. Our implementation is available at https://github.com/JohannesAck/OffPolicyCorrectedRewardModeling

Figures

Figures reproduced from arXiv: 2507.15507 by the authors.

Figure 1
Figure 1. Visualization of our approach in a 2D stateless task. On the top-left side, the true [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Our method and PPO with varied KL-regularization β. 6.2.1 Baseline comparison As baselines, we compare our proposed approach with PPO (Stiennon et al., 2020), DPO (Rafailov et al., 2023), WPO (Zhou et al., 2024), and Reinforcement Learning on Policy Synthetic Preference Generation (RLP-SPG) (Lang et al., 2024). RLP-SPG samples outputs from the policy and adds synthetic labels, which are used to retrain the RM. For R… view at source ↗
Figure 3
Figure 3. Training with different sizes of the preference-labeled dataset DRM. 0 20 40 60 80 −14 −12 −10 −8 DKL(π i |π 1 ) Gold Score k = 100, 000 k = 50, 000 k = 25, 000 k = 10, 000 [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗
Figures from the paper (4 more)
Figure 5
Figure 5. Figure 5: Comparison with baselines across three random seeds each with 95% confidence [PITH_FULL_IMAGE:figures/full_fig_p021_5.png]
Figure 6
Figure 6. Figure 6: Evaluation of our approach with different values for [PITH_FULL_IMAGE:figures/full_fig_p022_6.png]
Figure 7
Figure 7. Figure 7: Value loss during PPO training with k = 100, 000 for different random seeds. Note that during the early training stages the value function is still inaccurate, which might contribute to larger values for k performing better. E.3 k hyperparameter sensitivity We also inv…
Figure 8
Figure 8. Figure 8: Performance of DPO and WPO degrades when training for longer durations. [PITH_FULL_IMAGE:figures/full_fig_p024_8.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

43 extracted references · 28 canonical work pages

  1. [1]

    Back to Basics : Revisiting REINFORCE Style Optimization for Learning from Human Feedback in LLMs

    Arash Ahmadian, Chris Cremer, Matthias Gallé, Marzieh Fadaee, Julia Kreutzer, Olivier Pietquin, Ahmet Üstün, and Sara Hooker. Back to Basics : Revisiting REINFORCE Style Optimization for Learning from Human Feedback in LLMs . In ACL, 2024. URL https://aclanthology.org/2024.acl-long.662.pdf

    work page 2024

  2. [2]

    Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling

    Stella Biderman, Hailey Schoelkopf, Quentin Anthony, Herbie Bradley, Kyle O'Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, Aviya Skowron, Lintang Sutawika, and Oskar van der Wal. Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling . In ICML , 2023. URL https://proceedings.mlr.pre...

    work page 2023

  3. [3]

    Ralph Allan Bradley and Milton E. Terry. Rank Analysis of Incomplete Block Designs : I . The Method of Paired Comparisons . Biometrika, 39 0 (3/4): 0 324--345, 1952. URL https://www.jstor.org/stable/2334029

    arXiv 1952

  4. [4]

    Brown, Miljan Martic, Shane Legg, and Dario Amodei

    Paul Christiano, Jan Leike, Tom B. Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. In NeurIPS, 2017. URL https://proceedings.neurips.cc/paper_files/paper/2017/file/d5e2c0adad503c91f91df240d0cd4e49-Paper.pdf

    work page 2017

  5. [5]

    Reward Model Ensembles Help Mitigate Overoptimization

    Thomas Coste, Usman Anwar, Robert Kirk, and David Krueger. Reward Model Ensembles Help Mitigate Overoptimization . In ICLR, 2024. URL https://openreview.net/pdf?id=dcjtMYkpXx

    work page 2024

  6. [6]

    Fernando Hernandez-Garcia, Qingfeng Lan, Parash Rahman, A

    Shibhansh Dohare, J. Fernando Hernandez-Garcia, Qingfeng Lan, Parash Rahman, A. Rupam Mahmood, and Richard S. Sutton. Loss of plasticity in deep continual learning. Nature, 632 0 (8026): 0 768--774, 2024. URL https://www.nature.com/articles/s41586-024-07711-7

    work page 2024

  7. [7]

    RLHF Workflow : From Reward Modeling to Online RLHF

    Hanze Dong, Wei Xiong, Bo Pang, Haoxiang Wang, Han Zhao, Yingbo Zhou, Nan Jiang, Doyen Sahoo, Caiming Xiong, and Tong Zhang. RLHF Workflow : From Reward Modeling to Online RLHF . TMLR, 2024. URL https://openreview.net/pdf?id=a13aYUU9eU

    work page 2024

  8. [8]

    The Llama 3 Herd of Models , 2024

    Abhimanyu Dubey, Abhinav Jauhri, and et al. The Llama 3 Herd of Models , 2024. URL http://arxiv.org/abs/2407.21783

    arXiv 2024

Show all 43 references
  1. [9]

    Hashimoto

    Yann Dubois, Xuechen Li, Rohan Taori, Tianyi Zhang, Ishaan Gulrajani, Jimmy Ba, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Alpacafarm: A simulation framework for methods that learn from human feedback. In NeurIPS, 2023. URL https://proceedings.neurips.cc/paper_f...

    2023

  2. [10]

    Heller, Stephen Robert Pfohl, Deepak Ramachandran, Peter Shaw, and Jonathan Berant

    Jacob Eisenstein, Chirag Nagpal, Alekh Agarwal, Ahmad Beirami, Alexander Nicholas D'Amour, Krishnamurthy Dj Dvijotham, Adam Fisch, Katherine A. Heller, Stephen Robert Pfohl, Deepak Ramachandran, Peter Shaw, and Jonathan Berant. Helping or Herding ? Reward Model Ensembles Mitig...

    2024

  3. [11]

    The Perils of Optimizing Learned Reward Functions : Low Training Error Does Not Guarantee Low Regret , 2025

    Lukas Fluri, Leon Lang, Alessandro Abate, Patrick Forré, David Krueger, and Joar Skalse. The Perils of Optimizing Learned Reward Functions : Low Training Error Does Not Guarantee Low Regret , 2025. URL http://arxiv.org/abs/2406.15753

    2025 arXiv

  4. [12]

    Scaling Laws for Reward Model Overoptimization

    Leo Gao, John Schulman, and Jacob Hilton. Scaling Laws for Reward Model Overoptimization . In ICML, 2023. URL https://proceedings.mlr.press/v202/gao23h/gao23h.pdf

    2023

  5. [13]

    DeBERTa : Decoding -enhanced BERT with Disentangled Attention

    Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. DeBERTa : Decoding -enhanced BERT with Disentangled Attention . In ICLR, 2021. URL https://openreview.net/pdf?id=XPZIaotutsD

    2021

  6. [14]

    Genetic Algorithms and evolution strategies: Similarities and differences

    Frank Hoffmeister and Thomas Bäck. Genetic Algorithms and evolution strategies: Similarities and differences. In Hans-Paul Schwefel and Reinhard Männer (eds.), Parallel Problem Solving from Nature , pp.\ 455--469. Springer, 1991. URL https://link.springer.com/chapter/10.1007/B...

    1991 doi

  7. [15]

    The N + Implementation Details of RLHF with PPO : A Case Study on TL ; DR Summarization

    Shengyi Huang, Michael Noukhovitch, Arian Hosseini, Kashif Rasul, Weixun Wang, and Lewis Tunstall. The N + Implementation Details of RLHF with PPO : A Case Study on TL ; DR Summarization . In COLM, 2024. URL https://openreview.net/pdf?id=kHO2ZTa8e3

    2024

  8. [16]

    Semantic Uncertainty : Linguistic Invariances for Uncertainty Estimation in Natural Language Generation

    Lorenz Kuhn, Yarin Gal, and Sebastian Farquhar. Semantic Uncertainty : Linguistic Invariances for Uncertainty Estimation in Natural Language Generation . In ICLR , 2023. URL https://openreview.net/pdf?id=VD-AYtP0dve

    2023

  9. [17]

    Nathan Lambert, Valentina Pyatkin, Jacob Morrison, L. J. Miranda, Bill Yuchen Lin, Khyathi Chandu, Nouha Dziri, Sachin Kumar, Tom Zick, Yejin Choi, Noah A. Smith, and Hannaneh Hajishirzi. RewardBench : Evaluating Reward Models for Language Modeling , 2024. URL http://arxiv.org...

    2024 arXiv

  10. [18]

    Fine- Tuning Language Models with Reward Learning on Policy

    Hao Lang, Fei Huang, and Yongbin Li. Fine- Tuning Language Models with Reward Learning on Policy . In NAACL, 2024. URL https://aclanthology.org/2024.naacl-long.75.pdf

    2024

  11. [19]

    Policy Evaluation for Reinforcement Learning from Human Feedback : A Sample Complexity Analysis

    Zihao Li, Xiang Ji, Minshuo Chen, and Mengdi Wang. Policy Evaluation for Reinforcement Learning from Human Feedback : A Sample Complexity Analysis . In AISTATS , 2024. URL https://proceedings.mlr.press/v238/li24l.html

    2024

  12. [20]

    Skywork- Reward : Bag of Tricks for Reward Modeling in LLMs , 2024

    Chris Yuhao Liu, Liang Zeng, Jiacai Liu, Rui Yan, Jujie He, Chaojie Wang, Shuicheng Yan, Yang Liu, and Yahui Zhou. Skywork- Reward : Bag of Tricks for Reward Modeling in LLMs , 2024. URL http://arxiv.org/abs/2410.18451

    2024 arXiv

  13. [21]

    ProRL : Prolonged Reinforcement Learning Expands Reasoning Boundaries in Large Language Models , 2025

    Mingjie Liu, Shizhe Diao, Ximing Lu, Jian Hu, Xin Dong, Yejin Choi, Jan Kautz, and Yi Dong. ProRL : Prolonged Reinforcement Learning Expands Reasoning Boundaries in Large Language Models , 2025. URL http://arxiv.org/abs/2505.24864

    2025 arXiv

  14. [22]

    Decoupled Weight Decay Regularization

    Ilya Loshchilov and Frank Hutter. Decoupled Weight Decay Regularization . In ICLR, 2019. URL https://openreview.net/pdf?id=Bkg6RiCqY7

    2019

  15. [23]

    Foundations of Machine Learning

    Mehryar Mohri, Afshin Rostamizadeh, and Ameet Talwalkar. Foundations of Machine Learning. MIT Press, 2nd edition, 2018

    2018

  16. [24]

    The Primacy Bias in Deep Reinforcement Learning

    Evgenii Nikishin, Max Schwarzer, Pierluca D'Oro, Pierre-Luc Bacon, and Aaron Courville. The Primacy Bias in Deep Reinforcement Learning . In ICML , 2022. URL https://proceedings.mlr.press/v162/nikishin22a/nikishin22a.pdf

    2022

  17. [25]

    GPT -4 Technical Report , 2024

    OpenAI, Josh Achiam, Steven Adler, and et al. GPT -4 Technical Report , 2024. URL http://arxiv.org/abs/2303.08774

    2024 arXiv

  18. [26]

    Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and...

    2022

  19. [27]

    Qwen2.5 Technical Report , 2025

    Qwen, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le ...

    2025 arXiv

  20. [28]

    Manning, and Chelsea Finn

    Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, and Chelsea Finn. Direct Preference Optimization : Your Language Model is Secretly a Reward Model . In NeurIPS , 2023. URL https://papers.neurips.cc/paper_files/paper/2023/file/a85b405ed65c64...

    2023

  21. [29]

    WARM : On the Benefits of Weight Averaged Reward Models , 2024

    Alexandre Ramé, Nino Vieillard, Léonard Hussenot, Robert Dadashi, Geoffrey Cideron, Olivier Bachem, and Johan Ferret. WARM : On the Benefits of Weight Averaged Reward Models , 2024. URL http://arxiv.org/abs/2401.12187

    2024 arXiv

  22. [30]

    High- Dimensional Continuous Control Using Generalized Advantage Estimation

    John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. High- Dimensional Continuous Control Using Generalized Advantage Estimation . In ICLR , 2016. URL http://arxiv.org/abs/1506.02438

    2016 arXiv

  23. [31]

    Proximal Policy Optimization Algorithms , 2017

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal Policy Optimization Algorithms , 2017. URL http://arxiv.org/abs/1707.06347

    2017 arXiv

  24. [32]

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. DeepSeekMath : Pushing the Limits of Mathematical Reasoning in Open Language Models , 2024. URL http://arxiv.org/abs/2402.03300

    2024 arXiv

  25. [33]

    Improving predictive inference under covariate shift by weighting the log-likelihood function

    Hidetoshi Shimodaira. Improving predictive inference under covariate shift by weighting the log-likelihood function. Journal of Statistical Planning and Inference, 90 0 (2): 0 227--244, 2000. URL https://www.sciencedirect.com/science/article/pii/S0378375800001154

    2000

  26. [34]

    The Importance of Online Data : Understanding Preference Fine -tuning via Coverage

    Yuda Song, Gokul Swamy, Aarti Singh, Drew Bagnell, and Wen Sun. The Importance of Online Data : Understanding Preference Fine -tuning via Coverage . In NeurIPS, 2024. URL https://proceedings.neurips.cc/paper_files/paper/2024/file/16c628ab12dc4caca8e7712affa6c767-Paper-Conference.pdf

    2024

  27. [35]

    Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul Christiano

    Nisan Stiennon, Long Ouyang, Jeff Wu, Daniel M. Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul Christiano. Learning to summarize from human feedback. In NeurIPS , 2020. URL https://proceedings.neurips.cc/paper_files/paper/2020/file/1f89885d556929e98d3ef...

    2020

  28. [36]

    Reinforcement Learning : An Introduction (2nd edition)

    Richard S Sutton and Andrew G Barto. Reinforcement Learning : An Introduction (2nd edition) . MIT Press, 2018. URL http://incompleteideas.net/book/the-book-2nd.html

    2018

  29. [37]

    Statistical Learning Theory

    Vladimir Vapnik. Statistical Learning Theory. Wiley, 1998

    1998

  30. [38]

    Williams

    Ronald J. Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine Learning, 8 0 (3-4): 0 229--256, 1992. URL http://link.springer.com/10.1007/BF00992696

    1992 doi

  31. [39]

    Is DPO Superior to PPO for LLM Alignment ? A Comprehensive Study

    Shusheng Xu, Wei Fu, Jiaxuan Gao, Wenjie Ye, Weilin Liu, Zhiyu Mei, Guangju Wang, Chao Yu, and Yi Wu. Is DPO Superior to PPO for LLM Alignment ? A Comprehensive Study . In ICML, 2025. URL https://proceedings.mlr.press/v235/xu24h.html

    2025

  32. [40]

    Relative Density - Ratio Estimation for Robust Distribution Comparison

    Makoto Yamada, Taiji Suzuki, Takafumi Kanamori, Hirotaka Hachiya, and Masashi Sugiyama. Relative Density - Ratio Estimation for Robust Distribution Comparison . In NeurIPS , 2011. URL https://papers.neurips.cc/paper_files/paper/2011/file/d1f255a373a3cef72e03aa9d980c7eca-Paper.pdf

    2011

  33. [41]

    Improving Reinforcement Learning from Human Feedback with Efficient Reward Model Ensemble , 2024

    Shun Zhang, Zhenfang Chen, Sunli Chen, Yikang Shen, Zhiqing Sun, and Chuang Gan. Improving Reinforcement Learning from Human Feedback with Efficient Reward Model Ensemble , 2024. URL http://arxiv.org/abs/2401.16635

    2024 arXiv

  34. [42]

    WPO : Enhancing RLHF with Weighted Preference Optimization

    Wenxuan Zhou, Ravi Agrawal, Shujian Zhang, Sathish Reddy Indurthi, Sanqiang Zhao, Kaiqiang Song, Silei Xu, and Chenguang Zhu. WPO : Enhancing RLHF with Weighted Preference Optimization . In EMNLP 2024, 2024. URL https://aclanthology.org/2024.emnlp-main.475.pdf

    2024

  35. [43]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.