Pith. sign in

REVIEW 5 major objections 7 minor 59 references

Towards Reliable, Uncertainty-Aware Alignment

T0 review · 5 major / 7 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper claims that adding reward-model variance to the policy objective provably lowers the chance that RLHF produces a policy worse than the reference policy, and backs this with theory and LLM experiments.

desk verdict Plausible variance-aware RLHF idea, but the theoretical guarantee is invalid for real policies—simplex ignored and concentration bound misstated—so the headline claim doesn't stand. read the letter →

arxiv 2507.15906 v1 pith:5HKMIHUS submitted 2025-07-21 cs.LG cs.AI

Debangshu Banerjee , Kintan Saha , Aditya Gopalan This is my paper
classification cs.LGcs.AI
keywords RLHFrewardmodeluncertaintyvariance-awarepolicyoptimizationrobustalignmentensemblemodelsoveroptimizationproximalreferencerisk
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that a major reason RLHF alignment is unreliable is noise in the reward model, and that accounting for that noise makes alignment safer. It first shows empirically that ten identically configured reward models trained on the same preference data can disagree substantially on the same prompt-response pair. The authors then formulate a variance-aware policy optimization problem in which deviations from the reference policy are penalized by the reward covariance, and prove that, under a Gaussian noise model, the resulting policy is no more likely than the variance-unaware policy to score below the reference policy on the true reward. The practical algorithm is a variance-aware PPO objective, and the reported experiments show lower reward variance and statistically lower empirical risk of underperforming the reference policy across several LLM/reward-model combinations.

What carries the argument

The load-bearing object is a covariance-weighted quadratic trust region. The vanilla constraint $\|\pi-\pi_0\|_2^2\le\varepsilon$ is replaced by $\|\pi-\pi_0\|_\Sigma^2\le\tilde\varepsilon$, where $\Sigma=\mathrm{diag}(\sigma^2(x,y))$ is the covariance of reward estimates, so the update is pulled back toward $\pi_0$ most strongly on prompt-response pairs whose estimated rewards are most uncertain. In closed form, the updates are $\pi_1=\pi_0+\sqrt{\varepsilon/\hat R^\top\hat R}\,\hat R$ and $\pi_2=\pi_0+\sqrt{\tilde\varepsilon/\hat R^\top\Sigma^{-1}\hat R}\,\Sigma^{-1}\hat R$, making $\pi_2$ respond to the reward gradient after rescaling by inverse variance. Theorem 2.7 gives the matching distributional form $\pi^*(y|x)\propto\pi_0(y|x)\exp(\hat R(x,y)/(\beta\sigma^2(x,y)))$, connecting the method to a Sharpe-ratio-style reward-to-variability trade-off.

What would settle it

Run the closed-form updates (2)-(3) on a small finite problem, project $\pi_1$ and $\pi_2$ onto the probability simplex, and Monte-Carlo-sample $\hat R\sim\mathcal N(r^*,\Sigma)$; if the variance-aware policy's measured underperformance probability $P(\hat\pi_2^\top r^*\le\pi_0^\top r^*)$ is not below the vanilla one, the guarantee fails for actual policies.

Watch

Extended reading notes

Core claim

The central claim is that using reward-model variance in the policy update reduces the probability that alignment produces a policy worse than the reference policy. Let $d=\pi-\pi_0$ and model the estimated reward as $\hat R\sim\mathcal N(r^*,\Sigma)$. The vanilla policy is $\pi_1=\arg\max_{\|\pi-\pi_0\|_2^2\le\varepsilon}\pi^\top\hat R$ and the variance-aware policy is $\pi_2=\arg\max_{\|\pi-\pi_0\|_\Sigma^2\le\tilde\varepsilon}\pi^\top\hat R$. Theorem 2.4 asserts $P(\pi_2^\top r^*\le\pi_0^\top r^*)\le P(\pi_1^\top r^*\le\pi_0^\top r^*)$ when $\tilde\varepsilon=\lambda_{\min}(\Sigma)\varepsilon$. The proof reduces both underperformance events to Gaussian tail comparisons, using Cauchy-Schwarz to show the variance-aware tail is no larger.

Load-bearing premise

The load-bearing premise is that reward noise is Gaussian and that policies can be treated as arbitrary vectors inside the box $[-2,2]^{|\mathcal X||\mathcal Y|}$, while real language-model policies are probability distributions; the paper itself also flags that the practical method depends on reasonably calibrated variance estimates.

Editorial extensions

If this is right

  • If the Gaussian noise model holds, variance-aware alignment is a robustness improvement: for an equally sized feasible region it never increases the probability that the policy falls below the reference policy's true reward.
  • The variance-aware PPO objective used in the paper, $L(\theta)=\sum_{x,y}[\hat r(x,y)-\sigma^2(x,y)\log(\pi_\theta(y|x)/\pi_0(y|x))]$, should produce more reproducible alignment runs; the reported F-tests reject equal variance between vanilla and variance-aware policies in every configuration.
  • The benefit grows with the spread of reward variances across prompt-response pairs; when the prompted reward range is narrowed from $[1,100]$ to $[1,10]$, the measured variance reduction shrinks accordingly.
  • Theorem 2.7 yields a closed-form uncertainty-aware update rule: keep the reference policy's relative preferences and exponentially up-weight responses with high reward-to-variance ratio.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The theory's optimization domain is the box $[-2,2]^{|\mathcal X||\mathcal Y|}$, not the probability simplex, so the numerical guarantee in Theorem 2.4 should be read as a statement about the update mechanism rather than as a literal bound for language-model policies; a simplex-constrained version is the natural follow-up.
  • The same self-normalized concentration argument behind Theorem 2.2 should extend to heavier-tailed reward noise, so the risk-reduction claim is plausibly testable beyond Gaussian perturbations.
  • Because the variance-aware step is ridge-style shrinkage toward the reference policy, the idea can be transplanted into other trust-region alignment methods by swapping the KL or $\ell^2$ metric for the covariance-weighted metric defined here.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 7 minor

Summary. The paper addresses reward-model uncertainty in RLHF. It first presents empirical evidence that identically configured reward models disagree on the same prompt-response pairs (Appendix B). It then introduces a Gaussian noise model for the reward estimate bR with covariance Σ, states a high-probability bound (Theorem 2.2) meant to justify a variance-penalized surrogate objective, and proves (Theorem 2.4) that the policy obtained from a covariance-constrained program is less likely than the policy from a norm-constrained program to have true reward below that of a reference policy π0. Section 3 reports PPO-style fine-tuning experiments with GPT-2, Mistral-7B, and Qwen2.5-0.5B under ensemble and prompted reward models, claiming statistically significant variance reduction and risk reduction relative to vanilla PPO. The abstract claims that variance-aware policy optimization provably reduces the risk of outputting a worse policy than the default.

Significance. If the risk-reduction guarantee were valid for actual policies, this would be a useful theoretical complement to ensemble- and uncertainty-based RLHF methods and would justify a simple variance-weighted regularizer. The empirical study of reward-model disagreement (Appendix B) is a solid, reusable observation, and the experimental protocol with 80 policies per configuration and explicit F-, t-, and Z-tests is a genuine strength, as is the paper's candid Limitations section. However, the central Theorem 2.4 is proved for closed-form vectors that need not be probability distributions, Theorem 2.2's concentration bound is false in the stated form, and the prompted-reward experiments evaluate against a 'true reward' computed from the same model that produces the training signal. The paper therefore does not currently establish its headline claim.

major comments (5)
  1. [Section 2, Eqs. (2)-(3); Appendix A (Theorem 2.4 proof)] Theorem 2.4's proof maximizes the linear objective bR^T π over R^{|X||Y|} subject only to the norm constraints in Eqs. (2)-(3), but a policy must satisfy π(y|x) ≥ 0 and sum_y π(y|x) = 1 for every prompt x. The closed-form solutions derived in Appendix A, π1 = π0 + sqrt(ε/(bR^T bR))·bR and π2 = π0 + sqrt(ε~/(bR^T Σ^{-1} bR))·Σ^{-1}bR, are boundary points of an l2- or Σ-ellipsoid in R^n; they generally have negative entries and do not preserve per-prompt normalization, so they are not probability distributions and hence are not policies. The probabilities P(π1^T r* ≤ π0^T r*) and P(π2^T r* ≤ π0^T r*) are therefore computed for infeasible pseudopolicies, and the abstract's claim that variance-aware policy optimization provably reduces the risk of outputting a worse policy than the default is not established for actual LLM policies. Once the simplex constraints are imposed, the maximizers of Eqs. (2)-(3) are no longer the closed forms used in the proof, and the Gaussian comparison between bR^T r* and bR^T Σ^{-1} r* does not follow.
  2. [Section 2, Theorem 2.2] The concentration bound in Theorem 2.2 is false as stated. With n = |X||Y| and β = √n, the theorem claims the inequality holds with probability at least 1 - e^{-1} ≈ 0.632, whereas the proof's sufficient condition is ||bR - r*||_{Σ^{-1}} ≤ β; since ||bR - r*||²_{Σ^{-1}} is chi-square with n degrees of freedom, P(chi²_10 ≤ 10) ≈ 0.56 and P(chi²_n ≤ n) → 0.5 as n → ∞, so the claimed probability level is not attained at β = √n or in an interval around it. A correct statement would require a radius of order sqrt(n + 2√(n ln(1/δ)) + 2 ln(1/δ)) rather than sqrt(n ln(1/δ)). Because this theorem is the stated justification for the surrogate objective in Eq. (1), the theoretical motivation for the variance-aware objective is unsupported.
  3. [Section 3, Algorithm 1 and Eq. (5)] No theorem connects the implemented algorithm to the policy whose risk is bounded. Algorithm 1 and Eq. (5) optimize L(θ) = Σ_{x,y} (bR(x,y) - σ²(x,y) log(πθ/π0)), which is not the Lagrangian of the constrained problems in Eqs. (2)-(3) analyzed in Theorem 2.4; the closed-form policy π2 is never computed or optimized in the experiments. The only theoretical result resembling the implemented objective is Theorem 2.7's exponential-tilt policy π*(y|x) ∝ π0(y|x) exp(bR(x,y)/(βσ²(x,y))) (with β = 1), but no risk guarantee is proved for that policy. The experimental results in Section 3 therefore cannot be read as confirming the provable risk-reduction claim in the abstract.
  4. [Section 3.1, Algorithms 8-9] In the prompted-reward configurations, the true reward used for evaluation is r* = (a + b)/2, the midpoint of the interval returned by the same prompted model that supplies the noisy training reward bR ~ Uniform(a, b) and the variance (b - a)²/12. The training signal, the uncertainty estimate, and the ground-truth evaluation reward are thus all derived from a single LLM judgment, so the variance and risk reductions reported in Tables 1 and 3 partly measure alignment with that judge's interval statistics rather than with an independent ground truth. The ensemble configurations for GPT-2, which evaluate against the external FsfairX-LLaMA3-RM reward model, are not affected by this issue and should be emphasized accordingly.
  5. [Table 3] The significance tests in Table 3 are not reproducible from the reported numbers. For the QWEN-GEMINI-1.5-FLASH and QWEN-DEEPSEEK rows (empirical risk 0.24 vs. 0.29, n = 80 per group), a pooled two-proportion Z-test gives a one-sided p-value of about 0.24 rather than the reported 0.038 and 0.042; the MISTRAL-GEMINI-1.5-FLASH row (0.41 vs. 0.48) similarly yields p ≈ 0.21 rather than 0.028. The paper should specify the exact test statistic, the integer counts used, and whether any correction was applied for the seven simultaneous comparisons.
minor comments (7)
  1. [Section 3, risk definition] The displayed definition of empirical risk, P(bR ≤ π0^T r*), appears to involve the noisy reward bR rather than the policy's reward; this is presumably a typo and should be corrected.
  2. [Table 3] The column header 'polies' should read 'policies'.
  3. [Algorithm 1 and Appendix D] Algorithm 1 is labeled 'PPO', but the update it shows is a plain KL-penalized policy gradient without importance-ratio clipping, advantage estimation, or a value function; the pseudocode should be reconciled with the TRL-based PPO implementation referenced in Appendix D.
  4. [Section 2, Theorem 2.4 statement and proof] The scaling ε~ = λmin(Σ)·ε is motivated by nesting the feasible sets, but in the proof the step-size factors cancel, and the risk inequality holds for any positive ε and ε~ (within the unconstrained model used in the proof); the framing of the theorem should be corrected to state what the scaling actually does.
  5. [Eqs. (5)-(6) and Tables 9-10] The written objectives in Eqs. (5)-(6) use KL penalties with coefficient 1, while the reported 'Initial KL Coeff' values are 0.2 (vanilla) and 0.05 (variance-aware); the relation between the displayed equations and the implemented penalties should be stated.
  6. [Theorem 2.2 and Eqs. (1)-(3)] Theorem 2.2 bounds the supremum over the box d ∈ [-2,2]^{|X||Y|}, whereas Eqs. (2)-(3) impose no box constraints on π; the relationship between these feasible sets and the surrogate objective (1) should be clarified.
  7. [Table 4] The column 'Variance of std dev. across prompts-responses' does not state whether the variance is taken over the evaluation set, over ensemble heads, or over repeated samples; this should be specified.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity found: Theorem 2.4 is a self-contained algebraic consequence of the Gaussian noise model, and the experimental risk comparisons are not identities forced by construction.

full rationale

Walking the derivation chain, I find no step where a claimed prediction or first-principles result is equivalent to its own inputs by construction. Theorem 2.2 is derived from Assumption 2.1 via self-normalized Gaussian bounds and Cauchy–Schwarz; the proposed surrogate in Eq. (1) is motivated as a high-probability lower bound, not assumed to be the true objective. Theorem 2.4 solves the constrained problems in Eqs. (2) and (3) in closed form, and the risk comparison reduces to the sign events bR^T r* <= 0 and bR^T Sigma^{-1} r* <= 0. The distributions of these linear forms follow from the stated Gaussian model, and Cauchy–Schwarz yields the claimed ordering of tail probabilities. The scaling eps~ = lambda_min(Sigma) eps is not load-bearing: the underperformance events are independent of the feasible-region radius, so the risk reduction is not caused by merely shrinking the variance-aware feasible set. In the prompted-reward experiments, defining r* = (a+b)/2 and sampling bR ~ U[a,b] is a modeling choice, but the vanilla and variance-aware policies are both trained on the same sampled rewards and compared under this r*; the observed lower empirical risk is a statistical outcome, not a definitional identity. The only external derivation citation (Rafailov et al. for Theorem 2.7) is standard and not self-referential. The main caveats—that Eqs. (2)–(3) drop the simplex constraint on policies and that Algorithm 1's objective in Eq. (5) is not shown to be the Lagrangian of Eq. (3)—are validity and generalization gaps rather than circularity: they affect whether the theorem applies to the evaluated policies, but they do not make the theorem's internal derivation self-referential. Therefore the circularity score is 0.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The theory relies on a Gaussian noise model with known covariance and on unconstrained d-vectors that do not respect the probability simplex. The experiments introduce a self-defined ground truth for prompted rewards and hand-chosen regularization coefficients, which are the main extra commitments beyond the stated math.

free parameters (3)
  • ε~ = λmin(Σ)·ε scaling = λmin(Σ)·ε
    Chosen in Theorem 2.4 to make the variance-aware feasible set a subset of the vanilla one; the theorem's inequality depends on this choice.
  • β (Lagrange multiplier / KL coefficient) = 1 in Eq (5); 0.2/0.05/0.3 etc. in PPO tables
    Controls strength of the regularization; set by hand, and differs between vanilla and variance-aware runs in Tables 9 and 10.
  • ε (trust-region radius)
    Abstract constraint radius in Eqs (2)-(3); never specified in experiments.
assumptions (4)
  • domain assumption Assumption 2.1: reward noise is Gaussian, independent, with known per-pair variance Σ.
    Unrealistic for real reward models, and load-bearing for Theorems 2.2 and 2.4.
  • ad hoc to paper Unconstrained d-vector model ignores the probability simplex.
    Eqs (2)-(4) solve over all real d in [-2,2]^n, yielding policies that can have negative probabilities; this is necessary for the closed-form comparison in Theorem 2.4 but invalid for LLMs.
  • standard math Self-normalized Gaussian tail bound with P(||Z|| ≤ β) ≥ 1 - exp(-n/β^2).
    Invoked in the proof of Theorem 2.2; this bound is not a valid concentration inequality for standard Gaussians (e.g., n=10, β=√10).
  • ad hoc to paper Prompted interval midpoint is the ground-truth reward.
    In Section 3.1 and Algorithm 9, r* is defined as (a+b)/2 from the same model that supplies the interval, so no external preference signal is used.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Towards Reliable, Uncertainty-Aware Alignment." pith.science (2026). https://pith.science/paper/5HKMIHUS

@misc{pith2026250715906,
  author       = {Pith},
  title        = {Pith review of: Towards Reliable, Uncertainty-Aware Alignment},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/5HKMIHUS}},
  note         = {Machine review of arXiv:2507.15906}
}
read the original abstract

Alignment of large language models (LLMs) typically involves training a reward model on preference data, followed by policy optimization with respect to the reward model. However, optimizing policies with respect to a single reward model estimate can render it vulnerable to inaccuracies in the reward model. We empirically study the variability of reward model training on open-source benchmarks. We observe that independently trained reward models on the same preference dataset can exhibit substantial disagreement, highlighting the instability of current alignment strategies. Employing a theoretical model, we demonstrate that variability in reward model estimation can cause overfitting, leading to the risk of performance degradation. To mitigate this risk, we propose a variance-aware policy optimization framework for preference-based alignment. The key ingredient of the framework is a new policy regularizer that incorporates reward model variance estimates. We show that variance-aware policy optimization provably reduces the risk of outputting a worse policy than the default. Experiments across diverse LLM and reward model configurations confirm that our approach yields more stable and robust alignment than the standard (variance-unaware) pipeline.

Figures

Figures reproduced from arXiv: 2507.15906 by the authors.

Figure 1
Figure 1. Reward scores assigned by 10 reward models on the same prompt-response pair. The reward models are identical in that they are trained independently on the same dataset, with the same hyperparame￾ters and number of epochs. Despite this, we see a wide variation in the score assigned by each model. Challenges of Reward Model Reliability Reward models in RLHF can be inher￾ently noisy, posing challenges for robust alignm… view at source ↗
Figure 3
Figure 3. plots the expected return P3 i=1 πir ∗ i over 1000 samples Rbi ∼ N (r ∗ i , σ2 i ), for i = 1, 2, 3. Policies learned without variance information (cyan distribution) exhibit a 39% chance of underperforming the initial uniform policy, whereas this probability drops to 0.15% when variance is incorporated (red distribution). The base policy is a uniform distribution over the three arms and is marked as a purple vertic… view at source ↗
Figure 4
Figure 4. Distribution of policy returns under different reward variance settings. In both cases, the true reward vector r ∗ is fixed, and reward estimates Rb are sampled from a multivariate Gaussian with varying covariance matrices. While the variance-aware policy consistently exhibits lower return variance, the relative gain is substantially higher in the high-variability setting, highlighting its robustness under uncertain… view at source ↗
Figures from the paper (4 more)
Figure 5
Figure 5. Figure 5: True empirical reward distributions for GPT-2 fine-tuned with a custom ensemble. Shown are policies trained with the vanilla method, variance-aware method, and the reference policy. The true reward r ∗ is based on a larger LLAMA-3-8B model. Model Reward Model Vanilla P…
Figure 6
Figure 6. Figure 6: Empirical reward distributions for MISTRAL and QWEN models fine-tuned using prompted re￾ward models: GEMINI-1.5-FLASH, GEMINI-2.0-FLASH, and DEEPSEEK-V3. Results are shown for vanilla policies, variance-aware policies, and reference policies. The noisy reward used duri…
Figure 7
Figure 7. Figure 7: (e). Fine-tuning with these reward distributions reveals that as variance vari￾ability decreases, the benefit of the variance-aware method diminishes—evidenced by a declining variance ratio compared to the vanilla method. Results are summarized in [PITH_FULL_IMAGE:fig…
Figure 8
Figure 8. Figure 8: (Top Row) Distribution of sample variances in the rewards assigned to accepted responses. Each variance is computed across the outputs of 10 independently trained reward models. The median sample variance exceeds 3.81 in all datasets, with values reaching as high as 10…

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

59 extracted references · 13 canonical work pages

  1. [1]

    The claude 3 model family: Opus, sonnet, haiku

    Anthropic AI. The claude 3 model family: Opus, sonnet, haiku. https://www-cdn.anthropic.com/de8ba9b01c9ab7cbabf5c33b80b7bbc618857627/Model_Card_Claude_3.pdf, 2024 a

    work page 2024

  2. [2]

    The llama 3 herd of models, 2024 b

    Meta AI. The llama 3 herd of models, 2024 b . URL https://arxiv.org/abs/2407.21783

    arXiv 2024

  3. [3]

    Concrete problems in ai safety

    Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Man \'e . Concrete problems in ai safety. arXiv preprint arXiv:1606.06565, 2016

    arXiv 2016

  4. [4]

    Training a helpful and harmless assistant with reinforcement learning from human feedback

    Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862, 2022 a

    arXiv 2022

  5. [5]

    Constitutional ai: Harmlessness from ai feedback

    Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. Constitutional ai: Harmlessness from ai feedback. arXiv preprint arXiv:2212.08073, 2022 b

    arXiv 2022

  6. [6]

    Deep reinforcement learning from human preferences

    Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. Advances in neural information processing systems, 30, 2017

    2017

  7. [7]

    Reward model ensembles help mitigate overoptimization

    Thomas Coste, Usman Anwar, Robert Kirk, and David Krueger. Reward model ensembles help mitigate overoptimization. arXiv preprint arXiv:2310.02743, 2023

    arXiv 2023

  8. [8]

    Ultrafeedback: Boosting language models with high-quality feedback

    Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao, Wei Zhu, Yuan Ni, Guotong Xie, Zhiyuan Liu, and Maosong Sun. Ultrafeedback: Boosting language models with high-quality feedback. arXiv preprint arXiv:2310.01377, 2023

    arXiv 2023

Show all 59 references
  1. [9]

    Suphavadeeprasit

    Luigi Daniele. Suphavadeeprasit. Amplify-Instruct: Synthetically Generated Diverse Multi-turn Conversations for Effecient LLM Training. arXiv preprint arXiv:(coming soon), 2023

    2023

  2. [10]

    Gemini 2.0 flash: Next-generation multimodal ai model

    Google DeepMind. Gemini 2.0 flash: Next-generation multimodal ai model. https://cloud.google.com/vertex-ai/generative-ai/docs/models/gemini/2-0-flash, 2024. Accessed: 2025-05-08

    2024

  3. [11]

    Deepseek-v3 technical report

    DeepSeek-AI. Deepseek-v3 technical report. ArXiv, abs/2412.19437, 2024. URL https://api.semanticscholar.org/CorpusID:275118643

    2024 arXiv

  4. [12]

    RAFT : Reward ranked finetuning for generative foundation model alignment

    Hanze Dong, Wei Xiong, Deepanshu Goyal, Yihan Zhang, Winnie Chow, Rui Pan, Shizhe Diao, Jipeng Zhang, KaShun SHUM, and Tong Zhang. RAFT : Reward ranked finetuning for generative foundation model alignment. Transactions on Machine Learning Research, 2023. ISSN 2835-8856. URL ht...

    2023

  5. [13]

    Rlhf workflow: From reward modeling to online rlhf

    Hanze Dong, Wei Xiong, Bo Pang, Haoxiang Wang, Han Zhao, Yingbo Zhou, Nan Jiang, Doyen Sahoo, Caiming Xiong, and Tong Zhang. Rlhf workflow: From reward modeling to online rlhf. arXiv preprint arXiv:2405.07863, 2024

    2024 arXiv

  6. [14]

    Helping or herding? reward model ensembles mitigate but do not eliminate reward hacking

    Jacob Eisenstein, Chirag Nagpal, Alekh Agarwal, Ahmad Beirami, Alex D'Amour, DJ Dvijotham, Adam Fisch, Katherine Heller, Stephen Pfohl, Deepak Ramachandran, et al. Helping or herding? reward model ensembles mitigate but do not eliminate reward hacking. arXiv preprint arXiv:231...

    2023 arXiv

  7. [15]

    Understanding dataset difficulty with v-usable information

    Kawin Ethayarajh, Yejin Choi, and Swabha Swayamdipta. Understanding dataset difficulty with v-usable information. In International Conference on Machine Learning, pages 5988--6008. PMLR, 2022

    2022

  8. [16]

    Dropout as a bayesian approximation: Representing model uncertainty in deep learning

    Yarin Gal and Zoubin Ghahramani. Dropout as a bayesian approximation: Representing model uncertainty in deep learning. In International Conference on Machine Learning, 2016

    2016

  9. [17]

    Scaling laws for reward model overoptimization

    Leo Gao, John Schulman, and Jacob Hilton. Scaling laws for reward model overoptimization. In International Conference on Machine Learning, pages 10835--10866. PMLR, 2023

    2023

  10. [18]

    https://huggingface.co/google/gemma-2b, 2024

    Gemma. https://huggingface.co/google/gemma-2b, 2024

    2024

  11. [19]

    Beavertails: Towards improved safety alignment of llm via a human-preference dataset

    Jiaming Ji, Mickel Liu, Josef Dai, Xuehai Pan, Chi Zhang, Ce Bian, Boyuan Chen, Ruiyang Sun, Yizhou Wang, and Yaodong Yang. Beavertails: Towards improved safety alignment of llm via a human-preference dataset. Advances in Neural Information Processing Systems, 36, 2024

    2024

  12. [20]

    Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de Las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas ...

    doi arXiv

  13. [21]

    Reward design with language models

    Yoonho Kwon, Dilip Arumugam, Kimin Lee, and Chelsea Finn. Reward design with language models. arXiv preprint arXiv:2303.00001, 2023

    2023 arXiv

  14. [22]

    Rewardbench: Evaluating reward models for language modeling

    Nathan Lambert, Valentina Pyatkin, Jacob Morrison, LJ Miranda, Bill Yuchen Lin, Khyathi Chandu, Nouha Dziri, Sachin Kumar, Tom Zick, Yejin Choi, et al. Rewardbench: Evaluating reward models for language modeling. arXiv preprint arXiv:2403.13787, 2024

    2024 arXiv

  15. [23]

    Alpacaeval: An automatic evaluator of instruction-following models, 2023

    Xuechen Li, Tianyi Zhang, Yann Dubois, Rohan Taori, Ishaan Gulrajani, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto. Alpacaeval: An automatic evaluator of instruction-following models, 2023

    2023

  16. [24]

    Openorca: An open dataset of gpt augmented flan reasoning traces, 2023

    W Lian, B Goodson, E Pentland, et al. Openorca: An open dataset of gpt augmented flan reasoning traces, 2023

    2023

  17. [25]

    Reward uncertainty for exploration in preference-based reinforcement learning

    Xinran Liang, Katherine Shu, Kimin Lee, and Pieter Abbeel. Reward uncertainty for exploration in preference-based reinforcement learning. arXiv preprint arXiv:2205.12401, 2022

    2022 arXiv

  18. [26]

    Iterative prompting for estimating epistemic uncertainty

    Yifan Liu et al. Iterative prompting for estimating epistemic uncertainty. In NeurIPS, 2024

    2024

  19. [27]

    Uncertainty-aware reward model: Teaching reward models to know what is unknown

    Xingzhou Lou, Dong Yan, Wei Shen, Yuzi Yan, Jian Xie, and Junge Zhang. Uncertainty-aware reward model: Teaching reward models to know what is unknown. arXiv preprint arXiv:2410.00847, 2024

    2024 arXiv

  20. [28]

    Introducing meta llama 3: The most capable openly available llm to date

    AI Meta. Introducing meta llama 3: The most capable openly available llm to date. Meta AI, 2024

    2024

  21. [29]

    Montgomery and George C

    Douglas C. Montgomery and George C. Runger. Applied Statistics and Probability for Engineers. John Wiley & Sons, Hoboken, NJ, 6 edition, 2014. ISBN 9781118539712

    2014

  22. [30]

    Gpt-4 technical report

    OpenAI. Gpt-4 technical report. arxiv 2303.08774. View in Article, 2 0 (5), 2023

    2023 arXiv

  23. [31]

    Training language models to follow instructions with human feedback

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35: 0 2...

    2022

  24. [32]

    Language models are unsupervised multitask learners

    Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAI blog, 1 0 (8): 0 9, 2019

    2019

  25. [33]

    Direct preference optimization: Your language model is secretly a reward model

    Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36, 2024

    2024

  26. [34]

    Warm: On the benefits of weight averaged reward models

    Alexandre Ram \'e , Nino Vieillard, L \'e onard Hussenot, Robert Dadashi, Geoffrey Cideron, Olivier Bachem, and Johan Ferret. Warm: On the benefits of weight averaged reward models. arXiv preprint arXiv:2401.12187, 2024

    2024 arXiv

  27. [35]

    Trust region policy optimization

    John Schulman. Trust region policy optimization. arXiv preprint arXiv:1502.05477, 2015

    2015 arXiv

  28. [36]

    Proximal policy optimization algorithms

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017

    2017 arXiv

  29. [37]

    Mutual fund performance

    William F Sharpe. Mutual fund performance. The Journal of business, 39 0 (1): 0 119--138, 1966

    1966

  30. [38]

    Somanshu Singla, Zhen Wang, Tianyang Liu, Abdullah Ashfaq, Zhiting Hu, and Eric P. Xing. Dynamic rewarding with prompt optimization enables tuning-free self-alignment of language models. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen, editors, Proceedings of the 2024 Con...

    2024 doi

  31. [39]

    Llm-as-a-judge & reward model: What they can and cannot do, 2024

    Guijin Son, Hyunwoo Ko, Hoyoung Lee, Yewon Kim, and Seunghyeok Hong. Llm-as-a-judge & reward model: What they can and cannot do, 2024. URL https://arxiv.org/abs/2409.11239

    2024 arXiv

  32. [40]

    Learning to summarize with human feedback

    Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. Learning to summarize with human feedback. Advances in Neural Information Processing Systems, 33: 0 3008--3021, 2020

    2020

  33. [41]

    Policy gradient methods for reinforcement learning with function approximation

    Richard S Sutton, David McAllester, Satinder Singh, and Yishay Mansour. Policy gradient methods for reinforcement learning with function approximation. Advances in neural information processing systems, 12, 1999

    1999

  34. [42]

    Quantifying uncertainty in natural language explanations of large language models

    Sree Harsha Tanneru, Chirag Agarwal, and Himabindu Lakkaraju. Quantifying uncertainty in natural language explanations of large language models. arXiv preprint arXiv:2311.03533, 2023

    2023 arXiv

  35. [43]

    Gemini: a family of highly capable multimodal models

    Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805, 2023

    2023 arXiv

  36. [44]

    Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context

    Gemini Team, Petko Georgiev, Ving Ian Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.05530, 2024 a

    2024 arXiv

  37. [45]

    Gemma: Open models based on gemini research and technology

    Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivi \`e re, Mihir Sanjay Kale, Juliette Love, et al. Gemma: Open models based on gemini research and technology. arXiv preprint arXiv:2403.08295, 2024 b

    2024 arXiv

  38. [46]

    Llama 2: Open foundation and fine-tuned chat models

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023

    2023 arXiv

  39. [47]

    Trl: Transformer reinforcement learning

    Leandro von Werra, Younes Belkada, Lewis Tunstall, Edward Beeching, Tristan Thrush, Nathan Lambert, Shengyi Huang, Kashif Rasul, and Quentin Gallouédec. Trl: Transformer reinforcement learning. https://github.com/huggingface/trl, 2020

    2020

  40. [48]

    Helpsteer: Multi-attribute helpfulness dataset for steerlm

    Zhilin Wang, Yi Dong, Jiaqi Zeng, Virginia Adams, Makesh Narsimhan Sreedhar, Daniel Egert, Olivier Delalleau, Jane Polak Scowcroft, Neel Kant, Aidan Swope, et al. Helpsteer: Multi-attribute helpfulness dataset for steerlm. arXiv preprint arXiv:2311.09528, 2023

    2023 arXiv

  41. [49]

    B. L. Welch. The generalization of `student's' problem when several different population variances are involved. Biometrika, 34 0 (1/2): 0 28--35, 1947. ISSN 00063444. URL http://www.jstor.org/stable/2332510

    1947

  42. [50]

    Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mari...

    2020

  43. [51]

    Can llms express their uncertainty? an empirical evaluation of confidence elicitation in llms

    Miao Xiong, Zhiyuan Hu, Xinyang Lu, Yifei Li, Jie Fu, Junxian He, and Bryan Hooi. Can llms express their uncertainty? an empirical evaluation of confidence elicitation in llms. arXiv preprint arXiv:2306.13063, 2023

    2023 arXiv

  44. [52]

    Iterative preference learning from human feedback: Bridging theory and practice for rlhf under kl-constraint

    Wei Xiong, Hanze Dong, Chenlu Ye, Ziqi Wang, Han Zhong, Heng Ji, Nan Jiang, and Tong Zhang. Iterative preference learning from human feedback: Bridging theory and practice for rlhf under kl-constraint. ICML, 2024

    2024

  45. [53]

    Qwen2.5 technical report

    Qwen An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxin Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, ...

    2024 arXiv

  46. [54]

    Advancing llm reasoning generalists with preference trees

    Lifan Yuan, Ganqu Cui, Hanbin Wang, Ning Ding, Xingyao Wang, Jia Deng, Boji Shan, Huimin Chen, Ruobing Xie, Yankai Lin, et al. Advancing llm reasoning generalists with preference trees. arXiv preprint arXiv:2404.02078, 2024

    2024 arXiv

  47. [55]

    Uncertainty-penalized reinforcement learning from human feedback with diverse reward lora ensembles

    Yuanzhao Zhai, Han Zhang, Yu Lei, Yue Yu, Kele Xu, Dawei Feng, Bo Ding, and Huaimin Wang. Uncertainty-penalized reinforcement learning from human feedback with diverse reward lora ensembles. arXiv preprint arXiv:2401.00243, 2023

    2023 arXiv

  48. [56]

    Bayesian prompt ensembles: Model uncertainty estimation for black-box large language models

    Jianxiong Zhang et al. Bayesian prompt ensembles: Model uncertainty estimation for black-box large language models. Findings of ACL, 2024 a

    2024

  49. [57]

    Improving reinforcement learning from human feedback with efficient reward model ensemble

    Shun Zhang, Zhenfang Chen, Sunli Chen, Yikang Shen, Zhiqing Sun, and Chuang Gan. Improving reinforcement learning from human feedback with efficient reward model ensemble. arXiv preprint arXiv:2401.16635, 2024 b

    2024 arXiv

  50. [58]

    Judging llm-as-a-judge with mt-bench and chatbot arena

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36: 0 46595--46623, 2023

    2023

  51. [59]

    Fine-tuning language models from human preferences

    Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences. arXiv preprint arXiv:1909.08593, 2019

    1909 arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.