Pith. sign in

REVIEW 4 major objections 6 minor 43 references

Establishing Reliability Metrics for Reward Models in Large Language Models

T0 review · 4 major / 6 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper proposes RETA, a normalized metric that measures reward-model reliability through the average oracle quality of the top η quantile of ranked responses.

desk verdict A useful, well-defined reliability metric with a clean pipeline, but the estimator's claimed unbiasedness is unjustified and the validation never leaves the GPT-4 oracle. read the letter →

arxiv 2504.14838 v1 pith:NRF6PMWX submitted 2025-04-21 cs.AI

Yizhou Chen , Yawen Liu , Xuesi Wang , Qingtao Yu , Guangda Huzhang , Anxiang Zeng , Han Yu , Zhiming Zhou This is my paper · ORCID
classification cs.AI
keywords rewardmodelreliabilityRETAmetricbest-of-nsamplingoveroptimizationLLMalignmentevaluationquantile-basedoraclescoringbenchmarking
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper proposes a single number, Reliability at η (RETA), that measures how trustworthy a reward model is: take the top η fraction of responses the model ranks highest for a prompt, average the oracle-judged quality of those responses, and divide by the average quality of all responses. The central claim is that this ratio has a well-defined limiting value, converges quickly with finite response pools, and, once normalized, is largely insensitive to which prompts are included. RETA is meant to replace accuracy and best-of-n evaluations, which the paper argues either miss reliability or wobble with the ranking of a single response. Because the oracle labels a fixed response set once, every new reward model can be scored on the same pre-labeled pool without additional labeling cost, which makes reliability benchmarking affordable.

What carries the argument

The load-bearing object is the BETA subset — the top $\eta$ fraction of responses selected by the reward model $Y$ — together with the normalized average oracle score over that subset. Theorem 1 shows that as the response pool grows, this average converges to $\mathbb{E}_a[J_q(a)\mid Y_q(a)\ge \Theta(\eta)]/\mathbb{E}_a[J_q(a)]$, where $\Theta(\eta)$ is the upper-$\eta$ quantile of reward scores. The denominator $\mathbb{E}_a[J_q(a)]$ is what makes RETA interpretable: it puts the random baseline at 1 and removes much of the prompt-to-prompt variation in oracle scores. The estimation scheme chooses resampling sizes proportional to $N^{2/3}$, averaging over $n\in[3N^{2/3},5N^{2/3}]$, to balance bias from replacing the true quantile with the sample quantile against variance from Monte Carlo estimation.

What would settle it

Take the released benchmark response sets, re-score them with human annotators instead of GPT-4, recompute RETA for the ten tested reward models, and check whether the model rankings and RETA curves stay the same; if they change materially, RETA's validity rests on the GPT-4 oracle assumption.

Watch

Extended reading notes

Core claim

The paper's central discovery is Theorem 1: as the candidate pool grows, RETA converges to the conditional expectation of oracle quality among responses whose reward score is at or above the upper-$\eta$ quantile, divided by the unconditional mean oracle quality, $$\mathrm{RETA}_Y(\eta)=\frac{1}{|\mathcal{Q}|}\sum_{q\in\mathcal{Q}}\frac{\mathbb{E}_a[J_q(a)\mid Y_q(a)\ge \Theta(\eta)]}{\mathbb{E}_a[J_q(a)]}.$$ The value 1 is the random baseline, so values above 1 mean the reward model's top quantile beats average response quality, and a RETA curve across $\eta$ shows where that advantage decays. The paper further claims that the empirical estimator of Eq. (5), averaging over resampled subsets of size $n$ in $[3N^{2/3},5N^{2/3}]$, is asymptotically unbiased and that RETA(1/4) curves converge visibly to their limits; on the two constructed benchmarks it uses the curves to rank publicly available and in-house reward models and to identify which quantile is safest for selection.

Load-bearing premise

The validity of RETA depends on GPT-4's scores serving as a faithful oracle for genuine human preferences, with responses generated by Llama2-7B-Chat at temperature 1 representing the deployment distribution; if either premise fails, RETA measures agreement with GPT-4's judgment on that particular response distribution rather than true reliability.

Editorial extensions

If this is right

  • RETA lets a lab rank any number of reward models using one pre-labeled response set: each new RM only scores existing responses, so there is no incremental oracle cost.
  • The RETA curve gives an operational stopping rule for rejection sampling: the $\eta$ at which the curve stops rising is the quantile below which selecting even fewer responses no longer helps.
  • Accuracy alone can be misleading as a reliability signal: RAFT-3B, the highest test-accuracy RM in the study, is among the weakest under RETA, so RETA captures behavior that accuracy benchmarks do not.
  • The normalized metric is the right one for comparing across prompts: without the denominator, RETA drifts with prompt perplexity; with it, the relation flattens.
  • In the paper's experiments, ensembling several reward models keeps RETA close to the best member, pointing to ensembling as a practical mitigation for reward hacking.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: RETA is best read as reliability on a fixed reference-policy distribution, not as a universal property of the reward model; a model that looks reliable under Llama2-7B-Chat responses could fail on a different response distribution.
  • Editorial inference: the identical recipe — oracle-scored candidates, selector scoring, average quality of the top $\eta$ — applies to any selection system beyond reward models, such as retrieval rankers or LLM-as-a-judge pipelines, provided an oracle score is available for the candidate pool.
  • Editorial inference: the paper's future-work proposal implies a sharper test of the metric's usefulness: if training a reward model to maximize RETA does not improve downstream RLHF policy quality, then RETA would be measuring oracle agreement rather than alignment.
  • Editorial inference: because the oracle is GPT-4, RETA as implemented measures agreement with GPT-4's notion of quality; recomputing the same benchmark with human labels or multiple oracle models would reveal how much of the ranking is oracle-dependent.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes RETA (Reliable at η), a scalar metric for quantifying the reliability of reward models (RMs). RETA is defined as the limit, as the number of sampled responses per prompt grows, of the average oracle quality of the top η-quantile of responses ranked by the RM, normalized by the average oracle quality. The authors prove a limiting characterization (Theorem 1), propose a resampling-based estimator (Eq. 5), and build an end-to-end benchmarking pipeline that uses DPP-based prompt selection, Llama2-7B-Chat as the reference policy, and GPT-4 as the oracle. They benchmark 10 RMs on two datasets (Anthropic-Helpful and MT-Bench based) and show RETA curves, comparisons against BON and other metrics, and ablation of the normalization term. The paper also releases code and the benchmark.

Significance. If the proposed estimator and validation are sound, RETA addresses a genuinely important gap: a direct, budget-friendly reliability metric for RMs that does not require running RLHF. The quantile-average formulation is a sensible and principled improvement over best-of-n curves, which are unstable because they depend on a single selected response. The theoretical limit in Theorem 1 gives the metric a clear statistical interpretation, and the release of code and benchmark data supports reproducibility and community uptake. However, the paper's empirical validation is weakened by the use of GPT-4 both as the oracle and as the judge in the 'Win Rate' validation, and by an estimator whose unbiasedness is not rigorously established. These issues, while fixable, currently limit the confidence one can place in the reported rankings and in the claim that RETA captures human-aligned reliability.

major comments (4)
  1. [Sec. 3.3, Eq. (5)] The claim that the estimator in Eq. (5) is 'asymptotically unbiased' is not justified. The estimator is a ratio of two random quantities: the numerator is the resampled average of oracle scores over a random subset A of size n, and the denominator is the full-sample average over A_q. Linearity of expectation does not apply to ratios, and E[Σ_{a∈A} J_q(a) / Σ_{a∈A_q} J_q(a)] ≠ E[Σ_{a∈A} J_q(a)] / E[Σ_{a∈A_q} J_q(a)]. Moreover, the numerator and denominator are correlated because the subset is drawn without replacement from the same labeled set. The paper provides no bias or variance analysis; the statement in Appendix B.2 that the two adjacent order statistics 'will be arbitrarily close as n approaches infinity' is not a proof of unbiasedness. This is a load-bearing issue because the experimental RETA values in Figures 2–3 and Table 2 are computed with this estimator, and the pipeline's claimed validity depends on it. The authors should either provide a rigorous consistency and finite-sample bias analysis or replace the estimator with a provably unbiased one (e.g., a combinatorial estimator analogous to the BON sampler in Appendix B.3).
  2. [Appendix B.1, proof of Theorem 1] The proof of Theorem 1 contains an equality that is not correct as stated: p(X_(j−1:n) ≥ Y_q(a) ≥ X_(j+1:n)) = 1/n. The event involves the random variable a with density p(a) and the sample X_1,...,X_n; unless a is explicitly one of the n sample points and the probability is taken over the full n-sample including a, the probability that Y_q(a) falls between two adjacent order statistics is not 1/n. The notation conflates 'a' as an index of a sampled response with 'a' as an integration variable. A rigorous proof should use exchangeability or a conditioning argument on the empirical distribution of the pairs (Y_q(a), J_q(a)) and the standard consistency of sample quantiles. The limiting statement itself is plausible and likely true, but the proof as written does not establish it. The authors should rewrite the proof with clear probabilistic formalism.
  3. [Sec. 4.4 and Appendix A.1] The validation of RETA against 'Win Rate against GPT4' is circular. As the authors note in Appendix A.1, the Win Rate metric is 'a variant of the RETA metric using this oracle labeling scheme': it selects the top 64 of 256 responses by the RM and asks GPT-4 to judge their quality against GPT-4's own answer. Since the same GPT-4 oracle is used both to compute RETA and to produce the Win Rate labels, the agreement observed in Table 2 does not provide evidence that RETA reflects human preferences independent of GPT-4's biases. To substantiate the claim that RETA measures reliability with respect to actual human judgment, the authors should provide external validation with human annotations (or at least an independent oracle) on a subset of the data.
  4. [Sec. 3.1 and Appendix B.4] The benchmark's validity as a measure of RM reliability rests on two assumptions that are not empirically checked: (i) GPT-4 scores are a faithful proxy for human preferences, and (ii) Llama2-7B-Chat at temperature T=1 is a representative reference policy for the deployment distribution. The latter is especially concerning because several of the tested RMs (Starling-7B, RMv1-3, RM5H-7B, RMEns-3x7B) are trained from Llama2-7B, so a reference policy from the same model family may bias the comparisons. At minimum, the authors should report a sensitivity analysis with a different reference policy and/or a small human agreement study to show that the RETA rankings are robust to these choices.
minor comments (6)
  1. [Sec. 3.3] The notation E_A in Eq. (5) is undefined; please state that A is a uniformly random subset of size n drawn without replacement from A_q, and clarify that the expectation is with respect to this resampling distribution.
  2. [Sec. 3.3] The choice of the range [3N^(2/3), 5N^(2/3)] is said to be 'empirically selected', but no details are provided. Please describe the selection procedure and report sensitivity of the final RETA values to the constants 3 and 5.
  3. [Sec. 3.2, Theorem 1] The quantile function Θ(η) is defined as inf(x : F(x) ≥ 1−η), so 'continuity points of Θ' should be stated precisely with respect to the parameter η; also clarify that the theorem requires F to have no atom at the quantile or that the conditional expectation is defined appropriately in the presence of ties.
  4. [Appendix B.2] The statement that a_(⌊ηn⌋:n) and a_(⌊ηn⌋+1:n) 'will be arbitrarily close as n approaches infinity' is imprecise; the gap between adjacent order statistics is O_p(1/n) under a positive density, but the argument should be stated rigorously if it is used to justify asymptotic unbiasedness.
  5. [Table 2] In the 'Oracle' row, the Win Rate against GPT4 is reported as 52.93%, which seems surprising because the oracle's own selections are compared against GPT-4's answers. Please explain why the score is not near 100% and what this row represents.
  6. [Table 1] The DPP sampling cost is listed as O(|Q0|k), but the MCMC algorithm cited in the text has complexity O(|Q0|k log(1/ε)); the table omits the approximation factor ε.

Circularity Check

1 steps flagged · score 4.0 of 10

RETA is well defined and its limit theorem is self-contained, but its experimental validation is partly circular: the 'Win Rate against GPT4' ground truth is acknowledged as a variant of RETA sharing the same oracle and top-quantile selection.

  1. self definitional [Appendix A.1, cited in Sec. 4.4 and Table 2]
    "In addition, we introduce a metric, the Win Rate against GPT4, as a ground truth evaluation of the quality of the responses selected by the RM. We select the top 64 (out of 256) responses rated by the RM, and for each of them, we compare them with GPT4’s answer and ask GPT4 to determine which answer is better (or if they are a draw). ... Therefore, the Win Rate can be seen as a variant of the RETA metric using this oracle labeling scheme."

    The paper's only external validation of RETA is 'Win Rate against GPT4', which is constructed from the same GPT-4 oracle, the same reference-policy response set, and the same top-quantile selection (top 64 out of 256, i.e., eta = 1/4) as RETA. The paper itself concedes that Win Rate is a variant of RETA with a win/lose/draw oracle labeling scheme. Hence high agreement between RETA and Win Rate is substantially by construction and demonstrates robustness to the oracle scoring format, not that RETA independently tracks human-preference reliability. The central reliability claim therefore rests on the unvalidated premise that GPT-4 is a faithful human oracle, with the validation metric sharing RETA's key ingredients.

full rationale

The definition of RETA (Eq. 3) and its limiting characterization (Theorem 1, Eq. 4) are mathematically self-contained: the theorem follows from sample-quantile convergence and does not fit any parameter to a target result. The metric is not defined in terms of the quantity it is claimed to predict, nor is any fitted parameter renamed as a prediction. The main circularity is confined to the validation section: the 'Win Rate against GPT4' used as ground truth is explicitly described as a variant of RETA built from the same oracle and same top-quantile selection, so the reported alignment of RETA with Win Rate is partly by construction. This weakens the external-validity claim that RETA measures human-aligned reliability, but it does not affect the internal consistency of the metric or the proof of its limit. No load-bearing self-citation chains or imported uniqueness theorems were found, and the comparisons against BON, ranking metrics, and accuracy metrics are independent. Overall, the paper deserves a moderate circularity score because its core derivation is sound and only its validating evidence is partially self-referential.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

RETA is a scalar metric, not a new entity. The free parameters listed are estimation design choices, not fitted parameters of the metric itself.

free parameters (3)
  • n range constants 3 and 5 = 3 and 5 (empirically selected)
    Section 3.3: the final RETA estimate averages Eq. (5) over n in [3N^(2/3), 5N^(2/3)]; the constants are stated to be 'empirically selected'.
  • DPP eigenvalue scaling factor λ = Not reported
    Section 3.3: the kernel is λEᵀE with λ chosen to keep eigenvalues below 1; the exact value is not given.
  • Resampling count 200 = 200
    Section 3.3: 'we resampled 200 samples of A to evaluate the above expectation'; a hand-picked number affecting variance.
assumptions (4)
  • domain assumption Oracle scores J_q are bounded
    Required for Theorem 1; the implemented scores are bounded 1-10.
  • ad hoc to paper GPT-4 scores reflect human preferences
    The paper uses gpt-4-turbo as oracle (Sec. 3.1) and treats its scores as ground truth quality; false if GPT-4 has systematic biases.
  • domain assumption Responses are i.i.d. from the reference policy θ
    RETA's limit assumes a distribution over responses; in practice N=256 samples per prompt approximate this.
  • standard math Standard sample quantile convergence results hold
    Lemma 1 in Appendix B.1 relies on classical weak-convergence results for sample quantiles from [17,38].

how reviews work

0 comments
Cite this review

Pith. "Pith review of Establishing Reliability Metrics for Reward Models in Large Language Models." pith.science (2026). https://pith.science/paper/NRF6PMWX

@misc{pith2026250414838,
  author       = {Pith},
  title        = {Pith review of: Establishing Reliability Metrics for Reward Models in Large Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/NRF6PMWX}},
  note         = {Machine review of arXiv:2504.14838}
}
abstract

The reward model (RM) that represents human preferences plays a crucial role in optimizing the outputs of large language models (LLMs), e.g., through reinforcement learning from human feedback (RLHF) or rejection sampling. However, a long challenge for RM is its uncertain reliability, i.e., LLM outputs with higher rewards may not align with actual human preferences. Currently, there is a lack of a convincing metric to quantify the reliability of RMs. To bridge this gap, we propose the \textit{\underline{R}eliable at \underline{$\eta$}} (RETA) metric, which directly measures the reliability of an RM by evaluating the average quality (scored by an oracle) of the top $\eta$ quantile responses assessed by an RM. On top of RETA, we present an integrated benchmarking pipeline that allows anyone to evaluate their own RM without incurring additional Oracle labeling costs. Extensive experimental studies demonstrate the superior stability of RETA metric, providing solid evaluations of the reliability of various publicly available and proprietary RMs. When dealing with an unreliable RM, we can use the RETA metric to identify the optimal quantile from which to select the responses.

Figures

Figures reproduced from arXiv: 2504.14838 by the authors.

Figure 1
Figure 1. The benchmark building pipeline and the computation of the RETA metric. [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. The results on Reliability-on-Helpfulness benchmark: (a) The estimation of RETA( [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. The RETA curves on Reliability-on-Helpfulness benchmark. The x-axis is plotted on a [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figures from the paper (6 more)
Figure 4
Figure 4. Figure 4: The fitted curve of RETA metric versus prompt perplexity on Reliability-on-Helpfulness [PITH_FULL_IMAGE:figures/full_fig_p009_4.png]
Figure 5
Figure 5. Figure 5: The Hit Rate metric evaluated on Reliability-on-Helpfulness dataset: (a) The hit rate at n is calculated by comparing the top n responses generated by the RM with the ground truth set, which is defined as the η = 1/4 quantile of responses selected by the oracle. (b) Th…
Figure 6
Figure 6. Figure 6: The results on Reliability-on-Multi-Turn-Conversation benchmark: (a) The estimation of [PITH_FULL_IMAGE:figures/full_fig_p015_6.png]
Figure 7
Figure 7. Figure 7: The RETA curves on Reliability-on-Multi-Turn-Conversation benchmark. The x-axis is [PITH_FULL_IMAGE:figures/full_fig_p015_7.png]
Figure 8
Figure 8. Figure 8: The prompt for helpfulness oracle evaluation when building the Reliability-on-Helpfulness [PITH_FULL_IMAGE:figures/full_fig_p018_8.png]
Figure 9
Figure 9. Figure 9: The prompt for multi-turn oracle evaluation when building the Reliability-on-Multi-Turn [PITH_FULL_IMAGE:figures/full_fig_p019_9.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

43 extracted references · 15 canonical work pages

  1. [1]

    https://huggingface.co/OpenAssistant/oasst-rm-2.1-pythia-1

    Pythia 1.4b based reward model. https://huggingface.co/OpenAssistant/oasst-rm-2.1-pythia-1. 4b-epoch-2.5

    work page

  2. [2]

    https://huggingface.co/OpenAssistant/ reward-model-deberta-v3-large-v2

    Reward model trained from human feedback. https://huggingface.co/OpenAssistant/ reward-model-deberta-v3-large-v2

    work page

  3. [3]

    https://huggingface.co/IDEA-CCNL/Ziya-LLaMA-7B-Reward

    Ziya-llama-7b-reward. https://huggingface.co/IDEA-CCNL/Ziya-LLaMA-7B-Reward

    work page

  4. [4]

    Monte carlo markov chain algorithms for sampling strongly rayleigh distributions and determinantal point processes

    Nima Anari, Shayan Oveis Gharan, and Alireza Rezaei. Monte carlo markov chain algorithms for sampling strongly rayleigh distributions and determinantal point processes. In Conference on Learning Theory, pages 103–115. PMLR, 2016

    work page 2016

  5. [5]

    The claude 3 model family: Opus, sonnet, haiku

    Anthropic. The claude 3 model family: Opus, sonnet, haiku. 2024

    2024

  6. [6]

    On the asymptotic accuracy of the bootstrap under arbitrary resampling size

    Miguel A Arcones. On the asymptotic accuracy of the bootstrap under arbitrary resampling size. Annals of the Institute of Statistical Mathematics, 55:563–583, 2003

    work page 2003

  7. [7]

    Training a helpful and harmless assistant with reinforcement learning from human feedback

    Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862, 2022

    arXiv 2022

  8. [8]

    On the choice of m in the m out of n bootstrap and confidence bounds for extrema

    Peter J Bickel and Anat Sakov. On the choice of m in the m out of n bootstrap and confidence bounds for extrema. Statistica Sinica, pages 967–985, 2008

    work page 2008

Show all 43 references
  1. [9]

    Language models are few-shot learners

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901, 2020

    1901

  2. [10]

    Open problems and fundamental limitations of reinforcement learning from human feedback

    Stephen Casper, Xander Davies, Claudia Shi, Thomas Krendl Gilbert, Jérémy Scheurer, Javier Rando, Rachel Freedman, Tomasz Korbak, David Lindner, Pedro Freire, et al. Open problems and fundamental limitations of reinforcement learning from human feedback. arXiv preprint arXiv:2...

    2023 arXiv

  3. [11]

    Variance estimation for sample quantiles using the m out of n bootstrap

    KY Cheung and Stephen MS Lee. Variance estimation for sample quantiles using the m out of n bootstrap. Annals of the Institute of Statistical Mathematics, 57(2):279–290, 2005

    2005

  4. [12]

    Deep reinforcement learning from human preferences

    Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. Advances in neural information processing systems, 30, 2017

    2017

  5. [13]

    Reward model ensembles help mitigate overoptimization

    Thomas Coste, Usman Anwar, Robert Kirk, and David Krueger. Reward model ensembles help mitigate overoptimization. arXiv preprint arXiv:2310.02743, 2023

    2023 arXiv

  6. [14]

    Bert: Pre-training of deep bidirectional transformers for language understanding

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018

    2018 arXiv

  7. [15]

    Raft: Reward ranked finetuning for generative foundation model alignment

    Hanze Dong, Wei Xiong, Deepanshu Goyal, Rui Pan, Shizhe Diao, Jipeng Zhang, Kashun Shum, and Tong Zhang. Raft: Reward ranked finetuning for generative foundation model alignment. arXiv preprint arXiv:2304.06767, 2023

    2023 arXiv

  8. [16]

    Helping or herding? reward model ensembles mitigate but do not eliminate reward hacking

    Jacob Eisenstein, Chirag Nagpal, Alekh Agarwal, Ahmad Beirami, Alex D’Amour, DJ Dvi- jotham, Adam Fisch, Katherine Heller, Stephen Pfohl, Deepak Ramachandran, et al. Helping or herding? reward model ensembles mitigate but do not eliminate reward hacking. arXiv preprint arXiv:2...

    2023 arXiv

  9. [17]

    A modern approach to probability theory

    Bert E Fristedt and Lawrence F Gray. A modern approach to probability theory . Springer Science & Business Media, 2013

    2013

  10. [18]

    Scaling laws for reward model overoptimization

    Leo Gao, John Schulman, and Jacob Hilton. Scaling laws for reward model overoptimization. In International Conference on Machine Learning, pages 10835–10866. PMLR, 2023

    2023

  11. [19]

    X-risk analysis for ai research

    Dan Hendrycks and Mantas Mazeika. X-risk analysis for ai research. arXiv preprint arXiv:2206.05862, 2022. 10

    2022 arXiv

  12. [20]

    Cumulated gain-based evaluation of ir techniques

    Kalervo Järvelin and Jaana Kekäläinen. Cumulated gain-based evaluation of ir techniques. TOIS, 20(4):422–446, oct 2002

    2002

  13. [21]

    Simple and scalable predictive uncertainty estimation using deep ensembles

    Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. Advances in neural information processing systems, 30, 2017

    2017

  14. [22]

    The alignment ceiling: Objective mismatch in rein- forcement learning from human feedback

    Nathan Lambert and Roberto Calandra. The alignment ceiling: Objective mismatch in rein- forcement learning from human feedback. arXiv preprint arXiv:2311.00168, 2023

    2023 arXiv

  15. [23]

    Rewardbench: Evaluating reward models for language modeling

    Nathan Lambert, Valentina Pyatkin, Jacob Morrison, LJ Miranda, Bill Yuchen Lin, Khyathi Chandu, Nouha Dziri, Sachin Kumar, Tom Zick, Yejin Choi, et al. Rewardbench: Evaluating reward models for language modeling. arXiv preprint arXiv:2403.13787, 2024

    2024 arXiv

  16. [24]

    Efficient sampling for k-determinantal point processes

    Chengtao Li, Stefanie Jegelka, and Suvrit Sra. Efficient sampling for k-determinantal point processes. In Artificial Intelligence and Statistics, pages 1328–1337. PMLR, 2016

    2016

  17. [25]

    On the fragility of learned reward functions

    Lev McKinney, Yawen Duan, David Krueger, and Adam Gleave. On the fragility of learned reward functions. arXiv preprint arXiv:2301.03652, 2023

    2023 arXiv

  18. [26]

    Webgpt: Browser-assisted question-answering with human feedback

    Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christo- pher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, et al. Webgpt: Browser-assisted question-answering with human feedback. arXiv preprint arXiv:2112.09332, 2021

    2021 arXiv

  19. [27]

    Gpt-4 technical report

    R OpenAI. Gpt-4 technical report. ArXiv, 2303, 2023

    2023

  20. [28]

    Training language models to follow instructions with human feedback

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:2773...

    2022

  21. [29]

    The effects of reward misspecification: Mapping and mitigating misaligned models

    Alexander Pan, Kush Bhatia, and Jacob Steinhardt. The effects of reward misspecification: Mapping and mitigating misaligned models. arXiv preprint arXiv:2201.03544, 2022

    2022 arXiv

  22. [30]

    Improving language understanding by generative pre-training

    Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. Improving language understanding by generative pre-training. 2018

    2018

  23. [31]

    Warm: On the benefits of weight averaged reward models

    Alexandre Ramé, Nino Vieillard, Léonard Hussenot, Robert Dadashi, Geoffrey Cideron, Olivier Bachem, and Johan Ferret. Warm: On the benefits of weight averaged reward models. arXiv preprint arXiv:2401.12187, 2024

    2024 arXiv

  24. [32]

    Towards understanding sycophancy in language models

    Mrinank Sharma, Meg Tong, Tomasz Korbak, David Duvenaud, Amanda Askell, Samuel R Bowman, Newton Cheng, Esin Durmus, Zac Hatfield-Dodds, Scott R Johnston, et al. Towards understanding sycophancy in language models. arXiv preprint arXiv:2310.13548, 2023

    arXiv

  25. [33]

    A long way to go: Investigating length correlations in rlhf

    Prasann Singhal, Tanya Goyal, Jiacheng Xu, and Greg Durrett. A long way to go: Investigating length correlations in rlhf. arXiv preprint arXiv:2310.03716, 2023

    2023 arXiv

  26. [34]

    Learning to summarize with human feedback

    Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea V oss, Alec Radford, Dario Amodei, and Paul F Christiano. Learning to summarize with human feedback. Advances in Neural Information Processing Systems, 33:3008–3021, 2020

    2020

  27. [35]

    Stanford alpaca: An instruction-following llama model, 2023

    Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto. Stanford alpaca: An instruction-following llama model, 2023

    2023

  28. [36]

    Gemini: a family of highly capable multimodal models

    Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805, 2023

    2023 arXiv

  29. [37]

    Llama 2: Open foundation and fine-tuned chat models

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023

    2023 arXiv

  30. [38]

    Asymptotic statistics, volume 3

    Aad W Van der Vaart. Asymptotic statistics, volume 3. Cambridge university press, 2000. 11

    2000

  31. [39]

    Finetuned language models are zero-shot learners

    Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc V Le. Finetuned language models are zero-shot learners. arXiv preprint arXiv:2109.01652, 2021

    2021 arXiv

  32. [40]

    Dpp-based adversarial prompt searching for lanugage models

    Xu Zhang and Xiaojun Wan. Dpp-based adversarial prompt searching for lanugage models. arXiv preprint arXiv:2403.00292, 2024

    2024 arXiv

  33. [41]

    Judging llm-as-a-judge with mt-bench and chatbot arena

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36, 2024

    2024

  34. [42]

    Starling-7b: Improving llm helpfulness and harmlessness with rlaif, November 2023

    Banghua Zhu, Evan Frick, Tianhao Wu, Hanlin Zhu, and Jiantao Jiao. Starling-7b: Improving llm helpfulness and harmlessness with rlaif, November 2023. 12 A More Experimental Results A.1 Other Potential Metrics for Reliability Evaluation We also explore other potential metrics f...

    2023

  35. [43]

    The total labeling cost for this benchmark is $64.0

    to present two full conversations in a single evaluation template in which we ask the oracle to moderately focus on the second conversation. The total labeling cost for this benchmark is $64.0. The following observations can be made: • From Fig. 6, similar to the Reliability-o...

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.