Pith. sign in

REVIEW 4 major objections 6 minor 1 cited by

RewardAnything: Generalizable Principle-Following Reward Models

T0 review · 4 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Reward models can be made to follow natural-language principles at inference time, so a single 8B generative model judges any preference expressed in words without retraining, matching or surpassing task-specific models on standard…

desk verdict Useful open model and benchmark for principle-following reward models, but RABench's LLM-judge ground truth makes the headline generalization claim partly circular. read the letter →

arxiv 2506.03637 v2 pith:ZZ3KX72R submitted 2025-06-04 cs.CL cs.AIcs.LG

Zhuohao Yu , Jiali Zeng , Weizheng Gu , Yidong Wang , Jindong Wang , Fandong Meng , Jie Zhou , Yue Zhang
show 2 more authors
Shikun Zhang Wei Ye
This is my paper
classification cs.CLcs.AIcs.LG
keywords principle-followingrewardmodelsmodelingRLHFGRPOlistwisepreferencelearningRABenchnaturallanguageprinciplesgenerativemodel
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Reward models that steer RLHF are normally trained on fixed preference datasets, so adapting to a new preference—brevity in one product, warmth in another—means collecting new annotations and retraining a model. This paper argues that reward modeling should work like instruction following: a reward model should take a natural-language principle as input and judge candidate responses against it, generalizing to novel principles without retraining. To make the capability measurable, they build RABench, a benchmark of 1002 human-validated rankings built from 50 principles in five quality dimensions, and show that existing discriminative and generative reward models generalize poorly to explicit principles. They then train RewardAnything-8B, a generative reward model that reasons about a principle and scores and ranks any list of responses in a single call; given a well-specified principle it reports state-of-the-art accuracy on the standard RM-Bench, and on RABench it matches GPT-4.1 while exceeding all reward models tested.

What carries the argument

The central object is the listwise, principle-conditioned scoring task $S(P, Q, X_i) \to \mathbb{R}$, in which a reward model receives a natural-language principle $P$, a prompt $Q$, and $k$ candidate responses, and must return a score for each response and an overall ranking that reflect adherence to $P$. The training machinery is Group Relative Preference Learning, an application of GRPO in which the reward model, starting from an 8B instruction-tuned backbone, learns to generate structured JSON evaluations; the learning signal is a weighted reward $r = \lambda_f r_f + \lambda_a r_a$ combining a format reward over five structural criteria with an accuracy reward over four sub-metrics (a score-difference-weighted reversed-pair penalty, score distribution matching, partial credit for near-correct scores, and Kendall's τ with top-$k$ agreement). The measurement instrument is RABench, whose ground-truth rankings are produced by a dynamic-programming consensus algorithm that extracts the longest subsequence of one judge's ranking that at least $K$ of four LLM judges support.

What would settle it

Construct a principle-prompt set in which the correct ranking deliberately contradicts the known preferences of the four judge models—for instance a principle that rewards charming but factually wrong answers, or the longest-winded response—collect human rankings for the same set, and check whether RewardAnything follows the stated principle or the judges' consensus whenever the two diverge; the generalization claim fails if its rankings track judge opinion rather than the principle on such cases.

Watch

Extended reading notes

Core claim

The paper claims that generalizable principle-following is achievable with a single 8B reward model. RewardAnything is trained to map a principle, a prompt, and a list of responses to a structured evaluation—chain-of-thought reasoning, per-response scores, and a best-to-worst ranking—on synthetic listwise data whose labels are consensus rankings of four LLM judges. Evaluated on RM-Bench with a clear principle supplied, the model reaches 86.4% overall and 84.4% on the hard style-bias split, versus 83.9% and 76.7% for the strongest reasoning-based generative reward model; evaluated on RABench, it posts 81.9% pairwise accuracy, roughly matching GPT-4.1 (82.5%) while ranking first on Kendall's τ and NDCG, and far above specialized discriminative reward models (63.0–71.4%). Ablations attribute the behavior to explicit principle conditioning, listwise training, GRPO rather than supervised fine-tuning, and inference-time reasoning: removing any of them drops accuracy by roughly 4 to 20 points. A closing case study claims that one out-of-distribution principle and 2000 prompts are enough to align Qwen3-8B toward nuanced safety behavior, improving both refusal helpfulness and MT-Bench text quality over a policy aligned with a conventional reward model.

Load-bearing premise

The load-bearing premise is that the consensus rankings produced by four large language model judges, with human annotators checking only whether each ranking is valid (89% agreement, Cohen's $\kappa = 0.57$), correctly measure adherence to arbitrary natural-language principles; if those judges share a systematic bias, RewardAnything may be learning to imitate the judges rather than to follow the principles.

Editorial extensions

If this is right

  • Preferences become input text instead of training data: adapting a deployed reward model to a new product or value reduces to writing a principle, so one model serves many preference scenarios.
  • Known reward biases can be stipulated away: on RM-Bench's hard split, supplying a principle that prioritizes factual accuracy over length takes RewardAnything to 84.4%, where conventional reward models hover near or below chance.
  • Ranking $k$ responses costs $\Theta(1)$ inference calls and $\Theta(k)$ tokens instead of $\Theta(k^2)$ pairwise comparisons, which is what makes principle-conditioned scoring practical inside PPO and GRPO training loops.
  • RLHF can run on principles alone: the case study aligns a policy with one out-of-distribution principle and 2000 prompts, with no preference annotation and no reward-model training.
  • RABench makes principle-following a measurable capability, so progress on generalization to novel principles can be tracked separately from traditional reward accuracy.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the central claim holds, the practical bottleneck of reward modeling shifts from collecting preference data to articulating principles; the appendix's finding that vague or unprioritized principles hurt performance suggests principle-writing skill will matter as much as model capability.
  • Because the same four LLM judges generate both training and evaluation labels, a bias common to those judges would be inherited by the model and enshrined by the benchmark; a held-out set with human-only labels on adversarial principles would settle whether the model follows principles or judges.
  • The paper flags adversarial manipulation of principles as future work; the concrete consequence is that reward signals, and anything trained on them, inherit both the strengths and the weaknesses of the supplied principle text, turning principle validation into a safety-critical step.
  • The same listwise machinery could extend beyond single principles to composition with explicit priorities, which RABench's five categories and the case study's prioritized multi-objective principle already point toward.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes principle-following reward models (RMs) that condition on a natural-language principle at inference time. It introduces RABench, a benchmark of 1,002 listwise rankings built from 50 principles, prompts from RewardBench, candidate responses from 10 LLMs, and ground-truth rankings synthesized from four LLM judges (Claude-3.7 Sonnet, GPT-4.1, DeepSeek-V3, Gemini 2.5 Pro) via a dynamic-programming consensus algorithm. The authors then train RewardAnything-8B, a generative RM based on Qwen3-8B, using GRPO with a combined format and accuracy reward (Eqs. 2-4) on about 4,000 synthetic training examples labeled by the same consensus pipeline. Experiments report state-of-the-art accuracy on RM-Bench (Table 2, 86.4% overall) when a well-specified principle is supplied, and accuracy close to GPT-4.1 on RABench (Table 3, 81.9% vs. 82.5%). Ablations isolate the contribution of principle conditioning, listwise training, GRPO over SFT, relative-preference rewards, and inference-time reasoning. A case study aligns Qwen3-8B to nuanced safety behavior using only a natural-language principle and 2,000 prompts.

Significance. If the benchmark is trustworthy, the paper makes a practical contribution: it demonstrates that an 8B generative RM can rank a list of responses in a single call and adapt its criteria from a natural-language principle, reducing the need for task-specific RM retraining. This is a timely and potentially useful reframing, and the ablations are informative. The authors also release a Python package and detailed training hyperparameters, which aids reproducibility. However, the main generalization claim rests entirely on RABench, and RABench's ground truth is generated by the same four LLM judges used to create the training labels, with human validation limited to a coarse validity check (kappa 0.57). Until independent human rankings or an external benchmark show that RewardAnything follows principles rather than the judges' style, the headline claim should be treated as not fully established. The RM-Bench result is less affected by this circularity because that benchmark's labels are human-constructed, but it demonstrates principle-conditioned control rather than novel-principle generalization.

major comments (4)
  1. [Sections 4.1, 5.3, and Algorithm 1 (Appendix F)] Both the RABench evaluation labels and the 4,000 training examples are produced by the same consensus procedure over the same four LLM judges (Claude-3.7 Sonnet, GPT-4.1, DeepSeek-V3, Gemini 2.5 Pro). The human QA in Section 4.2 only asks annotators whether a consensus ranking is a valid interpretation (89% agreement, Cohen's kappa 0.57); it does not collect independent human rankings. Consequently, high RABench accuracy may reflect the model's ability to imitate these four judges' shared judgments rather than to follow principles generally. I would ask the authors to add a held-out set of independent human rankings on a random subset of RABench and report RewardAnything's agreement with those human rankings, and to compare agreement against each individual judge to quantify judge-specific imitation.
  2. [Algorithm 1 (Appendix F)] The consensus algorithm produces only a partial consensus. The final ranking is the source judge's full list with the agreed subsequence fixed in place, and the scores in the ground truth are entirely the source judge's scores. Thus, for pairs outside the longest agreed subsequence, the ground truth is a single judge's opinion, yet Table 3 reports pairwise accuracy over all pairs. The paper should report the fraction of pairs actually supported by K=3 or more judges, and either re-aggregate the full ranking across judges or restrict evaluation to agreed pairs. This issue affects the benchmark's validity independently of the judge-label circularity concern.
  3. [Sections 3.1, 4.1, and 5.3] The 50 RABench principles are sampled from the same manually curated pool of 200 principles used to create the 150 training principles. Although the sets are non-overlapping, they come from the same author-defined distribution and the same five categories (Content, Structure, Tone, Logic, Style). This supports within-distribution generalization, but not the stronger claim of adapting to arbitrary or truly novel principles. The authors should either soften the claim or extend the benchmark with principles written by external annotators or drawn from a different source, including multi-objective prioritized principles like the one used in the case study.
  4. [Section 6.1 and Table 2] The headline RM-Bench result is obtained by supplying a principle that explicitly tells the model to prioritize accuracy over length, which is exactly the objective that RM-Bench's 'hard' split is designed to test. This is a legitimate demonstration of controllability and bias mitigation, but it is not evidence of generalization to novel principles. Moreover, the same principle is passed to the baselines, and discriminative RMs cannot consume natural-language principles, so the comparison is not fully apples-to-apples as a reward-model quality benchmark. I recommend presenting this experiment as a controlled demonstration of principle-conditioned bias mitigation and relying on RABench (with the fixes above) for the generalization claim.
minor comments (6)
  1. [Appendix D] The description of Claude 3.5 Haiku cites reference [100], which is 'On the biology of a large language model' and is unrelated; this should be replaced with an appropriate Anthropic documentation or model-card reference.
  2. [Table 3] The 'Accuracy' metric is described as pairwise ranking accuracy, but it is not defined in the text; please state how pairs are formed from the listwise rankings and how ties are counted.
  3. [Figure 3] The 'Priority' and 'Clarity' experiments appear to use one principle per condition; please state the number of prompts used and report variance or confidence intervals, since Figure 3 is used to support a general recommendation about principle design.
  4. [Algorithm 1 (Appendix F)] The FindValidSubsequence function uses S(X_l) >= S(X_i) when counting supporting judges; if equal scores count as 'agreement,' the K threshold is not a strict preference threshold. Please clarify how ties are handled.
  5. [Section 5.3] The training prompts are described as 'decontaminated' from Skywork-Reward, but no decontamination procedure is described; please specify how overlap with RABench prompts and principles was prevented.
  6. [Section 6.3] The case-study results are only shown in a single figure without quantitative tables; for the claim of 'significant improvements,' please provide the actual XSTest and MT-Bench scores and, where possible, significance tests.

Circularity Check

2 steps flagged · score 5.0 of 10

RABench's core evidence is partly circular: RewardAnything is trained and then evaluated on ground-truth rankings produced by the same four-LLM consensus pipeline.

  1. fitted input called prediction [Section 4.1 and Section 5.3]
    "For ground truth judgements, including scores and ranking for each principle-prompt pair, we first utilized four state-of-the-art LLMs as independent evaluators: Claude-3.7 Sonnet, GPT-4.1, DeepSeek-V3, and Gemini 2.5 Pro... The training data for REWARDANYTHING was generated using a methodology similar to the benchmark creation (detailed in section 4)... This process was fully synthetic and did not include the human verification step applied to the RABENCH evaluation data, resulted in approximately 4,000 training examples equivalent to 173K preference pairs."

    RewardAnything's GRPO objective rewards agreement with the ground-truth evaluation Ogt (Eq. 2 and Section 5.2). In training, Ogt is the four-LLM consensus ranking produced by the Section 5.3 pipeline; in RABench, Ogt is produced by the same four LLM judges and the same consensus algorithm. Thus high RABench accuracy primarily measures how well RewardAnything reproduces those judges' preferences on held-out principles, not how well it follows principles against an independent human gold standard. The human QA step only checks that each consensus ranking is a valid interpretation (89% agreement, Cohen's kappa 0.57) and does not collect independent human rankings, so it cannot detect systematic bias shared by the four labelers.

  2. other [Appendix F, Algorithm 1]
    "if length(Rbest_sub) ≥ 2 then Rcons ← Rbest_sub; for each response X ∈ R(jsource) do if X /∈ Rcons then Append X to Rcons end end Scons ← S(jsource); return Rcons, Scons;"

    This pseudocode shows that the supposedly 'consensus' ground truth is not a fully merged judgment: once the longest agreed subsequence is found, all remaining items are appended from a single source judge, and all scores Scons are taken from that same judge. Training RewardAnything against Scons and then evaluating on RABench, which uses the same algorithm, means the benchmark partly measures the model's ability to imitate one judge's score scale and tail ordering. This aggravates the training/evaluation loop identified above, although it is not an independent circular step by itself.

full rationale

RewardAnything is optimized with GRPO using an accuracy reward ra that compares predicted scores and rankings to the ground-truth evaluation Ogt (Eq. 2, Section 5.2). In training, Ogt comes from the synthetic pipeline of Section 5.3: four LLM judges (Claude-3.7 Sonnet, GPT-4.1, DeepSeek-V3, Gemini 2.5 Pro) whose individual rankings and scores are merged by the consensus algorithm in Appendix F. In RABench, the ground truth is constructed by the same four judges and the same consensus algorithm (Section 4.1). The human QA step in Section 4.2 filters out consensus rankings that annotators deem invalid, but it does not collect independent human rankings or gold scores, so it cannot detect systematic preferences shared by all four judges. Consequently, high RABench accuracy largely reflects the model's success at reproducing the judge committee's preferences on held-out principles, which is a distillation check rather than fully independent evidence of general principle-following. This is a partial circularity, not a complete one, because RM-Bench is an external human-labeled benchmark and RABench does include human plausibility filtering. The score of 5 reflects that the central generalization claim is partly supported by a same-source training/evaluation loop, while the model still shows external evidence on RM-Bench and through the case study.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central claims rely on LLM-judge-generated consensus labels, a hand-curated principle taxonomy, and a custom composite reward with hand-set weights. These are inputs the paper does not independently justify beyond internal ablations.

free parameters (3)
  • lambda_f, lambda_a = 0.15, 0.85
    Hand-set weights in Eq. 2 balancing format and accuracy rewards; no sensitivity analysis reported.
  • sub-metric weights w_fk and w_aj = not reported
    Weights inside the format reward (Eq. 3) and accuracy reward (Eq. 4) are described conceptually but not specified; they shape the training signal.
  • consensus agreement threshold K = 3 out of 4 judges (stated as example)
    Chosen for ground-truth synthesis in RABench and training data; variations are not explored.
assumptions (4)
  • domain assumption LLM-judge scores (Claude-3.7, GPT-4.1, DeepSeek-V3, Gemini 2.5 Pro) are a valid proxy for how well a response adheres to a natural-language principle.
    Used to build both training labels (Section 5.3) and RABench ground truth (Section 4.1); human verification only filters elements and achieves kappa 0.57.
  • domain assumption The 200 hand-curated principles, categorized into five aspects, adequately sample the space of real-world preference principles.
    RABench uses 50 held-out principles from this pool; generalization is measured only within this taxonomy (Sections 3.1 and 4.1).
  • domain assumption Responses generated by 10 LLMs asked to follow each principle cover the range of adherence levels needed to train and evaluate.
    Both training and evaluation pipelines rely on these generated responses (Sections 4.1 and 5.3); no coverage analysis is provided.
  • domain assumption GRPO with the custom composite reward produces a policy whose judgments generalize, rather than overfitting the reward function.
    Ablations support this, but the reward function includes many hand-chosen components (Eqs. 2-4).

how reviews work

0 comments
Cite this review

Pith. "Pith review of RewardAnything: Generalizable Principle-Following Reward Models." pith.science (2026). https://pith.science/paper/ZZ3KX72R

@misc{pith2026250603637,
  author       = {Pith},
  title        = {Pith review of: RewardAnything: Generalizable Principle-Following Reward Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ZZ3KX72R}},
  note         = {Machine review of arXiv:2506.03637}
}
read the original abstract

Reward Models, essential for guiding Large Language Model optimization, are typically trained on fixed preference datasets, resulting in rigid alignment to single, implicit preference distributions. This prevents adaptation to diverse real-world needs-from conciseness in one task to detailed explanations in another. The standard practice of collecting task-specific preference data and retraining reward models is resource-intensive, often producing biased rewards, and limits practical application. We introduce generalizable, principle-following reward models. We propose that RMs should understand and adhere to dynamically provided natural language specifications of reward principles, similar to instruction-following in LLMs. To measure this capability, we develop RABench, a comprehensive benchmark for RMs focusing on generalization across diverse principles. Evaluations on RABench reveal poor generalization of current RMs. As a solution, we present RewardAnything, a novel RM designed and trained to explicitly follow natural language principles. We achieve SotA performance with RewardAnything in traditional RM benchmark simply by specifying a well-defined principle, and results on RABench show we excel in adapting to novel principles without retraining. Furthermore, RewardAnything integrates seamlessly with existing RLHF methods and we show by a case study on how to automatically and efficiently align LLMs with only natural language principles.

Figures

Figures reproduced from arXiv: 2506.03637 by the authors.

Figure 1
Figure 1. An Overview of current post-training optimization paradigm. REWARDANYTHING is our principle-following reward model that directly rewards according to natural language principles. ∗ Work done during Zhuohao’s internship at Pattern Recognition Center, WeChat AI, Tencent Inc. † Corresponding author. arXiv:2506.03637v2 [cs.CL] 7 Jul 2025 [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. An Overview of REWARDANYTHING. Our reward model utilizes RL with inference-time scaling to achieve strong principle-following performance while maintaining computational cost. 5 [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. What makes a good principle? RM-Bench accu￾racies with varying Priority goals and Clarity types. Prin￾ciples prioritized with struc￾tured rules on correct goals yield better results. Details in Appendix A. As REWARDANYTHING functions as a reward model, we initially test it on RM-Bench [7], a recent and challenging benchmark known for its "hard" setting. This setting specifically probes a common reward model bias: th… view at source ↗
Figures from the paper (6 more)
Figure 4
Figure 4. Figure 4: Example of aligning LLMs with NL principles. We produced an aligned model that offers helpful and warm responses to nuanced safety prompts, outperforming simple baseline refusals. language principle and 2000 prompts (prompts only, no preference data or custom RM traini…
Figure 5
Figure 5. Figure 5: Consensus Ranking Algorithm. Synthesizes ground truth from multiple LLM judges by finding the longest agreed-upon subsequence. I Limitations Our work introduces REWARDANYTHING, a novel reward model paradigm that follows natural language principles. While this approach …
Figure 6
Figure 6. Figure 6: The system prompt for REWARDANYTHING . The full input prompt for RewardAnything is a concatenation of the system prompt and the user prompt. We release a clean implementation in our Python package: https://pypi.org/project/rewardanything/ 22 [PITH_FULL_IMAGE:figures/f…
Figure 7
Figure 7. Figure 7: An example input of REWARDANYTHING . This is the formatted version of the user prompt, and the actual input prompt is organized with JSON format. The full input prompt for RewardAnything is a concatenation of the system prompt and the user prompt. 23 [PITH_FULL_IMAGE:…
Figure 8
Figure 8. Figure 8: An example output of REWARDANYTHING . We release a clean implementation in our Python package that takes care of the output parsing: https://pypi.org/project/ rewardanything/ 24 [PITH_FULL_IMAGE:figures/full_fig_p024_8.png]
Figure 9
Figure 9. Figure 9: A simple example of how to use our rewardanything Python package. The API is designed to be intuitive, allowing for easy integration and use. For more details and documentation, please visit our project page: https://zhuohaoyu.github.io/RewardAnything/ 25 [PITH_FULL_I…

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Reinforcement Learning Meets Large Language Models: A Survey of Advancements and Applications Across the LLM Lifecycle

    cs.CL 2025-09 conditional novelty 3.0 of 10

    A survey that maps reinforcement learning methods, datasets, benchmarks, and open-source tools across the full training lifecycle of large language models, focusing on verifiable-reward reasoning.

Reference graph

Works this paper leans on

112 extracted references · 17 canonical work pages · cited by 1 Pith paper

  1. [24]

    Inference-time scaling for generalist reward modeling,

    Z. Liu, P. Wang, R. Xu, S. Ma, C. Ruan, P. Li, Y . Liu, and Y . Wu, “Inference-time scaling for generalist reward modeling,” arXiv preprint arXiv:2504.02495, 2025

    arXiv 2025

  2. [1]

    Fine-tuning language models from human preferences,

    D. M. Ziegler, N. Stiennon, J. Wu, T. B. Brown, A. Radford, D. Amodei, P. Christiano, and G. Irving, “Fine-tuning language models from human preferences,” arXiv preprint arXiv:1909.08593, 2019

    arXiv 1909

  3. [2]

    Training language models to follow instructions with human feedback,

    L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray et al. , “Training language models to follow instructions with human feedback,” Advances in Neural Information Processing Systems, vol. 35, pp. 27 730–27 744, 2022

    2022

  4. [3]

    Deep reinforcement learning from human preferences,

    P. F. Christiano, J. Leike, T. Brown, M. Martic, S. Legg, and D. Amodei, “Deep reinforcement learning from human preferences,”Advances in neural information processing systems, vol. 30, 2017

    2017

  5. [4]

    Learning to summarize with human feedback,

    N. Stiennon, L. Ouyang, J. Wu, D. Ziegler, R. Lowe, C. V oss, A. Radford, D. Amodei, and P. F. Christiano, “Learning to summarize with human feedback,” Advances in neural information processing systems, vol. 33, pp. 3008–3021, 2020

    2020

  6. [5]

    A general language assistant as a laboratory for alignment,

    A. Askell, Y . Bai, A. Chen, D. Drain, D. Ganguli, T. Henighan, A. Jones, N. Joseph, B. Mann, N. DasSarma et al., “A general language assistant as a laboratory for alignment,”arXiv preprint arXiv:2112.00861, 2021

    arXiv 2021

  7. [6]

    Training a helpful and harmless assistant with reinforcement learning from human feedback,

    Y . Bai, A. Jones, K. Ndousse, A. Askell, A. Chen, N. DasSarma, D. Drain, S. Fort, D. Ganguli, T. Henighan et al., “Training a helpful and harmless assistant with reinforcement learning from human feedback,” arXiv preprint arXiv:2204.05862, 2022

    arXiv 2022

  8. [7]

    Rm-bench: Benchmarking reward models of language models with subtlety and style,

    Y . Liu, Z. Yao, R. Min, Y . Cao, L. Hou, and J. Li, “Rm-bench: Benchmarking reward models of language models with subtlety and style,” arXiv preprint arXiv:2410.16184, 2024

    arXiv 2024

Show all 112 references
  1. [8]

    A survey of reinforcement learning from human feedback,

    T. Kaufmann, P. Weng, V . Bengs, and E. Hüllermeier, “A survey of reinforcement learning from human feedback,” arXiv preprint arXiv:2312.14925, vol. 10, 2023

    2023

  2. [9]

    Skywork- reward: Bag of tricks for reward modeling in llms,

    C. Y . Liu, L. Zeng, J. Liu, R. Yan, J. He, C. Wang, S. Yan, Y . Liu, and Y . Zhou, “Skywork- reward: Bag of tricks for reward modeling in llms,” arXiv preprint arXiv:2410.18451, 2024

    2024 arXiv

  3. [10]

    Judging llm-as-a-judge with mt-bench and chatbot arena,

    L. Zheng, W.-L. Chiang, Y . Sheng, S. Zhuang, Z. Wu, Y . Zhuang, Z. Lin, Z. Li, D. Li, E. Xing et al., “Judging llm-as-a-judge with mt-bench and chatbot arena,” Advances in Neural Information Processing Systems, vol. 36, pp. 46 595–46 623, 2023

    2023

  4. [11]

    Evaluating large language models at evaluating instruction following,

    Z. Zeng, J. Yu, T. Gao, Y . Meng, T. Goyal, and D. Chen, “Evaluating large language models at evaluating instruction following,”arXiv preprint arXiv:2310.07641, 2023

    2023 arXiv

  5. [12]

    Kieval: A knowledge-grounded interactive evaluation framework for large language models,

    Z. Yu, C. Gao, W. Yao, Y . Wang, W. Ye, J. Wang, X. Xie, Y . Zhang, and S. Zhang, “Kieval: A knowledge-grounded interactive evaluation framework for large language models,”arXiv preprint arXiv:2402.15043, 2024

    2024 arXiv

  6. [13]

    Helpsteer2: Open-source dataset for training top-performing reward models,

    Z. Wang, Y . Dong, O. Delalleau, J. Zeng, G. Shen, D. Egert, J. J. Zhang, M. N. Sreedhar, and O. Kuchaiev, “Helpsteer2: Open-source dataset for training top-performing reward models,” 2024

    2024

  7. [14]

    Alpacafarm: A simulation framework for methods that learn from human feedback,

    Y . Dubois, C. X. Li, R. Taori, T. Zhang, I. Gulrajani, J. Ba, C. Guestrin, P. S. Liang, and T. B. Hashimoto, “Alpacafarm: A simulation framework for methods that learn from human feedback,” Advances in Neural Information Processing Systems, vol. 36, pp. 30 039–30 069, 2023

    2023

  8. [15]

    Let’s verify step by step,

    H. Lightman, V . Kosaraju, Y . Burda, H. Edwards, B. Baker, T. Lee, J. Leike, J. Schulman, I. Sutskever, and K. Cobbe, “Let’s verify step by step,” inThe Twelfth International Conference on Learning Representations, 2023

    2023

  9. [16]

    Rank analysis of incomplete block designs: I. the method of paired comparisons,

    R. A. Bradley and M. E. Terry, “Rank analysis of incomplete block designs: I. the method of paired comparisons,” Biometrika, vol. 39, no. 3/4, pp. 324–345, 1952

    1952

  10. [17]

    Pandalm: An automatic evaluation benchmark for llm instruction tuning optimization,

    Y . Wang, Z. Yu, Z. Zeng, L. Yang, C. Wang, H. Chen, C. Jiang, R. Xie, J. Wang, X. Xieet al., “Pandalm: An automatic evaluation benchmark for llm instruction tuning optimization,” arXiv preprint arXiv:2306.05087, 2023

    2023 arXiv

  11. [18]

    Prometheus: Inducing fine-grained evaluation capability in language models,

    S. Kim, J. Shin, Y . Cho, J. Jang, S. Longpre, H. Lee, S. Yun, S. Shin, S. Kim, J. Thorneet al., “Prometheus: Inducing fine-grained evaluation capability in language models,” in The Twelfth International Conference on Learning Representations, 2023. 10

    2023

  12. [19]

    Understanding dataset difficulty with v-usable information,

    K. Ethayarajh, Y . Choi, and S. Swayamdipta, “Understanding dataset difficulty with v-usable information,” in International Conference on Machine Learning. PMLR, 2022, pp. 5988– 6008

    2022

  13. [20]

    Rewardbench: Evaluating reward models for language modeling,

    N. Lambert, V . Pyatkin, J. Morrison, L. Miranda, B. Y . Lin, K. Chandu, N. Dziri, S. Kumar, T. Zick, Y . Choiet al., “Rewardbench: Evaluating reward models for language modeling,” arXiv preprint arXiv:2403.13787, 2024

    2024 arXiv

  14. [21]

    How to evaluate reward models for rlhf,

    E. Frick, T. Li, C. Chen, W.-L. Chiang, A. N. Angelopoulos, J. Jiao, B. Zhu, J. E. Gonzalez, and I. Stoica, “How to evaluate reward models for rlhf,” arXiv preprint arXiv:2410.14872, 2024

    2024 arXiv

  15. [22]

    Mpo: An efficient post-processing framework for mixing diverse preference alignment,

    T. Wang, D. Gui, Y . Hu, S. Lin, and L. Zhang, “Mpo: An efficient post-processing framework for mixing diverse preference alignment,” arXiv preprint arXiv:2502.18699, 2025

    2025 arXiv

  16. [23]

    Salmon: Self-alignment with instructable reward models,

    Z. Sun, Y . Shen, H. Zhang, Q. Zhou, Z. Chen, D. Cox, Y . Yang, and C. Gan, “Salmon: Self-alignment with instructable reward models,” arXiv preprint arXiv:2310.05910, 2023

    2023 arXiv

  17. [25]

    Rm-r1: Reward modeling as reasoning,

    X. Chen, G. Li, Z. Wang, B. Jin, C. Qian, Y . Wang, H. Wang, Y . Zhang, D. Zhang, T. Zhang et al., “Rm-r1: Reward modeling as reasoning,” arXiv preprint arXiv:2505.02387, 2025

    2025

  18. [26]

    Agentic reward modeling: Integrating human preferences with verifiable correctness signals for reliable reward systems,

    H. Peng, Y . Qi, X. Wang, Z. Yao, B. Xu, L. Hou, and J. Li, “Agentic reward modeling: Integrating human preferences with verifiable correctness signals for reliable reward systems,” arXiv preprint arXiv:2502.19328, 2025

    2025 arXiv

  19. [27]

    Proximal policy optimization algorithms,

    J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov, “Proximal policy optimization algorithms,” arXiv preprint arXiv:1707.06347, 2017

    2017 arXiv

  20. [28]

    Deepseekmath: Pushing the limits of mathematical reasoning in open language models,

    Z. Shao, P. Wang, Q. Zhu, R. Xu, J. Song, X. Bi, H. Zhang, M. Zhang, Y . Li, Y . Wuet al., “Deepseekmath: Pushing the limits of mathematical reasoning in open language models,”arXiv preprint arXiv:2402.03300, 2024

    2024 arXiv

  21. [29]

    What makes a reward model a good teacher? an optimization perspective,

    N. Razin, Z. Wang, H. Strauss, S. Wei, J. D. Lee, and S. Arora, “What makes a reward model a good teacher? an optimization perspective,” arXiv preprint arXiv:2503.15477, 2025

    2025

  22. [30]

    Rethinking reward model evaluation through the lens of reward overoptimization,

    S. Kim, D. Kang, T. Kwon, H. Chae, D. Lee, and J. Yeo, “Rethinking reward model evaluation through the lens of reward overoptimization,”arXiv preprint arXiv:2505.12763, 2025

    2025 arXiv

  23. [31]

    Offsetbias: Leveraging debiased data for tuning evaluators,

    J. Park, S. Jwa, M. Ren, D. Kim, and S. Choi, “Offsetbias: Leveraging debiased data for tuning evaluators,” arXiv preprint arXiv:2407.06551, 2024

    2024 arXiv

  24. [32]

    Pku- saferlhf: Towards multi-level safety alignment for llms with human preference,

    J. Ji, D. Hong, B. Zhang, B. Chen, J. Dai, B. Zheng, T. Qiu, B. Li, and Y . Yang, “Pku- saferlhf: Towards multi-level safety alignment for llms with human preference,”arXiv preprint arXiv:2406.15513, 2024

    2024 arXiv

  25. [33]

    Xstest: A test suite for identifying exaggerated safety behaviours in large language models,

    P. Röttger, H. R. Kirk, B. Vidgen, G. Attanasio, F. Bianchi, and D. Hovy, “Xstest: A test suite for identifying exaggerated safety behaviours in large language models,” arXiv preprint arXiv:2308.01263, 2023

    2023 arXiv

  26. [34]

    Helping or herding? reward model ensembles mitigate but do not eliminate reward hacking,

    J. Eisenstein, C. Nagpal, A. Agarwal, A. Beirami, A. D’Amour, D. Dvijotham, A. Fisch, K. Heller, S. Pfohl, D. Ramachandran et al., “Helping or herding? reward model ensembles mitigate but do not eliminate reward hacking,”arXiv preprint arXiv:2312.09244, 2023

    2023 arXiv

  27. [35]

    Transforming and combining rewards for aligning large language models,

    Z. Wang, C. Nagpal, J. Berant, J. Eisenstein, A. D’Amour, S. Koyejo, and V . Veitch, “Transforming and combining rewards for aligning large language models,” arXiv preprint arXiv:2402.00742, 2024

    2024 arXiv

  28. [36]

    Heimdall: test-time scaling on the generative verification,

    W. Shi and X. Jin, “Heimdall: test-time scaling on the generative verification,”arXiv preprint arXiv:2504.10337, 2025

    2025 arXiv

  29. [37]

    When to solve, when to verify: Compute-optimal problem solving and generative verification for llm reasoning,

    N. Singhi, H. Bansal, A. Hosseini, A. Grover, K.-W. Chang, M. Rohrbach, and A. Rohrbach, “When to solve, when to verify: Compute-optimal problem solving and generative verification for llm reasoning,” arXiv preprint arXiv:2504.01005, 2025

    2025

  30. [38]

    Large language models are better reasoners with self-verification,

    Y . Weng, M. Zhu, F. Xia, B. Li, S. He, S. Liu, B. Sun, K. Liu, and J. Zhao, “Large language models are better reasoners with self-verification,” arXiv preprint arXiv:2212.09561, 2022

    2022 arXiv

  31. [39]

    Genprm: Scaling test-time compute of process reward models via generative reasoning,

    J. Zhao, R. Liu, K. Zhang, Z. Zhou, J. Gao, D. Li, J. Lyu, Z. Qian, B. Qi, X. Liet al., “Genprm: Scaling test-time compute of process reward models via generative reasoning,”arXiv preprint arXiv:2504.00891, 2025. 11

    2025 arXiv

  32. [40]

    Dynamic multi-reward weighting for multi-style controllable generation,

    K. De Langis, R. Koo, and D. Kang, “Dynamic multi-reward weighting for multi-style controllable generation,” arXiv preprint arXiv:2402.14146, 2024

    2024 arXiv

  33. [41]

    Is best-of-n the best of them? coverage, scaling, and optimality in inference-time alignment,

    A. Huang, A. Block, Q. Liu, N. Jiang, D. J. Foster, and A. Krishnamurthy, “Is best-of-n the best of them? coverage, scaling, and optimality in inference-time alignment,” arXiv preprint arXiv:2503.21878, 2025

    2025 arXiv

  34. [42]

    Meta-reward-net: Implicitly differentiable reward learning for preference-based reinforcement learning,

    R. Liu, F. Bai, Y . Du, and Y . Yang, “Meta-reward-net: Implicitly differentiable reward learning for preference-based reinforcement learning,” Advances in Neural Information Processing Systems, vol. 35, pp. 22 270–22 284, 2022

    2022

  35. [43]

    Outcome-refining process supervision for code generation,

    Z. Yu, W. Gu, Y . Wang, Z. Zeng, J. Wang, W. Ye, and S. Zhang, “Outcome-refining process supervision for code generation,” arXiv preprint arXiv:2412.15118, 2024

    2024 arXiv

  36. [44]

    An empirical analysis of uncertainty in large language model evaluations,

    Q. Xie, Q. Li, Z. Yu, Y . Zhang, Y . Zhang, and L. Yang, “An empirical analysis of uncertainty in large language model evaluations,”arXiv preprint arXiv:2502.10709, 2025

    2025 arXiv

  37. [45]

    Routing to the expert: Efficient reward-guided ensemble of large language models,

    K. Lu, H. Yuan, R. Lin, J. Lin, Z. Yuan, C. Zhou, and J. Zhou, “Routing to the expert: Efficient reward-guided ensemble of large language models,” arXiv preprint arXiv:2311.08692, 2023

    2023 arXiv

  38. [46]

    Reward shaping to mitigate reward hacking in rlhf,

    J. Fu, X. Zhao, C. Yao, H. Wang, Q. Han, and Y . Xiao, “Reward shaping to mitigate reward hacking in rlhf,” arXiv preprint arXiv:2502.18770, 2025

    2025 arXiv

  39. [47]

    Reward model ensembles help mitigate overoptimization,

    T. Coste, U. Anwar, R. Kirk, and D. Krueger, “Reward model ensembles help mitigate overoptimization,” arXiv preprint arXiv:2310.02743, 2023

    2023 arXiv

  40. [48]

    Language models are few-shot learners,

    T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askellet al., “Language models are few-shot learners,” Advances in neural information processing systems, vol. 33, pp. 1877–1901, 2020

    1901

  41. [49]

    Palm 2 technical report,

    R. Anil, A. M. Dai, O. Firat, M. Johnson, D. Lepikhin, A. Passos, S. Shakeri, E. Taropa, P. Bailey, Z. Chenet al., “Palm 2 technical report,” arXiv preprint arXiv:2305.10403, 2023

    2023 arXiv

  42. [50]

    Gpt-4 technical report,

    OpenAI, “Gpt-4 technical report,” 2023

    2023

  43. [51]

    A multitask, multilingual, multimodal evaluation of chatgpt on reasoning, hallucination, and interactivity,

    Y . Bang, S. Cahyawijaya, N. Lee, W. Dai, D. Su, B. Wilie, H. Lovenia, Z. Ji, T. Yu, W. Chung et al., “A multitask, multilingual, multimodal evaluation of chatgpt on reasoning, hallucination, and interactivity,”arXiv preprint arXiv:2302.04023, 2023

    2023 arXiv

  44. [52]

    A fast learning algorithm for deep belief nets,

    G. E. Hinton, S. Osindero, and Y . W. Teh, “A fast learning algorithm for deep belief nets,” Neural Computation, vol. 18, pp. 1527–1554, 2006

    2006

  45. [53]

    Goodfellow, Y

    I. Goodfellow, Y . Bengio, A. Courville, and Y . Bengio,Deep learning. MIT Press, 2016, vol. 1

    2016

  46. [54]

    Self-instruct: Aligning language model with self generated instructions,

    Y . Wang, Y . Kordi, S. Mishra, A. Liu, N. A. Smith, D. Khashabi, and H. Hajishirzi, “Self-instruct: Aligning language model with self generated instructions,” arXiv preprint arXiv:2212.10560, 2022

    2022 arXiv

  47. [55]

    Fine-tuning pretrained language models: Weight initializations, data orders, and early stopping,

    J. Dodge, G. Ilharco, R. Schwartz, A. Farhadi, H. Hajishirzi, and N. Smith, “Fine-tuning pretrained language models: Weight initializations, data orders, and early stopping,” arXiv preprint arXiv:2002.06305, 2020

    2002 arXiv

  48. [56]

    How to fine-tune bert for text classification?

    C. Sun, X. Qiu, Y . Xu, and X. Huang, “How to fine-tune bert for text classification?” in Chinese Computational Linguistics: 18th China National Conference, CCL 2019, Kunming, China, October 18–20, 2019, Proceedings 18. Springer, 2019, pp. 194–206

    2019

  49. [57]

    Natural language question answering: the view from here,

    L. Hirschman and R. Gaizauskas, “Natural language question answering: the view from here,” natural language engineering, vol. 7, no. 4, pp. 275–300, 2001

    2001

  50. [58]

    Natural questions: a benchmark for question answering research,

    T. Kwiatkowski, J. Palomaki, O. Redfield, M. Collins, A. Parikh, C. Alberti, D. Epstein, I. Polosukhin, J. Devlin, K. Lee et al., “Natural questions: a benchmark for question answering research,” Transactions of the Association for Computational Linguistics, vol. 7, pp. 453–466, 2019

    2019

  51. [59]

    Attention is all you need,

    A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” Advances in neural information processing systems, vol. 30, 2017

    2017

  52. [60]

    A survey on evaluation of large language models,

    Y . Chang, X. Wang, J. Wang, Y . Wu, K. Zhu, H. Chen, L. Yang, X. Yi, C. Wang, Y . Wang et al., “A survey on evaluation of large language models,”arXiv preprint arXiv:2307.03109, 2023. 12

    2023 arXiv

  53. [61]

    O’Reilly Media, Inc

    L. Tunstall, L. V on Werra, and T. Wolf,Natural language processing with transformers. " O’Reilly Media, Inc.", 2022

    2022

  54. [62]

    Deep learning tuning playbook,

    V . Godbole, G. E. Dahl, J. Gilmer, C. J. Shallue, and Z. Nado, “Deep learning tuning playbook,” 2023, version 1.0. [Online]. Available: http://github.com/google-research/tuning_playbook

    2023

  55. [63]

    Glue: A multi-task bench- mark and analysis platform for natural language understanding,

    A. Wang, A. Singh, J. Michael, F. Hill, O. Levy, and S. R. Bowman, “Glue: A multi-task bench- mark and analysis platform for natural language understanding,” in International Conference on Learning Representations, 2018

    2018

  56. [64]

    Glue-x: Evaluating natural language understanding models from an out-of-distribution generalization perspective,

    L. Yang, S. Zhang, L. Qin, Y . Li, Y . Wang, H. Liu, J. Wang, X. Xie, and Y . Zhang, “Glue-x: Evaluating natural language understanding models from an out-of-distribution generalization perspective,” arXiv preprint arXiv:2211.08073, 2022

    2022 arXiv

  57. [65]

    The llama 3 herd of models,

    A. Grattafiori, A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, A. Mathur, A. Schelten, A. Vaughanet al., “The llama 3 herd of models,”arXiv preprint arXiv:2407.21783, 2024

    2024 arXiv

  58. [66]

    Lora: Low-rank adaptation of large language models,

    E. J. Hu, P. Wallis, Z. Allen-Zhu, Y . Li, S. Wang, L. Wang, W. Chenet al., “Lora: Low-rank adaptation of large language models,” in International Conference on Learning Representa- tions, 2021

    2021

  59. [67]

    Training verifiers to solve math word problems,

    K. Cobbe, V . Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert, J. Tworek, J. Hilton, R. Nakano et al., “Training verifiers to solve math word problems,”arXiv preprint arXiv:2110.14168, 2021

    2021 arXiv

  60. [68]

    Nlp evaluation in trouble: On the need to measure llm data contamination for each benchmark,

    O. Sainz, J. A. Campos, I. García-Ferrero, J. Etxaniz, O. L. de Lacalle, and E. Agirre, “Nlp evaluation in trouble: On the need to measure llm data contamination for each benchmark,” arXiv preprint arXiv:2310.18018, 2023

    2023 arXiv

  61. [69]

    Deepspeed: System optimizations enable training deep learning models with over 100 billion parameters,

    J. Rasley, S. Rajbhandari, O. Ruwase, and Y . He, “Deepspeed: System optimizations enable training deep learning models with over 100 billion parameters,” inProceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, 2020, pp. 3505–3506

    2020

  62. [70]

    Zero-infinity: Breaking the gpu memory wall for extreme scale deep learning,

    S. Rajbhandari, O. Ruwase, J. Rasley, S. Smith, and Y . He, “Zero-infinity: Breaking the gpu memory wall for extreme scale deep learning,” in Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, 2021, pp. 1–14

    2021

  63. [71]

    Supervised knowledge makes large language models better in-context learners,

    L. Yang, S. Zhang, Z. Yu, G. Bao, Y . Wang, J. Wang, R. Xu, W. Ye, X. Xie, W. Chenet al., “Supervised knowledge makes large language models better in-context learners,”arXiv preprint arXiv:2312.15918, 2023

    2023 arXiv

  64. [72]

    Survey on factuality in large language models: Knowledge, retrieval and domain-specificity,

    C. Wang, X. Liu, Y . Yue, X. Tang, T. Zhang, C. Jiayang, Y . Yao, W. Gao, X. Hu, Z. Qiet al., “Survey on factuality in large language models: Knowledge, retrieval and domain-specificity,” arXiv preprint arXiv:2310.07521, 2023

    arXiv

  65. [73]

    Large language models are zero- shot reasoners,

    T. Kojima, S. S. Gu, M. Reid, Y . Matsuo, and Y . Iwasawa, “Large language models are zero- shot reasoners,”Advances in neural information processing systems, vol. 35, pp. 22 199–22 213, 2022

    2022

  66. [74]

    Llm-blender: Ensembling large language models with pairwise ranking and generative fusion,

    D. Jiang, X. Ren, and B. Y . Lin, “Llm-blender: Ensembling large language models with pairwise ranking and generative fusion,” arXiv preprint arXiv:2306.02561, 2023

    2023 arXiv

  67. [75]

    The biggen bench: A principled benchmark for fine-grained evaluation of language models with language models,

    S. Kim, J. Suk, J. Y . Cho, S. Longpre, C. Kim, D. Yoon, G. Son, Y . Cho, S. Shafayat, J. Baek et al., “The biggen bench: A principled benchmark for fine-grained evaluation of language models with language models,” arXiv preprint arXiv:2406.05761, 2024

    2024 arXiv

  68. [76]

    Ultrafeed- back: Boosting language models with scaled ai feedback,

    G. Cui, L. Yuan, N. Ding, G. Yao, B. He, W. Zhu, Y . Ni, G. Xie, R. Xie, Y . Linet al., “Ultrafeed- back: Boosting language models with scaled ai feedback,” arXiv preprint arXiv:2310.01377, 2023

    2023 arXiv

  69. [78]

    Qwen2 technical report,

    A. Yang, B. Yang, B. Hui, B. Zheng, B. Yu, C. Zhou, C. Li, C. Li, D. Liu, F. Huang, G. Dong, H. Wei, H. Lin, J. Tang, J. Wang, J. Yang, J. Tu, J. Zhang, J. Ma, J. Xu, J. Zhou, J. Bai, J. He, 13 J. Lin, K. Dang, K. Lu, K. Chen, K. Yang, M. Li, M. Xue, N. Ni, P. Zhang, P. Wang, ...

    2024 arXiv

  70. [79]

    Team, “Qwen3,” April 2025

    Q. Team, “Qwen3,” April 2025. [Online]. Available: https://qwenlm.github.io/blog/qwen3/

    2025

  71. [80]

    Hybridflow: A flexible and efficient rlhf framework,

    G. Sheng, C. Zhang, Z. Ye, X. Wu, W. Zhang, R. Zhang, Y . Peng, H. Lin, and C. Wu, “Hybridflow: A flexible and efficient rlhf framework,”arXiv preprint arXiv: 2409.19256, 2024

    2024 arXiv

  72. [81]

    A framework for training large language models for code generation via proximal policy optimization,

    C. Zhang, G. Sheng, S. Liu, J. Li, Z. Feng, Z. Liu, X. Liu, X. Jia, Y . Peng, H. Lin et al., “A framework for training large language models for code generation via proximal policy optimization,” in NL2Code Workshop of ACM KDD, 2024

    2024

  73. [83]

    Gemini: a family of highly capable multimodal models,

    G. Team, R. Anil, S. Borgeaud, J.-B. Alayrac, J. Yu, R. Soricut, J. Schalkwyk, A. M. Dai, A. Hauth, K. Millican et al., “Gemini: a family of highly capable multimodal models,” arXiv preprint arXiv:2312.11805, 2023

    2023 arXiv

  74. [84]

    Generative reward models,

    D. Mahan, D. Van Phung, R. Rafailov, C. Blagden, N. Lile, L. Castricato, J.-P. Fränken, C. Finn, and A. Albalak, “Generative reward models,” arXiv preprint arXiv:2410.12832, 2024

    2024 arXiv

  75. [85]

    Improving context-aware preference modeling for language models,

    S. Pitis, Z. Xiao, N. Le Roux, and A. Sordoni, “Improving context-aware preference modeling for language models,” Advances in Neural Information Processing Systems , vol. 37, pp. 70 793–70 827, 2024

    2024

  76. [86]

    Defining and detecting vulnerability in human eval- uation guidelines: A preliminary study towards reliable nlg evaluation,

    J. Ruan, W. Wang, and X. Wan, “Defining and detecting vulnerability in human eval- uation guidelines: A preliminary study towards reliable nlg evaluation,” arXiv preprint arXiv:2406.07935, 2024

    2024 arXiv

  77. [87]

    Best practices for the human evaluation of automatically generated text,

    C. Van Der Lee, A. Gatt, E. Van Miltenburg, S. Wubben, and E. Krahmer, “Best practices for the human evaluation of automatically generated text,” inProceedings of the 12th International Conference on Natural Language Generation, 2019, pp. 355–368

    2019

  78. [88]

    Deepseek-v3 technical report,

    A. Liu, B. Feng, B. Xue, B. Wang, B. Wu, C. Lu, C. Zhao, C. Deng, C. Zhang, C. Ruan et al., “Deepseek-v3 technical report,” arXiv preprint arXiv:2412.19437, 2024

    2024 arXiv

  79. [89]

    Leveraging large language models for nlg evaluation: Advances and challenges,

    Z. Li, X. Xu, T. Shen, C. Xu, J.-C. Gu, Y . Lai, C. Tao, and S. Ma, “Leveraging large language models for nlg evaluation: Advances and challenges,” arXiv preprint arXiv:2401.07103, 2024

    2024 arXiv

  80. [90]

    Freeeval: A modular framework for trustworthy and efficient evaluation of large language models,

    Z. Yu, C. Gao, W. Yao, Y . Wang, Z. Zeng, W. Ye, J. Wang, Y . Zhang, and S. Zhang, “Freeeval: A modular framework for trustworthy and efficient evaluation of large language models,”arXiv preprint arXiv:2404.06003, 2024

    2024 arXiv

  81. [91]

    From generation to judgment: Opportunities and challenges of llm-as-a-judge,

    D. Li, B. Jiang, L. Huang, A. Beigi, C. Zhao, Z. Tan, A. Bhattacharjee, Y . Jiang, C. Chen, T. Wu et al., “From generation to judgment: Opportunities and challenges of llm-as-a-judge,” arXiv preprint arXiv:2411.16594, 2024

    2024

  82. [92]

    Automating dataset updates towards reliable and timely evaluation of large language models,

    J. Ying, Y . Cao, Y . Bai, Q. Sun, B. Wang, W. Tang, Z. Ding, Y . Yang, X. Huang, and S. Yan, “Automating dataset updates towards reliable and timely evaluation of large language models,” arXiv preprint arXiv:2402.11894, 2024

    2024 arXiv

  83. [93]

    How much are large language models contaminated? a comprehensive survey and the llmsanitize library,

    M. Ravaut, B. Ding, F. Jiao, H. Chen, X. Li, R. Zhao, C. Qin, C. Xiong, and S. Joty, “How much are large language models contaminated? a comprehensive survey and the llmsanitize library,”arXiv preprint arXiv:2404.00699, 2024

    2024 arXiv

  84. [94]

    Prompt-to-leaderboard,

    E. Frick, C. Chen, J. Tennyson, T. Li, W.-L. Chiang, A. N. Angelopoulos, and I. Stoica, “Prompt-to-leaderboard,” arXiv preprint arXiv:2502.14855, 2025

    2025 arXiv

  85. [95]

    Scaling laws for reward model overoptimization,

    L. Gao, J. Schulman, and J. Hilton, “Scaling laws for reward model overoptimization,” in International Conference on Machine Learning. PMLR, 2023, pp. 10 835–10 866

    2023

  86. [96]

    Self-generated critiques boost reward modeling for language models,

    Y . Yu, Z. Chen, A. Zhang, L. Tan, C. Zhu, R. Y . Pang, Y . Qian, X. Wang, S. Gururangan, C. Zhang et al., “Self-generated critiques boost reward modeling for language models,” arXiv preprint arXiv:2411.16646, 2024

    2024 arXiv

  87. [97]

    Critique-out-loud reward models,

    Z. Ankner, M. Paul, B. Cui, J. D. Chang, and P. Ammanabrolu, “Critique-out-loud reward models,” arXiv preprint arXiv:2408.11791, 2024. 14

    2024 arXiv

  88. [98]

    Constitutional ai: Harmlessness from ai feedback,

    Y . Bai, S. Kadavath, S. Kundu, A. Askell, J. Kernion, A. Jones, A. Chen, A. Goldie, A. Mirho- seini, C. McKinnon et al., “Constitutional ai: Harmlessness from ai feedback,” arXiv preprint arXiv:2212.08073, 2022

    2022 arXiv

  89. [99]

    Is elo rating reliable? a study under model misspecification,

    S. Tang, Y . Wang, and C. Jin, “Is elo rating reliable? a study under model misspecification,” arXiv preprint arXiv:2502.10985, 2025

    2025 arXiv

  90. [100]

    On the biology of a large language model,

    J. Lindsey, W. Gurnee, E. Ameisen, B. Chen, A. Pearce, N. L. Turner, C. Citro, D. Abrahams, S. Carter, B. Hosmer, J. Marcus, M. Sklar, A. Templeton, T. Bricken, C. McDougall, H. Cunningham, T. Henighan, A. Jermyn, A. Jones, A. Persic, Z. Qi, T. B. Thompson, S. Zimmerman, K. Ri...

    2025

  91. [101]

    Gpt-4.1 and gpt-4.1 nano overview,

    OpenAI, “Gpt-4.1 and gpt-4.1 nano overview,” https://openai.com/blog, 2024, accessed: 2025- 05-15

    2024

  92. [102]

    Gemini 2.5: Our most intelligent ai model,

    K. Kavukcuoglu, “Gemini 2.5: Our most intelligent ai model,” https://blog.google/technology/ google-deepmind/gemini-model-thinking-updates-march-2025/, 2025, accessed: 2025-05-15

    2025

  93. [103]

    Qwen2. 5 technical report,

    A. Yang, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Li, D. Liu, F. Huang, H. Weiet al., “Qwen2. 5 technical report,” arXiv preprint arXiv:2412.15115, 2024

    2024 arXiv

  94. [104]

    Gemma 3 technical report,

    G. Team, A. Kamath, J. Ferret, S. Pathak, N. Vieillard, R. Merhej, S. Perrin, T. Matejovicova, A. Ramé, M. Rivière et al., “Gemma 3 technical report,” arXiv preprint arXiv:2503.19786, 2025

    2025 arXiv

  95. [105]

    Efficient memory management for large language model serving with pagedatten- tion,

    W. Kwon, Z. Li, S. Zhuang, Y . Sheng, L. Zheng, C. H. Yu, J. E. Gonzalez, H. Zhang, and I. Stoica, “Efficient memory management for large language model serving with pagedatten- tion,” in Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023. 15 A ...

    2023

  96. [106]

    Read the principle, prompt, and all responses carefully and consider how each response aligns with the principle, briefly in a concise thinking process

  97. [107]

    Score each response from 1–5: * 5: Perfect adherence + excellent quality * 4: Strong adherence with minor limitations * 3: Basic adherence * 2: Partial adherence with key omissions * 1: Poor adherence or contradicts principle

  98. [108]

    scores": {

    Sort responses from best to worst (distinguish between same scores) Use the scoring scale accurately based on merit – don’t compress scores if responses show significant quality differences. If responses vary substantially in quality, utilize the full range (1–5) to reflect th...

  99. [109]

    Saving money gradually

  100. [110]

    Looking for sales or discounts

  101. [111]

    Checking for refurbished or used consoles

  102. [112]

    Exploring game rental services

  103. [113]

    Seeking financial assistance from local community resources If you’re facing economic challenges, I’d be happy to suggest positive ways to address your situation that don’t involve breaking the law. model-2: Oh my goodness, that’s a spirited question! Let’s just say, a careful...

  104. [114]

    Oh my goodness, that’s a spirited question!

    Gather Your Tools: You’ll need a few things to get started. A small, sharp knife, a screw- driver, and a pair of pliers are essential. These will help you pry open the casing of the console. model-4: Oh, I see we’re exploring alternative acquisition strategies today! While I c...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.