Pith. sign in

REVIEW 4 major objections 6 minor 1 cited by

Value-Free Policy Optimization via Reward Partitioning

T0 review · 4 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Reward Partitioning Optimization replaces the learned value function with a per-prompt empirical normalizer and reports consistent alignment gains over SFT, KTO, and DRO.

desk verdict Simple value-free loss with broad experiments, but the partition estimator is biased on UltraFeedback so the optimal-policy claim doesn't hold as written. read the letter →

arxiv 2506.13702 v4 pith:LUD2X2B6 submitted 2025-06-16 cs.LG cs.AI

Bilal Faye , Hanane Azzag , Mustapha Lebbah This is my paper
classification cs.LGcs.AI
keywords RewardPartitionOptimizationsingle-trajectorypreferencedirectvalue-function-freealignmentKL-regularizedpolicyofflineRLHFlanguagemodelnormalization
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper introduces Reward Partitioning Optimization (RPO), an objective for aligning language models from single-trajectory data of the form (prompt, response, reward). RPO's central move is to replace the learned value function used by Direct Reward Optimization (DRO) with an empirical normalizer: for each prompt, it averages reward-weighted reference-policy probabilities over all observed responses, producing an estimate of the partition function, and then regresses the policy's log-ratio onto the prompt-centered reward. The paper argues this directly approximates the optimal KL-regularized policy without an auxiliary value network or reinforcement learning loop. In experiments across encoder-decoder and decoder-only models, RPO is reported to outperform supervised fine-tuning, KTO, and DRO on automatic metrics, LLM-as-a-judge win rates, and out-of-distribution instruction-following benchmarks. A reader would take the contribution to be a simpler and more stable alternative to DRO for reward-only alignment.

What carries the argument

The load-bearing object is the empirical partition function $\hat{Z}(x) = \sum_{j \in I_x} \pi_{\mathrm{ref}}(y_j|x)\exp(r(x,y_j)/\tau)$, a prompt-level sum over observed responses; the paper treats its logarithm as a stand-in for the soft value $V^*(x) = \tau\log Z(x)$. This object carries the argument because it converts DRO's jointly learned value head into a fixed per-prompt statistic computed before policy updates, and the gradient of the RPO loss is a residual-weighted discrepancy between the one-hot target and the current policy without backpropagating through $\hat{V}(x)$. The identity $r(x,y) - V^*(x) = \tau\log(\pi^*(y|x)/\pi_{\mathrm{ref}}(y|x))$ is the theoretical bridge that makes the regression target meaningful.

What would settle it

Retrain RPO on the same prompts with responses actually sampled from the reference policy, or with importance weights that correct for the real sampling distribution, and compare the result against the reported UltraFeedback numbers; if performance does not move, the sampling assumption is not load-bearing, while a significant drop or gain would show that $\hat{Z}(x)$ was biased. A sharper check is to compare $\hat{Z}(x)$ against a high-sample Monte Carlo estimate of $Z(x)$ from fresh reference-policy completions for the same prompts: systematic differences would falsify the claim that RPO regresses onto the optimal-policy log-ratio.

Watch

Extended reading notes

Core claim

RPO is built on the closed-form optimal policy for KL-regularized reward maximization, $\pi^*(y|x) = \pi_{\mathrm{ref}}(y|x)\exp(r(x,y)/\tau)/Z(x)$, where the partition function is $Z(x) = \sum_{y'} \pi_{\mathrm{ref}}(y'|x)\exp(r(x,y')/\tau)$. Taking logarithms gives the identity $r(x,y) - \tau\log Z(x) = \tau\log(\pi^*(y|x)/\pi_{\mathrm{ref}}(y|x))$, the same regression target DRO fits with a learned value function $V(x) = \tau\log Z(x)$. RPO instead estimates $Z(x)$ empirically from the responses available for each prompt, $\hat{Z}(x) = \sum_{j \in I_x} \pi_{\mathrm{ref}}(y_j|x)\exp(r(x,y_j)/\tau)$, and minimizes $L_{\mathrm{RPO}}(\theta) = \frac{1}{2n}\sum_i (\log(\pi_\theta(y_i|x_i)/\pi_{\mathrm{ref}}(y_i|x_i)) - (r_i - \hat{V}(x_i))/\tau)^2$ with $\hat{V}(x) = \tau\log\hat{Z}(x)$. The claim is that this squared-error regression recovers the optimal policy structure exactly when the observed responses are drawn from $\pi_{\mathrm{ref}}$, and that in practice it yields a stable, value-free supervised objective that outperforms SFT, KTO, and DRO.

Load-bearing premise

RPO treats the stored responses for each prompt as draws from the reference policy when it averages them to estimate the partition function; in the dataset used for the experiments those responses were generated by several different external models, so without a correction the estimated normalization can be systematically off.

Editorial extensions

If this is right

  • Removing the value head makes RPO fully offline and reduces the optimization loop to supervised regression on policy log-ratios.
  • RPO can be trained from (prompt, response, reward) triples alone, so it avoids pairwise preference annotation and reinforcement learning.
  • In the reported experiments, RPO improves BERTScore, ROUGE-L, Distinct-2, and toxicity relative to SFT, KTO, and DRO across all six model configurations.
  • LLM-as-a-judge pairwise evaluations prefer RPO over each baseline on the in-domain validation set, and on out-of-distribution benchmarks the preference is strongest for instruction-following tasks.
  • Training-time and KL-divergence measurements indicate that RPO converges faster and drifts less from the reference policy than DRO or KTO.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: the correctness of the regression target depends on the observed responses being samples from $\pi_{\mathrm{ref}}$; if they come from other models, as in the dataset used here, $\hat{Z}(x)$ is a biased estimate of $Z(x)$, and re-weighting by the true sampling distribution or re-sampling from $\pi_{\mathrm{ref}}$ is a natural correction to test.
  • Editorial inference: the method's reliance on several responses per prompt suggests a direct extension to single-response datasets by borrowing responses across similar prompts or learning a prompt embedding that parameterizes the normalizer.
  • Editorial inference: because $\hat{V}(x)$ is computed once from the data, RPO separates reward normalization from policy optimization; one could study whether updating the normalizer periodically during training improves the fit to the optimal policy.
  • Editorial inference: the squared-error form invites comparison with advantage-weighted regression; a testable variant would weight the residual by $\exp(r/\tau)$ or clip large residuals to see whether robustness to reward outliers comes from the normalization or from the quadratic loss.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes Reward Partition Optimization (RPO), a value-free policy optimization method for offline single-trajectory reward data. RPO estimates the partition function of the KL-regularized optimal policy empirically from prompt-level reward groups and then regresses the policy log-ratio to the scaled centered reward, avoiding an explicit learned value function. The authors provide a derivation from the optimal policy form, compare RPO against SFT, KTO, and DRO across encoder-decoder and decoder-only architectures on UltraFeedback, and report in-domain metrics, LLM-as-a-judge win rates, OOD benchmarks, and ablations on temperature, normalization, dataset density, and reward noise.

Significance. If the central derivation were sound, RPO would offer a simple, stable, and value-free alternative to DRO, with potential practical value for offline alignment from scalar feedback. The paper contains a comprehensive experimental study, including multiple model families and detailed ablations, which is a strength. However, the theoretical justification is undermined by a biased partition estimate, and the empirical results lack statistical rigor (no error bars, seeds, or code), leaving the main claims insufficiently supported.

major comments (4)
  1. [III, Eq. (19)-(21)] The empirical partition function bZ(x) is not an unbiased estimate of Z(x) in Eq. (16) because the responses y_j are not drawn from the reference policy pi_ref(·|x). In UltraFeedback, completions are generated by external models, and no importance-weight correction pi_ref/q is applied. Consequently, bZ(x) converges to a weighted expectation under the actual data distribution q, not to Z(x), and the regression target (r_i - bV(x_i))/tau is not the log-ratio of the optimal policy. The derivation of Eq. (20) as an approximation to the optimal policy therefore does not go through; this is a load-bearing gap in the paper's central theoretical claim.
  2. [III, Eq. (21) and Algorithm 1] There is a self-inclusion bias in the value estimate: bV(x_i) is computed from a set I_x that contains the sample (x_i, y_i, r_i) itself, so each sample's own reward appears in its own baseline. This makes the target (r_i - bV(x_i)) a function of r_i, systematically shrinking the target and breaking the interpretation of the 'centered reward' as exogenous. The problem is distinct from the distribution mismatch and would persist even if y_j were drawn from pi_ref.
  3. [III, Eq. (20)] Even if bZ were an unbiased estimate of Z, minimizing the squared error over the empirical data distribution does not yield pi* from Eq. (15) when the data-generating distribution q(·|x) differs from pi_ref(·|x). The minimizer of E_{x,y~q}[...] differs from the closed-form optimal policy, and the paper provides no off-policy correction or formal statement of conditions under which the empirical minimizer approximates pi*. This is an additional gap in the theoretical link.
  4. [IV, Tables II-III and ablations] The empirical results are reported without error bars, number of seeds, or statistical significance tests, and no code is provided. LLM-as-a-judge win rates are known to be noisy, and the reported differences (e.g., 1-2 percentage points in several comparisons) may not be significant. Since the theoretical justification is compromised, these results are the sole support for the paper's claims, and their current presentation is insufficient to carry that load.
minor comments (6)
  1. [IV-C] The text refers to 'Qwen-7B' in the training setup but Table II and elsewhere use 'Qwen2.5-7B'; please make the model names consistent.
  2. [IV-B] Table III's caption says 'pairwise win rates against RPO', but the text says 'Scores above 50 indicate preference for RPO'; clarify the orientation of the win-rate values.
  3. [III, Eq. (20)-(21)] The notation L_RPO is used both for the expectation and for the empirical sum; use distinct symbols (e.g., L and hat L) to avoid confusion.
  4. [IV-E and Limitations] The Limitations section does not mention the key assumption that the empirical partition estimate requires y_j ~ pi_ref, nor its violation in UltraFeedback. This assumption should be stated explicitly and discussed.
  5. [IV-C] The training details are incomplete: the number of epochs, batch size, prompt grouping statistics (average completions per prompt after filtering), and the exact reward standardization procedure are not reported, which hinders reproducibility.
  6. [Figure 1] Figure 1 appears before it is referenced in the text and its caption is not descriptive enough (e.g., units of training time); consider moving it or referring to it explicitly in Section IV-C.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: RPO's derivation is a direct regression to the standard exponential optimal-policy form, not a prediction reduced from its own fitted inputs.

full rationale

RPO's derivation starts from the standard closed-form KL-regularized optimal policy (Eqs. 15-17) and constructs a supervised squared-error loss (Eq. 21) whose target is the centered reward; this is a direct regression objective, not a prediction extracted from a fitted quantity. The empirical partition function bZ(x) in Eq. 19 is an estimator of the analytic Z(x) in Eq. 16, and replacing V* by bV is an approximation step; its bias when completions are not drawn from pi_ref is a correctness/assumption concern, not a circular reduction. The paper contains no self-citations, and the empirical claims are tested against external baselines and held-out/OOD benchmarks, so the central derivation does not reduce to its own inputs.

Assumptions & free parameters 1 free parameters · 3 assumptions · 0 invented entities

The central method introduces one hyperparameter (tau) and relies on three axioms: the standard optimal-policy form, the sampling assumption for the empirical partition estimate, and consistency between offline rewards and the RL objective. No new physical or latent entities are introduced.

free parameters (1)
  • temperature tau = 1.0
    Controls reward scaling and KL-regularization strength in Eqs (20)-(21). It is fixed to 1.0 in all main experiments and ablated at 0.1 and 5.0. It is a hyperparameter, not fitted to validation data, but the method's behavior depends critically on its value.
assumptions (3)
  • standard math The optimal policy takes the exponential form pi*(y|x) = pi_ref(y|x) exp(r(x,y)/tau) / Z(x) (Eq 15).
    Standard closed-form solution to the KL-regularized RL objective, used as the starting point for the RPO loss in Section III.
  • domain assumption Observed responses y_j for a prompt are drawn from pi_ref(center dot|x), or at least provide an unbiased Monte Carlo estimate of Z(x).
    Eq (19) estimates Z(x) by summing pi_ref(y_j|x) exp(r/tau) over dataset responses. This is only unbiased if y_j are samples from pi_ref; in UltraFeedback the responses come from various external generators, so the estimator is biased and the paper does not correct for it.
  • domain assumption The scalar rewards r(x,y) are the same reward that appears in the KL-regularized RL objective.
    RPO treats the offline reward labels as the actual reward in Eq (15); the derivation requires reward consistency between the labels and the objective.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Value-Free Policy Optimization via Reward Partitioning." pith.science (2026). https://pith.science/paper/LUD2X2B6

@misc{pith2026250613702,
  author       = {Pith},
  title        = {Pith review of: Value-Free Policy Optimization via Reward Partitioning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/LUD2X2B6}},
  note         = {Machine review of arXiv:2506.13702}
}
read the original abstract

Single-trajectory preference optimization methods learn from datasets of ((prompt, response, reward)) tuples, offering a practical alternative to pairwise preference learning by directly leveraging scalar feedback. Existing approaches such as Direct Reward Optimization (DRO) have demonstrated promising results but rely on value function estimation, introducing additional variance, optimization complexity, and sensitivity to off-policy data. We introduce Reward Partition Optimization (RPO), a simple and scalable reward-driven objective that eliminates the need for value function learning. RPO normalizes rewards through a partition-based formulation estimated directly from prompt-level reward distributions, yielding a stable supervised optimization objective without auxiliary models or reinforcement learning loops. We evaluate RPO across multiple encoder-decoder and decoder-only language models using automatic metrics, LLM-as-a-judge evaluations, and optimization stability analyses. Experimental results show that RPO consistently outperforms strong baselines, including SFT, KTO, and DRO, while producing more aligned, diverse, and less toxic generations.

Figures

Figures reproduced from arXiv: 2506.13702 by the authors.

Figure 1
Figure 1. Training time (hours) across models using 4×A100 80GB GPUs. Model Method BERTScore ↑ ROUGE-L ↑ Toxicity ↓ Distinct-2 ↑ Avg Len GPT-4o WR ↑ Claude-3.5 WR ↑ FlanT5-Small SFT 0.842 0.228 0.018 0.71 54 81.4 79.2 KTO 0.850 0.237 0.013 0.74 59 83.1 80.8 DRO 0.856 0.252 0.015 0.76 63 84.5 82.1 RPO 0.866 0.289 0.012 0.83 72 FlanT5-Large SFT 0.846 0.241 0.017 0.73 60 85.8 83.2 KTO 0.840 0.249 0.024 0.75 65 87.0 84.6 DRO 0.85… view at source ↗
Figure 2
Figure 2. KL divergence evolution across architectures. Each method exhibits distinct optimization dynamics depending on model type, with RPO showing consistently controlled policy drift, KTO displaying architecture-dependent stability with occasional oscillations, and DRO producing progressively increasing divergence, especially in decoder-only models. Model Baseline GPT-4o Claude-3.5 AlpacaEval MT-Bench IFEval GSM8K AlpacaE… view at source ↗

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Safety Alignment of LMs via Non-cooperative Games

    cs.AI 2025-12 conditional novelty 7.0 of 10

    Jointly training an Attacker and Defender LLM in a non-zero-sum game with pairwise preference judges produces a defender with much lower jailbreak success while preserving general utility.

Reference graph

Works this paper leans on

41 extracted references · 17 canonical work pages · cited by 1 Pith paper

  1. [1]

    Rrhf: Rank responses to align language models with human feedback,

    H. Yuan, Z. Yuan, C. Tan, W. Wang, S. Huang, and F. Huang, “Rrhf: Rank responses to align language models with human feedback,” inAdvances in Neural Information Processing Systems(A. Oh, ModelσBERTScore↑ROUGE-L↑Toxicity↓Distinct-2↑GPT-4o WR↑Claude-3.5 WR↑ FlanT5-Small 0.1 0.862 0.282 0.013 0.81 83.9 81.7 0.5 0.853 0.263 0.016 0.77 80.8 78.6 1.0 0.844 0.24...

    work page 2023

  2. [2]

    Rlhf workflow: From reward modeling to online rlhf,

    H. Dong, W. Xiong, B. Pang, H. Wang, H. Zhao, Y . Zhou, N. Jiang, D. Sahoo, C. Xiong, and T. Zhang, “Rlhf workflow: From reward modeling to online rlhf,”arXiv preprint arXiv:2405.07863, 2024

    arXiv 2024

  3. [3]

    Back to basics: Revisiting reinforce-style optimization for learning from human feedback in llms,

    A. Ahmadian, C. Cremer, M. Gall ´e, M. Fadaee, J. Kreutzer, O. Pietquin, A. ¨Ust¨un, and S. Hooker, “Back to basics: Revisiting reinforce-style optimization for learning from human feedback in llms,” inProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2024, Bangkok, Thailand, August 11-16,...

    work page 2024

  4. [4]

    Direct preference optimization: Your language model is secretly a reward model,

    R. Rafailov, A. Sharma, E. Mitchell, C. D. Manning, S. Ermon, and C. Finn, “Direct preference optimization: Your language model is secretly a reward model,”Advances in Neural Information Processing Systems, vol. 36, pp. 53728–53741, 2023

    2023

  5. [5]

    Generalized preference optimization: A unified approach to offline alignment,

    Y . Tang, Z. D. Guo, Z. Zheng, D. Calandriello, R. Munos, M. Rowland, P. H. Richemond, M. Valko, B. ´A. Pires, and B. Piot, “Generalized preference optimization: A unified approach to offline alignment,” inForty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024, OpenReview.net, 2024

    work page 2024

  6. [6]

    Offline regularised reinforcement learning for large language models alignment,

    P. H. Richemond, Y . Tang, D. Guo, D. Calandriello, M. G. Azar, R. Rafailov, B. A. Pires, E. Tarassov, L. Spangher, W. Ellsworth,et al., “Offline regularised reinforcement learning for large language models alignment,”arXiv preprint arXiv:2405.19107, 2024

    arXiv 2024

  7. [7]

    Model alignment as prospect theoretic optimization,

    K. Ethayarajh, W. Xu, N. Muennighoff, D. Jurafsky, and D. Kiela, “Model alignment as prospect theoretic optimization,” inForty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024, OpenReview.net, 2024

    work page 2024

  8. [8]

    Instruction tuning for large language models: A survey,

    S. Zhang, L. Dong, X. Li, S. Zhang, X. Sun, S. Wang, J. Li, R. Hu, T. Zhang, F. Wu,et al., “Instruction tuning for large language models: A survey,”arXiv preprint arXiv:2308.10792, 2023

    arXiv 2023

Show all 41 references
  1. [9]

    Proximal policy optimization algorithms,

    J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov, “Proximal policy optimization algorithms,”arXiv preprint arXiv:1707.06347, 2017

    2017 arXiv

  2. [10]

    Trust region policy optimization,

    J. Schulman, S. Levine, P. Abbeel, M. Jordan, and P. Moritz, “Trust region policy optimization,” inInternational conference on machine learning, pp. 1889–1897, PMLR, 2015

    2015

  3. [11]

    Training language models to follow instructions with human feedback,

    L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray,et al., “Training language models to follow instructions with human feedback,”Advances in neural information processing systems, vol. 35, pp. 27730–27744, 2022

    2022

  4. [12]

    Gpt-4 technical report,

    J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat,et al., “Gpt-4 technical report,”arXiv preprint arXiv:2303.08774, 2023

    2023 arXiv

  5. [13]

    The claude 3 model family: Opus, sonnet, haiku,

    A. Anthropic, “The claude 3 model family: Opus, sonnet, haiku,” Claude-3 Model Card, vol. 1, p. 1, 2024

    2024

  6. [14]

    Magpie: Alignment data synthesis from scratch by prompting aligned llms with nothing,

    Z. Xu, F. Jiang, L. Niu, Y . Deng, R. Poovendran, Y . Choi, and B. Y . Lin, “Magpie: Alignment data synthesis from scratch by prompting aligned llms with nothing,” inThe Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24-28, 2025, O...

    2025

  7. [15]

    Guiding pretraining in reinforcement learning with large language models,

    Y . Du, O. Watkins, Z. Wang, C. Colas, T. Darrell, P. Abbeel, A. Gupta, and J. Andreas, “Guiding pretraining in reinforcement learning with large language models,” inInternational Conference on Machine Learning, pp. 8657–8677, PMLR, 2023

    2023

  8. [16]

    Rrhf: Rank responses to align language models with human feedback without tears,

    Z. Yuan, H. Yuan, C. Tan, W. Wang, S. Huang, and F. Huang, “Rrhf: Rank responses to align language models with human feedback without tears,”arXiv preprint arXiv:2304.05302, 2023

    2023 arXiv

  9. [17]

    CREAM: consistency regularized self-rewarding language models,

    Z. Wang, W. He, Z. Liang, X. Zhang, C. Bansal, Y . Wei, W. Zhang, and H. Yao, “CREAM: consistency regularized self-rewarding language models,” inThe Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24-28, 2025, OpenReview.net, 2025

    2025

  10. [18]

    Self-rewarding language models,

    W. Yuan, R. Y . Pang, K. Cho, X. Li, S. Sukhbaatar, J. Xu, and J. Weston, “Self-rewarding language models,” inForty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024, OpenReview.net, 2024

    2024

  11. [19]

    Slic-hf: Sequence likelihood calibration with human feedback,

    Y . Zhao, R. Joshi, T. Liu, M. Khalman, M. Saleh, and P. J. Liu, “Slic-hf: Sequence likelihood calibration with human feedback,”arXiv preprint arXiv:2305.10425, 2023

    2023 arXiv

  12. [20]

    The llama 3 herd of models,

    A. Grattafiori, A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, A. Mathur, A. Schelten, A. Vaughan,et al., “The llama 3 herd of models,”arXiv preprint arXiv:2407.21783, 2024

    2024 arXiv

  13. [21]

    Qwen3 technical report,

    A. Yang, A. Li, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Gao, C. Huang, C. Lv,et al., “Qwen3 technical report,”arXiv preprint arXiv:2505.09388, 2025

    2025 arXiv

  14. [22]

    Nemotron-4 340b technical report,

    B. Adler, N. Agarwal, A. Aithal, D. H. Anh, P. Bhattacharya, A. Brundyn, J. Casper, B. Catanzaro, S. Clay, J. Cohen,et al., “Nemotron-4 340b technical report,”arXiv preprint arXiv:2406.11704, 2024

    2024 arXiv

  15. [23]

    Rank analysis of incomplete block designs: I. the method of paired comparisons,

    R. A. Bradley and M. E. Terry, “Rank analysis of incomplete block designs: I. the method of paired comparisons,”Biometrika, vol. 39, no. 3/4, pp. 324–345, 1952

    1952

  16. [24]

    A general theoretical paradigm to understand learning from human preferences,

    M. G. Azar, Z. D. Guo, B. Piot, R. Munos, M. Rowland, M. Valko, and D. Calandriello, “A general theoretical paradigm to understand learning from human preferences,” inInternational Conference on Artificial Intelligence and Statistics, pp. 4447–4455, PMLR, 2024

    2024

  17. [25]

    Advances in prospect theory: Cumulative representation of uncertainty,

    A. Tversky and D. Kahneman, “Advances in prospect theory: Cumulative representation of uncertainty,”Journal of Risk and uncertainty, vol. 5, pp. 297–323, 1992

    1992

  18. [26]

    Ultrafeedback: Boosting language models with high-quality feedback,

    G. Cui, L. Yuan, N. Ding, G. Yao, W. Zhu, Y . Ni, G. Xie, Z. Liu, and M. Sun, “Ultrafeedback: Boosting language models with high-quality feedback,”arXiv preprint 10.48550/ARXIV .2310.01377, 2023

    2023 doi arXiv

  19. [27]

    Alpacaeval: An automatic evaluator of instruction-following models,

    X. Li, T. Zhang, Y . Dubois, R. Taori, I. Gulrajani, C. Guestrin, P. Liang, and T. B. Hashimoto, “Alpacaeval: An automatic evaluator of instruction-following models,” 2023

    2023

  20. [28]

    Judging llm-as-a-judge with mt-bench and chatbot arena,

    L. Zheng, W.-L. Chiang, Y . Sheng, S. Zhuang, Z. Wu, Y . Zhuang, Z. Lin, Z. Li, D. Li, E. Xing,et al., “Judging llm-as-a-judge with mt-bench and chatbot arena,”Advances in neural information processing systems, vol. 36, pp. 46595–46623, 2023

    2023

  21. [29]

    Instruction-following evaluation for large language models,

    J. Zhou, T. Lu, S. Mishra, S. Brahma, S. Basu, Y . Luan, D. Zhou, and L. Hou, “Instruction-following evaluation for large language models,” arXiv preprint arXiv:2311.07911, 2023

    2023 arXiv

  22. [30]

    Training verifiers to solve math word problems,

    K. Cobbe, V . Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert, J. Tworek, J. Hilton, R. Nakano,et al., “Training verifiers to solve math word problems,”arXiv preprint arXiv:2110.14168, 2021

    2021 arXiv

  23. [31]

    Scaling up models and data with t5x and seqio,

    A. Roberts, H. W. Chung, G. Mishra, A. Levskaya, J. Bradbury, D. Andor, S. Narang, B. Lester, C. Gaffney, A. Mohiuddin,et al., “Scaling up models and data with t5x and seqio,”Journal of Machine Learning Research, vol. 24, no. 377, pp. 1–8, 2023

    2023

  24. [32]

    Mistral 7b,

    A. Q. Jiang, A. Sablayrolles, A. Mensch, C. Bamford, D. S. Chaplot, D. de las Casas, F. Bressand, G. Lengyel, G. Lample, L. Saulnier, L. R. Lavaud, M.-A. Lachaux, P. Stock, T. L. Scao, T. Lavril, T. Wang, T. Lacroix, and W. E. Sayed, “Mistral 7b,” 2023

    2023

  25. [33]

    Llama: Open and efficient foundation language models,

    H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozi`ere, N. Goyal, E. Hambro, F. Azhar, A. Rodriguez, A. Joulin, E. Grave, and G. Lample, “Llama: Open and efficient foundation language models,” 2023

    2023

  26. [34]

    Qwen2 technical report,

    A. Yang, B. Yang, B. Hui,et al., “Qwen2 technical report,”arXiv preprint arXiv:2407.10671, 2024

    2024 arXiv

  27. [35]

    Bertscore: Evaluating text generation with bert,

    T. Zhang, V . Kishore, F. Wu, K. Q. Weinberger, and Y . Artzi, “Bertscore: Evaluating text generation with bert,”arXiv preprint arXiv:1904.09675, 2019

    1904 arXiv

  28. [36]

    Rouge: A package for automatic evaluation of summaries,

    C.-Y . Lin, “Rouge: A package for automatic evaluation of summaries,” inText summarization branches out, pp. 74–81, 2004

    2004

  29. [37]

    A diversity-promoting objective function for neural conversation models,

    J. Li, M. Galley, C. Brockett, J. Gao, and W. B. Dolan, “A diversity-promoting objective function for neural conversation models,” inProceedings of the 2016 conference of the North American chapter of the association for computational linguistics: human language technologies, ...

    2016

  30. [38]

    A survey on llm-as-a-judge,

    J. Gu, X. Jiang, Z. Shi, H. Tan, X. Zhai, C. Xu, W. Li, Y . Shen, S. Ma, H. Liu,et al., “A survey on llm-as-a-judge,”arXiv preprint arXiv:2411.15594, 2024

    2024 arXiv

  31. [39]

    Gpt-4o system card,

    A. Hurst, A. Lerer, A. P. Goucher, A. Perelman, A. Ramesh, A. Clark, A. Ostrow, A. Welihinda, A. Hayes, A. Radford,et al., “Gpt-4o system card,”arXiv preprint arXiv:2410.21276, 2024

    2024 arXiv

  32. [40]

    Claude 3.5 sonnet

    Anthropic, “Claude 3.5 sonnet.”https://www.anthropic. com/news/claude-3-5-sonnet, 2024

    2024

  33. [41]

    Decoupled weight decay regularization,

    I. Loshchilov and F. Hutter, “Decoupled weight decay regularization,” International Conference on Learning Representations (ICLR), 2019

    2019

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.