Pith. sign in

REVIEW 4 major objections 6 minor 21 references

Alignment can be recast as maximum-entropy policy optimization under a quality constraint, yielding a closed-form policy that provably beats RLHF on diversity without sacrificing quality.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 00:59 UTC pith:WGWJAW2V

load-bearing objection A competent theory-plus-experiments paper: the QEMPO-KL objective is new, the rest is mostly repackaged entropy-regularized RL, and the 'no quality loss' claim is weakened by missing error bars and an untested uniformity assumption. the 4 major comments →

arxiv 2602.15894 v2 pith:WGWJAW2V submitted 2026-02-11 cs.CL cs.LG

Quality-constrained Entropy Maximization Policy Optimization for LLM Diversity

Haihui Pan , Yuzhong Hong , Kaichen Zhang , Shaoke Lv , Junwei Bao , Hongfei Jiang , Yang Song This is my paper
classification cs.CL cs.LG
keywords LLM alignmentoutput diversityentropy maximizationquality constraintRLHFDPOpolicy optimizationclosed-form policy
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper tries to establish that the apparent trade-off between output quality and output diversity in LLM alignment is not fundamental: one can explicitly maximize diversity (entropy) subject to a reward constraint, and the optimal policy has a closed form. It proves that standard alignment objectives like RLHF only optimize the quality component, then derives two policies — QEMPO and QEMPO-KL — whose entropy is provably at least as high as RLHF's under stated conditions. Empirically, these policies improve diversity while matching or exceeding RLHF quality across four base models on MT-Bench, and in online math evaluations they improve pass@k at larger k. If true, this gives practitioners a simple, reference-free objective for alignment that does not force a quality-diversity sacrifice.

Core claim

On the paper's own terms, the central claim is that maximum-entropy alignment under a quality constraint has a closed-form solution, π_QEMPO(y|x) = exp(λr(x,y))/Z(x), and that including a KL constraint yields π_QEMPO-KL(y|x) = π_ref(y|x)^(λ2/(λ2+1)) exp(λ1 r(x,y)/(λ2+1))/Z(x). The paper proves that when 1/β = λ1/λ2, the entropy of π_QEMPO-KL is at least that of the RLHF policy (Prop 4), and under an additional condition π_QEMPO's entropy is at least π_QEMPO-KL's (Prop 6). These analytical forms are converted into offline DPO-style losses and online variance-weighted policy-gradient losses, and experiments report consistent diversity gains with no quality regression.

What carries the argument

The load-bearing object is the constrained maximum-entropy policy family derived from the alignment Lagrangian. By replacing the KL-to-reference term in RLHF with a direct entropy objective and keeping the reward expectation as a constraint, the paper obtains closed forms π ∝ exp(λr) and π ∝ π_ref^a exp(b r), which are then used to re-express the reward and build training objectives. The entropy-monotonicity lemma (Lemma 6.1) — that H(softmax(s z)) decreases with s — drives the proofs of the entropy ordering between RLHF, QEMPO-KL, and QEMPO.

Load-bearing premise

The proof that maximizing entropy is the right way to preserve diversity assumes the ideal policy spreads probability evenly over all satisfactory answers; if some satisfactory answers are much better than others, the entropy-maximizing policy will over-weight weak answers and may erode quality.

What would settle it

Take a task where one response is clearly correct and all others are flawed (e.g., a single-answer math problem). If QEMPO is trained on such data and its entropy rises while its pass@1 drops relative to RLHF, that would show the uniform-ideal-policy assumption fails. Concretely, run QEMPO's offline loss on UltraFeedback restricted to prompts with a unique best answer and compare reward-model score and diversity against RLHF.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Alignment can be decomposed into separate quality and diversity components, so a policy that optimizes only reward (as Policy Gradient does) leaves diversity on the table.
  • The closed-form QEMPO policy maximizes entropy given a reward threshold, so in principle diversity is guaranteed by construction rather than tuned by sampling temperature.
  • QEMPO-KL provably achieves at least as much entropy as RLHF when the inverse temperature matches the ratio of Lagrange multipliers.
  • The reference-free form of QEMPO removes the need for a reference model during training, cutting memory while increasing entropy.
  • On mathematical reasoning tasks, larger pass@k shows larger gains for QEMPO than RLHF, suggesting diversity helps on hard problems.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The paper's ideal-policy assumption — uniform mass over all satisfactory outputs — means the method is best suited to open-ended generation where many answers are equally good; on tasks with a single dominant correct answer, entropy maximization may dilute quality. This is not tested in the paper.
  • The analytical solution π ∝ exp(λr) is essentially a softmax over rewards; a natural testable extension is whether tuning sampling temperature on the RLHF policy alone reproduces QEMPO's diversity-quality trade-off, which would suggest the gains come from the reward-to-probability map rather than from the training objective.
  • If QEMPO's claims generalize, one would expect diversity gains to be largest when the reward model is noisy or the reference policy is poor; removing the reference term could then be a robustness feature, but the paper's theory only shows it under the uniform-π* assumption.
  • The online mode gates entropy optimization on correctness; a direct extension is to use a learned quality estimate instead of binary correctness so the method can be applied to open-ended tasks where verifiers are unavailable.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes QEMPO and QEMPO-KL, two policy-optimization objectives that maximize output entropy subject to quality constraints (and, for QEMPO-KL, an additional KL constraint to a reference policy). The theoretical sections derive closed-form solutions for these constrained problems, show that RLHF's optimal policy is the solution to a quality-constrained KL minimization, and prove conditional entropy orderings among RLHF, QEMPO-KL, and QEMPO. The paper also gives offline (DPO-style) and online (GRPO-style) training objectives. Experiments on MT-Bench report diversity/quality scores across four base models, and online experiments on math datasets report pass@k curves. The appendix contains proofs and simplified training code.

Significance. If the central claims hold, the paper provides a clean and explicitly formulated way to trade off diversity and quality in LLM alignment, with closed-form policies and simple preference-optimization objectives. The derivations of Propositions 1, 3, 4, and 5 are essentially correct (modulo minor typos), and the paper is unusually explicit about the assumptions behind its entropy-as-diversity framing. The main risk is that the load-bearing assumption of a uniform ideal policy over satisfactory outputs is not validated, and the proof of Proposition 6 is an unquantified approximation. The empirical evaluation is suggestive but lacks statistical grounding. The paper is a worthwhile contribution if these gaps are addressed; with the current state, the central claims are only conditionally established.

major comments (4)
  1. [Section 3.1, Eq. (22)] The proof of Proposition 2 assumes the ideal policy is uniform over satisfactory outputs, π*(y|x) = (1-ε)/|Y+| on Y+ and ε/|Y-| on Y-. This assumption is what equates diversity with entropy and drives the derived objectives. If the true distribution over high-quality outputs is non-uniform—e.g., some satisfactory responses are much stronger than others—then maximizing entropy under the mean-reward constraint will spread probability mass toward weak-but-valid outputs, potentially lowering quality even though the expected-reward constraint is satisfied. The paper only reports mean GPT-4o scores and pass@k, not the quality distribution, and it never tests the uniform-ideal assumption against a non-uniform target. Since the headline claim is 'diversity without sacrificing quality,' this assumption is load-bearing and should be either empirically supported or explicitly treated as a limitatio
  2. [Appendix, proof of Proposition 6 (Eqs. 72-79)] The proof of Proposition 6 is not rigorous. It requires λ2/λ1 λ to be 'sufficiently small' so that π_ref^{λ2/λ1 λ}→1, then applies Lemma 6.1 to the resulting expression. No error bound or quantitative condition is given for this approximation, and the transformation in Eq. (76)-(78) silently replaces π_QEMPO by a different normalized distribution. As stated, the proposition is not proven. Please provide a correct proof with explicit, quantified conditions, or downgrade the statement to an approximate or empirical claim. This matters because the entropy ordering among QEMPO and QEMPO-KL is one of the stated theoretical contributions.
  3. [Section 4.5 and Listings 3-4] The online objectives in Eqs. (8)-(9) are asserted without derivation, and the code in Listings 3-4 diverges from the text. Section 4.5 says that the variance term var(logπ) is computed 'only on data where the outputs are correct' and that when outputs contain errors the variance term is removed 'for such data points.' But the provided code zeroes the variance term for the entire group if any reward in the group is incorrect (sum(reward) == len(reward_list)). This makes the actual online algorithm ambiguous and reproducible results dependent on interpreting this discrepancy. Please align the description with the implementation, or vice versa, and provide a derivation of Eqs. (8)-(9).
  4. [Section 4.2, Table 2 and Section 4.5, Figure 3] The empirical evidence for the headline claim is based on single runs without variance estimates or significance tests. Several reported quality differences are very small (e.g., Qwen2.5-7B: RLHF MT-Bench 7.90 vs. QEMPO 7.92; Llama-3.2-1B: RLHF 5.68 vs. QEMPO 5.82), and diversity is measured without reporting sampling temperatures or other decoding details. Given that the paper claims QEMPO 'consistently' improves diversity without sacrificing quality, at least multiple seeds with standard deviations or a statistical test are needed to support that claim. As written, the empirical support is suggestive but not decisive.
minor comments (6)
  1. [Section 1, Section 2.1] Typo: 'Soling entropy maximization' should be 'Solving entropy maximization.' Also, πref is written as πref(x|y) in Section 2.1, but the correct conditional notation is πref(y|x).
  2. [Propositions 3 and 5] The statements say 'the analytical solution that minimizes this optimization objective' for problems that are explicitly maximization problems. The proofs correctly convert to minimization, but the wording should be 'maximizes.'
  3. [Appendix, proof of Proposition 3, Eq. (33)] The derivative has a typo: the term '-λ2πref(y|x)' should be '-λ2 ln πref(y|x)'. The final closed form is correct, but the intermediate derivative should be fixed.
  4. [Table 2] The SPL row under Qwen2.5-7B-Instruct is mislabeled as 'SPL(Qwen2.5-1.5B-Instruct)'. Also, Section 4.3 and Section 4.4 have duplicated heading numbers; the online experiment section should be renumbered.
  5. [Section 4.2.1, Table 3] The sensitivity analysis is only run on one base model and without repeated runs. This is acceptable for a preliminary ablation, but it would be stronger to report at least one additional model or a variance estimate.
  6. [Section 4.1.1] The diversity evaluation framework is cited, but the temperature and sampling procedure used to generate the 16 responses per query are not specified. Since the paper's main goal is diversity, this is an important experimental detail to report.

Circularity Check

2 steps flagged

Core QEMPO derivations are non-circular, but Proposition 2 and Corollary 3.1 are definitional tautologies; the central experimental claims remain independently evaluated.

specific steps
  1. self definitional [Section 3.1 / Appendix proof of Proposition 2, Eq. (22)]
    "According to the two attributes of π∗(y|x)—quality and diversity—we can infer that π∗(y|x) = (1−ϵ)/|Y+| for y∈Y+ and ϵ/|Y−| for y∈Y−. ... For diversity, we can use Hπ(Y|X) to represent it. This is because when the output distribution of π(y|x) is more uniform, Hπ(Y|X) becomes larger."

    The proposition's conclusion—that alignment decomposes into maximizing quality and diversity—is built into the stipulation that the ideal policy π* is uniform over satisfactory outputs. With that definition, the cross-entropy term Eπ[ln π*] is linear in P(y∈Y+) and the entropy term is literally Hπ, so the 'decomposition' follows by construction rather than being a derived property of alignment. The uniformity assumption is never empirically tested.

  2. self definitional [Section 3.1 / Appendix proof of Corollary 3.1]
    "If the reward function is defined as r(x, y) = 1_{y∈Y+}, then maxπ Σ_y r(x,y)π(y|x) = maxπ Σ_{y∈Y+} π(y|x). Based on the above analysis, we can conclude that Policy Gradient only optimizes the quality component in alignment tasks."

    Quality is defined as Σ_{y∈Y+} π(y|x), and the reward is set to the indicator of Y+. Policy Gradient's objective then equals this quality measure by definition. Claiming that Policy Gradient 'only optimizes quality' is therefore a restatement of the chosen reward function, not an independent finding about Policy Gradient methods.

full rationale

The central optimization results—Propositions 1, 3, 4, 5, and 6—are self-contained convex-analysis derivations. They solve constrained entropy/KL problems via Lagrange multipliers (standard exponential-family forms) and compare softmax temperatures; they do not assume their conclusions. Proposition 4's entropy ordering is conditional on 1/β = λ1/λ2, and the experiments set hyperparameters to that operating point, but the claim is a mathematical implication, not a fit to the diversity benchmark. The empirical diversity/quality results on MT-Bench and pass@k are measured with external metrics and do not reduce to the paper's inputs. The main weaknesses are (a) the untested uniform-ideal-policy assumption (Eq. 22), which is a validity risk for the 'entropy as diversity' framing, and (b) two definitional tautologies—the Proposition 2 decomposition and Corollary 3.1—where the conclusions are inscribed in the definitions of π* and the reward. The online objectives borrow from the authors' own GVPO paper, but that citation is an implementation detail and is not load-bearing for QEMPO's correctness. Overall, the empirical contributions stand on their own, so the circularity is limited to the motivational theory.

Axiom & Free-Parameter Ledger

4 free parameters · 5 axioms · 0 invented entities

No new physical or conceptual entities are introduced. The paper's contribution is a new optimization objective, not a new entity. The main free parameters are the Lagrange multiplier temperatures chosen by validation; the key domain assumption is the uniformity of the ideal policy over satisfactory outputs.

free parameters (4)
  • lambda (QEMPO) = 1/lambda = 4e-3 (ablation: 1e-3..1e-2)
    Trade-off between entropy and quality in the max-entropy policy; chosen by validation on UltraFeedback test loss.
  • lambda1 (QEMPO-KL) = 1/lambda1 = 4e-3 (ablation: 2e-3..6e-3)
    Inverse temperature for reward in the quality constraint; chosen by validation.
  • lambda2/lambda1 (QEMPO-KL) = 1e-2 (ablation: 6e-3..1.2e-2)
    Weight on KL constraint; chosen by validation, and set so lambda1/lambda2 = 1/beta (100) to satisfy Prop 4's condition.
  • beta (RLHF baseline) = 1e-2
    RLHF KL penalty from Zephyr setup; needed for the comparison condition in Prop 4.
axioms (5)
  • domain assumption The ideal policy pi* is uniform over the set of satisfactory outputs Y+ (pi*(y+|x) = (1-eps)/|Y+|)
    Used in Prop 2 (Eq. 22) to justify entropy maximization as the diversity objective; if the true distribution over good outputs is not uniform, entropy may over-weight mediocre outputs.
  • domain assumption The reward model r(x,y) accurately reflects output quality/preference
    The quality constraint E_pi[r] >= R and the offline/online objectives all use r as ground truth; the method inherits reward-model bias.
  • standard math Convexity/KKT conditions apply to the constrained entropy problems
    Props 1/3/5 use Lagrange multipliers and first-order optimality; entropy is concave and the feasible sets are convex, so the solutions are globally optimal.
  • standard math Bradley-Terry model of preferences for offline DPO-style objectives
    Offline objectives (Eqs. 6,7) substitute the derived reward expressions into the BT model, as in DPO.
  • domain assumption GVPO gradient-weighting method is valid for deriving online objectives
    Eqs. 8-9 are taken from Zhang et al. 2025 (same research group) without derivation in this paper; the online objective's correctness depends on GVPO's validity.

pith-pipeline@v1.3.0-alltime-deepseek · 16347 in / 15538 out tokens · 129087 ms · 2026-08-03T00:59:19.837624+00:00 · methodology

0 comments
read the original abstract

In many large language model (LLM) alignment applications, users expect not only high-quality outputs but also substantial diversity. However, existing methods often face a fundamental trade-off between these objectives: approaches that improve output quality tend to reduce diversity, while methods that increase diversity often do so at the expense of quality. In this work, we propose Quality-constrained Entropy Maximization Policy Optimization (QEMPO), a novel framework that enhances the diversity of LLM outputs while explicitly preserving output quality. QEMPO is grounded in a strong theoretical foundation: we derive a closed-form analytical solution that provably maximizes entropy-a principled measure of diversity-subject to a quality constraint, with guarantees on optimality under the defined objective. Leveraging this solution, QEMPO naturally supports both online and offline training settings. Empirical results demonstrate that QEMPO consistently improves output diversity without sacrificing quality, and in many cases yields gains in both dimensions compared to existing baselines, aligning with our theoretical guarantees.

Figures

Figures reproduced from arXiv: 2602.15894 by Haihui Pan, Hongfei Jiang, Junwei Bao, Kaichen Zhang, Shaoke Lv, Yang Song, Yuzhong Hong.

Figure 1
Figure 1. Figure 1: A schematic diagram of the output space distributions of three policies with varying quality [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Diversity and quality across different models and methods. Distinct colors represent [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Pass@k results of RLHF, QEMPO, and QEMPO-KL on different datasets. [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

21 extracted references · 17 linked inside Pith

  1. [1]

    The hyperfitting phenomenon: Sharpening and stabilizing llms for open-ended text generation.arXiv preprint arXiv:2412.04318,

    Fredrik Carlsson, Fangyu Liu, Daniel Ward, Murathan Kurfali, and Joakim Nivre. The hyperfitting phenomenon: Sharpening and stabilizing llms for open-ended text generation.arXiv preprint arXiv:2412.04318,

    Pith/arXiv arXiv

  2. [4]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948,

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948,

    Pith/arXiv arXiv

  3. [5]

    Benchmarking linguistic diversity of large language models.arXiv preprint arXiv:2412.10271,

    Yanzhu Guo, Guokan Shang, and Chlo´e Clavel. Benchmarking linguistic diversity of large language models.arXiv preprint arXiv:2412.10271,

    Pith/arXiv arXiv

  4. [6]

    Adam: A method for stochastic optimization.arXiv preprint arXiv:1412.6980,

    Diederik P Kingma. Adam: A method for stochastic optimization.arXiv preprint arXiv:1412.6980,

    Pith/arXiv arXiv

  5. [9]

    Llama 3.2: Revolutionizing edge ai and vision with open, customizable models.Meta AI Blog

    AI Meta. Llama 3.2: Revolutionizing edge ai and vision with open, customizable models.Meta AI Blog. Retrieved December, 20:2024,

    2024

  6. [10]

    One fish, two fish, but not the whole sea: Alignment reduces language models’ conceptual diversity.arXiv preprint arXiv:2411.04427,

    Sonia K Murthy, Tomer Ullman, and Jennifer Hu. One fish, two fish, but not the whole sea: Alignment reduces language models’ conceptual diversity.arXiv preprint arXiv:2411.04427,

    Pith/arXiv arXiv

  7. [12]

    Entrpo: trust region policy optimization method with entropy regularization.arXiv preprint arXiv:2110.13373,

    Sahar Roostaie and Mohammad Mehdi Ebadzadeh. Entrpo: trust region policy optimization method with entropy regularization.arXiv preprint arXiv:2110.13373,

    Pith/arXiv arXiv

  8. [13]

    Curiosity-driven reinforcement learning from human feedback.arXiv preprint arXiv:2501.11463,

    Haoran Sun, Yekun Chai, Shuohuan Wang, Yu Sun, Hua Wu, and Haifeng Wang. Curiosity-driven reinforcement learning from human feedback.arXiv preprint arXiv:2501.11463,

    Pith/arXiv arXiv

  9. [15]

    Zephyr: Direct distillation of lm alignment.arXiv preprint arXiv:2310.16944,

    Lewis Tunstall, Edward Beeching, Nathan Lambert, Nazneen Rajani, Kashif Rasul, Younes Belkada, Shengyi Huang, Leandro V on Werra, Cl´ementine Fourrier, Nathan Habib, et al. Zephyr: Direct distillation of lm alignment.arXiv preprint arXiv:2310.16944,

    Pith/arXiv arXiv

  10. [16]

    Improving diversity in language models: When temperature fails, change the loss.arXiv preprint arXiv:2508.09654,

    Alexandre Verine, Florian Le Bronnec, Kunhao Zheng, Alexandre Allauzen, Yann Chevaleyre, and Benjamin Negrevergne. Improving diversity in language models: When temperature fails, change the loss.arXiv preprint arXiv:2508.09654,

    Pith/arXiv arXiv

  11. [17]

    Diversity-oriented data augmentation with large language models.arXiv preprint arXiv:2502.11671,

    Zaitian Wang, Jinghan Zhang, Xinhao Zhang, Kunpeng Liu, Pengfei Wang, and Yuanchun Zhou. Diversity-oriented data augmentation with large language models.arXiv preprint arXiv:2502.11671,

    arXiv

  12. [18]

    Understanding the effects of rlhf on the quality and detectability of llm-generated texts.arXiv preprint arXiv:2503.17965,

    Beining Xu and Arkaitz Zubiaga. Understanding the effects of rlhf on the quality and detectability of llm-generated texts.arXiv preprint arXiv:2503.17965,

    Pith/arXiv arXiv

  13. [19]

    Qwen3 technical report.arXiv preprint arXiv:2505.09388,

    An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report.arXiv preprint arXiv:2505.09388,

    Pith/arXiv arXiv

  14. [20]

    Gvpo: Group variance policy optimization for large language model post-training.arXiv preprint arXiv:2504.19599,

    Kaichen Zhang, Yuzhong Hong, Junwei Bao, Hongfei Jiang, Yang Song, Dingqian Hong, and Hui Xiong. Gvpo: Group variance policy optimization for large language model post-training.arXiv preprint arXiv:2504.19599,

    arXiv

  15. [21]

    Balancing diversity and risk in llm sampling: How to select your method and parameter for open-ended text generation.arXiv preprint arXiv:2408.13586,

    Yuxuan Zhou, Margret Keuper, and Mario Fritz. Balancing diversity and risk in llm sampling: How to select your method and parameter for open-ended text generation.arXiv preprint arXiv:2408.13586,

    Pith/arXiv arXiv

  16. [1999]

    Qwen2.5 technical report.arXiv preprint arXiv:2412.15115,

    Qwen Team. Qwen2.5 technical report.arXiv preprint arXiv:2412.15115,

    Pith/arXiv arXiv

  17. [2014]

    Understanding the effects of rlhf on llm generalisation and diversity.arXiv preprint arXiv:2310.06452,

    Robert Kirk, Ishita Mediratta, Christoforos Nalmpantis, Jelena Luketina, Eric Hambro, Edward Grefenstette, and Roberta Raileanu. Understanding the effects of rlhf on llm generalisation and diversity.arXiv preprint arXiv:2310.06452,

    Pith/arXiv arXiv

  18. [2022]

    Does writing with language models reduce content diversity? arXiv preprint arXiv:2309.05196,

    Vishakh Padmakumar and He He. Does writing with language models reduce content diversity? arXiv preprint arXiv:2309.05196,

    Pith/arXiv arXiv

  19. [2023]

    Diverse preference optimization.arXiv preprint arXiv:2501.18101,

    Jack Lanchantin, Angelica Chen, Shehzaad Dhuliawala, Ping Yu, Jason Weston, Sainbayar Sukhbaatar, and Ilia Kulikov. Diverse preference optimization.arXiv preprint arXiv:2501.18101,

    Pith/arXiv arXiv

  20. [2024]

    xverify: Efficient answer verifier for reasoning model evaluations

    Ding Chen, Qingchen Yu, Pengyuan Wang, Wentao Zhang, Bo Tang, Feiyu Xiong, Xinchi Li, Minchuan Yang, and Zhiyu Li. xverify: Efficient answer verifier for reasoning model evaluations. arXiv preprint arXiv:2504.10481,

    arXiv

  21. [2025]

    Training verifiers to solve math word problems.arXiv preprint arXiv:2110.14168,

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems.arXiv preprint arXiv:2110.14168,

    Pith/arXiv arXiv