Pith. sign in

REVIEW 5 major objections 5 minor 1 cited by

ConfPO: Exploiting Policy Model Confidence for Critical Token Selection in Preference Optimization

T0 review · 5 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Low-confidence tokens carry the learning signal in preference optimization.

desk verdict A simple, zero-overhead token-selection recipe that looks real on length-controlled gains but whose headline attribution is undercut by unmatched hyperparameters and a corrupted appendix table. read the letter →

arxiv 2506.08712 v2 pith:YTYKBXQY submitted 2025-06-10 cs.CL cs.AIcs.LG

Hee Suk Yoon , Eunseop Yoon , Mark Hasegawa-Johnson , Sungwoong Kim , Chang D. Yoo This is my paper
classification cs.CLcs.AIcs.LG
keywords preferenceoptimizationtokenselectionpolicyconfidenceSimPODPOsurprisaloveroptimizationLLMalignment
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that in direct preference optimization, the tokens that actually drive human-alignment gains are the ones the policy model assigns low probability, and that these can be found with zero extra compute by reading the model's own confidence scores. It proposes ConfPO, which keeps only tokens whose probability is at or below the response's average token probability and plugs that selector into the SimPO and DPO objectives. Across two model families in base and instruct variants, this restricted training beats the uniform whole-token baseline on AlpacaEval 2 and Arena-Hard while using fewer than half of the tokens and no additional forward or backward passes. If the claim is right, preference updates should concentrate on surprising, information-bearing tokens rather than on predictable continuations, and the KL budget is spent where it matters.

What carries the argument

The load-bearing mechanism is the threshold selector s(yi) = 1(πθ(yi|x,y<i) ≤ τ), where τ is the arithmetic mean token probability of that response. It turns the uniform SimPO or DPO reward into a joint log-probability of only the selected tokens, so gradients flow only through low-confidence, high-surprisal tokens. The identity ||∇θ log πθ(yi|x,y<i)|| = ||∇θ πθ(yi|x,y<i)||/πθ(yi|x,y<i), together with a Shapley attribution showing the denominator dominates, is the theoretical rationale linking confidence to gradient size.

What would settle it

Compute the Spearman correlation between log πθ(yi|x,y<i) and ||∇θ log πθ(yi|x,y<i)|| on the full UltraFeedback training set, not just 10 sentences, at multiple checkpoints; if the correlation is not strongly negative throughout, the selector mis-targets. A second check: train ConfPO with the selection indicator detached via stop-gradient versus backpropagated; if the gains disappear when the mask is frozen, the dynamic masking, not the confidence signal, is what drives the improvement.

Watch

Extended reading notes

Core claim

The central discovery is Observation 3: training only on low-confidence tokens boosts alignment beyond training on all tokens, whereas training only on high-confidence tokens yields negligible or negative gains. The supporting evidence is Observation 1, that token-level gradient norms are long-tailed; Observation 2, that a token's log-probability is strongly negatively correlated with its gradient norm for both chosen and rejected responses; and a Shapley-value decomposition of the identity ||∇θ log πθ(yi|x,y<i)|| = ||∇θ πθ(yi|x,y<i)|| / πθ(yi|x,y<i) showing the denominator (the probability) dominates variation in the ratio. The paper converts this into the selector s(yi) = 1(πθ(yi|x,y<i) ≤ τ) with τ the per-response mean, inside the SimPO and DPO rewards (Eq. 13 and Eq. 15), and reports consistent benchmark gains and a higher alignment-versus-KL curve.

Load-bearing premise

The entire method rests on the premise that a token's predicted probability remains a reliable proxy for its gradient norm throughout training, because the selection mask is a step function built from that probability.

Editorial extensions

If this is right

  • ConfPO improves AlpacaEval 2 and Arena-Hard over SimPO on all four tested model configurations, with no change in compute cost.
  • The selector transfers to DPO, lifting AlpacaEval 2 scores on Mistral-7B and Llama-3-8B base models.
  • Optimizing fewer than half of the tokens moves the alignment-versus-squared-KL curve upward, indicating reduced overoptimization.
  • Random token selection at the same ratio underperforms full-token training, so the benefit comes from confidence-based selection, not mere sparsity.
  • The selected-token ratio grows from about 40% to about 49% during training, implying the method remains selective over the whole run.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural extension is to test whether the same confidence proxy identifies critical tokens in other training setups, such as pretraining losses or reward-model training, where per-token importance has also been observed to be concentrated.
  • The mean-probability threshold is the cheapest choice but is not necessarily the best; quantile-based or entropy-based thresholds could be compared without changing the method's zero-overhead nature.
  • If the confidence–gradient correlation weakens at larger scales or under different data distributions, the selector could be made adaptive by periodically re-estimating the correlation on a small sample.
  • The paper's theoretical account is an identity plus an attribution analysis, not a causal proof; the causal direction (low confidence causing high gradient, rather than co-occurrence) remains open to intervention-style tests.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper introduces ConfPO, a token-selection strategy for preference optimization that restricts the SimPO/DPO objective to tokens whose current policy probability is at or below the response-average probability. The method is motivated by three observations: token-level gradient norms are long-tailed; token confidence negatively correlates with gradient norm; and training only on low-confidence tokens improves alignment more than training on all tokens. The authors provide an identity-based rationale (Eq. 11) and a Shapley analysis (Fig. 6), then evaluate ConfPO on AlpacaEval 2 and Arena-Hard across Mistral-7B and Llama-3-8B in Base and Instruct configurations, reporting consistent LC/WR gains over SimPO and DPO at unchanged compute. Additional experiments include a random-selection control, a DPO variant, and overoptimization curves.

Significance. If the central claim holds, the paper offers a practically valuable, zero-overhead token-selection rule: it requires no auxiliary models, no extra forward/backward passes, and is simple to implement on top of standard DAAs. The inclusion of a random-selection control and the reporting of length-controlled (LC) win rates are the right first-order checks, and the public code release is a strength. However, the causal attribution 'confidence-based selection improves alignment' is currently entangled with hyperparameter tuning choices, and the Arena-Hard statistics in Table 7 contain duplicated and impossible confidence intervals. The central claim is defensible, but the experimental support needs to be repaired before the paper can be accepted.

major comments (5)
  1. [§6.3, Table 6, Appendix D] The comparison between ConfPO and SimPO is not hyperparameter-matched. Section 6.3 states that SimPO was retrained with the original paper's hyperparameters, while Appendix D reports that ConfPO searched beta in [1.0, 1.5, 2.0] and gamma in [0.5, 0.8, 1.2, 1.6, 2.0, 2.5] per model, and Section 6.3 further notes that ConfPO 'generally favors a lower optimal beta and a higher gamma compared to SimPO.' Since beta and gamma directly change the effective margin in Eq. (6), the observed LC gains in Table 1 (e.g., +0.3 to +2.1) could in principle be caused by the hyperparameter choice rather than by token selection. Please add a matched SimPO control trained with the same beta/gamma grid, and clarify whether the ConfPO-rand control in Table 2 used ConfPO's tuned hyperparameters; if it did, state this explicitly, because that would provide partial evidence against a pure hyperparameter explanation.
  2. [Table 7] The Arena-Hard 95% confidence intervals in Table 7 are duplicated and impossible. All four ConfPO rows report the same 95% CI high/low values (18.0/15.1) and the same length (699) as the SimPO Mistral-Base row, and for Llama-3-Instruct the reported Arena-Hard WR of 32.8 lies far outside the reported 95% CI of 15.1-18.0. These numbers cannot be correct and invalidate the Arena-Hard comparisons as reported. Please regenerate Table 7 from the evaluation logs, archive the outputs, and check the corresponding Arena-Hard values in Table 1 (e.g., Mistral-Instruct ConfPO is 22.4 in Table 1 but 22.7 in Table 7).
  3. [§4.2, Eq. (11), Figure 6] The 'theoretical rationale' for Observation 2 is an identity, not a causal or statistical proof. Eq. (11) is a chain-rule identity, and the Shapley analysis in Figure 6 is a descriptive attribution of the ratio r_i = b_i / c_i on the observed data; it cannot by itself establish that policy confidence is a reliable proxy for gradient norm across models, training stages, or DAAs. In addition, the empirical correlation in Figure 4 is computed from only 10 sampled sentences, which is too small to support a general claim of 'strong negative correlation.' Please report the correlation over a substantially larger token sample with confidence intervals, and either weaken the theoretical claim or provide a genuine argument for why the denominator dominates the gradient norm.
  4. [§5, Eq. (13)] The objective in Eq. (13) does not specify how the selection indicators s(y_i) and the thresholds tau are treated in the backward pass. Since s(y_i) = 1(pi_theta(y_i | x, y_<i) <= tau) and tau is the average token probability, the selected set is a function of the current policy theta; Figure 8 shows that the selected fraction drifts from about 40% to 49% during training. If the implementation stops gradients through the selection (as is standard for such hard masks), the paper should say so explicitly, for example by writing s(y_i) = sg(1(...)). If gradients are not stopped, the loss is discontinuous and the stated gradient analysis in Section 4 no longer applies. This is a load-bearing implementation detail for reproducibility.
  5. [Table 7, Llama-3-Base row] The raw win-rate gain for Llama-3-Base is substantially length-driven. Table 7 reports an average generation length of 2885 for ConfPO versus 1901 for SimPO, and the raw WR increases from 25.7 to 32.7, while the length-controlled LC increases only from 27.0 to 28.3. Since AlpacaEval 2's LC is explicitly designed to reduce length bias, the raw WR gap should not be presented as evidence of better alignment. Please either foreground LC in the discussion of this setting or analyze why ConfPO produces much longer outputs and whether that is a desirable property.
minor comments (5)
  1. [Table 3] The SFT row in Table 3 (8.4, 6.2, 1.3, 17.1) does not match the corresponding SFT entries in Table 1 for either Mistral-Base or Llama-3-Base; please correct this inconsistency.
  2. [§2, References] The citation 'Anonymous (2025)' in the Related Works section appears to be a double-blind placeholder for a real paper; please provide the actual author list and publication venue.
  3. [Eq. (7) and Eq. (12)] The text describes r(x,y) as the 'joint log probability of critical tokens,' but the formula divides the summed selected-token log probabilities by |y_s|, making it an average rather than a joint log probability; please align the wording with the equation.
  4. [Figure 7] The overoptimization plots lack axis labels and a precise definition of the plotted quantity; the text says only that ConfPO 'places higher' than SimPO. Please specify what is plotted on each axis and how the KL divergence is estimated.
  5. [Figure 4 caption] The caption does not state the sample size or the number of training checkpoints used for the Spearman correlations; the main text mentions 10 sampled sentences, which should be repeated in the caption for clarity.

Circularity Check

2 steps flagged · score 3.0 of 10

Core method is not construction-level circular: the selector uses the policy's own forward-pass probabilities and evaluation is on external benchmarks.

  1. self definitional [Section 4.2, Eq. 11 and Figure 6]
    "∥∇θ logπθ(yi|x, y<i)∥= ∥∇θπθ(yi|x, y<i)∥ / πθ(yi|x, y<i). (11) Thus, if token confidence πθ(yi|x, y<i) dominates this ratio, the log-probability gradient norm ∥∇θ logπθ(yi|x, y<i)∥ decreases as confidence increases, establishing the observed negative correlation."

    Eq. 11 is the chain-rule identity that defines the log-probability gradient; it holds for any differentiable policy, so it cannot by itself prove the negative correlation claimed in Observation 2. The subsequent Shapley analysis (Figure 6) attributes variation of r_i = b_i/c_i to the denominator c_i = π_θ(y_i|x,y_<i); since r_i equals b_i/c_i by that very definition, the conclusion that the denominator dominates is a restatement of the inverse relationship between probability and the ratio, not an independent derivation. The word 'establishing' thus elevates a tautology to a theoretical account; the proxy claim (low confidence approximately high gradient) actually rests on the measured Spearman correlation (Figure 4) and external validation (Tables 1-3).

  2. fitted input called prediction [Section 6.3 (Baselines), Appendix D (Implementation Details), Table 6]
    "For the SimPO baseline, we retrained the model using the hyperparameters specified in their original paper (Meng et al., 2024). ... Notably, we observe that ConfPO generally favors a lower optimal β and a higher γ compared to SimPO. ... We search the β in the range of [1.0, 1.5, 2.0] and γ in the range of [0.5, 0.8, 1.2, 1.6, 2.0, 2.5]."

    In the paper's own equations, the only structural difference between SimPO (Eq. 6) and ConfPO (Eq. 13) is the selection mask; Eq. 13 reduces to Eq. 6 when s(y_i) = 1 for all i. Yet the Table 1 gains are reported at ConfPO operating points whose β and γ were selected by searching the evaluation benchmarks (Appendix D), while the SimPO baseline was retrained at its original paper's hyperparameters without the same search budget. Because γ is a margin term inside the sigmoid (Eq. 6), and the paper states ConfPO 'favors a lower optimal β and a higher γ,' the +0.3 to +2.1 LC deltas conflate fitted hyperparameters with token selection; the attribution 'selective token optimization enhances alignment' is not isolated at the compared operating points.

full rationale

The core derivation is not construction-level circular. ConfPO's selector uses the current policy's own per-token probabilities (already computed in the forward pass), the threshold is the per-response average (Eq. 14), and the central claims are evaluated on external benchmarks (AlpacaEval 2, Arena-Hard) outside the method's fitted values; the random-selection control (Table 2) and the DPO extension (Table 3) give independent evidence that the selection rule, not merely token-count reduction, carries the effect. Self-citations (e.g., Yoon et al. 2024a in Section 3.3) support the 'not all tokens are equal' premise but are not load-bearing because the same sentence cites external work (Lin et al. 2024; Chen et al. 2024; Lai et al. 2024). Two soft spots keep the score above 2. First, Section 4.2's theoretical account of Observation 2 is the chain-rule identity (Eq. 11) plus a Shapley attribution of variation in b_i/c_i to the denominator; since Eq. 11 is the definition of the log-probability gradient, the theory restates the inverse relationship between gradient magnitude and probability rather than independently deriving it, and the empirical correlation (Figure 4) plus external results carry the proxy claim. Second, the headline Table 1 comparisons are made at hyperparameters selected for ConfPO by searching β and γ on the evaluation benchmarks (Appendix D: β in [1.0, 1.5, 2.0], γ in [0.5, 0.8, 1.2, 1.6, 2.0, 2.5]), while the SimPO baseline was retrained at its original paper's hyperparameters and not granted the same search; because γ directly changes the margin in Eq. 6, the per-model gains cannot be isolated as an effect of confidence-based token selection. These are rigor and attribution risks rather than a by-construction reduction of the result to its inputs, so the score is 3 rather than 6 or higher.

Assumptions & free parameters 4 free parameters · 6 assumptions · 0 invented entities

The only genuinely new moving part is the confidence-threshold selector, and its justification imports three empirical premises: gradients are long-tailed, confidence proxies gradient norm, and low-confidence tokens drive alignment. The chain-rule identity in Section 4.2 is exact, but the Shapley-based dominance claim is an empirical attribution, not a theorem, and the stop-gradient treatment of the selection function is an unstated assumption.

free parameters (4)
  • beta (ConfPO/SimPO temperature) = 1.0 (Llama-3), 1.5 (Mistral-Base), 2.0 (Mistral-Instruct)
    Searched over {1.0, 1.5, 2.0} on AlpacaEval 2 and Arena-Hard; tuned per model on the evaluation benchmarks used in Table 1.
  • gamma (target margin) = 0.5 (Mistral-Instruct), 1.6 (Mistral-Base), 2.0 (Llama-3-Base), 2.5 (Llama-3-Instruct)
    Searched over {0.5, 0.8, 1.2, 1.6, 2.0, 2.5} on the evaluation benchmarks.
  • learning rate = 3e-7, 5e-7, 6e-7, 2.51e-6 per setting
    Chosen per model on the evaluation benchmarks; standard practice but fitted nonetheless.
  • threshold rule (arithmetic mean) = arithmetic average; fixed 0.6 and geometric average rejected
    Selected by comparing variants on AlpacaEval 2 (Appendix B, Table 4); this is a design parameter fitted to the benchmark.
assumptions (6)
  • standard math Bradley-Terry pairwise preference model (Eq. 1) underlies the DPO/SimPO objective.
    Inherited from DPO and SimPO; standard modeling assumption in preference learning that pairwise preference probability is logistic in reward differences.
  • standard math Chain-rule identity ||grad log p|| = ||grad p|| / p (Eq. 11) is the basis of the gradient-confidence link.
    Algebraically exact; the paper uses it to argue that smaller token probability implies larger log-probability gradient norm, holding the numerator fixed.
  • domain assumption Token-level gradient norm proxies preference-relevant learning signal (Observation 1).
    The long-tailed gradient distribution is shown on a 10-sentence sample (Figure 3); the step from gradient magnitude to preference informativeness is assumed, not proven.
  • domain assumption Policy confidence (token probability) reliably proxies gradient norm throughout training (Observation 2).
    Supported by a correlation on 10 sampled sentences and a Shapley attribution (Figures 4 and 6); no proof of stability across scales or DAAs.
  • domain assumption Selection indicator s(yi) and threshold tau are constants under differentiation in Eq. 13.
    The stop-gradient treatment is unstated; the selected token set changes as the policy updates (Figure 8), so the objective is effectively non-stationary.
  • domain assumption PairRM-generated preference labels (Instruct settings) are valid proxies for human preference.
    The Instruct experiments use preference pairs scored by the PairRM reward model following Meng et al. (2024); human validity of these labels is assumed.

how reviews work

0 comments
Cite this review

Pith. "Pith review of ConfPO: Exploiting Policy Model Confidence for Critical Token Selection in Preference Optimization." pith.science (2026). https://pith.science/paper/YTYKBXQY

@misc{pith2026250608712,
  author       = {Pith},
  title        = {Pith review of: ConfPO: Exploiting Policy Model Confidence for Critical Token Selection in Preference Optimization},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/YTYKBXQY}},
  note         = {Machine review of arXiv:2506.08712}
}
read the original abstract

We introduce ConfPO, a method for preference learning in Large Language Models (LLMs) that identifies and optimizes preference-critical tokens based solely on the training policy's confidence, without requiring any auxiliary models or compute. Unlike prior Direct Alignment Algorithms (DAAs) such as Direct Preference Optimization (DPO), which uniformly adjust all token probabilities regardless of their relevance to preference, ConfPO focuses optimization on the most impactful tokens. This targeted approach improves alignment quality while mitigating overoptimization (i.e., reward hacking) by using the KL divergence budget more efficiently. In contrast to recent token-level methods that rely on credit-assignment models or AI annotators, raising concerns about scalability and reliability, ConfPO is simple, lightweight, and model-free. Experimental results on challenging alignment benchmarks, including AlpacaEval 2 and Arena-Hard, demonstrate that ConfPO consistently outperforms uniform DAAs across various LLMs, delivering better alignment with zero additional computational overhead.

Figures

Figures reproduced from arXiv: 2506.08712 by the authors.

Figure 1
Figure 1. Highlighted tokens—green for the chosen, red for the rejected—are those regarded low-confidence by the policy model, meaning their predicted probability falls below the sequence av￾erage. This strategy surfaces tokens that carry high informational value or mark key decision points, while skipping predictable con￾tinuations. For example, in “22” (tokenized as ‘2’, ‘2’), the first digit may be highlighted as uncertain… view at source ↗
Figure 2
Figure 2. Overview of the Confidence-guided Preference Optimization (ConfPO). Existing DAA algorithm (SimPO (Meng et al., 2024)) uses all the tokens uniformly for preference alignment. ConfPO selectively optimizes on few tokens based on the policy model’s confidence with no additional compute overhead. surprisal and the greater the information conveyed. Mod￾ern LLMs embody this principle by explicitly estimating P(wi |context… view at source ↗
Figure 3
Figure 3. Observation 1. Token-level gradient norm values (i.e., ∥∇θ log πθ(yi|x, y<i)∥) follow a long-tailed distribution for both the preferred response (left) and the unpreferred response (right). Each distribution is computed from 10 sampled sentences, and Llama-3-Base (8B) was used. This observation shows that not all tokens contribute equally to preference alignment. In this gradient equation, the sequence-level gradien… view at source ↗
Figures from the paper (8 more)
Figure 4
Figure 4. Figure 4: Observation 2. (a) Spearman correlation plot between the policy model’s log probability (confidence) for each token, log πθ(yi|x, y<i), and its gradient norm, ∥∇θ log πθ(yi|x, y<i)∥, over the course of training. A strong negative correlation emerges for tokens in both …
Figure 6
Figure 6. Figure 6: Absolute Shapley value decomposition of the gradient￾norm ratio ri = ∥∇θπθ(yi | x, y<i)∥/πθ(yi | x, y<i), illustrat￾ing the relative contributions of its numerator (bi) and denominator (ci). Results are shown for both Mistral-Base (7B) and Llama￾Base (8B), where ci con…
Figure 7
Figure 7. Figure 7: Results on Overoptimization. Each point in the plots represents a model trained with a different hyperparame￾ter. For both models—(a) Mistral-Base (7B) and (b) Llama-3-Base (8B)—the curve shows that ConfPO places higher than the SimPO, suggesting that it mitigates over…
Figure 8
Figure 8. Figure 8: Number of Tokens Selected During Training. For both models—(a) Mistral-Base (7B) and (b) Llama-3-Base (8B)—the number of tokens selected by our confidence-guided criteria (Eq. 13) gradually increases over the course of training. Notably, ConfPO optimizes fewer than hal…
Figure 9
Figure 9. Figure 9: Observation 1 for DPO [PITH_FULL_IMAGE:figures/full_fig_p013_9.png]
Figure 10
Figure 10. Figure 10: Observation 2 for DPO [PITH_FULL_IMAGE:figures/full_fig_p013_10.png]
Figure 11
Figure 11. Figure 11: Observation 3 for DPO. D. Implementation Details Following (Meng et al., 2024), we use a batch size of 128 and train for a single epoch for all preference optimization training. The maximum sequence length is set to 2048, with a maximum prompt length of 1800. We emplo…
Figure 12
Figure 12. Figure 12: Histogram showing the distribution of selected tokens with respect to the position in the sentence. To check for potential positional biases, we analyzed where low-confidence tokens are selected within a sequence. As shown in [PITH_FULL_IMAGE:figures/full_fig_p014_12.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Normalized Rewards for Preference Optimization

    cs.LG 2026-06 conditional novelty 5.0 of 10

    A regularization term that conserves the combined length-normalized probability of chosen and rejected responses reduces likelihood displacement in DPO/SimPO, improves AlpacaEval and benchmark outcomes, and acts prima...

Reference graph

Works this paper leans on

49 extracted references · 26 canonical work pages · cited by 1 Pith paper

  1. [1]

    Explaining individual predictions when features are dependent: More accurate approximations to shapley values

    Aas, K., Jullum, M., and L land, A. Explaining individual predictions when features are dependent: More accurate approximations to shapley values. Artificial Intelligence, 298: 0 103502, 2021

    work page 2021

  2. [2]

    Llama 3 model card

    AI@Meta. Llama 3 model card. 2024. URL https://github.com/meta-llama/llama3/blob/main/MODEL_CARD.md

    2024

  3. [3]

    Mitigating reward over-optimization in direct alignment algorithms with adaptive importance sampling, 2025

    Anonymous. Mitigating reward over-optimization in direct alignment algorithms with adaptive importance sampling, 2025. URL https://openreview.net/forum?id=H9dNX6TaRE

    work page 2025

  4. [4]

    G., Guo, Z

    Azar, M. G., Guo, Z. D., Piot, B., Munos, R., Rowland, M., Valko, M., and Calandriello, D. A general theoretical paradigm to understand learning from human preferences. In International Conference on Artificial Intelligence and Statistics, pp.\ 4447--4455. PMLR, 2024

    work page 2024

  5. [5]

    Bradley, R. A. and Terry, M. E. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39 0 (3/4): 0 324--345, 1952. ISSN 00063444, 14643510. URL http://www.jstor.org/stable/2334029

    arXiv 1952

  6. [6]

    Step-level value preference optimization for mathematical reasoning

    Chen, G., Liao, M., Li, C., and Fan, K. Step-level value preference optimization for mathematical reasoning. In Al-Onaizan, Y., Bansal, M., and Chen, Y.-N. (eds.), Findings of the Association for Computational Linguistics: EMNLP 2024, pp.\ 7889--7903, Miami, Florida, USA, November 2024. Association for Computational Linguistics. doi:10.18653/v1/2024.findi...

    doi:10.18653/v1/2024.findings-emnlp.463 2024

  7. [7]

    F., Leike, J., Brown, T., Martic, M., Legg, S., and Amodei, D

    Christiano, P. F., Leike, J., Brown, T., Martic, M., Legg, S., and Amodei, D. Deep reinforcement learning from human preferences. In Guyon, I., Luxburg, U. V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., and Garnett, R. (eds.), Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017

    work page 2017

  8. [8]

    Ultrafeedback: Boosting language models with high-quality feedback, 2023

    Cui, G., Yuan, L., Ding, N., Yao, G., Zhu, W., Ni, Y., Xie, G., Liu, Z., and Sun, M. Ultrafeedback: Boosting language models with high-quality feedback, 2023

    work page 2023

Show all 49 references
  1. [9]

    Enhancing chat language models by scaling high-quality instructional conversations

    Ding, N., Chen, Y., Xu, B., Qin, Y., Hu, S., Liu, Z., Sun, M., and Zhou, B. Enhancing chat language models by scaling high-quality instructional conversations. In Bouamor, H., Pino, J., and Bali, K. (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Lan...

    2023 doi

  2. [10]

    Dubois, Y., Galambosi, B., Liang, P., and Hashimoto, T. B. Length-controlled alpacaeval: A simple way to debias automatic evaluators. arXiv preprint arXiv:2404.04475, 2024

    2024 arXiv

  3. [11]

    Kto: Model alignment as prospect theoretic optimization

    Ethayarajh, K., Xu, W., Muennighoff, N., Jurafsky, D., and Kiela, D. Kto: Model alignment as prospect theoretic optimization. arXiv preprint arXiv:2402.01306, 2024

    2024 arXiv

  4. [12]

    Scaling laws for reward model overoptimization

    Gao, L., Schulman, J., and Hilton, J. Scaling laws for reward model overoptimization. In International Conference on Machine Learning, pp.\ 10835--10866. PMLR, 2023

    2023

  5. [13]

    Beyond imitation: Leveraging fine-grained quality signals for alignment

    Guo, G., Zhao, R., Tang, T., Zhao, X., and Wen, J.-R. Beyond imitation: Leveraging fine-grained quality signals for alignment. In The Twelfth International Conference on Learning Representations

  6. [14]

    A probabilistic earley parser as a psycholinguistic model

    Hale, J. A probabilistic earley parser as a psycholinguistic model. In Second meeting of the north american chapter of the association for computational linguistics, 2001

    2001

  7. [15]

    Reference-free monolithic preference optimization with odds ratio

    Hong, J., Lee, N., and Thorne, J. Reference-free monolithic preference optimization with odds ratio. arXiv e-prints, pp.\ arXiv--2403, 2024

    2024

  8. [16]

    and Levy, R

    Jaeger, T. and Levy, R. Speakers optimize information density through syntactic reduction. In Sch\" o lkopf, B., Platt, J., and Hoffman, T. (eds.), Advances in Neural Information Processing Systems, volume 19. MIT Press, 2006. URL https://proceedings.neurips.cc/paper_files/pap...

    2006

  9. [17]

    Q., Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D

    Jiang, A. Q., Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D. S., Casas, D. d. l., Bressand, F., Lengyel, G., Lample, G., Saulnier, L., et al. Mistral 7b. arXiv preprint arXiv:2310.06825, 2023

    2023 arXiv

  10. [18]

    Step-dpo: Step-wise preference optimization for long-chain reasoning of llms

    Lai, X., Tian, Z., Chen, Y., Yang, S., Peng, X., and Jia, J. Step-dpo: Step-wise preference optimization for long-chain reasoning of llms. arXiv preprint arXiv:2406.18629, 2024

    2024 arXiv

  11. [19]

    E., and Stoica, I

    Li*, T., Chiang*, W.-L., Frick, E., Dunlap, L., Zhu, B., Gonzalez, J. E., and Stoica, I. From live data to high-quality benchmarks: The arena-hard pipeline, April 2024. URL https://lmsys.org/blog/2024-04-19-arena-hard/

    2024

  12. [20]

    Li, X., Zhang, T., Dubois, Y., Taori, R., Gulrajani, I., Guestrin, C., Liang, P., and Hashimoto, T. B. Alpacaeval: An automatic evaluator of instruction-following models. https://github.com/tatsu-lab/alpaca_eval, 5 2023

    2023

  13. [21]

    Not all tokens are what you need for pretraining

    Lin, Z., Gou, Z., Gong, Y., Liu, X., yelong shen, Xu, R., Lin, C., Yang, Y., Jiao, J., Duan, N., and Chen, W. Not all tokens are what you need for pretraining. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URL https://openreview.net/for...

    2024

  14. [22]

    Provably mitigating overoptimization in RLHF : Your SFT loss is implicitly an adversarial regularizer

    Liu, Z., Lu, M., Zhang, S., Liu, B., Guo, H., Yang, Y., Blanchet, J., and Wang, Z. Provably mitigating overoptimization in RLHF : Your SFT loss is implicitly an adversarial regularizer. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URL ...

    2024

  15. [23]

    and Hutter, F

    Loshchilov, I. and Hutter, F. Decoupled weight decay regularization. In International Conference on Learning Representations, 2019. URL https://openreview.net/forum?id=Bkg6RiCqY7

    2019

  16. [24]

    Lundberg, S. M. and Lee, S.-I. A unified approach to interpreting model predictions. In Guyon, I., Luxburg, U. V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., and Garnett, R. (eds.), Advances in Neural Information Processing Systems, volume 30. Curran Associates, In...

    2017

  17. [25]

    Sim PO : Simple preference optimization with a reference-free reward

    Meng, Y., Xia, M., and Chen, D. Sim PO : Simple preference optimization with a reference-free reward. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URL https://openreview.net/forum?id=3Tzcot1LKb

    2024

  18. [26]

    and Frank, S

    Merkx, D. and Frank, S. L. Human sentence processing: Recurrence or attention? In Chersoni, E., Hollenstein, N., Jacobs, C., Oseki, Y., Pr \'e vot, L., and Santus, E. (eds.), Proceedings of the Workshop on Cognitive Modeling and Computational Linguistics, pp.\ 12--22, Online, ...

    2021 doi

  19. [27]

    F., Leike, J., and Lowe, R

    Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., Schulman, J., Hilton, J., Kelton, F., Miller, L., Simens, M., Askell, A., Welinder, P., Christiano, P. F., Leike, J., and Lowe, R. Training language models to f...

    2022

  20. [28]

    Owen, A. B. and Prieur, C. On shapley value for measuring importance of dependent inputs. SIAM/ASA Journal on Uncertainty Quantification, 5 0 (1): 0 986--1002, 2017

    2017

  21. [29]

    Disentangling length from quality in direct preference optimization

    Park, R., Rafailov, R., Ermon, S., and Finn, C. Disentangling length from quality in direct preference optimization. arXiv preprint arXiv:2403.19159, 2024

    2024 arXiv

  22. [30]

    B., Finn, C., and Niekum, S

    Rafailov, R., Chittepu, Y., Park, R., Sikchi, H., Hejna, J., Knox, W. B., Finn, C., and Niekum, S. Scaling laws for reward model overoptimization in direct alignment algorithms. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024 a . URL https...

    2024

  23. [31]

    D., Ermon, S., and Finn, C

    Rafailov, R., Sharma, A., Mitchell, E., Manning, C. D., Ermon, S., and Finn, C. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36, 2024 b

    2024

  24. [32]

    Shapley, L. S. A value for n-person games. In Kuhn, H. W. and Tucker, A. W. (eds.), Contributions to the Theory of Games II, pp.\ 307--317. Princeton University Press, Princeton, 1953

    1953

  25. [33]

    S., and Martin, A

    Slaats, S., Meyer, A. S., and Martin, A. E. Lexical surprisal shapes the time course of syntactic structure building. Neurobiology of Language, 5 0 (4): 0 942--980, 10 2024. ISSN 2641-4368. doi:10.1162/nol_a_00155. URL https://doi.org/10.1162/nol\_a\_00155

    2024 doi

  26. [34]

    Smith, N. J. and Levy, R. The effect of word predictability on reading time is logarithmic. Cognition, 128 0 (3): 0 302--319, 2013. ISSN 0010-0277. doi:https://doi.org/10.1016/j.cognition.2013.02.013. URL https://www.sciencedirect.com/science/article/pii/S0010027713000413

    2013 doi

  27. [35]

    Trl: Transformer reinforcement learning

    von Werra, L., Belkada, Y., Tunstall, L., Beeching, E., Thrush, T., Lambert, N., Huang, S., Rasul, K., and Gallouédec, Q. Trl: Transformer reinforcement learning. https://github.com/huggingface/trl, 2020

    2020

  28. [36]

    V., Murray, K., and Kim, Y

    Xu, H., Sharaf, A., Chen, Y., Tan, W., Shen, L., Durme, B. V., Murray, K., and Kim, Y. J. Contrastive preference optimization: Pushing the boundaries of LLM performance in machine translation. In Forty-first International Conference on Machine Learning, 2024. URL https://openr...

    2024

  29. [37]

    Selective preference optimization via token-level reward function estimation

    Yang, K., Liu, Z., Xie, Q., Huang, J., Min, E., and Ananiadou, S. Selective preference optimization via token-level reward function estimation. arXiv preprint arXiv:2408.13518, 2024

    2024

  30. [38]

    S., Hasegawa-Johnson, M

    Yoon, E., Yoon, H. S., Hasegawa-Johnson, M. A., and Yoo, C. D. Can video llms refuse to answer? alignment for answerability in video large language models. In The Thirteenth International Conference on Learning Representations

  31. [39]

    S., Eom, S., Han, G., Nam, D., Jo, D., On, K.-W., Hasegawa-Johnson, M., Kim, S., and Yoo, C

    Yoon, E., Yoon, H. S., Eom, S., Han, G., Nam, D., Jo, D., On, K.-W., Hasegawa-Johnson, M., Kim, S., and Yoo, C. TLCR : Token-level continuous reward for fine-grained reinforcement learning from human feedback. In Ku, L.-W., Martins, A., and Srikumar, V. (eds.), Findings of the...

    2024 doi

  32. [40]

    S., Tee, J

    Yoon, H. S., Tee, J. T. J., Yoon, E., Yoon, S., Kim, G., Li, Y., and Yoo, C. D. Esd: Expected squared difference as a tuning-free trainable calibration measure. arXiv preprint arXiv:2303.02472, 2023

    2023 arXiv

  33. [41]

    S., Yoon, E., Tee, J

    Yoon, H. S., Yoon, E., Tee, J. T. J., Hasegawa-Johnson, M., Li, Y., and Yoo, C. D. C-tpt: Calibrated test-time prompt tuning for vision-language models via text feature dispersion. arXiv preprint arXiv:2403.14119, 2024 b

    2024 arXiv

  34. [42]

    S., Kim, J., and Yoo, C

    Yoon, S., Yoon, E., Yoon, H. S., Kim, J., and Yoo, C. Information-theoretic text hallucination reduction for video-grounded dialogue. In Goldberg, Y., Kozareva, Z., and Zhang, Y. (eds.), Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp...

    2022 doi

  35. [43]

    Tpc: Test-time procrustes calibration for diffusion-based human image animation

    Yoon, S., Koo, G., Lee, Y., and Yoo, C. Tpc: Test-time procrustes calibration for diffusion-based human image animation. Advances in Neural Information Processing Systems, 37: 0 118654--118677, 2024 c

    2024

  36. [44]

    RRHF : Rank responses to align language models with human feedback

    Yuan, H., Yuan, Z., Tan, C., Wang, W., Huang, S., and Huang, F. RRHF : Rank responses to align language models with human feedback. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL https://openreview.net/forum?id=EdIGMCHk4l

    2023

  37. [45]

    Token-level direct preference optimization

    Zeng, Y., Liu, G., Ma, W., Yang, N., Zhang, H., and Wang, J. Token-level direct preference optimization. In Forty-first International Conference on Machine Learning

  38. [46]

    Zhao, Y., Joshi, R., Liu, T., Khalman, M., Saleh, M., and Liu, P. J. Slic-hf: Sequence likelihood calibration with human feedback. arXiv preprint arXiv:2305.10425, 2023

    2023 arXiv

  39. [47]

    P., Zhang, H., Gonzalez, J

    Zheng, L., Chiang, W.-L., Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., Lin, Z., Li, Z., Li, D., Xing, E. P., Zhang, H., Gonzalez, J. E., and Stoica, I. Judging llm-as-a-judge with mt-bench and chatbot arena, 2023

    2023

  40. [48]

    T-reg: Preference optimization with token-level reward regularization

    Zhou, W., Zhang, S., Zhao, L., and Meng, T. T-reg: Preference optimization with token-level reward regularization. arXiv preprint arXiv:2412.02685, 2024

    2024 arXiv

  41. [49]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.