{"id":"42bec4f3-1ae2-41e7-b6b5-bfe8b51ca759","arxiv_id":"2506.08712","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Training a policy to prefer responses by optimizing only its own low-confidence (high-surprisal) tokens improves alignment over uniform token optimization in SimPO and DPO.","lead":"ConfPO picks out the tokens an LLM is least confident about and applies preference optimization only to those tokens, instead of treating every token equally. The approach is claimed to improve instruction-following quality on AlpacaEval 2 and Arena-Hard with zero extra compute, as a drop-in change to existing methods like SimPO and DPO.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"ConfPO's reported gains may be an artifact of its tuned beta/gamma hyperparameters rather than confidence-based token selection; the paper lacks a matched SimPO baseline under identical hyperparameters.","rationale":"The reader's weakest_assumption focuses on the confidence-gradient correlation, but the missing matched hyperparameter control is more directly load-bearing for the empirical central claim. The paper explicitly reports different beta/gamma for ConfPO and admits these were chosen on the evaluation benchmarks, while SimPO used its original hyperparameters. This asymmetry can explain the main-table gains without invoking token selection. My proposed control is concrete and cheap to run because the code is public. I retain the reader's CONDITIONAL verdict, but with the condition made explicit: before the central claim is accepted, the authors must show that ConfPO with its hyperparameters beats SimPO with the same hyperparameters, and ideally that the benefit disappears when the selector is removed. This does not allege dishonesty; it identifies an omitted control that is essential for causal attribution. The reader noted hyperparameter tuning as one concern (point 3), so my agreement is partial rather than full: I am elevating it to the single most load-bearing issue and proposing a decisive test.","tokens_in":20066,"tokens_out":8121,"duration_ms":101454,"concrete_test":"Retrain the SimPO baseline on all four settings (Mistral-Base, Mistral-Instruct, Llama-3-Base, Llama-3-Instruct) using exactly the ConfPO hyperparameters from Table 6 (beta, gamma, learning rate, batch size, epochs, optimizer, LR schedule), with the same data and evaluation protocol, and report AlpacaEval 2 LC/WR and Arena-Hard WR. If tuned SimPO equals or exceeds ConfPO on LC (within 1 point), or if ConfPO with SimPO's original hyperparameters loses its advantage, the claim that low-confidence token selection is the cause of improvement is refuted. Additionally, run a comparable grid search over beta/gamma for SimPO with the same search budget; if the best SimPO matches ConfPO's best, the headline result is hyperparameter-driven.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim (Observation 3 / Table 1) attributes ConfPO's superiority to selecting low-confidence tokens. Section 6.3 states that ConfPO 'generally favors a lower optimal beta and a higher gamma compared to SimPO,' and Appendix D describes searching beta in [1.0, 1.5, 2.0] and gamma in [0.5, 0.8, 1.2, 1.6, 2.0, 2.5] on the evaluation benchmarks. The SimPO baselines were retrained with the original paper's hyperparameters, not given the same per-model tuning budget. A difference in beta/gamma alone changes the effective reward margin and length normalization in Eq. 6; with gamma = 2.0-2.5 (vs. typical SimPO gamma = 0.5-1.0), even the uniform objective becomes a different, arguably more aggressive algorithm. Because the tuned SimPO-with-same-hyperparameters control is absent, the observed LC/WR gains (e.g., +0.3 to +2.1 LC in Table 1) might be entirely due to hyperparameter choice rather than token selection. This is the most load-bearing gap: it directly attacks the causal attribution 'selective token optimization enhances alignment,' which is the paper's stated contribution.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces ConfPO, a token-selection strategy for preference optimization that restricts the SimPO/DPO objective to tokens whose current policy probability is at or below the response-average probability. The method is motivated by three observations: token-level gradient norms are long-tailed; token confidence negatively correlates with gradient norm; and training only on low-confidence tokens improves alignment more than training on all tokens. The authors provide an identity-based rationale (Eq. 11) and a Shapley analysis (Fig. 6), then evaluate ConfPO on AlpacaEval 2 and Arena-Hard across Mistral-7B and Llama-3-8B in Base and Instruct configurations, reporting consistent LC/WR gains over SimPO and DPO at unchanged compute. Additional experiments include a random-selection control, a DPO variant, and overoptimization curves.","tokens_in":20151,"tokens_out":6613,"duration_ms":77716,"significance":"If the central claim holds, the paper offers a practically valuable, zero-overhead token-selection rule: it requires no auxiliary models, no extra forward/backward passes, and is simple to implement on top of standard DAAs. The inclusion of a random-selection control and the reporting of length-controlled (LC) win rates are the right first-order checks, and the public code release is a strength. However, the causal attribution 'confidence-based selection improves alignment' is currently entangled with hyperparameter tuning choices, and the Arena-Hard statistics in Table 7 contain duplicated and impossible confidence intervals. The central claim is defensible, but the experimental support needs to be repaired before the paper can be accepted.","major_comments":[{"comment":"The comparison between ConfPO and SimPO is not hyperparameter-matched. Section 6.3 states that SimPO was retrained with the original paper's hyperparameters, while Appendix D reports that ConfPO searched beta in [1.0, 1.5, 2.0] and gamma in [0.5, 0.8, 1.2, 1.6, 2.0, 2.5] per model, and Section 6.3 further notes that ConfPO 'generally favors a lower optimal beta and a higher gamma compared to SimPO.' Since beta and gamma directly change the effective margin in Eq. (6), the observed LC gains in Table 1 (e.g., +0.3 to +2.1) could in principle be caused by the hyperparameter choice rather than by token selection. Please add a matched SimPO control trained with the same beta/gamma grid, and clarify whether the ConfPO-rand control in Table 2 used ConfPO's tuned hyperparameters; if it did, state this explicitly, because that would provide partial evidence against a pure hyperparameter explanation.","section":"§6.3, Table 6, Appendix D"},{"comment":"The Arena-Hard 95% confidence intervals in Table 7 are duplicated and impossible. All four ConfPO rows report the same 95% CI high/low values (18.0/15.1) and the same length (699) as the SimPO Mistral-Base row, and for Llama-3-Instruct the reported Arena-Hard WR of 32.8 lies far outside the reported 95% CI of 15.1-18.0. These numbers cannot be correct and invalidate the Arena-Hard comparisons as reported. Please regenerate Table 7 from the evaluation logs, archive the outputs, and check the corresponding Arena-Hard values in Table 1 (e.g., Mistral-Instruct ConfPO is 22.4 in Table 1 but 22.7 in Table 7).","section":"Table 7"},{"comment":"The 'theoretical rationale' for Observation 2 is an identity, not a causal or statistical proof. Eq. (11) is a chain-rule identity, and the Shapley analysis in Figure 6 is a descriptive attribution of the ratio r_i = b_i / c_i on the observed data; it cannot by itself establish that policy confidence is a reliable proxy for gradient norm across models, training stages, or DAAs. In addition, the empirical correlation in Figure 4 is computed from only 10 sampled sentences, which is too small to support a general claim of 'strong negative correlation.' Please report the correlation over a substantially larger token sample with confidence intervals, and either weaken the theoretical claim or provide a genuine argument for why the denominator dominates the gradient norm.","section":"§4.2, Eq. (11), Figure 6"},{"comment":"The objective in Eq. (13) does not specify how the selection indicators s(y_i) and the thresholds tau are treated in the backward pass. Since s(y_i) = 1(pi_theta(y_i | x, y_<i) <= tau) and tau is the average token probability, the selected set is a function of the current policy theta; Figure 8 shows that the selected fraction drifts from about 40% to 49% during training. If the implementation stops gradients through the selection (as is standard for such hard masks), the paper should say so explicitly, for example by writing s(y_i) = sg(1(...)). If gradients are not stopped, the loss is discontinuous and the stated gradient analysis in Section 4 no longer applies. This is a load-bearing implementation detail for reproducibility.","section":"§5, Eq. (13)"},{"comment":"The raw win-rate gain for Llama-3-Base is substantially length-driven. Table 7 reports an average generation length of 2885 for ConfPO versus 1901 for SimPO, and the raw WR increases from 25.7 to 32.7, while the length-controlled LC increases only from 27.0 to 28.3. Since AlpacaEval 2's LC is explicitly designed to reduce length bias, the raw WR gap should not be presented as evidence of better alignment. Please either foreground LC in the discussion of this setting or analyze why ConfPO produces much longer outputs and whether that is a desirable property.","section":"Table 7, Llama-3-Base row"}],"minor_comments":[{"comment":"The SFT row in Table 3 (8.4, 6.2, 1.3, 17.1) does not match the corresponding SFT entries in Table 1 for either Mistral-Base or Llama-3-Base; please correct this inconsistency.","section":"Table 3"},{"comment":"The citation 'Anonymous (2025)' in the Related Works section appears to be a double-blind placeholder for a real paper; please provide the actual author list and publication venue.","section":"§2, References"},{"comment":"The text describes r(x,y) as the 'joint log probability of critical tokens,' but the formula divides the summed selected-token log probabilities by |y_s|, making it an average rather than a joint log probability; please align the wording with the equation.","section":"Eq. (7) and Eq. (12)"},{"comment":"The overoptimization plots lack axis labels and a precise definition of the plotted quantity; the text says only that ConfPO 'places higher' than SimPO. Please specify what is plotted on each axis and how the KL divergence is estimated.","section":"Figure 7"},{"comment":"The caption does not state the sample size or the number of training checkpoints used for the Spearman correlations; the main text mentions 10 sampled sentences, which should be repeated in the caption for clarity.","section":"Figure 4 caption"}],"recommendation":"major_revision","confidential_remarks":"The paper has a simple and plausible core idea, and the random-selection control suggests the effect is not purely from reducing token count. However, the duplicated/impossible Arena-Hard CIs in Table 7 and the hyperparameter asymmetry between ConfPO and SimPO are serious enough that I cannot recommend acceptance in the current form. With a matched SimPO-tuned baseline, corrected Table 7, and a larger-scale correlation analysis, the paper could become a solid contribution."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: ConfPO is a clean idea — use the policy's own per-token probability below the response-average as a mask inside SimPO/DPO, and optimize only those tokens. The paper has genuine strengths: the random-selection control, the transfer to DPO, and the fact that length-controlled (LC) gains are positive in all four settings. If the effect is real, this is a useful drop-in improvement to the most-used DAA family.\n\nThe biggest problem is the missing matched baseline. Section 6.3 says SimPO was retrained with the original paper's hyperparameters, while ConfPO searched beta in [1.0, 1.5, 2.0] and gamma in [0.5, 0.8, 1.2, 1.6, 2.0, 2.5] on the eval benchmarks. The paper even admits ConfPO prefers a lower optimal beta and a higher gamma than SimPO. So the comparison conflates token selection with hyperparameter choice. A same-hyperparameter SimPO control is required before crediting the mask. This is load-bearing, not a nit.\n\nSecond, the raw win-rate gains are partly length artifacts on Llama-3-Base: WR rises +7.0 while LC rises only +1.3, and ConfPO outputs 2885 tokens versus SimPO's 1901. The paper does not discuss this discrepancy. Third, Table 7's Arena-Hard confidence intervals are duplicated across models and impossible — the reported intervals do not contain the reported win rates, and identical intervals appear for different models. That makes significance claims uncheckable. Fourth, the \"theoretical rationale\" in Section 4.2 is Eq. 11, an identity, plus a Shapley attribution computed on 10 sampled sentences. It is not a derivation. Also, the selection indicator in Eq. 13 has an unstated stop-gradient treatment, which matters because the selected-token set drifts from about 40% to 49% as training proceeds.\n\nCredit where due: the paper is honest in its limitation statement, ships code, includes a random-selection control that shows selection quality matters, and the central direction — low-confidence token optimization helps — survives in every setting on LC. The problem is not that the empirical claims are fabricated; it is that the causal attribution is under-supported by the current experiments.\n\nWho this is for: anyone working on token-level direct alignment algorithms will want to know about this recipe and the three gradient observations. It deserves a serious referee — the method is simple enough to reproduce, the question is important, and the paper is worth engaging with despite the gaps. I would send it to review with a strong request for a matched SimPO baseline, a corrected Table 7, and length-debiased reporting.","headline":"A simple, zero-overhead token-selection recipe that looks real on length-controlled gains but whose headline attribution is undercut by unmatched hyperparameters and a corrupted appendix table.","tokens_in":20903,"tokens_out":1911,"would_cite":false,"duration_ms":21809,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Low-confidence tokens carry the learning signal in preference optimization.","keywords":["preference optimization","token selection","policy confidence","SimPO","DPO","surprisal","overoptimization","LLM alignment"],"falsifier":"Compute the Spearman correlation between log πθ(yi|x,y<i) and ||∇θ log πθ(yi|x,y<i)|| on the full UltraFeedback training set, not just 10 sentences, at multiple checkpoints; if the correlation is not strongly negative throughout, the selector mis-targets. A second check: train ConfPO with the selection indicator detached via stop-gradient versus backpropagated; if the gains disappear when the mask is frozen, the dynamic masking, not the confidence signal, is what drives the improvement.","tokens_in":19686,"feed_emoji":"🎯","tokens_out":5113,"duration_ms":53198,"temperature":0.7,"pith_summary":"The paper tries to establish that in direct preference optimization, the tokens that actually drive human-alignment gains are the ones the policy model assigns low probability, and that these can be found with zero extra compute by reading the model's own confidence scores. It proposes ConfPO, which keeps only tokens whose probability is at or below the response's average token probability and plugs that selector into the SimPO and DPO objectives. Across two model families in base and instruct variants, this restricted training beats the uniform whole-token baseline on AlpacaEval 2 and Arena-Hard while using fewer than half of the tokens and no additional forward or backward passes. If the claim is right, preference updates should concentrate on surprising, information-bearing tokens rather than on predictable continuations, and the KL budget is spent where it matters.","feed_headline":"Low-confidence tokens alone beat full-token preference training","feed_subtitle":"A confidence-based selector improves LLM alignment on AlpacaEval 2 and Arena-Hard with zero extra compute.","key_machinery":"The load-bearing mechanism is the threshold selector s(yi) = 1(πθ(yi|x,y<i) ≤ τ), where τ is the arithmetic mean token probability of that response. It turns the uniform SimPO or DPO reward into a joint log-probability of only the selected tokens, so gradients flow only through low-confidence, high-surprisal tokens. The identity ||∇θ log πθ(yi|x,y<i)|| = ||∇θ πθ(yi|x,y<i)||/πθ(yi|x,y<i), together with a Shapley attribution showing the denominator dominates, is the theoretical rationale linking confidence to gradient size.","core_discovery":"The central discovery is Observation 3: training only on low-confidence tokens boosts alignment beyond training on all tokens, whereas training only on high-confidence tokens yields negligible or negative gains. The supporting evidence is Observation 1, that token-level gradient norms are long-tailed; Observation 2, that a token's log-probability is strongly negatively correlated with its gradient norm for both chosen and rejected responses; and a Shapley-value decomposition of the identity ||∇θ log πθ(yi|x,y<i)|| = ||∇θ πθ(yi|x,y<i)|| / πθ(yi|x,y<i) showing the denominator (the probability) dominates variation in the ratio. The paper converts this into the selector s(yi) = 1(πθ(yi|x,y<i) ≤ τ) with τ the per-response mean, inside the SimPO and DPO rewards (Eq. 13 and Eq. 15), and reports consistent benchmark gains and a higher alignment-versus-KL curve.","pith_inferences":["A natural extension is to test whether the same confidence proxy identifies critical tokens in other training setups, such as pretraining losses or reward-model training, where per-token importance has also been observed to be concentrated.","The mean-probability threshold is the cheapest choice but is not necessarily the best; quantile-based or entropy-based thresholds could be compared without changing the method's zero-overhead nature.","If the confidence–gradient correlation weakens at larger scales or under different data distributions, the selector could be made adaptive by periodically re-estimating the correlation on a small sample.","The paper's theoretical account is an identity plus an attribution analysis, not a causal proof; the causal direction (low confidence causing high gradient, rather than co-occurrence) remains open to intervention-style tests."],"forward_implications":["ConfPO improves AlpacaEval 2 and Arena-Hard over SimPO on all four tested model configurations, with no change in compute cost.","The selector transfers to DPO, lifting AlpacaEval 2 scores on Mistral-7B and Llama-3-8B base models.","Optimizing fewer than half of the tokens moves the alignment-versus-squared-KL curve upward, indicating reduced overoptimization.","Random token selection at the same ratio underperforms full-token training, so the benefit comes from confidence-based selection, not mere sparsity.","The selected-token ratio grows from about 40% to about 49% during training, implying the method remains selective over the whole run."],"supporting_citations":[{"why":"Supplies the SimPO objective and baseline that ConfPO extends and compares against.","marker":"(Meng et al., 2024)"},{"why":"Supplies DPO, the second objective ConfPO is applied to, and the implicit-reward formulation.","marker":"(Rafailov et al., 2024b)"},{"why":"Provides the prior claim that not all tokens are equally useful, which ConfPO transfers from pretraining to preference learning.","marker":"(Lin et al., 2024)"},{"why":"Establishes that DAAs suffer overoptimization, the curve ConfPO aims to shift.","marker":"(Rafailov et al., 2024a)"},{"why":"Defines reward-model overoptimization scaling laws that motivate the KL-efficiency argument.","marker":"(Gao et al., 2023)"},{"why":"Provides the dependent-feature Shapley estimator used in the theoretical rationale.","marker":"(Aas et al., 2021)"},{"why":"Defines the Shapley value used to attribute the gradient-norm ratio to probability versus numerator.","marker":"(Shapley, 1953)"}],"fun_headline_variants":["Low-confidence tokens alone win over full-token training","ConfPO: target uncertain tokens for better LLM alignment","Focus on policy uncertainty to boost preference learning","Zero-cost token selection beats full-token preference training","Align LLMs by optimizing only low-confidence tokens"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The entire method rests on the premise that a token's predicted probability remains a reliable proxy for its gradient norm throughout training, because the selection mask is a step function built from that probability.","fun_headline_variants_meta":{"raw":{"variants":["Low-confidence tokens alone win over full-token training","ConfPO: target uncertain tokens for better LLM alignment","Focus on policy uncertainty to boost preference learning","Zero-cost token selection beats full-token preference training","Align LLMs by optimizing only low-confidence tokens"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000711,"raw_usage":{"total_tokens":3182,"prompt_tokens":911,"completion_tokens":2271,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":527,"completion_tokens_details":{"reasoning_tokens":2197}},"tokens_in":527,"tokens_out":2271,"duration_ms":21310,"temperature":1.0,"reasoning_tokens":2197,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T05:06:07.098129+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compute the Spearman correlation between log πθ(yi|x,y<i) and ||∇θ log πθ(yi|x,y<i)|| on the full UltraFeedback training set, not just 10 sentences, at multiple checkpoints; if the correlation is not strongly negative throughout, the selector mis-targets. A second check: train ConfPO with the selection indicator detached via stop-gradient versus backpropagated; if the gains disappear when the mask is frozen, the dynamic masking, not the confidence signal, is what drives the improvement.","supporting_citations":[{"cited_title":"Sim PO : Simple preference optimization with a reference-free reward","cited_arxiv_id":null,"evidence_quote":"Supplies the SimPO objective and baseline that ConfPO extends and compares against."},{"cited_title":"Not all tokens are what you need for pretraining","cited_arxiv_id":null,"evidence_quote":"Provides the prior claim that not all tokens are equally useful, which ConfPO transfers from pretraining to preference learning."},{"cited_title":"Explaining individual predictions when features are dependent: More accurate approximations to shapley values","cited_arxiv_id":null,"evidence_quote":"Provides the dependent-feature Shapley estimator used in the theoretical rationale."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the Shapley value used to attribute the gradient-norm ratio to probability versus numerator."}],"review_version":1}