Not All Preferences are What You Need for Post-Training: Selective Alignment Strategy for Preference Optimization

Zhijin Dong

REVIEW 4 major objections 5 minor 13 references

Not All Preferences are What You Need for Post-Training: Selective Alignment Strategy for Preference Optimization

T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Training only the top 40% of tokens by alignment score beats standard DPO on preference benchmarks.

desk verdict A plausible simplification of SePO, but the core scoring signal is never validated against random token selection, so the empirical claim is under-supported. read the letter →

arxiv 2507.07725 v1 pith:XLRPO2HJ submitted 2025-07-10 cs.CL cs.AI

Zhijin Dong This is my paper

classification cs.CLcs.AI

keywords Selective-DPOpreferenceoptimizationtoken-levelselectiondirectalignmentreferencemodelknowledgedistillation

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that post-training alignment of language models should not treat every token in a preference pair equally. It proposes Selective-DPO, which scores each token by the gap between the current policy's log-probability and a reference model's log-probability, keeps the top 40% of tokens by that score, and applies the DPO loss only on those tokens. On Arena-Hard and MT-Bench, the resulting model outperforms standard DPO and distillation-based baselines for 0.5B and 3B models, with larger reference models giving larger gains. The paper also shows that the choice of reference model matters: a stronger reference model improves token selection accuracy and the final alignment quality.

What carries the argument

The load-bearing object is the alignment score in Eq. (6), the token-level log-probability difference between the reference model and the current policy, signed according to whether the token belongs to the winning or losing response. It selects the top $k\%$ of tokens, and the Selective-DPO loss in Eqs. (7)-(8) applies the standard DPO Bradley-Terry objective only to the log-ratio of chosen and rejected sequences computed over those selected tokens. The same score therefore does double duty: it identifies which tokens to train and which tokens to exclude when computing the preference reward.

What would settle it

A direct test would replace the alignment score with random token selection at the same 40% ratio; if random selection matches Selective-DPO's win rates, then the score itself is not doing the work. A second test would run Selective-DPO on an instruction-following benchmark such as IFEval and check whether the reported decline on objective metrics is consistent across seeds and tasks.

Watch

Extended reading notes

Core claim

The central claim is that a large part of preference data is uninformative for alignment, and that focusing optimization on a selected subset of tokens yields better alignment than optimizing the whole sequence. Selective-DPO defines a token-level alignment score $s(y_i) = (-1)^{I(y_i \in y_l)} [\log \pi_{\text{ref}}(y_i | x, y_{<i}) - \log \pi_\theta(y_i | x, y_{<i})]$, which is positive for tokens that the reference assigns higher probability than the current policy in the winning response, and for tokens the current policy over-weights in the losing response. Only the top $k\%$ of tokens by this score contribute to the reward $R(y)$ in the DPO-style loss. With the top 40% of tokens and reference models that are larger and instruct-tuned, the method raises the Arena-Hard win rate from 13.4% to 22.5% on the 0.5B model and from 16.6% to 26.9% on the 3B model relative to vanilla DPO.

Load-bearing premise

The method assumes that the token-level log-probability difference between the reference model and the current policy reliably identifies which tokens actually matter for preference alignment.

Editorial extensions

If this is right

If Selective-DPO generalizes, practitioners can reduce the effective training signal to 40% of tokens and improve alignment quality at lower compute.
Reference model quality becomes a first-class design choice: using a larger instruct-tuned reference is a cheap way to gain points on alignment benchmarks without enlarging the policy.
The token-selection ratio and the KL regularization coefficient become new hyperparameters with an observed optimum at 40% and $\beta = 0.01$ in this paper's ablations.
Other DPO variants such as SimPO and Step-DPO could be combined with the same selection mask and potentially inherit the reported gains.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The alignment score is effectively a per-token version of the DPO reward difference, so the selection mask may be interpretable as filtering the preference signal to the positions where the two models disagree.
The gain from larger reference models could reflect a better teacher signal rather than a property of selection itself, and a testable extension would be to hold the policy fixed, vary only the reference, and measure the stability of the token rankings.
The paper reports a decline on the instruction-following metric IFEval, which suggests the method optimizes subjective style at the expense of objective task adherence; an extension would be to add a small objective-metric regularization term to the selected tokens.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Take this paper for what it is: a modest, clearly-written variant of DPO that selects a top-k% subset of tokens to optimize, using the log-probability gap between the policy and a reference model as the importance score. The simplification over SePO—no oracle reward model, just a log-ratio—is legitimate and the derivation is clean. The paper also honestly acknowledges its debt to Rho-1 and SePO, and the related work is fine.

The core idea is plausible, and the empirical tables show consistent gains over DPO and Distill-DPO, with sensible ablations for k and beta. Using a stronger reference model does seem to help. That part is fine.

The problem is that the central assumption—that Eq. (6) identifies preference-critical tokens—is never directly tested. There is no control where a random 40% of tokens is masked, or where the worst-scoring tokens are used, under the same reference model and beta. Without those controls, the headline gain could come from the stronger reference model's regularizing effect or from any token subset, not from the scoring signal. The stress-test note is right about this, and it is a load-bearing unvalidated assumption.

The other issues are more mundane but add up: hyperparameters k and beta are tuned on Arena-Hard and MT-Bench, the same benchmarks used for the final results; no error bars or significance tests are reported; the admitted IFEval decline is mentioned in the limitations but never quantified; and model names are anonymized, making independent verification harder. These are fixable, but they are real.

I would not call this a fatally flawed paper. The idea is a reasonable variant, and the empirical evidence, though weak, is at least consistent. But as it stands, the paper claims more than it supports. The right referee would ask for the random/inverse mask controls, an evaluation protocol that separates tuning from testing, and error bars. If those come back clean, this could be a useful incremental contribution to the post-training toolbox.

For a reading group: worth a maybe, mainly to discuss the experimental design lesson. I would not cite it in my own work in the next year. That said, a serious editor should send it out: the idea is not obviously wrong, the writing is clear, and the evaluation is addressable. Give it to a referee who will ask the right questions.

Referee Report

4 major / 5 minor

Summary. This paper proposes Selective-DPO, a post-training alignment method that computes a token-level alignment score s(y_i) as the signed log-probability difference between a reference model and the current policy (Eq. 6), selects the top k% of tokens by this score, and optimizes a DPO-style loss (Eqs. 7-8) on only those tokens. The authors argue that not all tokens matter for preference optimization, that stronger reference models improve token selection, and that the method outperforms DPO and a distillation-based baseline. Experiments use the Skywork-Reward-Preference-80K-v0.2 dataset and evaluate on Arena-Hard and MT-Bench for 0.5B and 3B SFT models. The paper also reports ablations over k and beta and acknowledges an IFEval decline.

Significance. The proposal is simple and inexpensive, and the idea of token-level selection for preference optimization is timely; the code release is a plus. If the empirical claims held with proper controls, the method would be a useful contribution. However, the current evidence is not sufficient: the two hyperparameters are tuned on the same benchmarks used for the final comparison, no control experiment isolates the token-scoring mechanism from the effect of the stronger reference model, and the admitted degradation on IFEval limits the scope of the headline claim. The central mechanism of Eq. (6) is plausible but unvalidated.

major comments (4)

[Sec. 5.1, Tables 2 and 3] The final configuration (k=40%, beta=0.01) is selected by evaluating k and beta on Arena-Hard and MT-Bench, which are exactly the benchmarks reported in Table 1. This is selection on the test set, so the reported improvements in Table 1 may largely reflect tuning rather than a general advantage of Selective-DPO. Please report a held-out validation split or, at minimum, multiple seeds with mean and standard deviation and significance tests for the Table 1 comparisons.
[Sec. 4.1, Eq. (6), and Sec. 5.2, Table 1] The paper never validates that the score in Eq. (6) identifies preference-critical tokens. A necessary control is to compare top-40% selection against a random 40% token mask and a bottom-40% (inverse) mask under the same reference model and the same beta. Without these controls, the gains in Table 1 could be caused by the stronger reference model or by any token-subset regularization, not by the specific ranking produced by Eq. (6). Please add this experiment, and also report what the selected tokens look like (e.g., part-of-speech or position analysis) to support the claim that they are 'high-impact'.
[Sec. 6.1, Limitations] The paper admits that 'the IFEval metric showed a decline' but does not quantify the decline or report it in the main results. Since the abstract and introduction claim enhanced 'alignment fidelity' and 'overall effectiveness', the objective-metric decline must be quantified and discussed in the main results; otherwise the headline claim is overstated. Please report the IFEval numbers and discuss the trade-off between subjective preference benchmarks and objective instruction-following.
[Sec. 4.2 and Table 1] The claim that 'stronger reference models significantly improve token selection accuracy' is confounded: a larger instruct reference is also a stronger distillation teacher, and Selective-DPO with a 10B/33B reference may simply benefit from a distillation-like signal rather than from better token scoring. The comparison with Distill-DPO helps, but the differences are small (22.5 vs 20.3 for 0.5B; 26.9 vs 24.7 for 3B) and no uncertainty is reported. To support the 'token selection accuracy' interpretation, compare Selective-DPO with a strong reference against DPO regularized by the same reference and against a random-token-mask version using the same reference; otherwise the reference-quality claim is not separable from the choice of training objective.

minor comments (5)

[Title and running header] The running header 'Not All Prefrences are What You Need' contains a typo; the correct spelling is 'Preferences'.
[References] The MT-Bench evaluation is described correctly but cited as MT-Bench-101 [1]; please cite the original MT-Bench paper (Zheng et al., 2023). Similarly, Arena-Hard should cite its own paper rather than the Chatbot Arena paper [3].
[Sec. 5.1] The models are anonymized as '0.5B-SFT' and '3B-SFT', which prevents reproducibility; please specify at least the architecture, tokenizer, and pretraining data, even if the exact model names are withheld.
[Abstract and Sec. 5.2] The abstract claims the method 'reduces computational overhead', but no training time, FLOPs, or memory measurements are reported; the reduction is only in the number of optimized tokens, not in total training cost or inference cost.
[Tables 2 and 3] Tables 2 and 3 report single runs without variance; please include standard deviations across seeds or state the number of runs averaged.

Circularity Check

1 steps flagged · score 6.0 of 10

Reported best result is the grid-search maximum of Tables 2 and 3, making the headline 0.5B/10B row a fitted value rather than an independent prediction.

fitted input called prediction [Section 5.1 (Training Details), Table 2, Table 3]
"For Selective-DPO, we set the KL regularization coefficient β = 0.01, and retain the top 40% of tokens based on alignment scores computed from token-level log-likelihood differences between the policy and reference model. These hyperparameters are selected based on ablation studies and preliminary tuning."

Table 2 varies the selection ratio k and Table 3 varies β, both for the 0.5B-SFT model with the 10B-instruct reference, reporting Arena-Hard and MT-Bench scores. The main-result row for that same configuration in Table 1 is 22.5% Arena-Hard and 7.34 MT-Bench, which are exactly the maxima of Tables 2 and 3. Because the text states these hyperparameters were 'selected based on ablation studies and preliminary tuning,' the headline 0.5B/10B result is the argmax of the tuning grid on the same benchmarks that are later presented as validation. That number is therefore a fitted value, not an independent prediction, and the claimed improvement over DPO is partly forced by selection-on-test-set rather than by the token-selection mechanism itself.

full rationale

The core derivation is not definitionally circular: Eq. (6) defines a token-importance heuristic from the log-probability gap between π_ref and π_θ, and Selective-DPO is a masked version of the DPO reward. No load-bearing self-citation chain exists, and the method is not a renaming of a known result. The circularity is confined to the empirical evaluation: the specific hyperparameters (k=40%, β=0.01) that produce the best Table 1 row were selected by maximizing the same Arena-Hard and MT-Bench metrics on the same model/reference configuration, so that row's reported superiority reduces to a grid-search optimum. The 3B-SFT row is less directly fitted because no 3B ablation is shown, but it inherits the same test-selected hyperparameters. The paper also concedes an IFEval decline, further narrowing the claim to subjective benchmarks. I do not count the untested nature of Eq. (6) as circularity, since a missing control (random or inverse masks) is an experimental validity concern rather than a derivation that reduces to its inputs.

Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

The central claim rests on two fitted hyperparameters (top-k ratio and beta) and three axioms. The main domain assumption is that log-probability difference is a good importance signal; the main ad hoc assumption is that larger reference models are always better. No new physical or conceptual entities are introduced.

free parameters (2)

top_k_percent = 40%
Selected via ablation on Arena-Hard and MT-Bench (Table 2); the value 40% gives the best score on those test benchmarks.
beta = 0.01
Selected via ablation on Arena-Hard and MT-Bench (Table 3); the value 0.01 gives the best score on those test benchmarks.

assumptions (3)

standard math Preference pairs follow the Bradley-Terry model
Inherited from DPO in Eq. (2); this is a standard assumption for preference modeling.
domain assumption The token-level log-probability difference between the reference model and the current policy is a valid proxy for token importance in alignment
Core to the method, introduced in Eq. (6); the paper does not independently validate this proxy against other importance measures.
ad hoc to paper A stronger reference model provides more accurate token selection scores
Used to justify the 10B and 33B reference experiments in Section 4.2; this is asserted based on scaling laws and is not tested with controlled confounds.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Not All Preferences are What You Need for Post-Training: Selective Alignment Strategy for Preference Optimization." pith.science (2026). https://pith.science/paper/XLRPO2HJ

@misc{pith2026250707725,
  author       = {Pith},
  title        = {Pith review of: Not All Preferences are What You Need for Post-Training: Selective Alignment Strategy for Preference Optimization},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/XLRPO2HJ}},
  note         = {Machine review of arXiv:2507.07725}
}

read the original abstract

Post-training alignment of large language models (LLMs) is a critical challenge, as not all tokens contribute equally to model performance. This paper introduces a selective alignment strategy that prioritizes high-impact tokens within preference pairs, leveraging token-level log-probability differences between the current policy and a reference model. By focusing on these informative tokens, our approach reduces computational overhead and enhances alignment fidelity. We further explore the role of reference model quality, demonstrating that stronger reference models significantly improve token selection accuracy and overall optimization effectiveness. Comprehensive experiments on benchmarks such as Arena-Hard and MT-Bench validate the superiority of our Selective-DPO method over standard DPO and distillation-based baselines. Our findings highlight the importance of token-level optimization and reference model selection in advancing preference alignment for LLMs. The code is available at https://github.com/Dongzhijin/SDPO.

Figures

Figures reproduced from arXiv: 2507.07725 by the authors.

**Figure 1.** Workflow of the Selective Alignment Strategy for Preference Optimization. The method focuses on high-impact tokens within preference pairs, leveraging tokenlevel log-probability differences to score and select the most informative tokens for optimization. The process consists of three main steps: computing alignment scores, selecting top k% high-impact tokens, and optimizing the policy using selective-DPO loss. pre… view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

13 extracted references · 2 canonical work pages

[1]

In: Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)

Bai, G., Liu, J., Bu, X., He, Y., Liu, J., Zhou, Z., Lin, Z., Su, W., Ge, T., Zheng, B., Ouyang, W.: Mt-bench-101: A fine-grained benchmark for evaluating large language models in multi-turn dialogues. In: Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). p. 7421–7454. Association for Computat...

doi:10.18653/v1/2024.acl-long.401 2024
[2]

the method of paired comparisons

Bradley, R.A., Terry, M.E.: Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika 39(3/4), 324--345 (1952)

work page 1952
[3]

Chiang, W.L., Zheng, L., Sheng, Y., Angelopoulos, A.N., Li, T., Li, D., Zhang, H., Zhu, B., Jordan, M., Gonzalez, J.E., Stoica, I.: Chatbot arena: An open platform for evaluating llms by human preference (2024), https://arxiv.org/abs/2403.04132

arXiv 2024
[4]

Lai, X., Tian, Z., Chen, Y., Yang, S., Peng, X., Jia, J.: Step-dpo: Step-wise preference optimization for long-chain reasoning of llms (2024), https://arxiv.org/abs/2406.18629

arXiv 2024
[5]

Li, Y., Gu, Y., Dong, L., Wang, D., Cheng, Y., Wei, F.: Direct preference knowledge distillation for large language models (2025), https://arxiv.org/abs/2406.19774

arXiv 2025
[6]

Lin, Z., Gou, Z., Gong, Y., Liu, X., Shen, Y., Xu, R., Lin, C., Yang, Y., Jiao, J., Duan, N., Chen, W.: Rho-1: Not all tokens are what you need (2025), https://arxiv.org/abs/2404.07965

arXiv 2025
[7]

arXiv preprint arXiv:2410.18451 (2024)

Liu, C.Y., Zeng, L., Liu, J., Yan, R., He, J., Wang, C., Yan, S., Liu, Y., Zhou, Y.: Skywork-reward: Bag of tricks for reward modeling in llms. arXiv preprint arXiv:2410.18451 (2024)

arXiv 2024
[8]

Meng, Y., Xia, M., Chen, D.: Simpo: Simple preference optimization with a reference-free reward (2024), https://arxiv.org/abs/2405.14734

arXiv 2024

Show all 13 references

[9]

Rafailov, R., Sharma, A., Mitchell, E., Ermon, S., Manning, C.D., Finn, C.: Direct preference optimization: Your language model is secretly a reward model (2024), https://arxiv.org/abs/2305.18290

2024 arXiv
[10]

arXiv preprint arXiv:1707.06347 (2017), https://arxiv.org/abs/1707.06347

Schulman, J., Wolski, F., Dhariwal, P., Radford, A., Klimov, O.: Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347 (2017), https://arxiv.org/abs/1707.06347

2017 arXiv
[11]

Yang, K., Liu, Z., Xie, Q., Huang, J., Min, E., Ananiadou, S.: Selective preference optimization via token-level reward function estimation (2024), https://arxiv.org/abs/2408.13518

2024
[12]

, " * write output.state after.block = add.period write

ENTRY address author booktitle chapter doi edition editor eid howpublished institution journal key month note number organization pages publisher school series title type url volume year label INTEGERS output.state before.all mid.sentence after.sentence after.block FUNCTION in...
[13]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.

[1] [1]

In: Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)

Bai, G., Liu, J., Bu, X., He, Y., Liu, J., Zhou, Z., Lin, Z., Su, W., Ge, T., Zheng, B., Ouyang, W.: Mt-bench-101: A fine-grained benchmark for evaluating large language models in multi-turn dialogues. In: Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). p. 7421–7454. Association for Computat...

doi:10.18653/v1/2024.acl-long.401 2024

[2] [2]

the method of paired comparisons

Bradley, R.A., Terry, M.E.: Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika 39(3/4), 324--345 (1952)

work page 1952

[3] [3]

Chiang, W.L., Zheng, L., Sheng, Y., Angelopoulos, A.N., Li, T., Li, D., Zhang, H., Zhu, B., Jordan, M., Gonzalez, J.E., Stoica, I.: Chatbot arena: An open platform for evaluating llms by human preference (2024), https://arxiv.org/abs/2403.04132

arXiv 2024

[4] [4]

Lai, X., Tian, Z., Chen, Y., Yang, S., Peng, X., Jia, J.: Step-dpo: Step-wise preference optimization for long-chain reasoning of llms (2024), https://arxiv.org/abs/2406.18629

arXiv 2024

[5] [5]

Li, Y., Gu, Y., Dong, L., Wang, D., Cheng, Y., Wei, F.: Direct preference knowledge distillation for large language models (2025), https://arxiv.org/abs/2406.19774

arXiv 2025

[6] [6]

Lin, Z., Gou, Z., Gong, Y., Liu, X., Shen, Y., Xu, R., Lin, C., Yang, Y., Jiao, J., Duan, N., Chen, W.: Rho-1: Not all tokens are what you need (2025), https://arxiv.org/abs/2404.07965

arXiv 2025

[7] [7]

arXiv preprint arXiv:2410.18451 (2024)

Liu, C.Y., Zeng, L., Liu, J., Yan, R., He, J., Wang, C., Yan, S., Liu, Y., Zhou, Y.: Skywork-reward: Bag of tricks for reward modeling in llms. arXiv preprint arXiv:2410.18451 (2024)

arXiv 2024

[8] [8]

Meng, Y., Xia, M., Chen, D.: Simpo: Simple preference optimization with a reference-free reward (2024), https://arxiv.org/abs/2405.14734

arXiv 2024