Pith. sign in

REVIEW 2 major objections 4 minor 1 cited by

T-REG: Preference Optimization with Token-Level Reward Regularization

T0 review · 2 major / 4 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read T-REG adds self-generated token-level rewards as a regularizer to preference optimization, improving DPO on Alpaca Eval 2 by up to 3.8% and on Arena-Hard by up to 4.4%.

desk verdict Simple, well-ablated DPO add-on whose token-reward mechanism is plausible but under-validated; deserves refereeing. read the letter →

arxiv 2412.02685 v1 pith:7I3EVSTI submitted 2024-12-03 cs.CL cs.AIcs.LG

Wenxuan Zhou , Shujian Zhang , Lingxiao Zhao , Tao Meng This is my paper · ORCID
classification cs.CLcs.AIcs.LG
keywords token-levelrewardpreferenceoptimizationRLHFcontrastivepromptingcreditassignmentDPOinstructionfollowingalignment
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

T-REG is a training-time addition to preference optimization methods like DPO. Before each update, the model itself is asked to rewrite the chosen and rejected responses into 'better' and 'worse' versions, and the token-by-token probability differences between those two rewrites are turned into a per-token reward signal. That signal is added to the usual sequence-level preference loss as a weighted language-modeling regularizer, steering the model to raise the probability of high-reward tokens and lower the probability of low-reward ones. On instruction-following benchmarks, T-REG improves over DPO by up to 3.8% on Alpaca Eval 2 length-controlled win rate and by up to 4.4% on Arena-Hard, and it also helps SimPO and WPO. The paper's core claim is that these cheap self-generated token rewards provide better token-level credit assignment than the implicit token rewards of DPO alone.

What carries the argument

The central object is the self-generated token-level reward $\hat{r}(x, y_{<t}, y_t) = \sigma(\log(\pi_{\text{eval}}(y_t \mid x_{\text{better}}, y_{<t}) / \pi_{\text{eval}}(y_t \mid x_{\text{worse}}, y_{<t}))) - 0.5$, computed from the probability difference a causal language model assigns to token $y_t$ under a 'make this answer better' prompt versus a 'make this answer worse' prompt. This reward is inserted into a regularizer that is just a weighted log-likelihood over tokens — tokens with positive reward get their probability raised, negative ones lowered — and the regularizer is added to the sequence-level preference loss with a per-sequence weight equal to the sigmoid of the DPO reward margin. Two forward passes through the reward model produce rewards for all tokens, so the pipeline needs no trained reward model, no token-level annotators, and no extra parameters.

What would settle it

Construct a test set of prompts where the chosen response has been corrupted by local, token-level errors (e.g., a wrong number, a factually false word, or an instruction-violating capitalization), with human labels marking the bad token positions, then measure whether the regularized model's token-level rewards flag the corrupted tokens more often than DPO's implicit rewards do; if T-REG's token rewards are not more accurate on such a set, the paper's central credit-assignment claim collapses even if benchmark win rates still improve.

Watch

Extended reading notes

Core claim

Direct Preference Optimization implicitly redistributes a single sequence-level reward across tokens as the log-ratio of policy and reference probabilities times a KL coefficient, but this redistribution has no direct token-level supervision. T-REG adds a regularizer that compares that implicit token reward against a dense token reward produced by the LLM itself through contrastive prompting: an evaluation model (here, the reference model) is prompted to rewrite the response in a better direction and in a worse direction, and the reward for token $y_t$ is $\sigma(\log(\pi_{\text{eval}}(y_t \mid x_{\text{better}}, y_{<t}) / \pi_{\text{eval}}(y_t \mid x_{\text{worse}}, y_{<t}))) - 0.5$. This per-token reward is used as a weight on the token log-likelihood in a language-modeling loss, so that tokens judged good by the contrast are made more likely and tokens judged bad are made less likely, all while the sequence-level preference loss continues to enforce the pairwise ranking. The combination, balanced with a detached sequence weight derived from the DPO reward margin, yields consistent gains on Alpaca Eval 2 and Arena-Hard over DPO and SimPO, and qualitative cases show the learned token rewards catch instruction-following errors that DPO misses.

Load-bearing premise

The method's success hinges on the assumption that the contrast between an answer rewritten to be 'better' and an answer rewritten to be 'worse' produces a per-token probability difference that is an accurate and stable measure of each token's quality.

Editorial extensions

If this is right

  • T-REG turns any LLM that can rewrite its own outputs into a source of dense token-level supervision, so token-level credit assignment is no longer tied to a separately trained credit-assignment model or to AI annotators.
  • The regularizer transfers across preference optimizers: applying it on top of SimPO gives further gains, and applying it to WPO on hybrid data reaches a 78.0% length-controlled win rate on Alpaca Eval 2, suggesting it is a general add-on to sequence-level alignment losses.
  • Because the rewards come from the reference model, which is fixed during training, they can be computed before optimization begins, so the per-step training cost need not include the contrastive-prompt forward passes.
  • The sequence-weighting term prevents the regularizer from overruling the preference signal, acting as a per-example gate that down-weights regularization in cases where the DPO margin is already large in the preferred direction.
  • On the models and benchmarks tested, T-REG outperforms alternative token-level methods such as RTO, SePO, and TDPO, achieving consistent positive gains on both general questions and harder expert-level queries.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural next test is to build a token-level reward benchmark, for instance by corrupting individual tokens with factual errors or instruction violations and labeling the bad positions, and then measure whether T-REG's learned token rewards beat DPO's implicit rewards at detecting those corruptions; the paper's limitations section notes that no such quantitative evaluation exists.
  • The contrastive-prompt signal is only as good as the model's self-refinement ability, so the method's benefit may grow with the base model's rewrite quality and could shrink for models that cannot meaningfully revise their own outputs.
  • The same regularization recipe should extend naturally to step-level or span-level rewards for math and coding, where intermediate reasoning steps carry credit, because the weighted language-modeling loss only needs a per-token score.
  • Because the regularizer is effectively a token-selective SFT that avoids the degeneration caused by plain SFT on entire preferred responses, it could be combined with loss-filtering or token-selection strategies to further sharpen which tokens are reinforced.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 4 minor

Summary. The paper proposes T-REG, an extension of DPO/SimPO that adds a token-level regularization term to preference optimization. Token rewards are computed by Eq. (3) as a sigmoid-recentered log-probability ratio between outputs prompted to be "better" and "worse" (Fig. 2), and the regularizer in Eq. (1) is a weighted language-modeling loss applied to both preferred and dispreferred responses with a sequence-level weight (Eq. (2)). Experiments on Alpaca Eval 2 and Arena-Hard with Llama-3-8B-Instruct and Gemma-2-9B-it report consistent gains over DPO and SimPO, up to +3.8 and +4.4 points respectively, with ablations in Table 2 supporting the design choices. The paper claims that the method improves token-level credit assignment, but this is validated only through three qualitative examples in Section 4.3, and the Limitations section acknowledges the absence of a quantitative token-level reward evaluation.

Significance. If the claimed effects are genuine, T-REG is an attractive add-on to preference optimization: it requires no extra reward model, no token-level annotations, and only two extra forward passes per sequence. The ablations in Table 2, especially the comparison with SFT on y_w and with DPO-derived rewards, are informative and show that the method is not a tautology. The main risk is the validity of Eq. (3) as a token-reward signal; the paper's own Limitations section concedes that no systematic quantitative study of token-level reward accuracy is provided. Because the central claim is specifically about token-level credit assignment, this gap is load-bearing.

major comments (2)
  1. [Section 3.3, Eq. (3), Fig. 2] The load-bearing premise of the regularizer is that Eq. (3) measures the contribution of token y_t to response quality. However, the contrastive prompts x_better and x_worse in Fig. 2 both contain the full reference answer {answer} and explicitly instruct the model to "closely resemble the original answer." Hence when π_eval computes π_eval(y_t | x_better, y_<t) and π_eval(y_t | x_worse, y_<t), the target token and later tokens are already visible in the prompt, and the model is rewarded for copying. The log-ratio therefore conflates "this token appears in a better revision" with "this token is easy to copy from the prompt." The paper's only validation is three qualitative examples in Sec. 4.3, and the Limitations explicitly acknowledge that no systematic quantitative study of token-level reward accuracy is provided. As written, the benchmark gains in Table 1 cannot be attributed to token-level credit assignment; at minimum, the authors should validate Eq. (3) with a prompt template that does not reveal the answer, compare against a copying-only baseline, or substantially soften the credit-assignment claim.
  2. [Section 4.1, Table 1] Section 4.1 states that α is searched in {0.1, 0.25, 0.5}, but it does not say whether the search is performed on a held-out validation split or directly on Alpaca Eval 2 and Arena-Hard. If the evaluation benchmarks were used for model selection, the reported gains are optimistic. Moreover, Table 1 reports no seeds or error bars, and the caption's p<0.05 claim is not backed by a description of the statistical test or the number of runs. Please report the selected α values for each model, the validation protocol, and seed-level variability.
minor comments (4)
  1. [Table 1, SimPO-REG row] The statement that T-REG "consistently outperforms" the baselines is contradicted by one reported cell: SimPO-REG on Gemma-2-9B-it has an Alpaca Eval 2 "Win Rate vs GPT-4" of 70.5 versus SimPO's 70.7. Please qualify this claim.
  2. [Section 4.1] The chosen α values are never reported in the main text, tables, or appendix. Please include them per model and per benchmark for reproducibility.
  3. [Section 4.3, Fig. 3] The qualitative case study would be more convincing if it reported numerical token-reward values or a larger sample; three hand-picked examples cannot establish the precision of token-level credit assignment.
  4. [Limitations] The Limitations state that no benchmarks currently exist for evaluating token-level rewards, but process-supervision datasets (e.g., PRM800K) and token-level human annotation efforts exist in the literature. Please acknowledge these and explain why they are not directly applicable to the instruction-following setting.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity; T-REG's derivation chain is self-contained and its benchmark claims are externally evaluated.

full rationale

I find no circular step in the claimed derivation chain. The token-level reward in Eq. (3) is defined as a sigmoid-recentered log-probability ratio of a single evaluation model under two revision prompts; it is an auxiliary, self-supervised signal and is not fitted to, nor defined in terms of, the benchmark targets. The regularizer in Eq. (1) is a weighted language-modeling loss that is added to the DPO loss in Eq. (2); neither equation reduces to the paper's conclusion of improved alignment. The main results are measured on external benchmarks (Alpaca Eval 2 and Arena-Hard), and the ablation table (Table 2) shows that replacing the self-generated rewards with DPO-derived rewards, removing sequence weighting, or adding unweighted SFT degrades results, which is inconsistent with the claim being forced by construction. The self-citations (notably WPO, Zhou et al. 2024a, in the hybrid-data experiment) are confined to an auxiliary result and are not load-bearing for the central DPO/SimPO gains. The Limitations passage explicitly concedes that token-level reward accuracy is not quantitatively evaluated; that is an empirical-evidence gap and a potential correctness risk, but it is not a circularity, because the token-level supervision is not derived from the same target it is used to predict. The skeptical concern that the revision prompts contain the full answer and may reward copying is a validity threat to Eq. (3), not an identity between the output and the input of the derivation.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The central claim rests on two domain assumptions: DPO's implicit token-level reward decomposition (from Rafailov et al. 2024) and the validity of contrastive-prompt probability differences as dense rewards, which is not quantitatively validated. The only numeric free parameters are alpha and the 0.5 recenter offset, both selected by hand or by search on the evaluation benchmarks. No invented entities are introduced.

free parameters (2)
  • alpha regularization strength = searched in {0.1, 0.25, 0.5}; per-model value not reported
    Controls the weight of the token-level regularization term in Eq. (2); no held-out validation is described.
  • Reward recenter offset 0.5 = 0.5
    Hand-chosen constant in Eq. (3) to map the sigmoid ratio into [-0.5, 0.5]; changes the regularization scale and sign threshold.
assumptions (4)
  • domain assumption DPO implicitly defines token-level rewards as beta * log(pi(yt|x,y<t)/pi_ref(yt|x,y<t))
    Used in Section 3.2 to construct Lreg; taken from Rafailov et al. (2024), not re-derived in this paper.
  • domain assumption Contrastive-prompt probability ratios are valid dense token-level rewards
    Eq. (3) assumes P(yt|x_better,y<t) vs P(yt|x_worse,y<t) is a meaningful token-level reward; no quantitative validation is provided beyond three qualitative cases.
  • domain assumption The reference model pi_ref can serve as an evaluator without additional training
    Section 3.3 uses pi_eval = pi_ref; this assumes the SFT reference model's revised-token probabilities are informative for credit assignment.
  • standard math Bradley-Terry preference model
    Standard RLHF assumption used to define LDPO and the sequence-level preference objective.

how reviews work

0 comments
Cite this review

Pith. "Pith review of T-REG: Preference Optimization with Token-Level Reward Regularization." pith.science (2026). https://pith.science/paper/7I3EVSTI

@misc{pith2026241202685,
  author       = {Pith},
  title        = {Pith review of: T-REG: Preference Optimization with Token-Level Reward Regularization},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/7I3EVSTI}},
  note         = {Machine review of arXiv:2412.02685}
}
read the original abstract

Reinforcement learning from human feedback (RLHF) has been crucial in aligning large language models (LLMs) with human values. Traditionally, RLHF involves generating responses to a query and using a reward model to assign a reward to the entire response. However, this approach faces challenges due to its reliance on a single, sparse reward, which makes it challenging for the model to identify which parts of the sequence contribute most significantly to the final reward. Recent methods have attempted to address this limitation by introducing token-level rewards. However, these methods often rely on either a trained credit assignment model or AI annotators, raising concerns about the quality and reliability of the rewards. In this paper, we propose token-level reward regularization (T-REG), a novel approach that leverages both sequence-level and token-level rewards for preference optimization. Harnessing the self-refinement capabilities of LLMs, our method uses contrastive prompting to enable LLMs to self-generate token-level rewards. These self-generated rewards then act as reward regularization, guiding the model to more effectively distribute sequence-level rewards across tokens. This facilitates better token-level credit assignment and enhances alignment performance. Experiments on the instruction following benchmarks, including Alpaca Eval 2 and Arena-Hard, show that our method consistently outperforms baseline methods by up to 3.8% and 4.4%, respectively. We will release the code and models at https://github.com/wzhouad/T-REG.

Figures

Figures reproduced from arXiv: 2412.02685 by the authors.

Figure 1
Figure 1. Overview of the Token-Level Reward Regularization (T-REG). Existing DPO directly optimizes the [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Prompt template for revising the output to either a better or worse one. To make the output [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Case study on the token-level rewards learned [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. ConfPO: Exploiting Policy Model Confidence for Critical Token Selection in Preference Optimization

    cs.CL 2025-06 conditional novelty 6.0 of 10

    Training a policy to prefer responses by optimizing only its own low-confidence (high-surprisal) tokens improves alignment over uniform token optimization in SimPO and DPO.

Reference graph

Works this paper leans on

62 extracted references · 6 canonical work pages · cited by 1 Pith paper

  1. [1]

    Mohammad Gheshlaghi Azar, Zhaohan Daniel Guo, Bilal Piot, Remi Munos, Mark Rowland, Michal Valko, and Daniele Calandriello. 2024. A general theoretical paradigm to understand learning from human preferences. In International Conference on Artificial Intelligence and Statistics, pages 4447--4455. PMLR

    2024

  2. [2]

    Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. 2022. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862

    arXiv 2022

  3. [3]

    Ralph Allan Bradley and Milton E Terry. 1952. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3/4):324--345

    1952

  4. [4]

    Alex J Chan, Hao Sun, Samuel Holt, and Mihaela van der Schaar. 2024. Dense reward for free in reinforcement learning from human feedback. arXiv preprint arXiv:2402.00782

    arXiv 2024

  5. [5]

    Zhao Chen, Vijay Badrinarayanan, Chen-Yu Lee, and Andrew Rabinovich. 2018. Gradnorm: Gradient normalization for adaptive loss balancing in deep multitask networks. In International conference on machine learning, pages 794--803. PMLR

    2018

  6. [6]

    Zhipeng Chen, Kun Zhou, Wayne Xin Zhao, Junchen Wan, Fuzheng Zhang, Di Zhang, and Ji-Rong Wen. 2024. Improving large language models via fine-grained reinforcement learning with minimum editing constraint. arXiv preprint arXiv:2401.06081

    arXiv 2024

  7. [7]

    Wei-Lin Chiang, Lianmin Zheng, Ying Sheng, Anastasios Nikolas Angelopoulos, Tianle Li, Dacheng Li, Hao Zhang, Banghua Zhu, Michael Jordan, Joseph E Gonzalez, et al. 2024. Chatbot arena: An open platform for evaluating llms by human preference. arXiv preprint arXiv:2403.04132

    arXiv 2024

  8. [8]

    Sayak Ray Chowdhury, Anush Kini, and Nagarajan Natarajan. 2024. Provably robust dpo: Aligning language models with noisy feedback. arXiv preprint arXiv:2403.00409

    arXiv 2024

Show all 62 references
  1. [9]

    Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. 2017. Deep reinforcement learning from human preferences. Advances in neural information processing systems, 30

    2017

  2. [10]

    Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. 2024. Scaling instruction-finetuned language models. Journal of Machine Learning Research, 25(70):1--53

    2024

  3. [11]

    Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao, Wei Zhu, Yuan Ni, Guotong Xie, Zhiyuan Liu, and Maosong Sun. 2023. Ultrafeedback: Boosting language models with high-quality feedback. arXiv preprint arXiv:2310.01377

    2023 arXiv

  4. [12]

    Hanze Dong, Wei Xiong, Deepanshu Goyal, Rui Pan, Shizhe Diao, Jipeng Zhang, Kashun Shum, and Tong Zhang. 2023. Raft: Reward ranked finetuning for generative foundation model alignment. arXiv preprint arXiv:2304.06767

    2023 arXiv

  5. [13]

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783

    2024 arXiv

  6. [14]

    Yann Dubois, Bal \'a zs Galambosi, Percy Liang, and Tatsunori B Hashimoto. 2024. Length-controlled alpacaeval: A simple way to debias automatic evaluators. arXiv preprint arXiv:2404.04475

    2024 arXiv

  7. [15]

    Logan Engstrom, Andrew Ilyas, Shibani Santurkar, Dimitris Tsipras, Firdaus Janoos, Larry Rudolph, and Aleksander Madry. 2019. Implementation matters in deep rl: A case study on ppo and trpo. In International conference on learning representations

    2019

  8. [16]

    Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky, and Douwe Kiela. 2024. Kto: Model alignment as prospect theoretic optimization. arXiv preprint arXiv:2402.01306

    2024 arXiv

  9. [17]

    Leo Gao, John Schulman, and Jacob Hilton. 2023. Scaling laws for reward model overoptimization. In International Conference on Machine Learning, pages 10835--10866. PMLR

    2023

  10. [18]

    Amelia Glaese, Nat McAleese, Maja Tr e bacz, John Aslanides, Vlad Firoiu, Timo Ewalds, Maribeth Rauh, Laura Weidinger, Martin Chadwick, Phoebe Thacker, et al. 2022. Improving alignment of dialogue agents via targeted human judgements. arXiv preprint arXiv:2209.14375

    2022 arXiv

  11. [19]

    Geyang Guo, Ranchi Zhao, Tianyi Tang, Wayne Xin Zhao, and Ji-Rong Wen. 2023. Beyond imitation: Leveraging fine-grained quality signals for alignment. arXiv preprint arXiv:2311.04072

    2023 arXiv

  12. [20]

    Shangmin Guo, Biao Zhang, Tianlin Liu, Tianqi Liu, Misha Khalman, Felipe Llinares, Alexandre Rame, Thomas Mesnard, Yao Zhao, Bilal Piot, et al. 2024. Direct language model alignment from online ai feedback. arXiv preprint arXiv:2402.04792

    2024 arXiv

  13. [21]

    James Y Huang, Wenxuan Zhou, Fei Wang, Fred Morstatter, Sheng Zhang, Hoifung Poon, and Muhao Chen. 2024. Offset unlearning for large language models. arXiv preprint arXiv:2404.11045

    2024 arXiv

  14. [22]

    Jungo Kasai, Keisuke Sakaguchi, Ronan Le Bras, Hao Peng, Ximing Lu, Dragomir Radev, Yejin Choi, and Noah A Smith. 2022. Twist decoding: Diverse generators guide each other. arXiv preprint arXiv:2205.09273

    2022 arXiv

  15. [23]

    Taehyeon Kim, Joonkee Kim, Gihun Lee, and Se-Young Yun. 2024. Instructive decoding: Instruction-tuned large language models are self-refiner from noisy instructions. In The Twelfth International Conference on Learning Representations

    2024

  16. [24]

    Nathan Lambert, Valentina Pyatkin, Jacob Morrison, LJ Miranda, Bill Yuchen Lin, Khyathi Chandu, Nouha Dziri, Sachin Kumar, Tom Zick, Yejin Choi, et al. 2024. Rewardbench: Evaluating reward models for language modeling. arXiv preprint arXiv:2403.13787

    2024 arXiv

  17. [25]

    Harrison Lee, Samrat Phatale, Hassan Mansoor, Thomas Mesnard, Johan Ferret, Kellie Lu, Colton Bishop, Ethan Hall, Victor Carbune, Abhinav Rastogi, et al. 2023. Rlaif: Scaling reinforcement learning from human feedback with ai feedback. arXiv preprint arXiv:2309.00267

    2023 arXiv

  18. [26]

    Tianle Li, Wei-Lin Chiang, Evan Frick, Lisa Dunlap, Tianhao Wu, Banghua Zhu, Joseph E Gonzalez, and Ion Stoica. 2024. From crowdsourced data to high-quality benchmarks: Arena-hard and benchbuilder pipeline. arXiv preprint arXiv:2406.11939

    2024 arXiv

  19. [27]

    Xiang Lisa Li, Ari Holtzman, Daniel Fried, Percy Liang, Jason Eisner, Tatsunori Hashimoto, Luke Zettlemoyer, and Mike Lewis. 2022. Contrastive decoding: Open-ended text generation as optimization. arXiv preprint arXiv:2210.15097

    2022 arXiv

  20. [28]

    Hashimoto

    Xuechen Li, Tianyi Zhang, Yann Dubois, Rohan Taori, Ishaan Gulrajani, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023. Alpacaeval: An automatic evaluator of instruction-following models. https://github.com/tatsu-lab/alpaca_eval

    2023

  21. [29]

    Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. 2023. Let's verify step by step. arXiv preprint arXiv:2305.20050

    2023 arXiv

  22. [30]

    Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. 2023. https://doi.org/10.18653/v1/2023.emnlp-main.153 G -eval: NLG evaluation using gpt-4 with better human alignment . In Proceedings of the 2023 Conference on Empirical Methods in Natural Language ...

    2023 doi

  23. [31]

    Ximing Lu, Sean Welleck, Jack Hessel, Liwei Jiang, Lianhui Qin, Peter West, Prithviraj Ammanabrolu, and Yejin Choi. 2022. Quark: Controllable text generation with reinforced unlearning. Advances in neural information processing systems, 35:27591--27609

    2022

  24. [32]

    Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, et al. 2024. Self-refine: Iterative refinement with self-feedback. Advances in Neural Information Processing Systems, 36

    2024

  25. [33]

    Yu Meng, Mengzhou Xia, and Danqi Chen. 2024. Simpo: Simple preference optimization with a reference-free reward. arXiv preprint arXiv:2405.14734

    2024 arXiv

  26. [34]

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 3...

    2022

  27. [35]

    Rafael Rafailov, Joey Hejna, Ryan Park, and Chelsea Finn. 2024. From r to q^* : Your language model is secretly a q-function. arXiv preprint arXiv:2404.12358

    2024 arXiv

  28. [36]

    Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. 2023. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36

    2023

  29. [37]

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347

    2017 arXiv

  30. [38]

    Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivi \`e re, Mihir Sanjay Kale, Juliette Love, et al. 2024. Gemma: Open models based on gemini research and technology. arXiv preprint arXiv:2403.08295

    2024 arXiv

  31. [39]

    Lewis Tunstall, Edward Beeching, Nathan Lambert, Nazneen Rajani, Kashif Rasul, Younes Belkada, Shengyi Huang, Leandro von Werra, Cl \'e mentine Fourrier, Nathan Habib, et al. 2023. Zephyr: Direct distillation of lm alignment. arXiv preprint arXiv:2310.16944

    2023 arXiv

  32. [40]

    Jonathan Uesato, Nate Kushman, Ramana Kumar, Francis Song, Noah Siegel, Lisa Wang, Antonia Creswell, Geoffrey Irving, and Irina Higgins. 2022. Solving math word problems with process-and outcome-based feedback. arXiv preprint arXiv:2211.14275

    2022 arXiv

  33. [41]

    Binghai Wang, Rui Zheng, Lu Chen, Yan Liu, Shihan Dou, Caishuang Huang, Wei Shen, Senjie Jin, Enyu Zhou, Chenyu Shi, et al. 2024 a . Secrets of rlhf in large language models part ii: Reward modeling. arXiv preprint arXiv:2401.06080

    2024 arXiv

  34. [42]

    Fei Wang, Wenxuan Zhou, James Y Huang, Nan Xu, Sheng Zhang, Hoifung Poon, and Muhao Chen. 2024 b . mdpo: Conditional preference optimization for multimodal large language models. arXiv preprint arXiv:2406.11839

    2024 arXiv

  35. [43]

    Haoxiang Wang, Yong Lin, Wei Xiong, Rui Yang, Shizhe Diao, Shuang Qiu, Han Zhao, and Tong Zhang. 2024 c . Arithmetic control of llms for diverse user preferences: Directional preference alignment with multi-objective rewards. In ACL

    2024

  36. [44]

    Haoxiang Wang, Wei Xiong, Tengyang Xie, Han Zhao, and Tong Zhang. 2024 d . Interpretable preferences via multi-objective reward modeling and mixture-of-experts. arXiv preprint arXiv:2406.12845

    2024 arXiv

  37. [45]

    Tianlu Wang, Ilia Kulikov, Olga Golovneva, Ping Yu, Weizhe Yuan, Jane Dwivedi-Yu, Richard Yuanzhe Pang, Maryam Fazel-Zarandi, Jason Weston, and Xian Li. 2024 e . Self-taught evaluators. arXiv preprint arXiv:2408.02666

    2024 arXiv

  38. [46]

    Zeqiu Wu, Yushi Hu, Weijia Shi, Nouha Dziri, Alane Suhr, Prithviraj Ammanabrolu, Noah A Smith, Mari Ostendorf, and Hannaneh Hajishirzi. 2024. Fine-grained human feedback gives better rewards for language model training. Advances in Neural Information Processing Systems, 36

    2024

  39. [47]

    Wei Xiong, Hanze Dong, Chenlu Ye, Ziqi Wang, Han Zhong, Heng Ji, Nan Jiang, and Tong Zhang. 2024. https://arxiv.org/abs/2312.11456 Iterative preference learning from human feedback: Bridging theory and practice for rlhf under kl-constraint . Preprint, arXiv:2312.11456

    2024 arXiv

  40. [48]

    Haoran Xu, Amr Sharaf, Yunmo Chen, Weiting Tan, Lingfeng Shen, Benjamin Van Durme, Kenton Murray, and Young Jin Kim. 2024. Contrastive preference optimization: Pushing the boundaries of llm performance in machine translation. In Forty-first International Conference on Machine Learning

    2024

  41. [49]

    Kailai Yang, Zhiwei Liu, Qianqian Xie, Jimin Huang, Erxue Min, and Sophia Ananiadou. 2024 a . Selective preference optimization via token-level reward function estimation. arXiv preprint arXiv:2408.13518

    2024

  42. [50]

    Shentao Yang, Shujian Zhang, Congying Xia, Yihao Feng, Caiming Xiong, and Mingyuan Zhou. 2024 b . Preference-grounded token-level guidance for language model fine-tuning. Advances in Neural Information Processing Systems, 36

    2024

  43. [51]

    Eunseop Yoon, Hee Suk Yoon, SooHwan Eom, Gunsoo Han, Daniel Wontae Nam, Daejin Jo, Kyoung-Woon On, Mark A Hasegawa-Johnson, Sungwoong Kim, and Chang D Yoo. 2024. Tlcr: Token-level continuous reward for fine-grained reinforcement learning from human feedback. arXiv preprint arX...

    2024 arXiv

  44. [52]

    Yongcheng Zeng, Guoqing Liu, Weiyu Ma, Ning Yang, Haifeng Zhang, and Jun Wang. 2024. Token-level direct preference optimization. arXiv preprint arXiv:2404.11999

    2024 arXiv

  45. [53]

    Xuandong Zhao, Xianjun Yang, Tianyu Pang, Chao Du, Lei Li, Yu-Xiang Wang, and William Yang Wang. 2024 a . Weak-to-strong jailbreaking on large language models. arXiv preprint arXiv:2401.17256

    2024 arXiv

  46. [54]

    Yao Zhao, Rishabh Joshi, Tianqi Liu, Misha Khalman, Mohammad Saleh, and Peter J Liu. 2023. Slic-hf: Sequence likelihood calibration with human feedback. arXiv preprint arXiv:2305.10425

    2023 arXiv

  47. [55]

    Zhengyue Zhao, Xiaoyun Zhang, Kaidi Xu, Xing Hu, Rui Zhang, Zidong Du, Qi Guo, and Yunji Chen. 2024 b . Adversarial contrastive decoding: Boosting safety alignment of large language models via opposite prompt optimization. arXiv preprint arXiv:2406.16743

    2024

  48. [56]

    Chujie Zheng, Ziqi Wang, Heng Ji, Minlie Huang, and Nanyun Peng. 2024. Weak-to-strong extrapolation expedites alignment. arXiv preprint arXiv:2404.16792

    2024 arXiv

  49. [57]

    Han Zhong, Guhao Feng, Wei Xiong, Li Zhao, Di He, Jiang Bian, and Liwei Wang. 2024. Dpo meets ppo: Reinforced token optimization for rlhf. arXiv preprint arXiv:2404.18922

    2024 arXiv

  50. [58]

    Wenxuan Zhou, Ravi Agrawal, Shujian Zhang, Sathish Reddy Indurthi, Sanqiang Zhao, Kaiqiang Song, Silei Xu, and Chenguang Zhu. 2024 a . Wpo: Enhancing rlhf with weighted preference optimization. arXiv preprint arXiv:2406.11827

    2024 arXiv

  51. [59]

    Zhanhui Zhou, Zhixuan Liu, Jie Liu, Zhichen Dong, Chao Yang, and Yu Qiao. 2024 b . Weak-to-strong search: Align large language models via searching over small language models. arXiv preprint arXiv:2405.19262

    2024 arXiv

  52. [60]

    Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. 2019. Fine-tuning language models from human preferences. arXiv preprint arXiv:1909.08593

    2019 arXiv

  53. [61]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  54. [62]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.