Pith. sign in

REVIEW 4 major objections 6 minor 50 references

Preference-Oriented Supervised Fine-Tuning: Favoring Target Model Over Aligned Large Language Models

T0 review · 4 major / 6 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper introduces PoFT, an SFT objective that makes aligned LLMs' length-normalized likelihoods act as per-sample quality weights, and reports stable gains over cross-entropy SFT across datasets, base models, and noise levels.

desk verdict PoFT is a genuinely new SFT objective with sound math and consistent empirical gains, but the quality-filtering story is under-supported and the gains are modest. read the letter →

arxiv 2412.12865 v1 pith:UCUKNY5A submitted 2024-12-17 cs.CL

Yuchen Fan , Yuzhong Hong , Qiushi Wang , Junwei Bao , Hongfei Jiang , Yang Song This is my paper · ORCID
classification cs.CL
keywords preference-orientedsupervisedfine-tuninginstructiontuningdataqualityBradley-TerrymodellargelanguagemodelsnoiserobustnesssoftfilteringDPOcombination
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Standard supervised fine-tuning (SFT) trains a large language model with a cross-entropy loss that treats every instruction-response pair equally, so low-quality pairs drag the model down. This paper proposes PoFT, a fine-tuning objective that instead asks the model to beat a set of already-aligned reference LLMs on each training pair, using length-normalized likelihood as the score. The Bradley-Terry term makes the gradient of each sample carry an automatic weight that is low when the reference models give that sample a low score, turning the loss into a soft data-quality filter. The authors report that PoFT consistently outperforms cross-entropy SFT across multiple datasets and base models, degrades more gracefully on noisy and quality-limited data, and combines with data filtering and with DPO. The point of the paper is that preference modeling between models, rather than between responses, is a practical way to make SFT robust to imperfect data.

What carries the argument

The central object is a Bradley-Terry (BT) preference term applied between models rather than between candidate responses. The reward is the length-normalized log-likelihood $r_\theta(x,y) = \frac{1}{T_0(y)}\log p_\theta(y\mid x)$ for the target model and $r_j(x,y)=\frac{1}{T_j(y)}\log p_j(y\mid x)$ for each reference LLM, and the loss is $-\log\sigma\!\left(\frac{1}{M}\sum_{j=1}^M (r_\theta(x,y)-r_j(x,y))\right)$. Its gradient, $\nabla_\theta \mathcal{L}_{\mathrm{PoFT}} = -\frac{1}{T_0(y)}\frac{1}{p_\theta(y\mid x)}\,\tau\,\nabla_\theta p_\theta(y\mid x)$, carries the coefficient $\tau = \frac{G}{G + p_\theta(y\mid x)^{1/T_0(y)}}$, where $G$ is the geometric mean of the references' normalized likelihoods. That $\tau$ is the dynamic weight that down-weights low-quality samples, and the paper's derivation of it is the argument that PoFT is CE with soft quality weighting rather than a fundamentally different training paradigm.

What would settle it

Train two copies of the same model on the same noisy dataset, one with PoFT using the real reference scores and one with the same reference scores randomly permuted across samples; if the permuted version matches PoFT's robustness, the quality-proxy mechanism is not what drives the gain. A cheaper check is to measure the correlation between the reference likelihoods and human quality judgments on a sample of the training set, and look for a positive correlation.

Watch

Extended reading notes

Core claim

The paper's central claim is that PoFT's objective—maximizing the log-probability that the target model's normalized likelihood beats the average normalized likelihood of the reference LLMs on the same $\langle x, y\rangle$—yields the same gradient as cross-entropy SFT multiplied by a per-sample coefficient $\tau$, and that this coefficient is precisely what makes training stable on noisy data. When the references assign a sample a high likelihood, $\tau$ is large and the sample contributes strongly; when they assign a low likelihood, $\tau$ shrinks and the sample's influence fades. Because the method needs only $\langle x, y\rangle$ pairs and no chosen/rejected triples, the paper presents PoFT as an SFT method rather than a preference-alignment method, and it shows empirically that the gains persist across UltraChat, OpenHermes, and ShareGPT on Mistral-7B and Llama-3-8B, with the largest gains on OpenHermes where the reference score distribution is widest.

Load-bearing premise

The load-bearing premise is that an aligned LLM's length-normalized likelihood on a training sample measures that sample's quality; if low-quality samples receive high likelihoods, the weighting would amplify noise instead of suppressing it.

Editorial extensions

If this is right

  • PoFT improves final-checkpoint and epoch-averaged scores over cross-entropy SFT on the Open LLM Leaderboard, MT-Bench, and AlpacaEval 2.0 across datasets and base models.
  • On quality-limited datasets (Alpaca, Dolly) and on datasets with injected noise, PoFT models avoid the late-epoch performance collapse that SFT shows, so the objective is more stable when data quality cannot be guaranteed.
  • PoFT is orthogonal to explicit data filtering: combining it with IFD, Instag, or Deita yields higher scores than either approach alone.
  • PoFT can serve as the SFT stage before DPO; the two-step PoFT-then-DPO pipeline beats SFT-then-DPO on the reported benchmarks.
  • The per-sample weighting makes PoFT act as a soft filter, so its benefit is largest when the reference models' preference scores on the training set are widely spread rather than concentrated.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the reference-likelihood quality proxy holds, PoFT offers a low-cost way to inject quality information into SFT without human annotation; a natural extension is to test whether a single, smaller reference model gives the same benefit as the ensemble of three.
  • The same model-vs-model preference objective should transfer to other instruction-tuning settings, such as vision-language models, as long as a reference model can assign a likelihood to the target output.
  • A telling control experiment would be to shuffle or randomize the reference scores: if PoFT keeps its advantage, the gain comes from the BT term's implicit regularization rather than from quality assessment; the paper does not report this control.
  • The bi-PoFT variant, which explicitly flips the preference for labeled noise data, suggests that providing even coarse noise labels could tune the filter; this is an extension the authors flag for future work.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes PoFT, a supervised fine-tuning objective that applies the Bradley-Terry model to a preference between the target model and a set of fixed aligned LLMs, using length-normalized log-likelihoods as reward functions. The derived gradient (Eqs. 6–8 and Appendix A) is cross-entropy scaled by a per-sample coefficient tau, which the authors interpret as a data-quality weight from the aligned LLMs. The empirical sections report improvements over CE SFT on UltraChat200k, OpenHermes, and ShareGPT with Mistral-7B and Llama-3-8B, plus ablations on quality-limited data, synthetic noise, data filtering, and integration with DPO. The paper claims that PoFT is more robust to low-quality data than standard SFT and that the reference likelihoods provide data-quality assessments.

Significance. If the empirical claims hold with proper statistical support, PoFT is a simple and potentially useful contribution: it adds no trainable parameters, is easy to integrate into existing SFT pipelines, and the code is provided. The gradient derivation is essentially correct (apart from an appendix typo), and the synthetic-noise and filtering experiments are a reasonable first step toward the robustness claim. However, the core quality-proxy mechanism and the statistical strength of the main improvements need additional support, so the paper is not yet ready for acceptance.

major comments (4)
  1. [Section 4.2, Table 1] The central claim of 'stable and consistent improvements' is not yet supported statistically. The reported standard deviations are across training epochs, not across random seeds, so they do not measure run-to-run variability, and no significance tests or confidence intervals are provided. Several differences are small or within the reported across-epoch variability (e.g., Llama-3-8B on UltraChat: 65.90 vs 65.71 last-epoch; average 65.88 vs 65.65; ShareGPT average 65.45 vs 65.34), and in the ShareGPT row the PoFT run has higher across-epoch std than the SFT run (0.19 vs 0.11). Please report multiple seeds with mean and variance, or otherwise provide a statistical basis for the comparison.
  2. [Section 3.2, Eq. (8)] The paper's mechanism claim that PoFT 'incorporates assessment information on data quality' is load-bearing but not validated. The weight tau in Eq. (8) depends on the target model's own likelihood p_theta as well as the reference likelihoods, and it decreases as p_theta grows; the observed robustness could therefore be a reference-relative confidence regularizer rather than a quality filter. The paper does not compare against a control that isolates the target-confidence term, and the reference-likelihood proxy is never checked against independent quality labels on real SFT data. Please provide such a check (e.g., correlation with human or model-based quality annotations) or revise the claimed mechanism.
  3. [Appendix A, Eq. (11)] The displayed chain for the PoFT gradient contains an incorrect intermediate expression: the denominator should be p_theta(y|x)^{1/T0} + R, where R is the geometric mean of the reference likelihood terms, not p_theta(y|x)^{1/T0} + tau (with tau already the full coefficient defined in Eq. (8)). The final gradient in Eq. (7) is correct, so this is a derivational typo, but it should be fixed because it is part of the formal derivation.
  4. [Section 4.1, Table 1] One of the reference models, Zephyr-7B-sft-full, was itself trained on UltraChat200k, the same dataset used in the UltraChat experiments. The reference likelihoods are therefore not an independent quality signal for that dataset. The paper should report the UltraChat results with the Zephyr reference excluded, or explicitly discuss the possible circularity, in order to support the claim that the reference models provide external quality assessments.
minor comments (6)
  1. [Figure 3] The caption labels are inconsistent: it lists '(c) Performances ... with Alpaca. (d) Performances ... with Dolly. (d) Preference score distributions ...' with a duplicated (d), and the main text refers to Figure 3b/3c and 3d/3e in a way that does not match the subfigure numbering.
  2. [Appendix B.1] The training settings are not fully specified; 'learning rate is varied from 5e-6 to 5e-5' and 'three to four epochs' are not enough to reproduce Tables 1–5. Please give the exact hyperparameters used for each row.
  3. [Table 2] MT-Bench and AlpacaEval results are reported only for Mistral-7B; to support the claim that improvements hold across base models, these evaluations should also be reported for Llama-3-8B or the claim should be limited to the Open LLM Leaderboard.
  4. [Abstract and Conclusion] The abstract and conclusion use 'prove' for empirical integration results; since these are experimental demonstrations, a weaker verb such as 'show' would be more accurate.
  5. [Section 3.2, Eq. (6)] The notation 'LLM j ∈ D_LLM, |D| = M' is confusing; the set and its size are later mixed with the dataset D_SFT. Please use distinct symbols.
  6. [Section 4.1, Table 1] The sentence about Zephyr-7B-sft-full, 'by adjusting hyper-parameters, it could achieve better performance (see the fourth row of Table 1)', is vague because the fourth row is the Mistral-7B SFT baseline; please clarify what is being compared.

Circularity Check

1 steps flagged · score 2.0 of 10

PoFT is largely self-contained; the main circularity is localized to the UltraChat experiments, where a reference model used as the quality oracle was itself trained on the same UltraChat200k data, making its likelihood partly a memorization score rather than an independent quality signal.

  1. other [Section 4.1 Settings, Model; Table 1 (UltraChat rows)]
    "For aligned LLMs, we adopt zephyr-7b-sft-full (Tunstall et al. 2023b), Llama-3-8B-Instruct (AI@Meta 2024), and Yi-6B-Chat (AI et al. 2024). Notably, Zephyr-7B-sft-full, derived from Mistral-7B-v0.1, trained on UltraChat200k."

    In the UltraChat200k experiments, one of the reference models whose length-normalized likelihood defines the dynamic weight tau in Eq. 8 was itself SFT-trained on the exact same UltraChat200k samples. Its likelihood on those samples is therefore partly a fitted/memorized value rather than an independent assessment of data quality. Using that likelihood as the 'quality' signal to weight the same training data makes the UltraChat quality-assessments partially circular by data overlap, even though the final benchmark scores are external and not forced by this construction.

full rationale

The central derivation is self-contained: PoFT's loss in Eq. 6 is defined directly from the target likelihood and the fixed reference likelihoods, and its gradient in Eq. 7 is just the CE gradient scaled by tau in Eq. 8. No parameter is fitted to the evaluation benchmarks, and the main empirical claims are tested against external benchmarks (Open LLM Leaderboard, MT-Bench, AlpacaEval 2.0), so the claimed improvements do not reduce to the loss definition. There is no load-bearing self-citation chain, no imported uniqueness theorem, and no fitted parameter renamed as a benchmark prediction. The only notable circularity is the localized UltraChat confound: Zephyr-7B-sft-full, one of the three aligned reference LLMs, was trained on UltraChat200k, so its likelihood on that dataset is partly memorization rather than clean quality evidence. Additionally, the paper defines 'data quality' operationally as aligned-LLM likelihood, making statements such as 'incorporating assessment information on data quality' partly definitional; however, the robustness claims also rest on synthetic-noise experiments and data-filtering comparisons, which provide independent empirical support. Overall, the circularity is minor and does not undermine the central derivation.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The central claims rest on four domain assumptions; no free parameters are fitted to the benchmarks, and PoFT introduces no new objects or entities.

assumptions (4)
  • domain assumption Aligned LLM likelihood on an SFT sample is a valid indicator of data quality
    The method's robustness claim rests on the premise that p_j(y|x) reflects sample quality; invoked in Section 3.2 and throughout the noise experiments, but not directly validated.
  • ad hoc to paper The target model should be preferred to aligned LLMs on all SFT data (P goes to 1)
    This imposed preference defines the objective (Eq. 6); it is a modeling choice rather than an empirically grounded relationship.
  • domain assumption Length-normalized log-likelihoods are comparable across models with different tokenizers
    Section 3.2 justifies using 1/T log p(y|x); assumes normalization accounts for tokenizer mismatch.
  • domain assumption Hand-crafted corruption (random matching, character edits) mimics real low-quality SFT data
    Section 4.3 builds noise data this way to draw conclusions about robustness to real quality-limited data.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Preference-Oriented Supervised Fine-Tuning: Favoring Target Model Over Aligned Large Language Models." pith.science (2026). https://pith.science/paper/UCUKNY5A

@misc{pith2026241212865,
  author       = {Pith},
  title        = {Pith review of: Preference-Oriented Supervised Fine-Tuning: Favoring Target Model Over Aligned Large Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/UCUKNY5A}},
  note         = {Machine review of arXiv:2412.12865}
}
read the original abstract

Alignment, endowing a pre-trained Large language model (LLM) with the ability to follow instructions, is crucial for its real-world applications. Conventional supervised fine-tuning (SFT) methods formalize it as causal language modeling typically with a cross-entropy objective, requiring a large amount of high-quality instruction-response pairs. However, the quality of widely used SFT datasets can not be guaranteed due to the high cost and intensive labor for the creation and maintenance in practice. To overcome the limitations associated with the quality of SFT datasets, we introduce a novel \textbf{p}reference-\textbf{o}riented supervised \textbf{f}ine-\textbf{t}uning approach, namely PoFT. The intuition is to boost SFT by imposing a particular preference: \textit{favoring the target model over aligned LLMs on the same SFT data.} This preference encourages the target model to predict a higher likelihood than that predicted by the aligned LLMs, incorporating assessment information on data quality (i.e., predicted likelihood by the aligned LLMs) into the training process. Extensive experiments are conducted, and the results validate the effectiveness of the proposed method. PoFT achieves stable and consistent improvements over the SFT baselines across different training datasets and base models. Moreover, we prove that PoFT can be integrated with existing SFT data filtering methods to achieve better performance, and further improved by following preference optimization procedures, such as DPO.

Figures

Figures reproduced from arXiv: 2412.12865 by the authors.

Figure 1
Figure 1. The overall modeling framework of PoFT. By [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Preference scores generated by aligned LLMs [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Analysis and model performances on quality-limited data. (a) Preference score distributions of data-limited datasets [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Performance of Mistral-7B trained with different [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: Performances of bi-PoFT models training with [PITH_FULL_IMAGE:figures/full_fig_p009_5.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

50 extracted references · 14 canonical work pages

  1. [1]

    AI, .; :; Young, A.; Chen, B.; Li, C.; Huang, C.; Zhang, G.; Zhang, G.; Li, H.; Zhu, J.; Chen, J.; Chang, J.; Yu, K.; Liu, P.; Liu, Q.; Yue, S.; Yang, S.; Yang, S.; Yu, T.; Xie, W.; Huang, W.; Hu, X.; Ren, X.; Niu, X.; Nie, P.; Xu, Y.; Liu, Y.; Wang, Y.; Cai, Y.; Gu, Z.; Liu, Z.; and Dai, Z. 2024. Yi: Open Foundation Models by 01.AI. arXiv:2403.04652

    arXiv 2024

  2. [2]

    S., Sondos Mahmoud Bsharat

    Aidar Myrzakhan, Z. S., Sondos Mahmoud Bsharat. 2024. Open-LLM-Leaderboard: From Multi-choice to Open-style Questions for LLMs Evaluation, Benchmark, and Arena. arXiv preprint arXiv:2406.07545

    arXiv 2024

  3. [3]

    AI@Meta. 2024. Llama 3 Model Card

    2024

  4. [4]

    Bai, Y.; Jones, A.; Ndousse, K.; Askell, A.; Chen, A.; DasSarma, N.; Drain, D.; Fort, S.; Ganguli, D.; Henighan, T.; Joseph, N.; Kadavath, S.; Kernion, J.; Conerly, T.; El-Showk, S.; Elhage, N.; Hatfield-Dodds, Z.; Hernandez, D.; Hume, T.; Johnston, S.; Kravec, S.; Lovitt, L.; Nanda, N.; Olsson, C.; Amodei, D.; Brown, T.; Clark, J.; McCandlish, S.; Olah, ...

    arXiv 2022

  5. [5]

    A.; and Terry, M

    Bradley, R. A.; and Terry, M. E. 1952. Rank Analysis of Incomplete Block Designs: I. The Method of Paired Comparisons. Biometrika, 39: 324

    1952

  6. [6]

    Chen, L.; Li, S.; Yan, J.; Wang, H.; Gunaratna, K.; Yadav, V.; Tang, Z.; Srinivasan, V.; Zhou, T.; Huang, H.; and Jin, H. 2024 a . AlpaGasus: Training A Better Alpaca with Fewer Data. arXiv:2307.08701

    arXiv 2024

  7. [7]

    Chen, Z.; Deng, Y.; Yuan, H.; Ji, K.; and Gu, Q. 2024 b . Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Models. arXiv:2401.01335

    arXiv 2024

  8. [8]

    Clark, P.; Cowhey, I.; Etzioni, O.; Khot, T.; Sabharwal, A.; Schoenick, C.; and Tafjord, O. 2018. Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge. arXiv: Artificial Intelligence,arXiv: Artificial Intelligence

    work page 2018

Show all 50 references
  1. [9]

    Cobbe, K.; Kosaraju, V.; Bavarian, M.; Hilton, J.; Nakano, R.; Hesse, C.; and Schulman, J. 2021. Training Verifiers to Solve Math Word Problems. Cornell University - arXiv,Cornell University - arXiv

    2021

  2. [10]

    Conover, M.; Hayes, M.; Mathur, A.; Xie, J.; Wan, J.; Shah, S.; Ghodsi, A.; Wendell, P.; Zaharia, M.; and Xin, R. 2023. Free dolly: Introducing the world’s first truly open instruction-tuned llm. Company Blog of Databricks

    2023

  3. [11]

    Cui, G.; Yuan, L.; Ding, N.; Yao, G.; Zhu, W.; Ni, Y.; Xie, G.; Liu, Z.; and Sun, M. 2023. UltraFeedback: Boosting Language Models with High-quality Feedback. arXiv:2310.01377

    2023 arXiv

  4. [12]

    Dao, T. 2023. FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning. arXiv:2307.08691

    2023 arXiv

  5. [13]

    Ding, N.; Chen, Y.; Xu, B.; Qin, Y.; Zheng, Z.; Hu, S.; Liu, Z.; Sun, M.; and Zhou, B. 2023. Enhancing Chat Language Models by Scaling High-quality Instructional Conversations. arXiv:2305.14233

    2023 arXiv

  6. [14]

    Dong, H.; Xiong, W.; Goyal, D.; Zhang, Y.; Chow, W.; Pan, R.; Diao, S.; Zhang, J.; Shum, K.; and Zhang, T. 2023. RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignment. arXiv:2304.06767

    2023 arXiv

  7. [15]

    Du, Q.; Zong, C.; and Zhang, J. 2023. MoDS: Model-oriented Data Selection for Instruction Tuning. arXiv:2311.15653

    2023 arXiv

  8. [16]

    Gao, L.; Tow, J.; Abbasi, B.; Biderman, S.; Black, S.; DiPofi, A.; Foster, C.; Golding, L.; Hsu, J.; Le Noac'h, A.; Li, H.; McDonell, K.; Muennighoff, N.; Ociepa, C.; Phang, J.; Reynolds, L.; Schoelkopf, H.; Skowron, A.; Sutawika, L.; Tang, E.; Thite, A.; Wang, B.; Wang, K.; a...

    2023

  9. [17]

    Hendrycks, D.; Burns, C.; Basart, S.; Zou, A.; Mazeika, M.; Song, D.; and Steinhardt, J. 2020. Measuring Massive Multitask Language Understanding. Cornell University - arXiv,Cornell University - arXiv

    2020

  10. [18]

    Houlsby, N.; Giurgiu, A.; Jastrzebski, S.; Morrone, B.; Laroussilhe, Q.; Gesmundo, A.; Attariyan, M.; and Gelly, S. 2019. Parameter-Efficient Transfer Learning for NLP. International Conference on Machine Learning,International Conference on Machine Learning

    2019

  11. [19]

    Ivison, H.; Bhagia, A.; Wang, Y.; Hajishirzi, H.; and Peters, M. 2023. HINT: Hypernetwork Instruction Tuning for Efficient Zero- & Few-Shot Generalisation. arXiv:2212.10315

    2023 arXiv

  12. [20]

    J., H.; Shen, Y.; Wallis, P.; Allen-Zhu, Z.; Li, Y.; Wang, S.; and Chen, W. 2021. LoRA: Low-Rank Adaptation of Large Language Models. arXiv: Computation and Language,arXiv: Computation and Language

    2021

  13. [21]

    Q.; Sablayrolles, A.; Mensch, A.; Bamford, C.; Chaplot, D

    Jiang, A. Q.; Sablayrolles, A.; Mensch, A.; Bamford, C.; Chaplot, D. S.; de las Casas, D.; Bressand, F.; Lengyel, G.; Lample, G.; Saulnier, L.; Lavaud, L. R.; Lachaux, M.-A.; Stock, P.; Scao, T. L.; Lavril, T.; Wang, T.; Lacroix, T.; and Sayed, W. E. 2023. Mistral 7B. arXiv:2310.06825

    2023 arXiv

  14. [22]

    Li, M.; Chen, L.; Chen, J.; He, S.; Gu, J.; and Zhou, T. 2024 a . Selective Reflection-Tuning: Student-Selected Data Recycling for LLM Instruction-Tuning. arXiv:2402.10110

    2024 arXiv

  15. [23]

    Li, M.; Zhang, Y.; He, S.; Li, Z.; Zhao, H.; Wang, J.; Cheng, N.; and Zhou, T. 2024 b . Superfiltering: Weak-to-Strong Data Filtering for Fast Instruction-Tuning. arXiv:2402.00530

    2024 arXiv

  16. [24]

    Li, M.; Zhang, Y.; Li, Z.; Chen, J.; Chen, L.; Cheng, N.; Wang, J.; Zhou, T.; and Xiao, J. 2024 c . From Quantity to Quality: Boosting LLM Performance with Self-Guided Data Selection for Instruction Tuning. arXiv:2308.12032

    2024 arXiv

  17. [25]

    Li, X.; Yu, P.; Zhou, C.; Schick, T.; Levy, O.; Zettlemoyer, L.; Weston, J.; and Lewis, M. 2024 d . Self-Alignment with Instruction Backtranslation. arXiv:2308.06259

    2024 arXiv

  18. [26]

    Li, X.; Zhang, T.; Dubois, Y.; Taori, R.; Gulrajani, I.; Guestrin, C.; Liang, P.; and Hashimoto, T. B. 2023. AlpacaEval: An Automatic Evaluator of Instruction-following Models. https://github.com/tatsu-lab/alpaca_eval

    2023

  19. [27]

    Lin, S.; Hilton, J.; and Evans, O. 2022. TruthfulQA: Measuring How Models Mimic Human Falsehoods. arXiv:2109.07958

    2022 arXiv

  20. [28]

    Liu, W.; Zeng, W.; He, K.; Jiang, Y.; and He, J. 2024. What Makes Good Data for Alignment? A Comprehensive Study of Automatic Data Selection in Instruction Tuning. arXiv:2312.15685

    2024 arXiv

  21. [29]

    Lu, K.; Yuan, H.; Yuan, Z.; Lin, R.; Lin, J.; Tan, C.; Zhou, C.; and Zhou, J. 2024. \#InsTag: Instruction Tagging for Analyzing Supervised Fine-tuning of Large Language Models. In The Twelfth International Conference on Learning Representations

    2024

  22. [30]

    Meng, Y.; Xia, M.; and Chen, D. 2024. SimPO: Simple Preference Optimization with a Reference-Free Reward. arXiv:2405.14734

    2024 arXiv

  23. [31]

    Mukherjee, S.; Mitra, A.; Jawahar, G.; Agarwal, S.; Palangi, H.; and Awadallah, A. 2023. Orca: Progressive Learning from Complex Explanation Traces of GPT-4. arXiv:2306.02707

    2023 arXiv

  24. [32]

    OpenAI; Achiam, J.; Adler, S.; Agarwal, S.; Ahmad, L.; Akkaya, I.; Aleman, F. L.; Almeida, D.; Altenschmidt, J.; Altman, S.; Anadkat, S.; Avila, R.; Babuschkin, I.; Balaji, S.; Balcom, V.; Baltescu, P.; Bao, H.; Bavarian, M.; Belgum, J.; Bello, I.; Berdine, J.; Bernadett-Shapi...

    2024 arXiv

  25. [33]

    Ouyang, L.; Wu, J.; Jiang, X.; Almeida, D.; Wainwright, C. L.; Mishkin, P.; Zhang, C.; Agarwal, S.; Slama, K.; Ray, A.; Schulman, J.; Hilton, J.; Kelton, F.; Miller, L.; Simens, M.; Askell, A.; Welinder, P.; Christiano, P.; Leike, J.; and Lowe, R. 2022. Training language model...

    2022 arXiv

  26. [34]

    D.; and Finn, C

    Rafailov, R.; Sharma, A.; Mitchell, E.; Ermon, S.; Manning, C. D.; and Finn, C. 2023. Direct Preference Optimization: Your Language Model is Secretly a Reward Model. arXiv:2305.18290

    2023 arXiv

  27. [35]

    Y.; Ruwase, O.; Yang, S.; Zhang, M.; Li, D.; and He, Y

    Ren, J.; Rajbhandari, S.; Aminabadi, R. Y.; Ruwase, O.; Yang, S.; Zhang, M.; Li, D.; and He, Y. 2021. ZeRO-Offload: Democratizing Billion-Scale Model Training. arXiv:2101.06840

    2021 arXiv

  28. [36]

    Sakaguchi, K.; Le Bras, R.; Bhagavatula, C.; and Choi, Y. 2020. WinoGrande: An Adversarial Winograd Schema Challenge at Scale. Proceedings of the AAAI Conference on Artificial Intelligence, 8732–8740

    2020

  29. [37]

    Schick, T.; and Schütze, H. 2021. Exploiting Cloze-Questions for Few-Shot Text Classification and Natural Language Inference. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume

    2021

  30. [38]

    shareAI. 2023. ShareGPT-Chinese-English-90k Bilingual Human-Machine QA Dataset. https://huggingface.co/datasets/shareAI/ShareGPT-Chinese-English-90k

    2023

  31. [39]

    Song, F.; Yu, B.; Li, M.; Yu, H.; Huang, F.; Li, Y.; and Wang, H. 2024. Preference Ranking Optimization for Human Alignment. arXiv:2306.17492

    2024 arXiv

  32. [40]

    Taori, R.; Gulrajani, I.; Zhang, T.; Dubois, Y.; Li, X.; Guestrin, C.; Liang, P.; and Hashimoto, T. B. 2023. Stanford Alpaca: An Instruction-following LLaMA model. https://github.com/tatsu-lab/stanford_alpaca

    2023

  33. [41]

    Teknium. 2023. OpenHermes 2.5: An Open Dataset of Synthetic Data for Generalist LLM Assistants

    2023

  34. [42]

    M.; and Wolf, T

    Tunstall, L.; Beeching, E.; Lambert, N.; Rajani, N.; Huang, S.; Rasul, K.; Rush, A. M.; and Wolf, T. 2023 a . The Alignment Handbook. https://github.com/huggingface/alignment-handbook

    2023

  35. [43]

    M.; and Wolf, T

    Tunstall, L.; Beeching, E.; Lambert, N.; Rajani, N.; Rasul, K.; Belkada, Y.; Huang, S.; von Werra, L.; Fourrier, C.; Habib, N.; Sarrazin, N.; Sanseviero, O.; Rush, A. M.; and Wolf, T. 2023 b . Zephyr: Direct Distillation of LM Alignment. arXiv:2310.16944

    2023 arXiv

  36. [44]

    A.; Khashabi, D.; and Hajishirzi, H

    Wang, Y.; Kordi, Y.; Mishra, S.; Liu, A.; Smith, N. A.; Khashabi, D.; and Hajishirzi, H. 2023. Self-Instruct: Aligning Language Models with Self-Generated Instructions. arXiv:2212.10560

    2023 arXiv

  37. [45]

    Xu, C.; Sun, Q.; Zheng, K.; Geng, X.; Zhao, P.; Feng, J.; Tao, C.; and Jiang, D. 2023. WizardLM: Empowering Large Language Models to Follow Complex Instructions. arXiv:2304.12244

    2023 arXiv

  38. [46]

    Yuan, Z.; Yuan, H.; Tan, C.; Wang, W.; Huang, S.; and Huang, F. 2023. RRHF: Rank Responses to Align Language Models with Human Feedback without tears. arXiv:2304.05302

    2023 arXiv

  39. [47]

    Zellers, R.; Holtzman, A.; Bisk, Y.; Farhadi, A.; and Choi, Y. 2019. HellaSwag: Can a Machine Really Finish Your Sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics

    2019

  40. [48]

    P.; Zhang, H.; Gonzalez, J

    Zheng, L.; Chiang, W.-L.; Sheng, Y.; Zhuang, S.; Wu, Z.; Zhuang, Y.; Lin, Z.; Li, Z.; Li, D.; Xing, E. P.; Zhang, H.; Gonzalez, J. E.; and Stoica, I. 2023. Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. arXiv:2306.05685

    2023 arXiv

  41. [49]

    , " * write output.state after.block = add.period write newline

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint howpublished institution isbn journal key month note number organization pages publisher school series title type volume year label extra.label sort.label short.list INTEGERS output.state before.all...

  42. [50]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.