{"id":"da31dcde-2f18-45a9-aa88-93ff85336ed9","arxiv_id":"2412.12865","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"PoFT is a new supervised fine-tuning loss that makes the target model outrank aligned reference LLMs on the same data, improving robustness and slightly raising benchmark scores.","lead":"PoFT is a new fine-tuning loss for LLMs that trains the target model to assign higher likelihood than a set of aligned reference LLMs on the same instruction-response data. It is designed to make supervised fine-tuning more robust to low-quality data and shows small but consistent benchmark gains over standard cross-entropy training.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"PoFT's gradient is CE down-weighted by tau=ref/(ref+target); the paper never validates that reference likelihood measures quality, so the robustness gains could be a confidence regularizer rather than a quality filter.","rationale":"The reader identified the reference-likelihood-as-quality assumption as the weakest point; this stress-test agrees and sharpens it. The mathematical form of the PoFT objective shows that the gradient is CE scaled by a factor that depends on both the reference likelihood and the target's own likelihood, making the loss equivalent to CE plus a saturating penalty on overconfidence relative to the reference. This opens a concrete alternative explanation for the reported gains: a regularizer that limits the target model from becoming too confident on any sample, rather than a mechanism that detects low-quality data. The paper's evidence for the quality interpretation is suggestive but not decisive. The synthetic-noise experiments create noise by random matching and character corruption, which guarantees low reference likelihood, so the filter trivially identifies those samples; this does not demonstrate that real low-quality instruction-response pairs are assigned low likelihood. The filtering experiment in Figure 4 shows that high-likelihood data train better, but that is a correlation with likelihood, not with independently measured quality. The lack of multiple-seed runs is a secondary concern, but the consistent direction across six dataset/backbone combinations and the public code base provide some empirical support. Conditional acceptance is appropriate: the method is promising and reproducible, but the central conceptual claim needs direct validation. No change to the reader's verdict is required.","tokens_in":16044,"tokens_out":7260,"duration_ms":72056,"concrete_test":"Obtain per-sample quality scores for a real SFT dataset (e.g., Alpaca, Dolly, or a subset of OpenHermes) using human ratings or a strong LLM judge on a 1-5 scale. Compute the geometric-mean length-normalized reference likelihood R used in Eq. 8 for each sample with the paper's three reference LLMs, and measure the Spearman rank correlation between R and the quality scores. If the correlation is weak or negative, the quality-proxy assumption fails and the robustness claim must be reframed as regularization rather than quality filtering. As a second check, train a static-weight CE baseline with per-sample weight proportional to R (without the target-model denominator); if it matches PoFT within noise, the dynamic BT term is not the source of the gains.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central robustness claim is that PoFT's dynamic per-sample weighting, driven by aligned-LLM likelihoods, down-weights low-quality data and thereby stabilizes SFT on quality-limited datasets. This rests on the assumption that length-normalized reference likelihood is a valid proxy for data quality. The paper supports this only indirectly: preference-score distribution plots, synthetic noise constructed to have low likelihood, and filtering experiments showing that higher-likelihood data train better. None of these establish a correlation with independent quality labels on real SFT data.\n\nMore specifically, from Eq. 6-8 the PoFT loss is L = -log(sigma((1/M)sum(r_theta - r_j))) = log(1 + R / p_theta^{1/T0}), with R = (prod_j p_j^{1/T_j})^{1/M} a constant w.r.t. theta. The gradient is tau * grad(L_CE), where tau = R/(R + p_theta^{1/T0}) in (0,1). This is not simply a fixed quality weight: it depends on the target model's own likelihood and decreases as p_theta grows, so the loss saturates when the target becomes confident relative to the reference. The improvements could therefore stem from a reference-relative confidence regularizer that prevents overfitting, independent of whether R encodes quality. The claimed mechanism—'incorporating assessment information on data quality'—is unvalidated and potentially mischaracterized.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes PoFT, a supervised fine-tuning objective that applies the Bradley-Terry model to a preference between the target model and a set of fixed aligned LLMs, using length-normalized log-likelihoods as reward functions. The derived gradient (Eqs. 6–8 and Appendix A) is cross-entropy scaled by a per-sample coefficient tau, which the authors interpret as a data-quality weight from the aligned LLMs. The empirical sections report improvements over CE SFT on UltraChat200k, OpenHermes, and ShareGPT with Mistral-7B and Llama-3-8B, plus ablations on quality-limited data, synthetic noise, data filtering, and integration with DPO. The paper claims that PoFT is more robust to low-quality data than standard SFT and that the reference likelihoods provide data-quality assessments.","tokens_in":16322,"tokens_out":8320,"duration_ms":69963,"significance":"If the empirical claims hold with proper statistical support, PoFT is a simple and potentially useful contribution: it adds no trainable parameters, is easy to integrate into existing SFT pipelines, and the code is provided. The gradient derivation is essentially correct (apart from an appendix typo), and the synthetic-noise and filtering experiments are a reasonable first step toward the robustness claim. However, the core quality-proxy mechanism and the statistical strength of the main improvements need additional support, so the paper is not yet ready for acceptance.","major_comments":[{"comment":"The central claim of 'stable and consistent improvements' is not yet supported statistically. The reported standard deviations are across training epochs, not across random seeds, so they do not measure run-to-run variability, and no significance tests or confidence intervals are provided. Several differences are small or within the reported across-epoch variability (e.g., Llama-3-8B on UltraChat: 65.90 vs 65.71 last-epoch; average 65.88 vs 65.65; ShareGPT average 65.45 vs 65.34), and in the ShareGPT row the PoFT run has higher across-epoch std than the SFT run (0.19 vs 0.11). Please report multiple seeds with mean and variance, or otherwise provide a statistical basis for the comparison.","section":"Section 4.2, Table 1"},{"comment":"The paper's mechanism claim that PoFT 'incorporates assessment information on data quality' is load-bearing but not validated. The weight tau in Eq. (8) depends on the target model's own likelihood p_theta as well as the reference likelihoods, and it decreases as p_theta grows; the observed robustness could therefore be a reference-relative confidence regularizer rather than a quality filter. The paper does not compare against a control that isolates the target-confidence term, and the reference-likelihood proxy is never checked against independent quality labels on real SFT data. Please provide such a check (e.g., correlation with human or model-based quality annotations) or revise the claimed mechanism.","section":"Section 3.2, Eq. (8)"},{"comment":"The displayed chain for the PoFT gradient contains an incorrect intermediate expression: the denominator should be p_theta(y|x)^{1/T0} + R, where R is the geometric mean of the reference likelihood terms, not p_theta(y|x)^{1/T0} + tau (with tau already the full coefficient defined in Eq. (8)). The final gradient in Eq. (7) is correct, so this is a derivational typo, but it should be fixed because it is part of the formal derivation.","section":"Appendix A, Eq. (11)"},{"comment":"One of the reference models, Zephyr-7B-sft-full, was itself trained on UltraChat200k, the same dataset used in the UltraChat experiments. The reference likelihoods are therefore not an independent quality signal for that dataset. The paper should report the UltraChat results with the Zephyr reference excluded, or explicitly discuss the possible circularity, in order to support the claim that the reference models provide external quality assessments.","section":"Section 4.1, Table 1"}],"minor_comments":[{"comment":"The caption labels are inconsistent: it lists '(c) Performances ... with Alpaca. (d) Performances ... with Dolly. (d) Preference score distributions ...' with a duplicated (d), and the main text refers to Figure 3b/3c and 3d/3e in a way that does not match the subfigure numbering.","section":"Figure 3"},{"comment":"The training settings are not fully specified; 'learning rate is varied from 5e-6 to 5e-5' and 'three to four epochs' are not enough to reproduce Tables 1–5. Please give the exact hyperparameters used for each row.","section":"Appendix B.1"},{"comment":"MT-Bench and AlpacaEval results are reported only for Mistral-7B; to support the claim that improvements hold across base models, these evaluations should also be reported for Llama-3-8B or the claim should be limited to the Open LLM Leaderboard.","section":"Table 2"},{"comment":"The abstract and conclusion use 'prove' for empirical integration results; since these are experimental demonstrations, a weaker verb such as 'show' would be more accurate.","section":"Abstract and Conclusion"},{"comment":"The notation 'LLM j ∈ D_LLM, |D| = M' is confusing; the set and its size are later mixed with the dataset D_SFT. Please use distinct symbols.","section":"Section 3.2, Eq. (6)"},{"comment":"The sentence about Zephyr-7B-sft-full, 'by adjusting hyper-parameters, it could achieve better performance (see the fourth row of Table 1)', is vague because the fourth row is the Mistral-7B SFT baseline; please clarify what is being compared.","section":"Section 4.1, Table 1"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The core idea is worth your time: PoFT applies a Bradley-Terry preference between the target model and a fixed set of aligned LLMs on the same (x,y) pairs, and the gradient reduces to cross-entropy with a per-sample weight tau. That derivation checks out, and the objective is genuinely new, not just a re-labeling of DPO or SimPO. The empirical pattern is consistent: PoFT beats CE SFT on all three datasets and both backbones, with the largest gaps on OpenHermes, and it composes cleanly with DPO and with IFD/Instag/Deita filtering. The authors also did the right sanity checks: ablating the reference set, comparing against data distillation, and showing the weight distribution. That is a solid piece of work.\n\nThe soft spots are real but not fatal. The biggest one is the claim that aligned-LLM likelihood measures data quality. The stress-test note is right: tau in Eq. 8 is not a fixed quality weight; it depends on the target's own likelihood and saturates as the target becomes confident. So PoFT may be acting as a reference-relative confidence regularizer, which prevents overfitting, rather than as a quality filter. The paper never directly validates the quality-proxy assumption with independent quality labels on real SFT data. The synthetic noise experiments show PoFT is robust, but they don't prove the mechanism. I'd soften the 'incorporating assessment information on data quality' language and add an ablation that manipulates reference likelihoods independently of any plausible quality signal.\n\nSecond, the reported gains are often small (0.2 to 0.5 points on UltraChat and ShareGPT), and the standard deviations are across epochs, not seeds. That matters. The OpenHermes gains are more convincing, but the paper would be stronger with at least three seeds for the main comparisons. The UltraChat circularity with Zephyr-7B-sft-full is minor since Zephyr is just one of three references and the method still helps on OpenHermes and ShareGPT, but it should be disclosed more prominently.\n\nOverall, this is a serious contribution to the SFT-objective literature. The math is sound, the experiments are reproducible in principle (public data, linked code), and the failure to validate the quality assumption is a fixable gap, not a fundamental flaw. I'd send it to review and ask for the mechanism validation and seed variance.\n\nWho it's for: anyone working on instruction tuning, data curation, or preference optimization. I'd bring it to a reading group and would cite it if I were writing about soft data weighting or SFT objectives.","headline":"PoFT is a genuinely new SFT objective with sound math and consistent empirical gains, but the quality-filtering story is under-supported and the gains are modest.","tokens_in":16871,"tokens_out":1237,"would_cite":true,"duration_ms":13237,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper introduces PoFT, an SFT objective that makes aligned LLMs' length-normalized likelihoods act as per-sample quality weights, and reports stable gains over cross-entropy SFT across datasets, base models, and noise levels.","keywords":["preference-oriented supervised fine-tuning","instruction tuning","data quality","Bradley-Terry model","large language models","noise robustness","soft data filtering","DPO combination"],"falsifier":"Train two copies of the same model on the same noisy dataset, one with PoFT using the real reference scores and one with the same reference scores randomly permuted across samples; if the permuted version matches PoFT's robustness, the quality-proxy mechanism is not what drives the gain. A cheaper check is to measure the correlation between the reference likelihoods and human quality judgments on a sample of the training set, and look for a positive correlation.","tokens_in":15834,"feed_emoji":"🎯","tokens_out":9107,"duration_ms":71657,"temperature":0.7,"pith_summary":"Standard supervised fine-tuning (SFT) trains a large language model with a cross-entropy loss that treats every instruction-response pair equally, so low-quality pairs drag the model down. This paper proposes PoFT, a fine-tuning objective that instead asks the model to beat a set of already-aligned reference LLMs on each training pair, using length-normalized likelihood as the score. The Bradley-Terry term makes the gradient of each sample carry an automatic weight that is low when the reference models give that sample a low score, turning the loss into a soft data-quality filter. The authors report that PoFT consistently outperforms cross-entropy SFT across multiple datasets and base models, degrades more gracefully on noisy and quality-limited data, and combines with data filtering and with DPO. The point of the paper is that preference modeling between models, rather than between responses, is a practical way to make SFT robust to imperfect data.","feed_headline":"Aligned LLMs grade training data in this new fine-tuning loss","feed_subtitle":"PoFT beats cross-entropy SFT on uneven datasets and pairs with DPO for extra gains.","key_machinery":"The central object is a Bradley-Terry (BT) preference term applied between models rather than between candidate responses. The reward is the length-normalized log-likelihood $r_\\theta(x,y) = \\frac{1}{T_0(y)}\\log p_\\theta(y\\mid x)$ for the target model and $r_j(x,y)=\\frac{1}{T_j(y)}\\log p_j(y\\mid x)$ for each reference LLM, and the loss is $-\\log\\sigma\\!\\left(\\frac{1}{M}\\sum_{j=1}^M (r_\\theta(x,y)-r_j(x,y))\\right)$. Its gradient, $\\nabla_\\theta \\mathcal{L}_{\\mathrm{PoFT}} = -\\frac{1}{T_0(y)}\\frac{1}{p_\\theta(y\\mid x)}\\,\\tau\\,\\nabla_\\theta p_\\theta(y\\mid x)$, carries the coefficient $\\tau = \\frac{G}{G + p_\\theta(y\\mid x)^{1/T_0(y)}}$, where $G$ is the geometric mean of the references' normalized likelihoods. That $\\tau$ is the dynamic weight that down-weights low-quality samples, and the paper's derivation of it is the argument that PoFT is CE with soft quality weighting rather than a fundamentally different training paradigm.","core_discovery":"The paper's central claim is that PoFT's objective—maximizing the log-probability that the target model's normalized likelihood beats the average normalized likelihood of the reference LLMs on the same $\\langle x, y\\rangle$—yields the same gradient as cross-entropy SFT multiplied by a per-sample coefficient $\\tau$, and that this coefficient is precisely what makes training stable on noisy data. When the references assign a sample a high likelihood, $\\tau$ is large and the sample contributes strongly; when they assign a low likelihood, $\\tau$ shrinks and the sample's influence fades. Because the method needs only $\\langle x, y\\rangle$ pairs and no chosen/rejected triples, the paper presents PoFT as an SFT method rather than a preference-alignment method, and it shows empirically that the gains persist across UltraChat, OpenHermes, and ShareGPT on Mistral-7B and Llama-3-8B, with the largest gains on OpenHermes where the reference score distribution is widest.","pith_inferences":["If the reference-likelihood quality proxy holds, PoFT offers a low-cost way to inject quality information into SFT without human annotation; a natural extension is to test whether a single, smaller reference model gives the same benefit as the ensemble of three.","The same model-vs-model preference objective should transfer to other instruction-tuning settings, such as vision-language models, as long as a reference model can assign a likelihood to the target output.","A telling control experiment would be to shuffle or randomize the reference scores: if PoFT keeps its advantage, the gain comes from the BT term's implicit regularization rather than from quality assessment; the paper does not report this control.","The bi-PoFT variant, which explicitly flips the preference for labeled noise data, suggests that providing even coarse noise labels could tune the filter; this is an extension the authors flag for future work."],"forward_implications":["PoFT improves final-checkpoint and epoch-averaged scores over cross-entropy SFT on the Open LLM Leaderboard, MT-Bench, and AlpacaEval 2.0 across datasets and base models.","On quality-limited datasets (Alpaca, Dolly) and on datasets with injected noise, PoFT models avoid the late-epoch performance collapse that SFT shows, so the objective is more stable when data quality cannot be guaranteed.","PoFT is orthogonal to explicit data filtering: combining it with IFD, Instag, or Deita yields higher scores than either approach alone.","PoFT can serve as the SFT stage before DPO; the two-step PoFT-then-DPO pipeline beats SFT-then-DPO on the reported benchmarks.","The per-sample weighting makes PoFT act as a soft filter, so its benefit is largest when the reference models' preference scores on the training set are widely spread rather than concentrated."],"supporting_citations":[{"why":"Supplies the pairwise ranking model used to define the preference between target model and aligned LLMs.","marker":"Bradley and Terry 1952"},{"why":"Establishes the DPO framework that PoFT combines with in two-step training.","marker":"Rafailov et al. 2023"},{"why":"Provides the length-normalization argument used to build the reward functions from log-likelihoods.","marker":"Meng, Xia, and Chen 2024"},{"why":"Supplies the Zephyr aligned model and the UltraChat-based SFT baseline PoFT is compared against.","marker":"Tunstall et al. 2023b"},{"why":"Introduces the UltraChat200k dataset used as a primary training set.","marker":"Ding et al. 2023"},{"why":"Introduces the OpenHermes dataset where PoFT shows its largest gains.","marker":"Teknium 2023"},{"why":"Provides the IFD filtering method that PoFT is combined with and compared against.","marker":"Li et al. 2024c"},{"why":"Provides the Instag filtering method used in the integration experiments.","marker":"Lu et al. 2024"},{"why":"Provides the Deita filtering method used in the integration experiments.","marker":"Liu et al. 2024"}],"fun_headline_variants":["PoFT: LLM fine-tuning that lets reference models grade data","New loss: target model must beat aligned LLMs on same data","Grading your SFT data with aligned LLMs: PoFT","PoFT outperforms SFT by leveraging reference model scores","PoFT: reference-aware SFT that improves on noisy data"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that an aligned LLM's length-normalized likelihood on a training sample measures that sample's quality; if low-quality samples receive high likelihoods, the weighting would amplify noise instead of suppressing it.","fun_headline_variants_meta":{"raw":{"variants":["PoFT: LLM fine-tuning that lets reference models grade data","New loss: target model must beat aligned LLMs on same data","Grading your SFT data with aligned LLMs: PoFT","PoFT outperforms SFT by leveraging reference model scores","PoFT: reference-aware SFT that improves on noisy data"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001089,"raw_usage":{"total_tokens":4588,"prompt_tokens":1020,"completion_tokens":3568,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":636,"completion_tokens_details":{"reasoning_tokens":3480}},"tokens_in":636,"tokens_out":3568,"duration_ms":21578,"temperature":1.0,"reasoning_tokens":3480,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T13:38:53.138267+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train two copies of the same model on the same noisy dataset, one with PoFT using the real reference scores and one with the same reference scores randomly permuted across samples; if the permuted version matches PoFT's robustness, the quality-proxy mechanism is not what drives the gain. A cheaper check is to measure the correlation between the reference likelihoods and human quality judgments on a sample of the training set, and look for a positive correlation.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Introduces the OpenHermes dataset where PoFT shows its largest gains."}],"review_version":1}