Pith. sign in

REVIEW 4 major objections 6 minor 14 references

Assessing Robustness to Spurious Correlations in Post-Training Language Models

T0 review · 4 major / 6 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read No single post-training strategy wins all spurious-data tests.

desk verdict A useful controlled comparison of SFT/DPO/KTO under synthetic spurious correlations; the math results are solid, but the docQA evidence rests on an unvalidated GPT-4o-mini judge and the headline task-dependence claim is not yet secure. read the letter →

arxiv 2505.05704 v1 pith:35XKAWGM submitted 2025-05-09 cs.CL cs.AI

Julia Shuieh , Prasann Singhal , Apaar Shanker , John Heyer , George Pu , Samuel Denton This is my paper · ORCID
classification cs.CLcs.AI
keywords spuriouscorrelationspost-trainingalignmentsupervisedfine-tuningdirectpreferenceoptimizationKTOshortcutlearningdocument-groundedQAmathematicalreasoning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks whether three common post-training methods — supervised fine-tuning (SFT), Direct Preference Optimization (DPO), and KTO — keep working when the training data contains spurious correlations: features that look like signals of correctness but do not actually cause it. The authors build synthetic training sets in three domains (arithmetic word problems, constrained instruction-following, and document-grounded question answering), inject two kinds of artifacts (a token that is always present in correct answers, or a narrowing of the answer distribution), and compare the methods at 10% and 90% contamination. They find that DPO and KTO generally beat SFT on math reasoning, while SFT scores higher on document QA, and that increasing spuriousness does not always hurt performance. The paper concludes that no single post-training strategy is universally best; the choice should depend on the task and the kind of shortcut present.

What carries the argument

The engine of the study is a controlled synthetic benchmark that injects spurious shortcuts into three task families. Feature Ambiguity makes a non-required token or pattern (a date, a keyword, a sentence-final word) perfectly correlated with the chosen response, so a model can appear correct by reproducing the pattern. Distributional Narrowness restricts the set of correct answers (for instance, forcing all math answers into the range 1–5), so a model can succeed by guessing from a narrow interval. Each task is trained at 10% and 90% contamination with SFT, DPO, or KTO, and accuracy is measured on wider test distributions to see which objective resists the shortcut.

What would settle it

Re-run the document-QA evaluations with a judge that cannot see the spurious feature — for example, extract the fact-bearing span and use rule-based matching against the gold answer instead of the GPT-4o-mini judgment — and compare the SFT versus DPO/KTO accuracy rankings. If SFT's lead vanishes or reverses, the paper's main docQA conclusion is an evaluation artifact. A second check would remove the spurious token from the context on a held-out test set so the shortcut cannot be exploited; if SFT still wins, that would support the paper's claim.

Watch

Extended reading notes

Core claim

The central claim is that robustness to spurious correlations in post-trained language models is conditional, not a property of the algorithm alone. Across 162 checkpoints spanning three Llama 3.x sizes, three post-training methods, and two spuriousness ratios, the authors observe that preference-based objectives (DPO and KTO) consistently outperform SFT on mathematical reasoning tasks — for word-inclusion bias and restricted-answer-range settings their accuracies are roughly 0.75–0.81 versus 0.54–0.64 — while SFT keeps a clear lead on document-grounded QA such as word-inclusion and date-inclusion settings, with accuracies around 0.40–0.48 versus 0.30–0.34 for DPO/KTO. The paper also reports that the effect of raising contamination from 10% to 90% is task-specific: some settings degrade, some stay flat, and a few even improve. From this the authors conclude that the best post-training method depends on the target task and the nature of the spurious correlation.

Load-bearing premise

The paper's central measurement assumption is that the evaluation metrics, especially the GPT-4o-mini judge used for document QA, measure genuine task correctness and are not themselves swayed by the injected spurious features, so that the reported SFT advantage is not an artifact of the judge rewarding dates or keywords.

Editorial extensions

If this is right

  • A default preference for DPO/KTO over SFT, or the reverse, is not justified when spurious correlations are a concern; the right choice depends on the task and the artifact.
  • In math reasoning with restricted answer ranges or token-inclusion biases, DPO and KTO maintain higher accuracy than SFT even at 90% contamination.
  • In context-heavy document QA, SFT preserves factual correctness better than DPO/KTO under date- and keyword-inclusion biases.
  • Instruction-following tasks with a spurious sentence-final token are extremely brittle: SFT scores essentially zero and DPO/KTO barely improve.
  • A higher spuriousness ratio does not always lower accuracy; some configurations are stable or improve, so contamination level alone is not a reliable predictor of failure.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because SFT and the preference methods win on complementary task types, a hybrid pipeline that first supervised-fine-tunes on the task then applies DPO/KTO could outperform either alone; the paper does not test this.
  • The docQA finding hinges on the GPT-4o-mini judge; a rule-based factual check that ignores the injected dates and keywords would show whether SFT's lead is genuine or an artifact of the judge's own sensitivity to the spurious feature.
  • A practical diagnostic follows from the Feature Ambiguity / Distributional Narrowness taxonomy: practitioners who audit their training data for perfect token–correctness correlations and restricted answer ranges can pick the post-training method accordingly.
  • The paper calls the model-size effect inconclusive; a larger systematic sweep across more seeds and scales could turn the observed 70B advantage into a firm claim.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The manuscript proposes a controlled evaluation of three post-training algorithms (SFT, DPO, KTO) under injected spurious correlations. It builds training sets from QuAC (docQA), GSM8K (math), and CoLLIE-style instruction-following, manipulates spuriousness via Feature Ambiguity and Distributional Narrowness at 10% and 90% contamination ratios, and evaluates 162 checkpoints from Llama-3.x 3B/8B/70B models. The headline finding is that no post-training method dominates: preference-based methods are relatively robust in math reasoning, while SFT is stronger in document-grounded QA; the paper concludes that method choice should depend on task and spurious-feature type.

Significance. If the findings are validated, this is a useful empirical contribution to the alignment literature: it provides a controlled, multi-task benchmark for shortcut robustness and cautions against defaulting to SFT or preference optimization. The paper has concrete strengths: the aggregate accuracies in Table A.3 are internally consistent with the per-model values; the contamination protocol is described in enough detail to be reproduced on public datasets; and the task/artifact distinction (FA vs DN) is a sensible organizing axis. The main limitation is that the central task-dependence claim rests on docQA results measured by an unvalidated LLM judge, and the numbers contain a direct counterexample (docQA omission at 90%). With judge validation and a few control conditions, the study could support the claimed conclusions.

major comments (4)
  1. [Section 4.3, Table 1] The docQA evaluations are performed by GPT-4o mini, yet the injected spurious features are exactly the kinds of surface signals (specific keywords, dates, answer position) that an LLM judge could reward when judging correctness. No validation of the judge against human labels or against a test set without the injected feature is reported. Because the abstract's claim that SFT 'maintains stronger performance in complex, context-intensive tasks' is driven by docQA, the SFT-vs-DPO/KTO ranking in those settings may reflect judge bias rather than robustness. Please report judge agreement and rerun the docQA evaluation with the spurious feature removed or with human-annotated correctness.
  2. [Section 5.1, observation 2; Table A.3 (Docqa omission 90%)] The claim that SFT outperforms preference methods in docQA is contradicted by the omission setting: at 90% spuriousness, SFT falls to 0.108 while DPO reaches 0.284 and KTO 0.200. Since the abstract's task-dependence conclusion uses the docQA-vs-math inversion as its main evidence, this internal counterexample needs to be explicitly reconciled, for example by excluding omission or by explaining why SFT collapses specifically under distributional narrowness.
  3. [Section 4.3 / Section 5] No clean-data baseline is included. The 10%-vs-90% contrast can measure sensitivity to contamination ratio, but 'robustness' is only meaningful relative to training on uncontaminated data; without such a baseline, one cannot tell whether a method's performance at 10% is already degraded by the spurious feature. Please train and evaluate a no-artifact control for each task.
  4. [Table A.3 and Section 5] Each reported accuracy is a single run with no error bars, multiple seeds, or significance tests. Several method differences are small (for example, instruction tiny constraints at 90%: KTO 0.399 vs DPO 0.371), so the qualitative claims about method ordering are not statistically grounded. At minimum, provide bootstrap confidence intervals or multiple training seeds for the aggregate results.
minor comments (6)
  1. [Section 5.1, item 5] There is a typo: 'DPO/KTO and remain more robust' should read 'DPO/KTO remain more robust.'
  2. [Table 2] The row 'Math restricted range answer 10% SFT 0.538' is duplicated; please remove the duplicate row.
  3. [Section 1] Calling the benchmark 'synthetic' is imprecise because the tasks are built on real datasets (QuAC, GSM8K, CoLLIE) with synthetic artifact injection; consider naming it 'controlled contamination over existing tasks.'
  4. [Section 4.3] The number of evaluation examples per condition is not reported, and the text does not explicitly confirm that the evaluation sets are disjoint from the training sets; this information is needed to interpret the accuracies.
  5. [Appendix A.2] SFT is trained for 3 epochs whereas DPO and KTO are trained for 1 epoch, which confounds training budget with algorithm; the paper should discuss or equalize compute when comparing methods.
  6. [Figure captions] The captions do not explain how arrows or markers map to conditions; please make the legends explicit and refer to Table A.3 for exact numerical values.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: the paper's conclusions are empirical summaries of controlled evaluations, not derivations from fitted parameters or self-citations.

full rationale

The paper is an empirical benchmark study. There is no derivation chain: Section 3 constructs synthetic training sets by injecting spurious features, Section 4.3 evaluates checkpoints on held-out wider distributions, and Section 5 reports measured accuracies in Table A.3. The central claim ('no single post-training strategy universally outperforms in all scenarios') is an empirical summary of those measurements, not a quantity computed from an assumed relationship. No fitted parameter is renamed as a prediction, no uniqueness theorem is imported from prior author work, and no ansatz is smuggled in via citation. The only same-author citation (Singhal et al., 2024, in Section 2) is background on length over-optimization and is not load-bearing: the paper's conclusions do not invoke it as an axiom or use it to prohibit alternative methods. The GPT-4o-mini judge used for docQA (Section 4.3) raises a legitimate measurement-validity concern, but that is a correctness risk rather than a circularity: the paper does not define docQA correctness as whatever the judge outputs, and no fitted value is reintroduced as a prediction. Accordingly, no step reduces to its own input by construction, and the appropriate finding is a non-finding on circularity.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The central claim rests on domain assumptions about what the synthetic spurious features do, how correctness is measured, and how comparable the training configurations are. No free parameters are fitted to data in the usual sense, and no new theoretical entities are introduced.

assumptions (4)
  • domain assumption The injected spurious features are not causally related to task correctness, so accuracy changes between 10% and 90% spuriousness are attributable to shortcut learning rather than task difficulty.
    This underpins the entire design; if the 'spurious' feature, such as dates in docQA or range 1-5 in math, changes the task distribution in ways that also affect difficulty, the 10% versus 90% comparison is confounded. Invoked throughout Section 3 and Table 1.
  • domain assumption The evaluation metrics measure the target skill and are not themselves influenced by the injected spurious features.
    Section 4.3 states docQA uses GPT-4o-mini correctness judgments and math and instruction use rule-based checks, but no validation or agreement statistics are reported. If the judge rewards dates or keywords, the SFT-versus-preference gaps in docQA could be artifacts.
  • domain assumption The three training methods are compared under equivalent data and hyperparameter choices such that observed differences reflect algorithmic properties.
    Section 4.2 and Appendix A.2 list different hyperparameters per method, for example SFT 3 epochs versus DPO and KTO 1 epoch, with different learning rates. Without ablations or equated budgets, some differences could be due to training configuration rather than the objective.
  • domain assumption Llama-3.1 and Llama-3.2 instruct models are representative of post-trained language models.
    The study only uses Llama 3.x instruct models, yet the conclusion 'no universal winner' is generalized beyond this family. The authors acknowledge this limitation in the Discussion.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Assessing Robustness to Spurious Correlations in Post-Training Language Models." pith.science (2026). https://pith.science/paper/35XKAWGM

@misc{pith2026250505704,
  author       = {Pith},
  title        = {Pith review of: Assessing Robustness to Spurious Correlations in Post-Training Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/35XKAWGM}},
  note         = {Machine review of arXiv:2505.05704}
}
read the original abstract

Supervised and preference-based fine-tuning techniques have become popular for aligning large language models (LLMs) with user intent and correctness criteria. However, real-world training data often exhibits spurious correlations -- arising from biases, dataset artifacts, or other "shortcut" features -- that can compromise a model's performance or generalization. In this paper, we systematically evaluate three post-training algorithms -- Supervised Fine-Tuning (SFT), Direct Preference Optimization (DPO), and KTO (Kahneman-Tversky Optimization) -- across a diverse set of synthetic tasks and spuriousness conditions. Our tasks span mathematical reasoning, constrained instruction-following, and document-grounded question answering. We vary the degree of spurious correlation (10% vs. 90%) and investigate two forms of artifacts: "Feature Ambiguity" and "Distributional Narrowness." Our results show that the models often but not always degrade under higher spuriousness. The preference-based methods (DPO/KTO) can demonstrate relative robustness in mathematical reasoning tasks. By contrast, SFT maintains stronger performance in complex, context-intensive tasks. These findings highlight that no single post-training strategy universally outperforms in all scenarios; the best choice depends on the type of target task and the nature of spurious correlations.

Figures

Figures reproduced from arXiv: 2505.05704 by the authors.

Figure 1
Figure 1. docQA tasks: arrows connect each model’s accuracy at 10% to its accuracy at 90% spuri [PITH_FULL_IMAGE:figures/full_fig_p005_1.png] view at source ↗
Figure 2
Figure 2. math tasks: each point is a (model, method) at 10% or 90% spuriousness, with arrows [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. instruction tasks: performance for 10% vs. 90% spurious data. Models and methods [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

14 extracted references · 4 canonical work pages

  1. [1]

    Q u AC : Question answering in context

    Eunsol Choi, He He, Mohit Iyyer, Mark Yatskar, Wen-tau Yih, Yejin Choi, Percy Liang, and Luke Zettlemoyer. Q u AC : Question answering in context. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp.\ 2174--2184, Brussels, Belgium, October-November 2018. Association for Computational Linguistics. doi:10.18653/v1/D...

    doi:10.18653/v1/d18-1241 2018

  2. [2]

    Training verifiers to solve math word problems

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021

    arXiv

  3. [3]

    The llama 3 herd of models

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024

    arXiv 2024

  4. [4]

    Length-controlled alpacaeval: A simple way to debias automatic evaluators

    Yann Dubois, Bal'azs Galambosi, Percy Liang, and Tatsunori Hashimoto. Length-controlled alpacaeval: A simple way to debias automatic evaluators. Conference on Language Modeling, 2024. URL https://api.semanticscholar.org/CorpusID:269004605

    work page 2024

  5. [5]

    Kto: Model alignment as prospect theoretic optimization

    Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky, and Douwe Kiela. Kto: Model alignment as prospect theoretic optimization. arXiv preprint arXiv:2402.01306, 2024

    arXiv 2024

  6. [6]

    Smith, and Hanna Hajishirzi

    Nathan Lambert, Valentina Pyatkin, Jacob Daniel Morrison, Lester James Validad Miranda, Bill Yuchen Lin, Khyathi Raghavi Chandu, Nouha Dziri, Sachin Kumar, Tom Zick, Yejin Choi, Noah A. Smith, and Hanna Hajishirzi. Rewardbench: Evaluating reward models for language modeling. ArXiv, abs/2403.13787, 2024. URL https://api.semanticscholar.org/CorpusID:268537409

    arXiv 2024

  7. [7]

    Thomas McCoy, Ellie Pavlick, and Tal Linzen

    R. Thomas McCoy, Ellie Pavlick, and Tal Linzen. Right for the wrong reasons: Diagnosing syntactic heuristics in natural language inference. In Annual Meeting of the Association for Computational Linguistics, 2019. URL https://api.semanticscholar.org/CorpusID:59599752

    work page 2019

  8. [8]

    Disentangling length from quality in direct preference optimization

    Ryan Park, Rafael Rafailov, Stefano Ermon, and Chelsea Finn. Disentangling length from quality in direct preference optimization. In Annual Meeting of the Association for Computational Linguistics, 2024. URL https://api.semanticscholar.org/CorpusID:268733207

    work page 2024

Show all 14 references
  1. [9]

    Direct preference optimization: Your language model is secretly a reward model

    Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36: 0 53728--53741, 2023

    2023

  2. [10]

    A long way to go: Investigating length correlations in rlhf

    Prasann Singhal, Tanya Goyal, Jiacheng Xu, and Greg Durrett. A long way to go: Investigating length correlations in rlhf. Conference on Language Modeling, 2024. URL https://api.semanticscholar.org/CorpusID:263672200

    2024

  3. [11]

    Ziegler, Ryan J

    Nisan Stiennon, Long Ouyang, Jeff Wu, Daniel M. Ziegler, Ryan J. Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul Christiano. Learning to summarize from human feedback. ArXiv, abs/2009.01325, 2020. URL https://api.semanticscholar.org/CorpusID:221665105

    2009 arXiv

  4. [12]

    Finetuned language models are zero-shot learners

    Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc V Le. Finetuned language models are zero-shot learners. arXiv preprint arXiv:2109.01652, 2021

    2021 arXiv

  5. [13]

    Collie: Systematic construction of constrained text generation tasks

    Shunyu Yao, Howard Chen, Austin W Hanjie, Runzhe Yang, and Karthik Narasimhan. Collie: Systematic construction of constrained text generation tasks. arXiv preprint arXiv:2307.08689, 2023

    2023 arXiv

  6. [14]

    Instruction tuning for large language models: A survey

    Shengyu Zhang, Linfeng Dong, Xiaoya Li, Sen Zhang, Xiaofei Sun, Shuhe Wang, Jiwei Li, Runyi Hu, Tianwei Zhang, Fei Wu, et al. Instruction tuning for large language models: A survey. arXiv preprint arXiv:2308.10792, 2023

    2023

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.