{"id":"01c4c4a2-3b54-4fbf-8aec-0548ba54c2c9","arxiv_id":"2505.05704","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Across synthetic spurious-correlation tasks, DPO and KTO are more robust in math reasoning while SFT is stronger in document QA, and higher spuriousness does not uniformly degrade performance.","lead":"Researchers trained Llama models with three alignment methods, SFT, DPO, and KTO, on synthetic datasets where correct answers were correlated with irrelevant clues such as dates, keywords, or narrow answer ranges. The results show no single method wins everywhere: preference-based methods look more robust in math, SFT does better on document QA, and higher spuriousness does not always hurt accuracy.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"DocQA rankings may be artifacts of a GPT-4o-mini judge that rewards injected spurious tokens; since no judge validation or feature-absent test set is reported, the SFT-vs-DPO/KTO pattern and the task-dependence claim are not yet secure.","rationale":"The paper's headline claim is a negative universal, so it is supported by almost any heterogeneous result table; the useful and load-bearing part is the positive pattern that preference-based methods excel in math while SFT excels in context-heavy QA. That pattern is established by comparing rule-based math scores with judge-based docQA scores. The docQA judge is unvalidated, and the injected artifacts are precisely the surface cues an LLM judge could be biased by, making the measurement layer the weakest link. I agree with the reader's weakest assumption and add that the omission result already provides an internal counterexample to the SFT-in-docQA generalization, which reinforces the need for a feature-blind evaluation. The near-zero accuracy in two instruction tasks is a further reason to temper secondary conclusions, but it is not the main threat to the central argument. The proposed human-judgment check with masked features would settle whether the task-dependence pattern is real or an artifact of the evaluation protocol; until then, CONDITIONAL is the appropriate verdict.","tokens_in":11842,"tokens_out":5276,"duration_ms":58645,"concrete_test":"Hold out a clean docQA test subset where the injected feature is not a valid cue (e.g., questions whose gold answers contain no date and no target keyword). Have two human annotators judge 100 responses per method and docQA condition, with target keywords and dates masked in the model responses, and compare those human judgments against the original GPT-4o-mini judgments. If the human-based SFT-vs-preference ranking differs in any docQA condition, the central pattern is an evaluation artifact; if it matches, the judge concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The docQA results that drive the SFT-advantage pattern are measured with a GPT-4o-mini judge (Section 4.3 and Table 1), while the injected spurious features are exactly the surface tokens a biased judge might reward: keywords, dates, and answer position. The paper reports no validation of this judge against human labels or against a test set where the spurious feature is absent, so the higher SFT scores in word-inclusion, date, and late-spurious docQA could reflect the judge rewarding spurious surface form rather than factual correctness. This matters because the inversion between math (rule-based, preference methods win) and docQA (judge-based, SFT wins) is the main evidence for the task-dependence claim. The internal counterexample already present, omission at 90% where DPO 0.284 and KTO 0.200 beat SFT 0.108, further shows the SFT-in-docQA pattern is fragile, so the headline claim cannot be accepted until the judge confound is resolved.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript proposes a controlled evaluation of three post-training algorithms (SFT, DPO, KTO) under injected spurious correlations. It builds training sets from QuAC (docQA), GSM8K (math), and CoLLIE-style instruction-following, manipulates spuriousness via Feature Ambiguity and Distributional Narrowness at 10% and 90% contamination ratios, and evaluates 162 checkpoints from Llama-3.x 3B/8B/70B models. The headline finding is that no post-training method dominates: preference-based methods are relatively robust in math reasoning, while SFT is stronger in document-grounded QA; the paper concludes that method choice should depend on task and spurious-feature type.","tokens_in":11977,"tokens_out":4773,"duration_ms":46826,"significance":"If the findings are validated, this is a useful empirical contribution to the alignment literature: it provides a controlled, multi-task benchmark for shortcut robustness and cautions against defaulting to SFT or preference optimization. The paper has concrete strengths: the aggregate accuracies in Table A.3 are internally consistent with the per-model values; the contamination protocol is described in enough detail to be reproduced on public datasets; and the task/artifact distinction (FA vs DN) is a sensible organizing axis. The main limitation is that the central task-dependence claim rests on docQA results measured by an unvalidated LLM judge, and the numbers contain a direct counterexample (docQA omission at 90%). With judge validation and a few control conditions, the study could support the claimed conclusions.","major_comments":[{"comment":"The docQA evaluations are performed by GPT-4o mini, yet the injected spurious features are exactly the kinds of surface signals (specific keywords, dates, answer position) that an LLM judge could reward when judging correctness. No validation of the judge against human labels or against a test set without the injected feature is reported. Because the abstract's claim that SFT 'maintains stronger performance in complex, context-intensive tasks' is driven by docQA, the SFT-vs-DPO/KTO ranking in those settings may reflect judge bias rather than robustness. Please report judge agreement and rerun the docQA evaluation with the spurious feature removed or with human-annotated correctness.","section":"Section 4.3, Table 1"},{"comment":"The claim that SFT outperforms preference methods in docQA is contradicted by the omission setting: at 90% spuriousness, SFT falls to 0.108 while DPO reaches 0.284 and KTO 0.200. Since the abstract's task-dependence conclusion uses the docQA-vs-math inversion as its main evidence, this internal counterexample needs to be explicitly reconciled, for example by excluding omission or by explaining why SFT collapses specifically under distributional narrowness.","section":"Section 5.1, observation 2; Table A.3 (Docqa omission 90%)"},{"comment":"No clean-data baseline is included. The 10%-vs-90% contrast can measure sensitivity to contamination ratio, but 'robustness' is only meaningful relative to training on uncontaminated data; without such a baseline, one cannot tell whether a method's performance at 10% is already degraded by the spurious feature. Please train and evaluate a no-artifact control for each task.","section":"Section 4.3 / Section 5"},{"comment":"Each reported accuracy is a single run with no error bars, multiple seeds, or significance tests. Several method differences are small (for example, instruction tiny constraints at 90%: KTO 0.399 vs DPO 0.371), so the qualitative claims about method ordering are not statistically grounded. At minimum, provide bootstrap confidence intervals or multiple training seeds for the aggregate results.","section":"Table A.3 and Section 5"}],"minor_comments":[{"comment":"There is a typo: 'DPO/KTO and remain more robust' should read 'DPO/KTO remain more robust.'","section":"Section 5.1, item 5"},{"comment":"The row 'Math restricted range answer 10% SFT 0.538' is duplicated; please remove the duplicate row.","section":"Table 2"},{"comment":"Calling the benchmark 'synthetic' is imprecise because the tasks are built on real datasets (QuAC, GSM8K, CoLLIE) with synthetic artifact injection; consider naming it 'controlled contamination over existing tasks.'","section":"Section 1"},{"comment":"The number of evaluation examples per condition is not reported, and the text does not explicitly confirm that the evaluation sets are disjoint from the training sets; this information is needed to interpret the accuracies.","section":"Section 4.3"},{"comment":"SFT is trained for 3 epochs whereas DPO and KTO are trained for 1 epoch, which confounds training budget with algorithm; the paper should discuss or equalize compute when comparing methods.","section":"Appendix A.2"},{"comment":"The captions do not explain how arrows or markers map to conditions; please make the legends explicit and refer to Table A.3 for exact numerical values.","section":"Figure captions"}],"recommendation":"major_revision","confidential_remarks":"The manuscript seems within scope for a venue that accepts empirical robustness/alignment studies, but the lack of judge validation and of uncertainty quantification makes it unsuitable for acceptance at this stage. I recommend major revision rather than rejection because the identified issues are addressable with additional experiments. The authors should also consider releasing code and the judge prompts to support reproducibility."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear [name],\n\nThis is a solid empirical paper that does something genuinely useful: it builds a synthetic benchmark injecting two types of spurious correlation (feature ambiguity and distributional narrowness) at 10% and 90% contamination into three task families, then compares SFT, DPO, and KTO on Llama-3.1/3.2 at three sizes. The math results are rule-based and internally consistent: DPO/KTO consistently beat SFT on arithmetic word problems at both spuriousness levels, with large gaps (e.g., restricted-range 10%: SFT 0.54 vs DPO 0.79/KTO 0.80). That part of the paper holds up.\n\nThe soft spot is the document-QA evidence, which drives the task-dependence conclusion. DocQA accuracy is judged by GPT-4o-mini, and the injected spurious features are exactly the surface tokens that judge could reward: dates, keywords, answer position. The paper reports no validation of the judge against human labels or on a feature-absent test set, so the SFT advantage in docQA (word-inclusion 10%: SFT 0.47 vs DPO/KTO ~0.31) may be partly an evaluation artifact. The paper itself contains a counterexample: in the omission condition at 90%, DPO 0.284 and KTO 0.200 beat SFT 0.108. So the abstract's blanket claim that 'SFT maintains stronger performance in complex, context-intensive tasks' is too broad.\n\nOther issues: point estimates only — no error bars, no multiple seeds, no significance tests — and no clean-data baseline, so we cannot tell if the 10% condition is already degraded. The appendix data/code link is redacted, so reproducibility is limited. The instruction-following results are so low that they mostly show floor effects.\n\nMy take: the contribution is real and worth refereeing, but the measurement layer needs attention. The authors should validate the docQA judge, add a feature-absent evaluation, provide multiple seeds with variance, and soften the claims. The math results alone support the 'no universal winner' conclusion; the math-vs-docQA task-dependence claim will be secure only after the judge confound is resolved.\n\nI would send this to a serious venue with major-revision expectations. It is also a good reading-group case study on how automatic evaluators can confound empirical claims.\n\nBest.","headline":"A useful controlled comparison of SFT/DPO/KTO under synthetic spurious correlations; the math results are solid, but the docQA evidence rests on an unvalidated GPT-4o-mini judge and the headline task-dependence claim is not yet secure.","tokens_in":12588,"tokens_out":3347,"would_cite":true,"duration_ms":30821,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"No single post-training strategy wins all spurious-data tests.","keywords":["spurious correlations","post-training alignment","supervised fine-tuning","direct preference optimization","KTO","shortcut learning","document-grounded QA","mathematical reasoning"],"falsifier":"Re-run the document-QA evaluations with a judge that cannot see the spurious feature — for example, extract the fact-bearing span and use rule-based matching against the gold answer instead of the GPT-4o-mini judgment — and compare the SFT versus DPO/KTO accuracy rankings. If SFT's lead vanishes or reverses, the paper's main docQA conclusion is an evaluation artifact. A second check would remove the spurious token from the context on a held-out test set so the shortcut cannot be exploited; if SFT still wins, that would support the paper's claim.","tokens_in":11597,"feed_emoji":"⚖️","tokens_out":12563,"duration_ms":106463,"temperature":0.7,"pith_summary":"This paper asks whether three common post-training methods — supervised fine-tuning (SFT), Direct Preference Optimization (DPO), and KTO — keep working when the training data contains spurious correlations: features that look like signals of correctness but do not actually cause it. The authors build synthetic training sets in three domains (arithmetic word problems, constrained instruction-following, and document-grounded question answering), inject two kinds of artifacts (a token that is always present in correct answers, or a narrowing of the answer distribution), and compare the methods at 10% and 90% contamination. They find that DPO and KTO generally beat SFT on math reasoning, while SFT scores higher on document QA, and that increasing spuriousness does not always hurt performance. The paper concludes that no single post-training strategy is universally best; the choice should depend on the task and the kind of shortcut present.","feed_headline":"No single post-training strategy wins all spurious-data tests","feed_subtitle":"Controlled synthetic experiments show DPO/KTO lead on math, SFT on document QA.","key_machinery":"The engine of the study is a controlled synthetic benchmark that injects spurious shortcuts into three task families. Feature Ambiguity makes a non-required token or pattern (a date, a keyword, a sentence-final word) perfectly correlated with the chosen response, so a model can appear correct by reproducing the pattern. Distributional Narrowness restricts the set of correct answers (for instance, forcing all math answers into the range 1–5), so a model can succeed by guessing from a narrow interval. Each task is trained at 10% and 90% contamination with SFT, DPO, or KTO, and accuracy is measured on wider test distributions to see which objective resists the shortcut.","core_discovery":"The central claim is that robustness to spurious correlations in post-trained language models is conditional, not a property of the algorithm alone. Across 162 checkpoints spanning three Llama 3.x sizes, three post-training methods, and two spuriousness ratios, the authors observe that preference-based objectives (DPO and KTO) consistently outperform SFT on mathematical reasoning tasks — for word-inclusion bias and restricted-answer-range settings their accuracies are roughly 0.75–0.81 versus 0.54–0.64 — while SFT keeps a clear lead on document-grounded QA such as word-inclusion and date-inclusion settings, with accuracies around 0.40–0.48 versus 0.30–0.34 for DPO/KTO. The paper also reports that the effect of raising contamination from 10% to 90% is task-specific: some settings degrade, some stay flat, and a few even improve. From this the authors conclude that the best post-training method depends on the target task and the nature of the spurious correlation.","pith_inferences":["Because SFT and the preference methods win on complementary task types, a hybrid pipeline that first supervised-fine-tunes on the task then applies DPO/KTO could outperform either alone; the paper does not test this.","The docQA finding hinges on the GPT-4o-mini judge; a rule-based factual check that ignores the injected dates and keywords would show whether SFT's lead is genuine or an artifact of the judge's own sensitivity to the spurious feature.","A practical diagnostic follows from the Feature Ambiguity / Distributional Narrowness taxonomy: practitioners who audit their training data for perfect token–correctness correlations and restricted answer ranges can pick the post-training method accordingly.","The paper calls the model-size effect inconclusive; a larger systematic sweep across more seeds and scales could turn the observed 70B advantage into a firm claim."],"forward_implications":["A default preference for DPO/KTO over SFT, or the reverse, is not justified when spurious correlations are a concern; the right choice depends on the task and the artifact.","In math reasoning with restricted answer ranges or token-inclusion biases, DPO and KTO maintain higher accuracy than SFT even at 90% contamination.","In context-heavy document QA, SFT preserves factual correctness better than DPO/KTO under date- and keyword-inclusion biases.","Instruction-following tasks with a spurious sentence-final token are extremely brittle: SFT scores essentially zero and DPO/KTO barely improve.","A higher spuriousness ratio does not always lower accuracy; some configurations are stable or improve, so contamination level alone is not a reliable predictor of failure."],"supporting_citations":[{"why":"It provides the GSM8K arithmetic word problems used as the basis of the math task and its spurious variants.","marker":"Cobbe et al., 2021"},{"why":"It supplies the QuAC document-grounded QA corpus used for the docQA task.","marker":"Choi et al., 2018"},{"why":"It provides the CoLLIE benchmark that inspires the constrained instruction-following tasks.","marker":"Yao et al., 2023"},{"why":"It defines Direct Preference Optimization, one of the two preference-based methods compared against SFT.","marker":"Rafailov et al., 2023"},{"why":"It defines KTO, the other preference-based method compared.","marker":"Ethayarajh et al., 2024"},{"why":"It provides the Llama 3.1/3.2 model family used for all post-training runs.","marker":"Dubey et al., 2024"},{"why":"It establishes the concept of spurious heuristics in NLP data that motivates the artifact design.","marker":"McCoy et al., 2019"},{"why":"It supplies the LLM-evaluator methodology underlying the docQA judge.","marker":"Dubois et al., 2024"}],"fun_headline_variants":["Post-training robustness to spurious data is task-dependent","DPO/KTO beat SFT on math, lose on doc QA under spuriousness","No one post-training method survives all spurious correlations","Spurious data: best post-training method shifts by task","Conditional robustness: pick post-training by task and bias type"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper's central measurement assumption is that the evaluation metrics, especially the GPT-4o-mini judge used for document QA, measure genuine task correctness and are not themselves swayed by the injected spurious features, so that the reported SFT advantage is not an artifact of the judge rewarding dates or keywords.","fun_headline_variants_meta":{"raw":{"variants":["Post-training robustness to spurious data is task-dependent","DPO/KTO beat SFT on math, lose on doc QA under spuriousness","No one post-training method survives all spurious correlations","Spurious data: best post-training method shifts by task","Conditional robustness: pick post-training by task and bias type"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000182,"raw_usage":{"total_tokens":1330,"prompt_tokens":987,"completion_tokens":343,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":603,"completion_tokens_details":{"reasoning_tokens":256}},"tokens_in":603,"tokens_out":343,"duration_ms":3828,"temperature":1.0,"reasoning_tokens":256,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T22:58:54.384756+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the document-QA evaluations with a judge that cannot see the spurious feature — for example, extract the fact-bearing span and use rule-based matching against the gold answer instead of the GPT-4o-mini judgment — and compare the SFT versus DPO/KTO accuracy rankings. If SFT's lead vanishes or reverses, the paper's main docQA conclusion is an evaluation artifact. A second check would remove the spurious token from the context on a held-out test set so the shortcut cannot be exploited; if SFT still wins, that would support the paper's claim.","supporting_citations":[{"cited_title":"Length-controlled alpacaeval: A simple way to debias automatic evaluators","cited_arxiv_id":null,"evidence_quote":"It supplies the LLM-evaluator methodology underlying the docQA judge."}],"review_version":1}