{"id":"b07a4ab1-7bb6-4006-93c0-7924ad27a37e","arxiv_id":"2505.21011","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Predicate frequency is systematically biased in NLI entailment data, fine-tuned LLMs increasingly rely on this frequency cue, and the cue correlates with WordNet hypernym frequency.","lead":"This paper measures how often verbs occur in NLI sentences and finds that, in positive entailment examples, the hypothesis verb tends to be more common than the premise verb. It then shows that large language models, especially after NLI fine-tuning, use this frequency difference as a shortcut and fail more often when the pattern is reversed.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The cons/adv split likely confounds frequency direction with semantic difficulty; the AUC gap and fine-tuning amplification need a matched control before the frequency-reliance claim is supported.","rationale":"The reader's weakest assumption identifies the same load-bearing vulnerability as my stress-test: the frequency-consistent/adversarial split is not shown to be matched on semantic difficulty. I agree with the reader's conditional assessment. The central empirical pattern in Table 4, lower AUC on Levy/Holt_adv, only licenses a frequency explanation if the two subsets differ in frequency direction and in no other way that affects inferability. Since hypernyms tend to be more frequent, Levy/Holt_adv likely contains entailments that move from a more general premise predicate to a more specific hypothesis predicate, which is a less prototypical and arguably harder inference direction. The paper also does not report whether the adv examples have lower annotator agreement or are otherwise less natural. The fine-tuning amplification result inherits the same concern because the increased gap could simply reflect improved performance on the easier, typical subset. A human-judgment control would directly test whether the adv subset is intrinsically harder. I therefore retain the reader's CONDITIONAL verdict: the paper is promising but the central claim should not be accepted until a matched-difficulty control is supplied. The code link and reproducible prompt templates are a positive feature, but they do not substitute for the missing control.","tokens_in":9654,"tokens_out":6617,"duration_ms":83475,"concrete_test":"Collect human entailment judgments on the same Levy/Holt samples used in Table 4 and compute human AUC separately for Levy/Holt_cons and Levy/Holt_adv, then compare the human cons-minus-adv gap with the model gaps. If humans show a comparable gap (e.g., greater than about 5 AUC points), the two subsets are not matched on semantic difficulty and the frequency-reliance interpretation is not supported; if humans are roughly flat across the two subsets while LLMs show the gap, the frequency-reliance claim survives this confound.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing step in §4.3 is the construction of Levy/Holt_cons and Levy/Holt_adv directly from the signed bias Bias(hypo,prem)=Freq(hypo)-Freq(prem), with no evidence that the two partitions are matched on anything except this sign. Table 4 then shows lower AUC on Levy/Holt_adv for every model. If the adv subset also contains pairs that are semantically less natural, harder to annotate, or more often reverse the WordNet hypernym direction, the AUC gap is predicted even by a model that has no frequency heuristic at all. The fine-tuning amplification claim has the same problem: EG-tuned models improve cons AUC by roughly 7-20 points while adv AUC stays approximately flat, which is also what would happen if fine-tuning simply makes models better on the easier, more typical half of the data without any specifically frequency-based mechanism. No balance statistics are reported for predicate length, word-frequency magnitude, hypernym direction, annotator agreement, or any independent measure of inferential difficulty, so the paper's central attribution to frequency learning is not yet isolated.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper investigates whether LLMs learn frequency-based shortcuts during NLI fine-tuning. Using WordFreq, the authors compute a bias score Bias(hypo, prem) = Freq(hypo) - Freq(prem) on verbal predicates and report that positive NLI examples tend to have higher-frequency hypotheses than premises. They evaluate base and LoRA fine-tuned LLMs on Levy/Holt, partition the test set into frequency-consistent and frequency-adversarial subsets, and report lower AUC on the adversarial subset, with larger drops after fine-tuning on Entailment Graphs, RTE, or MNLI. Finally, they show that WordNet hypernyms are more frequent than hyponyms and that hypernyms occur more often in hypotheses of entailment pairs, proposing that frequency bias tracks a generalization gradient from specific to general predicates.","tokens_in":9824,"tokens_out":4957,"duration_ms":49067,"significance":"If the claims hold, the paper makes a useful contribution by connecting dataset-level frequency statistics to LLM inference behavior and to the known fragility of fine-tuned models on adversarial examples. The study has concrete strengths: it evaluates several LLM families and scales, it uses an external corpus (WordFreq) rather than fitting the bias to model predictions, it includes an independent WordNet-based check of the frequency-entailment correlation, and the analysis code is released. The central limitation is that the consistent/adversarial split is not matched on independent measures of inferential difficulty, so the observed AUC gap has an alternative explanation that is not ruled out by the reported experiments.","major_comments":[{"comment":"The load-bearing comparison between Levy/Holt_cons and Levy/Holt_adv is confounded: the split is defined by the sign of Bias(hypo, prem) = Freq(hypo) - Freq(prem), and no evidence is provided that the two subsets are matched on semantic difficulty, naturalness, hypernym direction, or any model-independent measure of inferential complexity. As a result, the lower AUC on the adversarial subset could be produced by a model with no frequency heuristic at all if that subset is simply harder. The fine-tuning amplification claim has the same problem: the cons AUC gains of roughly 7–20 points with flat adv AUC are also what would be expected if fine-tuning improves performance on the easier, more typical half of the data. Please add matched controls (e.g., balance the subsets on human accuracy, WordNet hypernym direction, predicate frequency magnitude, sentence length, and annotator agreement, or compare against a semantic-only baseline) before attributing the gap to frequency learning.","section":"§2.2, §4.3, Table 4"},{"comment":"The verbal-predicate extraction procedure is not specified, and the paper is internally inconsistent about whether sentence-level datasets are included. Table 1 reports frequency bias for RTE and MNLI, but Appendix C states that MNLI and RTE 'lack clearly defined predicates' and therefore excludes them from the hyponym/hypernym analysis. Please specify exactly how verbal predicates were identified and extracted in sentence-level premises and hypotheses (including multi-predicate sentences and negation), and either justify the inclusion of RTE/MNLI in Table 1 or move them to a clearly labeled exploratory analysis.","section":"§2.1, Table 1, Appendix C"},{"comment":"The paper uses language such as 'significantly increased reliance' and 'consistent findings' without reporting any significance tests, confidence intervals, or variability across seeds. All AUC values in Tables 4 and 7 are point estimates from single runs. Please report per-instance or per-run variability (e.g., bootstrap confidence intervals for AUC, paired tests across the consistent/adversarial subsets) so that the reader can assess whether the reported gaps are reliable.","section":"Abstract, §4.3, Tables 2–4"}],"minor_comments":[{"comment":"Table 6 is hard to interpret: the column header 'Hypernmys Hyponyms' is a typo, and the table does not show whether the counts refer to hypernyms in hypotheses versus hyponyms in premises, or to total counts in each class. Please make the table self-contained and align it with the text's claim about where hypernyms appear.","section":"Table 6 and §4.4"},{"comment":"The verbs 'prove' and 'demonstrate' are used too strongly for correlational behavioral evidence, e.g., in §4.3 and the conclusion. I recommend replacing them with 'provide evidence for' or 'are consistent with'.","section":"Throughout"},{"comment":"There are several typos and formatting inconsistencies, including 'hypotheis' and 'datsets' in Table 1, 'we reports' in §4.2, and inconsistent notation for 'Levy/Holt_cons' / 'Levy/Holt_adv'. These should be cleaned up.","section":"Tables 1–3, §4.2"},{"comment":"The fine-tuning description omits standard details such as batch size, learning-rate schedule, number of training examples, and any seed averaging; including these would improve reproducibility.","section":"§3.2, Appendix E"}],"recommendation":"major_revision","confidential_remarks":"I concur with the stress-test concern: the consistent/adversarial partition in §4.3 needs a matched control before the paper's central attribution to frequency learning is supported. The paper cites the authors' own EG dataset and prior inference studies heavily; the dependence on Cheng et al. (2025) for the training data and prompt setup should be made explicit in the main text. This is a fixable weakness, so I recommend major revision rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: the frequency bias is real, the WordNet correlation is a nice external check, and the behavioral split is suggestive. But the main claim—that fine-tuning gains come from learning this bias—needs matched controls before it can carry the weight the authors put on it.\n\nThe paper does something useful: it quantifies a predicate-frequency asymmetry in NLI datasets (hypotheses more frequent than premises on positive instances), shows that standard and fine-tuned LLMs have lower AUC on the frequency-adversarial half of Levy/Holt, and connects the bias to hyponym/hypernym frequency in WordNet. That last point is genuinely original as far as I know, and it gives the frequency story a plausible mechanism: generalization from specific to general aligns with a high-frequency direction. The experiments cover several model families and NLI training sets, and the code is linked. That is real evidence.\n\nThe soft spot is the split. Levy/Holt_cons and Levy/Holt_adv are defined by the sign of Bias(hypo,prem) alone. The paper doesn't report whether the two halves are matched on anything else—semantic naturalness, annotator agreement, predicate length, or WordNet hypernym direction. If the adversarial half contains less natural or genuinely harder inference pairs, the AUC gap appears without any frequency heuristic. The fine-tuning amplification claim has the same problem: fine-tuning could simply improve the easier, more typical half more, and the gap widens without any frequency-specific mechanism. The paper needs a matched control, such as a second split on a semantic difficulty proxy that is orthogonal to frequency, or at least balance statistics on the two halves.\n\nTwo smaller issues. The paper says \"prove\" repeatedly, but there are no confidence intervals or significance tests anywhere. The numbers are consistent across models, which helps, but the overclaim is unnecessary. And the verb-extraction step for sentence-level datasets (RTE, MNLI) is unspecified; Table 1 reports frequencies for those, and the reader can't tell how predicates were pulled out of full sentences. The appendix acknowledges the issue by analyzing hyponym counts only on EG and Levy/Holt, which is fine, but the main table still needs a protocol.\n\nWho is this for? Anyone working on NLI debiasing, shortcut learning, or benchmark design will want to know this frequency bias exists and is learnable. It deserves a serious referee, but the revision needs matched controls, statistics, and a precise extraction protocol before the causal story holds.","headline":"A real and measurable frequency bias in NLI data, but the paper's attribution of fine-tuning gains to frequency learning is not yet isolated from semantic difficulty.","tokens_in":10340,"tokens_out":2485,"would_cite":true,"duration_ms":25754,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"LLMs exploit a word-frequency bias in NLI inference, and fine-tuning on NLI data amplifies that reliance.","keywords":["frequency bias","natural language inference","LLM fine-tuning","shortcut learning","hyponym-hypernym entailment","Levy/Holt","entailment graphs","robustness"],"falsifier":"A concrete check would be to train on a version of the NLI data in which premise and hypothesis predicate frequencies are balanced (by resampling or rewriting), then measure AUC on the adversarial half: if the gap between the consistent and adversarial subsets disappears while overall AUC stays high, the frequency-bias explanation is unsupported.","tokens_in":9450,"feed_emoji":"📉","tokens_out":8044,"duration_ms":70450,"temperature":0.7,"pith_summary":"This paper claims that large language models learn a frequency pattern from natural-language-inference data rather than a general inference rule. In entailment-labeled examples, the predicate in the hypothesis tends to be more frequent in general language than the predicate in the premise; the reverse holds for non-entailment examples. The authors show that models answer correctly when a test case follows this bias and fail when it goes against it, and that fine-tuning on NLI corpora deepens the dependence. They also show that hypernyms are more frequent than their hyponyms in WordNet, so following the frequency direction often amounts to performing hyponym-to-hypernym entailment, which explains why the shortcut can help on benchmarks even though it is not robust.","feed_headline":"NLI fine-tuning amplifies LLMs' word-frequency shortcut","feed_subtitle":"Hypothesis predicates are more frequent than premise predicates in entailment data; reverse the pattern and accuracy collapses.","key_machinery":"The central object is the frequency-bias measure $Bias(hypo, prem)=Freq(hypo)-Freq(prem)$, where $Freq$ is the average WordFreq frequency of the verbal predicates in a statement. The paper splits the Levy/Holt test set into frequency-consistent ($Levy/Holt_{\\mathrm{cons}}$) and frequency-adversarial ($Levy/Holt_{\\mathrm{adv}}$) subsets and compares AUC between them, before and after fine-tuning. WordNet hyponym-hypernym pairs supply the semantic anchor: because hypernyms are more frequent, the bias direction aligns with the specific-to-general entailment relation that dominates positive NLI examples.","core_discovery":"The central claim is that gains from fine-tuning LLMs on NLI corpora are largely the acquisition of a frequency heuristic. In Entail instances the hypothesis predicate is more frequent than the premise predicate, while in No-Entail instances the pattern flips. Both standard and fine-tuned LLMs make correct predictions when the test case is consistent with this bias, and their accuracy drops on frequency-adversarial cases: for example, LLaMA-3-8B fine-tuned on entailment graphs falls from 85.2 AUC on the frequency-consistent split to 62.5 on the adversarial split. Using WordNet hyponym-hypernym pairs, the paper shows hypernyms are more frequent than hyponyms, making the frequency bias a proxy for a generalization gradient that supports entailment from specific to general concepts; this is offered as the reason NLI training can improve performance while leaving models fragile when the frequency direction is reversed.","pith_inferences":["An untested extension would be a debiasing procedure that balances predicate frequencies between premises and hypotheses during training; the paper's account predicts the robustness gap on adversarial cases would close without a large loss in overall accuracy.","A causal intervention, such as rewriting adversarial examples to reverse the frequency direction while preserving meaning, would directly test whether the model tracks frequency rather than semantics.","The paper restricts its frequency measure to verbal predicates; the same mechanism may operate on noun and adjective frequencies, which could make the bias even more pervasive in sentence-level NLI.","The WordNet correlation suggests frequency bias is not a pure artifact but a semi-valid heuristic that becomes harmful only when the frequency gradient opposes the semantic one."],"forward_implications":["If the claim holds, NLI benchmark gains from fine-tuning are partly illusory: they reflect learning the dataset's frequency direction, not a general inference rule, so performance on reversed-frequency entailments should stay low.","The AUC gap between the frequency-consistent and frequency-adversarial halves becomes a diagnostic for shortcut reliance and should be reported alongside overall accuracy.","Fine-tuning on logically valid counterfactual entailment graphs still widens the gap, so even semantically sound training data can carry frequency regularities that models pick up.","Because hypernyms are more frequent than hyponyms, the frequency heuristic partially aligns with a genuine entailment relation, which explains why a shortcut can help in-distribution without being robust."],"supporting_citations":[{"why":"Supplies the WordFreq n-gram frequency data used to compute every predicate-frequency measurement.","marker":"Speer, 2022"},{"why":"Supplies WordNet, from which the paper extracts hyponym-hypernym pairs to connect frequency to entailment.","marker":"Miller, 1994"},{"why":"Provides the Levy/Holt test set that the paper splits into frequency-consistent and frequency-adversarial subsets.","marker":"Levy and Dagan, 2016"},{"why":"Companion source for the Levy/Holt dataset and its directional entailment pairs.","marker":"Holt, 2019"},{"why":"Introduces the counterfactual entailment-graph training data and fine-tuning setup whose gains the paper reinterprets as frequency learning.","marker":"Cheng et al., 2025"},{"why":"Prior evidence that LLMs memorize and use dataset artifacts as shortcuts, which this paper extends to frequency patterns.","marker":"Mckenna et al., 2023a"},{"why":"Supplies the prompt templates used for fine-tuning and few-shot inference in the experiments.","marker":"Schmitt and Schütze, 2021"}],"fun_headline_variants":["Reverse the frequency bias, and NLI-tuned LLMs' accuracy collapses","Fine-tuned LLMs learn a frequency shortcut, not NLI logic","LLM NLI gains after fine-tuning trace to a word-frequency bias","Frequency bias explains NLI tuning gains; reverse it, and LLMs fail","LLMs become frequency detectors after NLI fine-tuning"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that splitting Levy/Holt into frequency-consistent and frequency-adversarial subsets isolates the frequency direction while holding semantic difficulty fixed; if the adversarial examples are also semantically harder, the accuracy gap could arise without the model relying on frequency.","fun_headline_variants_meta":{"raw":{"variants":["Reverse the frequency bias, and NLI-tuned LLMs' accuracy collapses","Fine-tuned LLMs learn a frequency shortcut, not NLI logic","LLM NLI gains after fine-tuning trace to a word-frequency bias","Frequency bias explains NLI tuning gains; reverse it, and LLMs fail","LLMs become frequency detectors after NLI fine-tuning"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000929,"raw_usage":{"total_tokens":3957,"prompt_tokens":904,"completion_tokens":3053,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":520,"completion_tokens_details":{"reasoning_tokens":2959}},"tokens_in":520,"tokens_out":3053,"duration_ms":21656,"temperature":1.0,"reasoning_tokens":2959,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T13:42:41.542231+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A concrete check would be to train on a version of the NLI data in which premise and hypothesis predicate frequencies are balanced (by resampling or rewriting), then measure AUC on the adversarial half: if the gap between the consistent and adversarial subsets disappears while overall AUC stays high, the frequency-bias explanation is unsupported.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the Levy/Holt test set that the paper splits into frequency-consistent and frequency-adversarial subsets."},{"cited_title":"Probabilistic Models of Relational Implication","cited_arxiv_id":"1907.12048","evidence_quote":"Companion source for the Levy/Holt dataset and its directional entailment pairs."},{"cited_title":"Neutralizing Bias in LLM Reasoning using Entailment Graphs","cited_arxiv_id":"2503.11614","evidence_quote":"Introduces the counterfactual entailment-graph training data and fine-tuning setup whose gains the paper reinterprets as frequency learning."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the prompt templates used for fine-tuning and few-shot inference in the experiments."}],"review_version":1}