{"id":"f4b2a6ff-c230-48a7-80b3-4927774aed3e","arxiv_id":"2504.16188","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"FinNLI is a 21,304-pair financial NLI benchmark with an expert-annotated test set where the best model reaches 78.62% macro F1, exposing a gap in current models' financial reasoning.","lead":"This paper introduces FinNLI, the first natural language inference dataset built from real financial texts like SEC filings, annual reports, and earnings calls. It shows that today's best language models still get only about 78% macro F1 on the expert-labeled test set, leaving clear room for improvement in financial reasoning.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Systematic label noise in train/dev (neutral labels agree only 66.5%) can depress fine-tuned PLM scores and overstate FinNLI difficulty; a noise-corrected retraining check is needed.","rationale":"The paper is a strong, transparent dataset contribution. The expert-annotated test set (3,304 examples, Fleiss kappa 88.31%) is the right gold standard, the generation pipeline is documented in detail, z-filtering is applied, and the evaluation covers zero-shot NLI models, fine-tuned PLMs, general LLMs, and financial LLMs. The reported test-set scores are therefore meaningful as measurements of the models on this particular expert-labeled set. The weakest link is the supervised training signal: Section 3.3 explicitly uses Llama-generated labels as gold for train/dev, and Section 4.1 reports only 76.84% agreement with experts. The class-wise agreement rates (contradiction 90.86%, entailment 75.46%, neutral 66.52%) show the noise is not random; the paper's own qualitative analysis and Table 13 identify a specific direction (generated entailment relabelled neutral). This systematic noise can bias fine-tuned PLMs toward overpredicting entailment, consistent with the confusion matrix in Table 11, and can lower their macro F1 on the expert test set. Thus the paper's fine-tuned PLM results, which are a central pillar of the 'challenging benchmark' and 'room for improvement' claims, are confounded. This does not invalidate the test set or the zero-shot/LLM results; it means the claim needs a noise-corrected ablation or an explicit caveat. I agree with the reader's weakest assumption and keep the verdict CONDITIONAL. The no-release issue also supports conditionality.","tokens_in":24470,"tokens_out":9445,"duration_ms":90580,"concrete_test":"Estimate the label-noise transition matrix T from the 626 R4 expert annotations: T[i,j] = P(expert label = j | generated label = i). Retrain RoBERTa-Large on the official 16,200-example training set using a noise-corrected loss that incorporates T (e.g., forward correction by multiplying predicted probabilities by T), then evaluate macro F1 on the expert test set. If the corrected model's macro F1 exceeds 74.57% by more than one standard deviation, or if the neutral-to-entailment confusion in the corrected model drops markedly, the paper's PLM ceiling and 'room for improvement' conclusion are partly due to systematic training-label noise rather than the intrinsic difficulty of FinNLI.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's supervised fine-tuning results are the main evidence that FinNLI is hard for PLMs, but the training and development labels are Llama-3.1-70B generations, not expert labels (Section 3.3: 'We use the generated labels as the gold standard for training and development'). Section 4.1 reports R4 agreement of only 76.84%, and the per-class breakdown shows the noise is strongly class-dependent: Llama's entailment labels agree with experts 75.46% of the time and neutral labels only 66.52%, while contradiction labels agree 90.86%. The paper itself notes the dominant failure mode: entailment hypotheses that append information absent from the premise are relabelled to neutral. With 5,400 entailment and 5,400 neutral training instances per class, this implies roughly 1,325 entailment labels and 1,807 neutral labels are wrong, and the errors are not symmetric. A model trained on these labels is explicitly taught to treat unsupported inferences as entailment, which is exactly the error pattern reported in Table 11 (large neutral-to-entailment confusion). The reported 74.57% macro F1 for RoBERTa-Large may therefore underestimate what the same architecture achieves when trained on clean labels, and the 'substantial room for improvement' claim is partly an artifact of the training supervision, not of the benchmark task itself. The expert test set remains credible (Fleiss kappa 88.31%), so the benchmark is not invalidated; but the fine-tuned PLM results, a central component of the difficulty argument, are confounded by systematic label noise.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces FinNLI, a multi-genre financial natural language inference dataset containing 21,304 premise–hypothesis pairs drawn from annual reports, SEC filings, and earnings call transcripts. Hypotheses are generated by GPT-4 and Llama 3.1 70B under varied roles and writing styles, passed through z-filtering to mitigate spurious correlations, and a 3,304-instance test set is labeled by finance experts. The authors evaluate zero-shot general-domain NLI models, fine-tuned PLMs, and general-purpose and finance-domain LLMs, reporting that general-domain NLI models degrade sharply on FinNLI, that the best fine-tuned PLM (RoBERTa-Large) achieves 74.57% macro F1, that the best LLM (Llama 3.1 70B) achieves 78.62%, and that instruction-tuned financial LLMs underperform general-purpose models.","tokens_in":24854,"tokens_out":10307,"duration_ms":97137,"significance":"If the benchmark is sound, FinNLI fills a real gap: financial NLI is an underexplored task with clear downstream relevance, and the paper’s pipeline—real-world premises, multi-model generation, artifact filtering, and expert test annotation—is a plausible template for domain-specific NLI dataset construction. The paper earns credit for a transparent evaluation with standard deviations over three runs, a high expert inter-annotator agreement (Fleiss κ = 88.31%), an explicit error analysis, and confidence-based subgroup reporting. The central difficulty and domain-shift claims are, however, currently undercut by two design issues: the train/dev labels are LLM-generated with class-dependent noise, and the test set is adversarially sampled rather than representative. Both issues are fixable and do not invalidate the expert-annotated test set, but the paper’s headline quantitative claims need to be re-examined under those controls.","major_comments":[{"comment":"","section":"§3.3, §4.1, Table 4"},{"comment":"","section":"§3.4, §6.1, abstract"},{"comment":"","section":"§4.2, data availability"}],"minor_comments":[{"comment":"","section":"§6.1 vs Table 4"},{"comment":"","section":"Table 7, Appendix A.1"},{"comment":"","section":"§7.1"},{"comment":"","section":"Figure 3, Appendix G.1"},{"comment":"","section":"Appendix G.1 vs §6.2"}],"recommendation":"major_revision","confidential_remarks":"The expert-annotated test set is the strongest part of the submission, and I do not see a reason to reject. The label-noise concern raised in the stress-test is real and lands on a central claim; the revision must include a noise-sensitivity analysis rather than a paragraph of discussion. I would also treat the absence of a public data release as a blocking issue for a benchmark paper, even though it is straightforward to fix. The paper is within scope for the journal and, with the proposed controls, could become a useful resource."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know about FinNLI. First, it is a real gap-filler: the first NLI benchmark for finance, with premises from SEC filings, annual reports, and earnings calls, and a test set of 3,304 instances annotated by finance experts with almost perfect inter-annotator agreement. Second, the headline numbers for fine-tuned PLMs are weaker evidence than they look. The training and development labels are Llama-3.1-70B generations with only 76.84% agreement with experts, and the noise is class-dependent—neutral labels agree just 66.52%. That is not random noise.\n\nWhat the paper does well: the generation pipeline is careful. Z-filtering, prompt refinement via expert feedback, and the adversarial-like sampling of test instances are all sensible. Evaluation is transparent: three runs, standard deviations, error analysis, confidence subgroups. The domain-shift finding is solid—zero-shot general NLI models drop about 20 points on FinNLI, and that is measured on the expert test set, independent of the noisy train labels.\n\nThe load-bearing concern is exactly the label noise. With 5,400 entailment and 5,400 neutral training examples, agreement rates imply roughly 1,300 entailment and 1,800 neutral labels are wrong. The errors are not symmetric: Llama tends to write entailment hypotheses that append information the premise doesn't support, and experts relabel them to neutral. Fine-tuning on these labels explicitly teaches the model to treat unsupported inferences as entailment—which is exactly the neutral-to-entailment confusion in the error analysis. So the 74.57% macro F1 for RoBERTa-Large likely understates what the same architecture would get with clean labels, and the \"substantial room for improvement\" claim is partly an artifact of the training supervision. This is fixable: retrain on a cleaned sample, or relabel a subset, to see how much the difficulty estimate moves. The dataset is also not released, which makes the benchmark hard to adopt, and a random baseline is missing, which matters for interpreting the difficulty claim. Self-preference bias (Llama generating many test hypotheses and scoring best) is real but not fatal, since the test labels are expert gold.\n\nWho is this for? Anyone building financial NLI systems or studying domain shift. It deserves a serious referee and revision, not desk rejection. My recommendation: accept with major revision, and require either dataset release or a concrete plan, plus a noise-corrected retraining check. If those land, the benchmark will be a standard resource.","headline":"A genuinely useful first financial NLI benchmark with a credible expert test set, but the fine-tuned PLM results are confounded by systematic label noise in the training data.","tokens_in":25333,"tokens_out":2866,"would_cite":true,"duration_ms":26618,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"FinNLI, a new benchmark for financial natural-language inference drawn from SEC filings, annual reports, and earnings calls, shows that the best current large language model reaches only 78.62% macro F1.","keywords":["financial natural language inference","benchmark dataset","domain shift","large language models","spurious correlations","expert annotation","financial reasoning","dataset generation"],"falsifier":"Retrain the best fine-tuned model on FinNLI's training set after finance experts re-label every instance, and compare macro F1 on the same expert-annotated test set. If expert relabeling does not move scores materially, the LLM-label noise is benign and the plateau is genuine; if scores climb past the best LLM's 78.62%, part of the claimed difficulty is an artifact of training on labels experts reject.","tokens_in":24313,"feed_emoji":"📊","tokens_out":9667,"duration_ms":80854,"temperature":0.7,"pith_summary":"FinNLI is a benchmark of 21,304 premise–hypothesis pairs for natural-language inference in finance, paired with a 3,304-instance test set that finance experts annotated with almost perfect agreement. The paper's central claim is that financial NLI is a difficult, largely unsolved task: the best fine-tuned PLM reaches 74.57% macro F1 and the best LLM reaches 78.62%, while general-domain NLI models drop roughly 20 accuracy points when they move from MNLI to FinNLI. The authors attribute the drop to domain shift—financial jargon, numerical and temporal reasoning, and specialized knowledge—and argue the dataset's design, including z-filtering against spurious correlations and premise sampling from three real financial genres, makes the difficulty genuine rather than an artifact. They also report that finance-specific instruction-tuned LLMs perform worst, which suggests narrow instruction tuning does not transfer to a novel reasoning task. A reader should care because FinNLI provides a concrete yardstick for improving how models reason about financial text, with implications for risk assessment, reporting, and fact-checking.","feed_headline":"Best LLM scores 78.62% on new financial NLI benchmark","feed_subtitle":"General NLP models lose ~20 points on finance text, and even the top model leaves wide room for improvement.","key_machinery":"The load-bearing object is FinNLI itself, produced by a five-stage pipeline: multi-genre premise sampling; LLM hypothesis generation with role and writing-style conditioning; z-filtering to suppress spurious correlations; iterative prompt refinement driven by a general-domain NLI model and expert feedback; and three-expert annotation with majority voting. Z-filtering is the mechanism that defends the benchmark's difficulty claim: it rejects samples whose n-grams, lexical overlap, length ratios, or financial-term density correlate too strongly with labels, so models cannot solve the task by exploiting surface shortcuts. The expert-annotated test set anchors validity, giving every reported score a human-judged target.","core_discovery":"FinNLI is constructed by sampling premise sentences from SEC filings, annual reports, and earnings call transcripts, then prompting GPT-4 and Llama 3.1 70B to generate one entailment, one neutral, and one contradiction hypothesis per premise under randomized roles and writing styles. A z-filtering step removes instances whose task-independent features are strongly correlated with labels, and the final test set is annotated by three finance experts per instance, with majority labels as gold. The paper claims that the resulting benchmark is valid and hard: expert labels show a Fleiss κ of 88.31%, yet the best fine-tuned PLM (RoBERTa-Large) scores 74.57% macro F1 and the best LLM (Llama 3.1 70B) scores 78.62%. General-domain NLI models drop from roughly 90% accuracy on MNLI to about 70% on FinNLI, and finance-instruction-tuned FinMA models fall to 23.20% and 29.46%, below their base Llama-2 model. The authors' conclusion is that financial NLI remains an open problem: models over-infer implicit information, struggle with financial and mathematical reasoning, and most often mistake neutral hypotheses for entailment.","pith_inferences":["Beyond the paper: because training and development labels come from Llama 3.1 70B with only 76.84% agreement with expert annotators, part of the measured difficulty could reflect training-label noise; re-annotating the training set by experts would show whether the 74.57% fine-tuned PLM ceiling rises.","Beyond the paper: the two-generator test set invites a sharper generalization check—evaluating models on expert-written hypotheses never produced by an LLM—which would separate genuine financial reasoning from generation-pattern memorization.","Beyond the paper: the authors' observation that Llama 3.1 70B scores similarly on both generator subsets while fine-tuned RoBERTa prefers GPT-4-written instances suggests generator bias is partly a fine-tuning effect, a claim that could be tested by controlling for premise genre and hypothesis length."],"forward_implications":["General-domain NLI systems cannot be assumed to transfer to finance; the roughly 20-point accuracy drop from MNLI to FinNLI means domain-specific evaluation is a prerequisite for deploying NLI in financial applications.","Fine-tuned PLMs remain competitive with much larger models: RoBERTa-Large at 74.57% outperforms most evaluated LLMs, so task-specific fine-tuning is still a strong route for financial NLI.","Finance-specific instruction tuning can hurt rather than help: FinMA 7B and 30B score below their base Llama-2 model, implying that instruction-tuning data built for financial sentiment-style tasks does not generalize to novel reasoning tasks.","The plateau near 78.62% leaves clear headroom, and progress on FinNLI should carry over to downstream financial question answering, fact-checking, and risk assessment, since the authors frame NLI as underpinning those tasks."],"supporting_citations":[{"why":"Provides MNLI, the general-domain NLI benchmark whose roughly 90% accuracy drops by about 20 points on FinNLI, establishing the domain-shift claim.","marker":"Williams et al., 2018"},{"why":"Supplies SNLI, the other general-domain NLI training source, used for comparison in the dataset statistics and model evaluations.","marker":"Bowman et al., 2015"},{"why":"Provides the z-filtering algorithm that removes spurious correlations during dataset generation.","marker":"Wu et al., 2022"},{"why":"Introduces Llama 3.1 70B, the best LLM baseline at 78.62% and the generator of the training and development sets.","marker":"Dubey et al., 2024"},{"why":"Introduces GPT-4, the second hypothesis generator whose test instances are harder for zero-shot NLI models.","marker":"OpenAI, 2023"},{"why":"Provides FinMA 7B and 30B, the finance-instruction-tuned LLMs whose poor generalization is a central negative result.","marker":"Xie et al., 2023"},{"why":"Provides BART-Large-MNLI, the zero-shot NLI baseline that also serves as a fine-tuned comparison model.","marker":"Lewis et al., 2020a"},{"why":"Provides RoBERTa-Base and RoBERTa-Large, the PLMs whose fine-tuned results anchor the 74.57% best-PLM score.","marker":"Liu et al., 2019"},{"why":"Provides the GNLI synthetic NLI comparison and the 84.97% agreement figure used to contextualize FinNLI's annotation quality.","marker":"Hosseini et al., 2024"},{"why":"Provides FiLM, the financially pre-trained PLM baseline whose marginal gain is used to assess domain-specific pretraining.","marker":"Choe et al., 2023"}],"fun_headline_variants":["Expert-checked FinNLI benchmark: best LLM hits 78.62%, finance-tuned models fail","Domain shift in financial NLI: models drop ~20 points, best score 78.62%","New FinNLI dataset exposes LLM weakness in financial reasoning","Finance-tuned LLMs score lowest on FinNLI, below their base models"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The training and development sets treat Llama 3.1 70B's automatically generated labels as gold, yet finance experts agree with those labels only 76.84% of the time, so the benchmark's measured difficulty partly rests on the assumption that this label noise is random and harmless.","fun_headline_variants_meta":{"raw":{"variants":["Expert-checked FinNLI benchmark: best LLM hits 78.62%, finance-tuned models fail","Domain shift in financial NLI: models drop ~20 points, best score 78.62%","New FinNLI dataset exposes LLM weakness in financial reasoning","Finance-tuned LLMs score lowest on FinNLI, below their base models"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000786,"raw_usage":{"total_tokens":3469,"prompt_tokens":949,"completion_tokens":2520,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":565,"completion_tokens_details":{"reasoning_tokens":2427}},"tokens_in":565,"tokens_out":2520,"duration_ms":15990,"temperature":1.0,"reasoning_tokens":2427,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T11:10:08.088568+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Retrain the best fine-tuned model on FinNLI's training set after finance experts re-label every instance, and compare macro F1 on the same expert-annotated test set. If expert relabeling does not move scores materially, the LLM-label noise is benign and the plateau is genuine; if scores climb past the best LLM's 78.62%, part of the claimed difficulty is an artifact of training on labels experts reject.","supporting_citations":[],"review_version":1}