Pith. sign in

REVIEW 3 major objections 4 minor 56 references

Two spellings of the same physical quantity—95°C or 368.15 K—are routinely misread as conflicting claims by fine-tuned neural fact-checkers, and training on symbolically generated label-preserving rewrites closes this blind spot, lifting pr

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 13:34 UTC pith:4KHHUU5W

load-bearing objection Typed-quantity verification and training-time symbolic augmentation are genuinely worth knowing about, but the headline numbers sit on a cross-validation split that ignores the 741 source evidence sentences; treat the gains as provisional until the split is grouped. the 3 major comments →

arxiv 2607.16212 v1 pith:4KHHUU5W submitted 2026-05-19 cs.AI

Symbolic Augmentation Closes a Canonical-Equivalence Blind Spot in Neural Fact-Checkers

Genpei Zhang This is my paper
classification cs.AI
keywords typed-quantity verificationunit hallucinationcanonical equivalencedata augmentationsymbolic verificationfact-checkingscientific claim verificationfive-class taxonomy
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper argues that neural fact-checkers fail not on paraphrases but on a narrow structural case: two spellings of the same physical quantity, such as 95°C and 368.15 K, are treated as conflicting claims. It proposes generating training data by running a symbolic unit verifier in reverse, so that every rewrite is guaranteed to keep its label. Trained on these rewrite families, the encoder's accuracy on canonical-equivalence probes rises from 36.5% to 98.2% while its in-distribution macro-F1 improves slightly. The authors further compare five places where symbolic information can enter the pipeline and find that only training-time augmentation helps; auxiliary features, logit ensembling, and silver-label teaching do not. The broader claim is that the right way to combine symbolic and learned verification is at the data layer, not the model layer.

Core claim

A fine-tuned encoder that scores macro-F1 = 0.899 on five-class typed-quantity verification collapses to 36.5% accuracy when a gold-correct summary restates its quantity in canonical-equivalent form (95°C as 368.15 K, 0.5 mol/L as 500 mmol/L). The paper shows the failure is structural: 99 of 115 probe errors land in SCALEERROR, meaning the model reads a physically identical surface as an altered magnitude. The proposed fix, Symbolic Augmentation, inverts the modules of a symbolic verifier into deterministic, label-preserving rewrite rules; training on the resulting data raises the probe to 98.2%, slightly improves macro-F1, matches a closed-frontier LLM at no inference cost, and transfers to

What carries the argument

Symbolic Augmentation: the modules of a deterministic symbolic verifier (quantity extraction, unit canonicalization, comparator and trigger detection) are run in reverse to define rewrite families (canonical-equivalence, comparator synonyms, unit-name paraphrase, scientific notation) that, applied to gold CORRECT items, generate new training pairs whose labels are correct by construction. The canonical-equivalence family, derived from the unit registry, is the one that closes the blind spot.

Load-bearing premise

The 1500 benchmark items derive from 741 unique evidence sentences, but the cross-validation is stratified by gold label, not by evidence sentence, so rewrites of the same evidence can appear in both train and test folds and inflate the reported gains.

What would settle it

Split the data by the 741 evidence sentences so no source's rewrites straddle the train/test boundary, retrain, and re-run the probes; if the F1-probe gain of 36.5→98.2 collapses, the blind-spot fix was partly a memorization artifact of the split scheme.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Fine-tuned encoders can reach closed-frontier-level robustness on units at negligible inference cost, if trained on label-preserving symbolic rewrites.
  • The four family-wide probes give a transferable diagnostic: any verifier can be probed for canonicalization blind spots before deployment.
  • The five integration-point comparison suggests that rule-based and learned components should be combined at training time, not inference time.
  • The gain transfers out of distribution: binary macro-F1 on an external scientific fact-checking benchmark rises from 0.791 to 0.828 with zero target-domain training data.
  • Scientific-notation rewrites also improve via cross-family transfer: F1 augmentation alone lifts the notation probe from 68% to 96%.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The reverse-verifier recipe is general: any symbolic verifier whose stages expose deterministic rewrites (unit conversion, arithmetic, temporal normalization) could generate label-preserving training data for the same blind spot in other modalities.
  • If the evidence-sentence grouping were honored in cross-validation, the headline gains might shrink; the paper's tables stratify by label only, so a regrouping could change the headline numbers.
  • The probe suite is a reusable diagnostic: before deployment, run canonical-equivalence probes on any fact-checker to find whether numeric surface variation is misread as contradiction.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper introduces a five-class typed-quantity verification task, a 1500-item benchmark built from PMC and arXiv evidence sentences and labeled by two LLM annotators with adjudication, and a fine-tuned ModernBERT encoder. It reports that the encoder reaches macro-F1 = 0.899 on the benchmark but collapses to 36.5% accuracy on canonical-equivalence probe rewrites (e.g., 95°C versus 368.15 K). The central proposal is Symbolic Augmentation: deterministic, label-preserving rewrite rules derived from the modules of a symbolic verifier (UGV) are used to generate augmented training data for the encoder. The paper claims this lifts F1-probe accuracy to 98.2%, slightly improves macro-F1 to 0.902, transfers to SciFact-Open (binary macro-F1 0.791 to 0.828), and matches a closed-frontier LLM at no inference cost. It further reports negative results for feature-level integration, silver-label teaching, and inference-time ensembling, concluding that training-time augmentation is the only effective integration point.

Significance. If the empirical claims hold, the paper makes a useful contribution: it identifies a concrete and plausible failure mode in neural fact-checkers, proposes a clean mechanism (inverting symbolic verifier rules into label-preserving augmentation), and provides an external transfer check. The deterministic nature of the rewrite rules and the explicit per-family probes are strengths, as is the attempt to compare five integration points. However, the headline numbers rest on a cross-validation protocol that does not separate items by source evidence sentence, and the benchmark/probe use LLMs in ways that create circularity risks. These issues are load-bearing for the central claims and require correction before the quantitative conclusions can be accepted.

major comments (3)
  1. [§5.1, App. D, App. G] The 1500 benchmark items are derived from only 741 unique evidence sentences (App. D), yet the five-fold cross-validation is stratified only by gold label (App. G). Nothing prevents summaries from the same evidence sentence from appearing in both training and test folds. A test summary whose evidence appeared in training can be classified by evidence-level lexical memory rather than by generalization to unseen evidence. This inflates the headline macro-F1 (Table 1, 0.899), the F1 probe gain (Table 2, 36.5% to 98.2%), and the SciFact-Open comparison is the only result not affected. The F1 probe is especially sensitive: probe items are rewrites of test CORRECT items, while augmentation adds rewrites of training CORRECT items; when those items share an evidence sentence, the augmented model has effectively seen near-identical canonical-equivalent surfaces at train time. Please re-run the ev
  2. [App. D] The annotation pipeline is not as described. The rewriter is claude-haiku-4-5, and annotator A is also claude-haiku-4-5; only annotator B is from a different model family. The manuscript claims 'two independent LLM annotators' and 'drawn from different model families,' but annotator A is the same checkpoint family as the generator, so the inter-annotator agreement (Krippendorff's alpha = 0.882) partially reflects a model labeling its own outputs. The human spot-check is only 50 items (94% agreement), which is too small to rule out systematic bias. Please either replace annotator A with a model family not used for rewriting or provide a substantially larger human validation set, and revise the circularity discussion accordingly.
  3. [Abstract vs. App. C] The abstract claims the probe set uses a 'disjoint perturbation generator' to address the self-labeling circularity concern, but Appendix C states that 'Rule definitions and probe generation share code' with the augmentation families. The probes are therefore not disjoint from the augmentation distribution; they are generated by the same transformations the model is trained on. The 98.2% probe result can be read as interpolation to the training augmentation distribution rather than as evidence of robustness to an independent perturbation family. Please either remove the 'disjoint' claim or design a held-out probe that uses surface transformations not present in the augmentation families.
minor comments (4)
  1. [§5.3] The text says the all-four augmentation result is 'statistically indistinguishable' from baseline, but no variance estimate, confidence interval, or significance test is reported. With only one 5-fold pooled number, this claim is unsupported.
  2. [Table 1] The fine-tuned encoder is evaluated with 5-fold pooled out-of-fold predictions, while the closed-frontier LLMs are evaluated once on the full 1500-item set. The comparison in Table 1 is therefore not under identical protocols; this should be stated explicitly when comparing macro-F1 values.
  3. [Table 3] Table 3 reports a single-fold probe evaluation, while Table 2 reports 5-fold pooled results. The relationship between the two is not explained clearly; state why the single fold is representative, or report both with confidence intervals.
  4. [§5.4 / App. H] The logit ensemble uses a single fixed alpha = 0.25. Since the conclusion is that ensembling fails, at least a small alpha sweep or a statement that the result is robust to alpha is needed to rule out a tuning artifact.

Circularity Check

2 steps flagged

The headline robustness gain is measured with a probe generated by the same code as the augmentation, and the five-fold split is not grouped by source evidence, so the 36.5→98.2 probe claim and macro-F1 0.899 are not fully independent of the training inputs.

specific steps
  1. self definitional [Abstract; §4.3; Appendix C]
    ""A small probe set with a disjoint perturbation generator and a 50-item human spot-check (94% agreement) address the self-labeling circularity concern." ... "Rule definitions and probe generation share code (Appendix C), so a probe and its corresponding augmentation test exactly the same transformation.""

    The F1-probe that defines the headline blind-spot measurement is generated by the same code as the training augmentation. Thus the reported closure (36.5%→98.2%) measures the model's exposure to the augmentation transformation itself; the probe is not an independent, disjoint test of canonical-equivalence generalization. The abstract's 'disjoint perturbation generator' is contradicted by Appendix C. The §4.3 acknowledgment ('share code ... test exactly the same transformation') makes the dual role explicit, but it means the probe gain is partly by construction rather than an external check.

  2. other [Appendix D; Appendix G; Table 2]
    ""The 1500 benchmark items derive from 741 unique evidence sentences" ... "stratified five-fold cross-validation, stratified by gold label""

    Because the split is stratified by gold label, not by the 741 source evidence sentences, test items can share their evidence with training items. The F1 augmentation is applied to gold CORRECT training items, producing F1-rewrites of the same evidence that later appears in test CORRECT items and in the F1 probe. The headline macro-F1 (0.899) and probe gain (36.5→98.2) therefore do not establish generalization to unseen evidence unless the paper provides a group-split guarantee, which it does not. The evaluation protocol allows the model to score by evidence-level memory of training rewrites.

full rationale

Most of the paper is not circular: there is no fitted parameter disguised as a prediction, no self-citation chain, and the integration-point comparison (Table 4) plus the SciFact-Open transfer (0.791→0.828) give independent evidence that training-time augmentation has a real, transferable effect. However, the central headline numbers rest on two self-generated measurements: the F1 probe shares code with the augmentation, and the five-fold CV is not grouped by evidence sentence. These make the strongest claim ('closes essentially all of the blind spot') partly by construction, and the abstract's 'disjoint perturbation generator' is contradicted by Appendix C. The paper is self-aware about the shared code but does not address the evidence-level leakage in the Limitations. Score 5 reflects partial circularity/leakage in the headline measurement, with independent external support preventing a higher score.

Axiom & Free-Parameter Ledger

3 free parameters · 4 axioms · 0 invented entities

No new physical entities are introduced; UNITGRAPHVERIFIER is a deterministic software pipeline, and the taxonomy is a labeling scheme, not an entity requiring independent falsifiable evidence.

free parameters (3)
  • canonical-equivalence tolerance = 1%
    Section 3.2 and Appendix B: the SCALEERROR check is suppressed when canonicalized values agree within 1%; this hand-set threshold defines where CORRECT ends and SCALEERROR begins, and it shapes both gold labels and F1 probe variants.
  • logit ensemble weight alpha = 0.25
    Appendix H: the only positive weight tried for blending UGV's one-hot output with the encoder's softmax in the inference-time ensemble; no sweep is reported.
  • augmentation caps = 3 per source (F1), 4 per source (all-four)
    Appendix G: caps on how many rewritten items each gold CORRECT training item may generate; controls augmented set size and diversity, no sensitivity analysis.
axioms (4)
  • domain assumption LLM-generated labels approximate ground truth well enough to train and evaluate on.
    Section 5.1/Appendix D: labels come from claude-haiku/gpt-5.4-mini with claude-opus adjudication; only 50 items are human-checked (94% by the author). If the two LLM annotators share systematic biases, macro-F1 is not measuring real errors.
  • domain assumption The five-class priority ordering resolves multi-error items correctly.
    Section 3.3/Appendix A: UNSUPPORTED > UNIT > SCALE > RELATION > CORRECT is asserted, and it actively relabels 116/1500 items; a different ordering changes the gold set.
  • ad hoc to paper Deterministic rewrite rules from UGV modules are label-preserving.
    Section 4.2: Symbolic Augmentation's guarantee relies on each inverted verifier rule preserving the CORRECT label; if canonicalization or synonym substitution ever changes the physical claim, augmented training data and probes inherit wrong labels.
  • ad hoc to paper Probe variants generated by the same code as augmentation are a valid measure of the blind spot.
    Section 4.3/Appendix C: probes and augmentation share rewrite code; this ensures alignment but tests only the exact trained transformation and not generalization to other canonical-equivalence rewrites.

pith-pipeline@v1.3.0-alltime-deepseek · 19967 in / 16905 out tokens · 185546 ms · 2026-08-02T13:34:29.428752+00:00 · methodology

0 comments
read the original abstract

Large language models hallucinate numbers and units when summarizing scientific text, a failure mode that can silently invert a scientific claim. We recast the detection of such errors as typed verification: we introduce a five-class typed-quantity error taxonomy and a 1500-item benchmark, rewritten from PMC and arXiv sources and labeled by two independent LLM annotators with adjudication (Krippendorff's alpha = 0.882). A ModernBERT encoder fine-tuned on this benchmark reaches macro-F1 = 0.899, far above any off-the-shelf neural fact-checker, yet four probes expose a sharp structural blind spot: on canonical-equivalent rewrites of physically equivalent quantities (e.g., 95{\deg}C and 368.15 K) its accuracy collapses to 36.5%. We propose Symbolic Augmentation, a training-time framework that runs the modules of a symbolic verifier in reverse to generate label-preserving augmented training data. The augmentation lifts canonical-equivalence robustness to 98.2% while slightly improving in-distribution accuracy (macro-F1: 0.899 to 0.902); the augmented encoder matches a closed-frontier LLM at no inference cost and transfers to an external benchmark (SciFact-Open binary macro-F1: 0.791 to 0.828). Two negative results sharpen the claim: symbolic features as auxiliary encoder inputs add nothing, and symbolic silver labels scale negatively under teacher noise. Together these results identify training-time augmentation as the right integration point between symbolic and learned components.

Figures

Figures reproduced from arXiv: 2607.16212 by Genpei Zhang.

Figure 1
Figure 1. Figure 1: Overview of the typed-quantity verification task and the Symbolic Augmentation framework. [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Per-family probe accuracy before and after [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Prediction flow on the F1 probe, all 181 vari￾ants gold CORRECT. The canonical encoder (left) sends 99 of its 115 errors to SCALE ERROR; Symbolic Aug￾mentation (right) recovers 114 of the 115. Integration strategy macro-F1 ∆ Canonical fine-tuning 0.899 — Inference-time integration Logit ensemble 0.895 −0.004 Agreement filter 0.888 −0.011 Feature-level integration UGV output as features 0.879 −0.020 Silver-… view at source ↗
Figure 4
Figure 4. Figure 4: The annotation pipeline. Each rewritten sum [PITH_FULL_IMAGE:figures/full_fig_p014_4.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

56 extracted references · 9 linked inside Pith

  1. [1]

    Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations) , pages=

    Ai2 scholar qa: Organized literature synthesis with attribution , author=. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations) , pages=

  2. [2]

    arXiv preprint arXiv:2411.14199 , year=

    Openscholar: Synthesizing scientific literature with retrieval-augmented lms , author=. arXiv preprint arXiv:2411.14199 , year=

    Pith/arXiv arXiv

  3. [3]

    Nature , volume=

    Autonomous chemical research with large language models , author=. Nature , volume=. 2023 , publisher=

    2023

  4. [4]

    Nature machine intelligence , volume=

    Augmenting large language models with chemistry tools , author=. Nature machine intelligence , volume=. 2024 , publisher=

    2024

  5. [5]

    Reproductive biomedicine online , volume=

    Artificial intelligence in scientific writing: a friend or a foe? , author=. Reproductive biomedicine online , volume=. 2023 , publisher=

    2023

  6. [6]

    arXiv preprint arXiv:2403.07183 , year=

    Monitoring ai-modified content at scale: A case study on the impact of chatgpt on ai conference peer reviews , author=. arXiv preprint arXiv:2403.07183 , year=

    Pith/arXiv arXiv

  7. [7]

    ACM computing surveys , volume=

    Survey of hallucination in natural language generation , author=. ACM computing surveys , volume=. 2023 , publisher=

    2023

  8. [8]

    arXiv 2023 , author=

    Siren’s song in the AI ocean: A survey on hallucination in large language models. arXiv 2023 , author=. arXiv preprint arXiv:2309.01219 , year=

    Pith/arXiv arXiv 2023

  9. [9]

    Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing , pages=

    Minicheck: Efficient fact-checking of llms on grounding documents , author=. Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing , pages=

    2024

  10. [10]

    2024 , howpublished =

    Bespoke-. 2024 , howpublished =

    2024

  11. [11]

    Proceedings of the 2023 conference on empirical methods in natural language processing , pages=

    Selfcheckgpt: Zero-resource black-box hallucination detection for generative large language models , author=. Proceedings of the 2023 conference on empirical methods in natural language processing , pages=

    2023

  12. [12]

    arXiv preprint arXiv:2307.13528 , year=

    FacTool: Factuality Detection in Generative AI--A Tool Augmented Framework for Multi-Task and Multi-Domain Scenarios , author=. arXiv preprint arXiv:2307.13528 , year=

    Pith/arXiv arXiv

  13. [13]

    FactCG: Enhancing fact checkers with graph-based multi-hop data , author=. Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers) , pages=

    2025

  14. [14]

    Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=

    Smarter, better, faster, longer: A modern bidirectional encoder for fast, memory efficient, and long context finetuning and inference , author=. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=

  15. [15]

    tasksource: A Large Collection of NLP tasks with a Structured Dataset Preprocessing Framework

    Sileo, Damien. tasksource: A Large Collection of NLP tasks with a Structured Dataset Preprocessing Framework. Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024). 2024

    2024

  16. [16]

    arXiv preprint arXiv:2505.16973 , year=

    VERIFASTSCORE: Speeding up long-form factuality evaluation , author=. arXiv preprint arXiv:2505.16973 , year=

    arXiv

  17. [17]

    arXiv preprint arXiv:2412.07724 , year=

    Granite guardian , author=. arXiv preprint arXiv:2412.07724 , year=

    Pith/arXiv arXiv

  18. [18]

    arXiv preprint arXiv:2407.08488 , year=

    Lynx: An open source hallucination evaluation model , author=. arXiv preprint arXiv:2407.08488 , year=

    Pith/arXiv arXiv

  19. [19]

    Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies , pages=

    Representing numbers in NLP: a survey and a vision , author=. Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies , pages=

    2021

  20. [20]

    Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=

    NumGLUE: A suite of fundamental yet challenging mathematical reasoning tasks , author=. Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=

  21. [21]

    Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) , pages=

    Fact or fiction: Verifying scientific claims , author=. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) , pages=

    2020

  22. [22]

    Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=

    Generating scientific claims for zero-shot scientific fact checking , author=. Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=

  23. [23]

    arXiv preprint arXiv:2412.09560 , year=

    Foundational large language models for materials research , author=. arXiv preprint arXiv:2412.09560 , year=

    Pith/arXiv arXiv

  24. [24]

    Nature Machine Intelligence , pages=

    A family of large language models for materials research with insights into model adaptability in continued pretraining , author=. Nature Machine Intelligence , pages=. 2026 , publisher=

    2026

  25. [25]

    2024 , howpublished =

    quantulum3: A. 2024 , howpublished =

    2024

  26. [26]

    2024 , howpublished=

    Grecco, Hern. 2024 , howpublished=

    2024

  27. [27]

    Educational and psychological measurement , volume=

    A coefficient of agreement for nominal scales , author=. Educational and psychological measurement , volume=. 1960 , publisher=

    1960

  28. [28]

    Personnel psychology , volume=

    Content analysis: An introduction to its methodology , author=. Personnel psychology , volume=. 2004 , publisher=

    2004

  29. [29]

    Advances in large margin classifiers , volume=

    Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods , author=. Advances in large margin classifiers , volume=. 1999 , publisher=

    1999

  30. [30]

    Proceedings of the eighth ACM SIGKDD international conference on Knowledge discovery and data mining , pages=

    Transforming classifier scores into accurate multiclass probability estimates , author=. Proceedings of the eighth ACM SIGKDD international conference on Knowledge discovery and data mining , pages=

  31. [31]

    International conference on machine learning , pages=

    On calibration of modern neural networks , author=. International conference on machine learning , pages=. 2017 , organization=

    2017

  32. [32]

    Findings of the Association for Computational Linguistics: ACL 2024 , pages=

    Acueval: Fine-grained hallucination evaluation and correction for abstractive summarization , author=. Findings of the Association for Computational Linguistics: ACL 2024 , pages=

    2024

  33. [33]

    Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=

    FineSurE: Fine-grained summarization evaluation using LLMs , author=. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=

  34. [34]

    Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=

    Hallulens: Llm hallucination benchmark , author=. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=

  35. [35]

    Proceedings of the 19th International Workshop on Semantic Evaluation (SemEval-2025) , pages=

    SemEval-2025 task 3: Mu-SHROOM, the multilingual shared-task on hallucinations and related observable overgeneration mistakes , author=. Proceedings of the 19th International Workshop on Semantic Evaluation (SemEval-2025) , pages=

    2025

  36. [36]

    arXiv preprint arXiv:2403.17169 , year=

    Quantemp: A real-world open-domain benchmark for fact-checking numerical claims , author=. arXiv preprint arXiv:2403.17169 , year=

    Pith/arXiv arXiv

  37. [37]

    arXiv preprint arXiv:2510.22055 , year=

    A Benchmark for Open-Domain Numerical Fact-Checking Enhanced by Claim Decomposition , author=. arXiv preprint arXiv:2510.22055 , year=

    arXiv

  38. [38]

    Working Notes of CLEF , year=

    Overview of the CLEF-2025 CheckThat! lab task 3 on fact-checking numerical claims , author=. Working Notes of CLEF , year=

    2025

  39. [39]

    arXiv preprint arXiv:2507.06195 , year=

    DS@ GT at CheckThat! 2025: evaluating context and tokenization strategies for numerical fact verification , author=. arXiv preprint arXiv:2507.06195 , year=

    Pith/arXiv arXiv 2025

  40. [40]

    arXiv preprint arXiv:2509.11492 , year=

    ClaimIQ at CheckThat! 2025: comparing prompted and fine-tuned language models for verifying numerical claims , author=. arXiv preprint arXiv:2509.11492 , year=

    arXiv 2025

  41. [41]

    Empirical Methods in Natural Language Processing (EMNLP) 2025 , pages=

    Think Right, Not More: Test-Time Scaling for Numerical Claim Verification , author=. Empirical Methods in Natural Language Processing (EMNLP) 2025 , pages=. 2025 , organization=

    2025

  42. [42]

    Proceedings of the Second Workshop on Scholarly Document Processing , pages=

    Overview and insights from the SCIVER shared task on scientific claim verification , author=. Proceedings of the Second Workshop on Scholarly Document Processing , pages=

  43. [43]

    arXiv preprint arXiv:2512.22396 , year=

    HalluMat: Detecting Hallucinations in LLM-Generated Materials Science Content Through Multi-Stage Verification , author=. arXiv preprint arXiv:2512.22396 , year=

    arXiv

  44. [44]

    arXiv preprint arXiv:2601.06519 , year=

    MedRAGChecker: Claim-Level Verification for Biomedical Retrieval-Augmented Generation , author=. arXiv preprint arXiv:2601.06519 , year=

    arXiv

  45. [45]

    Transactions of the Association for Computational Linguistics , volume=

    Tabver: Tabular fact verification with natural logic , author=. Transactions of the Association for Computational Linguistics , volume=. 2024 , publisher=

    2024

  46. [46]

    arXiv preprint arXiv:2512.01224 , year=

    CoSineVerifier: Tool-Augmented Answer Verification for Computation-Oriented Scientific Questions , author=. arXiv preprint arXiv:2512.01224 , year=

    arXiv

  47. [47]

    Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing , pages=

    Explain, edit, generate: rationale-sensitive counterfactual data augmentation for multi-hop fact verification , author=. Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing , pages=

    2023

  48. [48]

    Proceedings of the 2023 conference on empirical methods in natural language processing , pages=

    LLM-powered data augmentation for enhanced cross-lingual performance , author=. Proceedings of the 2023 conference on empirical methods in natural language processing , pages=

    2023

  49. [49]

    Transactions of the Association for Computational Linguistics , volume=

    An empirical survey of data augmentation for limited data learning in nlp , author=. Transactions of the Association for Computational Linguistics , volume=. 2023 , publisher=

    2023

  50. [50]

    Computational Linguistics , volume=

    Probing classifiers: Promises, shortcomings, and advances , author=. Computational Linguistics , volume=

  51. [51]

    Proceedings of the 58th annual meeting of the association for computational linguistics , pages=

    Beyond accuracy: Behavioral testing of NLP models with CheckList , author=. Proceedings of the 58th annual meeting of the association for computational linguistics , pages=

  52. [52]

    Transactions of the association for computational linguistics , volume=

    A primer in BERTology: What we know about how BERT works , author=. Transactions of the association for computational linguistics , volume=

  53. [53]

    Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=

    Dually Self-Improved Counterfactual Data Augmentation Using Large Language Model , author=. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=

  54. [54]

    Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing , pages=

    From Input Perception to Predictive Insight: Modeling Model Blind Spots Before They Become Errors , author=. Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing , pages=

    2025

  55. [55]

    Proceedings of the 29th Conference on Computational Natural Language Learning , pages=

    Beyond accuracy: Revisiting out-of-distribution generalization in NLI models , author=. Proceedings of the 29th Conference on Computational Natural Language Learning , pages=

  56. [56]

    arXiv preprint arXiv:2506.13342 , year=

    Verifying the verifiers: Unveiling pitfalls and potentials in fact verifiers , author=. arXiv preprint arXiv:2506.13342 , year=

    arXiv