Pith. sign in

REVIEW 4 major objections 4 minor 1 cited by

DS@GT at CheckThat! 2025: Evaluating Context and Tokenization Strategies for Numerical Fact Verification

T0 review · 4 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Numerical fact-checking is limited by the quality of retrieved evidence, not by how much context the model sees or how numbers are tokenized.

desk verdict Useful negative result on R2L tokenization and long context for numerical fact verification, but the headline claim about evidence quality being the bottleneck is asserted rather than demonstrated. read the letter →

arxiv 2507.06195 v1 pith:TPIKDZFL submitted 2025-07-08 cs.CL

Maximilian Heil , Aleksandar Pramov This is my paper
classification cs.CL
keywords numericalfactverificationQuanTempdatasetModernBERTright-to-lefttokenizationcontextwindowevidenceretrievalveracitypredictionnaturallanguageinference
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Automated fact-checking of numerical claims fails for a different reason than many model-centered approaches assume. This paper shows, through a controlled ablation on numerical claims and an evidence corpus of over 400,000 snippets, that giving the veracity model more retrieved evidence (a longer context window) or changing the order in which numbers are tokenized (right-to-left instead of left-to-right) does not improve veracity prediction. The authors attribute the limiting factor to the quality of the retrieved evidence: when they rebuilt the evidence-retrieval pipeline themselves, performance fell relative to the original benchmark, and no model-side intervention recovered the gap. If the claim holds, effort in numerical fact-checking should shift from bigger inputs and special number tokenizers to stronger evidence retrieval and reranking.

What carries the argument

The load-bearing machinery is an ablation that varies exactly two model-side factors while holding the classifier architecture fixed. Evidence is retrieved by decomposing each claim into sub-questions with GPT-4o-mini, running BM25 sparse retrieval, and reranking with a cross-encoder; the reconstructed pipeline is then fed to a ModernBERT natural-language-inference classifier. The experiment contrasts one evidence snippet with a 256-token context against three snippets with a 1,024-token context, and swaps the tokenizer to right-to-left digit order. The drop from the organizer's benchmark macro-F1 (0.75 train, 0.56 validation) to the recreated pipeline (0.56 train, 0.52 validation) carries the paper's inference about evidence quality.

What would settle it

Use oracle evidence (the snippets the benchmark labels as supporting each claim) and the paper's BM25-plus-reranker evidence for the same claims, keeping classifier, context length, and tokenizer fixed. If longer context or R2L tokenization improves macro-F1 only when oracle evidence is used, the paper's claim is refuted; if it never improves even with oracle evidence, the claim is strengthened.

Watch

Extended reading notes

Core claim

The paper's central claim is that in the absence of high-quality evidence, neither a longer context window (three evidence snippets at 1,024 tokens versus one snippet at 256 tokens) nor right-to-left tokenization of numbers improves veracity prediction for numerical and temporal claims. This directly contradicts earlier findings on arithmetic reasoning, where right-to-left number tokenization helped. The authors further argue that the performance drop between the original benchmark pipeline and their recreated retrieval pipeline is a sign that evidence quality is the dominant bottleneck, since identical classifiers perform worse when fed evidence from their own BM25-plus-reranker retrieval. Their best system, a ModernBERT classifier with one evidence snippet per decomposed sub-question and a 1,024-token context, reaches a macro-averaged F1 of 0.57 on validation and 0.52 on test.

Load-bearing premise

The paper never directly measures evidence quality; its key conclusion rests on the assumption that its recreated retrieval pipeline differs from the original only in evidence quality, not in claim decomposition, reranker choice, or training details.

Editorial extensions

If this is right

  • Veracity systems for number-heavy claims should concentrate effort on retrieval and reranking quality rather than on enlarging the input window.
  • R2L tokenization, which helps arithmetic reasoning, does not transfer to natural-language inference over numerical claims; tokenizer choices need task-specific validation.
  • Supplying additional weak evidence snippets neither helps nor hurts: with poor retrieval, three snippets behave like one.
  • A single evidence snippet per decomposed sub-question with a 1,024-token context can match the benchmark's validation macro-F1 of 0.57.
  • LoRA-style parameter-efficient fine-tuning and focal loss do not recover the gap left by weak evidence.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the paper is right, the fastest way to confirm the bottleneck is to hold classifier and retrieval fixed and swap in oracle evidence; the paper does not run this control.
  • Task-family transfer is the deeper lesson: a tokenization change that helps arithmetic can fail on numerical NLI, so results should not be carried across benchmarks without re-testing.
  • Part of the ceiling may be label-side rather than evidence-side: the paper's own embedding plot shows True and Conflicting claims overlapping heavily, suggesting coarse label mapping could cap any retrieval improvement.
  • A practical test of the paper's implication would gate evidence by reranker confidence, adding a snippet only when it clears a threshold, rather than always feeding three.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. This CLEF 2025 working-notes paper describes a system for Task 3 of CheckThat! on numerical and temporal fact verification over the QuanTemp corpus. The authors reimplement the QuanTemp evidence pipeline (GPT-4o-mini claim decomposition, BM25 retrieval, MiniLM cross-encoder reranking) and fine-tune ModernBERT as the NLI/veracity classifier, with MathRoBERTa and few-shot GPT-4o-mini baselines. They report ablations that vary input context length (256 vs. 1,024 tokens, with 1 vs. 3 evidence snippets), left-to-right vs. right-to-left numerical tokenization, and the use of PEFT or focal loss. The headline conclusion is that neither longer context nor R2L tokenization improves veracity prediction, and the authors attribute this to poor evidence quality from their recreated retrieval pipeline. Their best system reaches a validation macro-F1 of 0.57 and a reported test macro-F1 of 0.52/0.58 in different parts of the text, ranking 4th out of 10 submissions.

Significance. If the central conclusion were fully supported, the paper would provide a useful negative result for numerical fact verification: that model-side interventions such as longer context windows and altered numerical tokenization cannot compensate for weak retrieved evidence. The paper has strengths worth acknowledging: it is evaluated on a public shared-task benchmark, it reports a systematic set of ablations, it reproduces a substantial retrieval pipeline, and it makes code available. Its results also complement prior work by suggesting that the R2L-tokenization benefit observed in arithmetic reasoning does not transfer to NLI on numerical claims. However, the paper's main attribution of performance differences to 'evidence quality' is not supported by any direct measurement, and the reported numbers are internally inconsistent. As a result, the significance of the headline claim is currently limited by methodological and reporting issues rather than by the absence of a clear evaluation setup.

major comments (4)
  1. [Section 5, Table 1] The central 'evidence quality is the bottleneck' conclusion is based on the Benchmark (0.56) vs. Our-Data (0.52) comparison, but at least four pipeline components change simultaneously: the claim-decomposition prompt and sampling parameters, the BM25 retrieval depth over decomposed sub-claims, the reranker model, and the training schedule (5 vs. 7 epochs in Table 2). The paper never directly measures evidence quality, for example by computing recall of the official QuanTemp gold evidence, reranker MRR, or overlap between the retrieved and organizer-provided evidence sets. Without such a measure, the paper cannot distinguish poorer evidence from other pipeline differences, so the abstract and conclusion statements about evidence quality as 'the dominant bottleneck' are unsupported as stated.
  2. [Section 5, Tables 1 and 3] The manuscript contains irreconcilable numeric discrepancies in the main result: the text states a validation macro-F1 drop from 0.63 to a test value of 0.58, while Table 1 lists the Submission validation macro-F1 as 0.57 and Table 3 lists the test macro-F1 as 0.52. The text also describes the validation result as 0.57 in the preceding paragraph. The authors must identify which checkpoint and which table row correspond to each reported number, and correct the inconsistent values, because the discrepancy prevents a reader from verifying the claimed validation-to-test generalization drop.
  3. [Section 5, RQ1] RQ1 asks whether longer context improves veracity prediction, but the proposed comparison conflates two factors: Short-Context uses 1 evidence snippet and a 256-token window, whereas Long-Context uses 3 evidence snippets and a 1,024-token window. Any performance difference (or lack of it) could be due to the number of evidences rather than the context-window size. The experiment should either hold the evidence count fixed while varying the token budget, or the paper should explicitly acknowledge that RQ1 as tested is about the joint effect of evidence quantity and context length.
  4. [Section 5, Focal Loss] The paper claims that focal-loss results are 'not statistically significant' from the cross-entropy Submission results, but no statistical test, confidence interval, or variance estimate is reported anywhere, and each configuration appears to be a single run. With differences on the order of 0.01-0.05 in macro-F1, and no seed variation, the statement is not verifiable. The absence of repeated runs or significance testing also weakens the RQ2/RQ3 conclusions, which are based on the same single-run comparison.
minor comments (4)
  1. [Table 2] The sentence 'epochs until fine-tuning is finished (early-stopping after 2 epochs)' is ambiguous with respect to the epoch counts of 5-7 listed in the table; please clarify whether the early-stopping patience is 2 epochs and report the actual stopping epoch, or explain why the epoch counts include the patience window.
  2. [References] The paper cites GPT-4o-mini with reference [12], which is the GPT-4 technical report; please provide the correct citation or documentation for GPT-4o-mini.
  3. [Figure 1] The claim of 'a noticeable separation of False claims from the other two categories' is based only on a qualitative inspection of a UMAP projection; consider adding a quantitative measure such as silhouette scores or pairwise centroid distances, or soften the wording to reflect the exploratory nature of the plot.
  4. [References] References [6] and [13] list the author 'A. Anand' twice; please correct the author lists.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper is an external benchmark evaluation with held-out test data, and its central bottleneck claim is a confounded interpretation rather than a derivation from its own inputs.

full rationale

The paper reports a shared-task system built on the QuanTemp dataset and evaluates it against an external held-out benchmark. No parameter is fitted to the target result and then renamed as a prediction; the R2L tokenization and context-length ablations are genuine experimental manipulations, and the submission was chosen by validation performance in the standard way. The main interpretive claim — that poorer evidence quality is the bottleneck — rests on comparing the organizer's Benchmark run with the authors' recreated Our-Data run. That comparison is confounded because claim decomposition, BM25 depth, reranker choice, and fine-tuning epochs change simultaneously, and evidence quality is never directly measured. This is a validity or attribution weakness, not circularity: the compared F1 numbers are independently produced and are not defined in terms of one another. There are effectively no self-citations by the paper's own authors, no imported uniqueness theorem, and no ansatz smuggled in through citation. The internally inconsistent F1 reporting (e.g., text vs. tables) is a reporting problem, not a circular reduction. Therefore no specific circular step can be exhibited, and the appropriate circularity score is 0.

Assumptions & free parameters 6 free parameters · 2 assumptions · 0 invented entities

The paper introduces no new mathematical derivations, fitted constants, or invented entities. The listed items are configuration choices and implicit domain assumptions on which the empirical conclusions depend.

free parameters (6)
  • context_window_short = 256
    Hand-picked context window size for the short-context condition in RQ1.
  • context_window_long = 1024
    Hand-picked context window size for the long-context condition in RQ1.
  • rerank_k = 1-3
    Number of reranked evidence snippets kept per sub-claim, varied by ablation condition.
  • focal_loss_gamma = not reported
    Focal loss modulating factor used in the Focal-Loss run; value is not stated in the paper.
  • logit_bias_alpha = not reported
    Scaling strength for the label prior in GPT-4o-mini few-shot classification; value is not stated.
  • lora_rank = not reported
    Rank of LoRA adapters in the PEFT run; value is not stated.
assumptions (2)
  • domain assumption The recreated BM25 plus cross-encoder reranking pipeline provides a fair approximation of the original QuanTemp retrieval setup.
    The paper attributes performance differences to evidence quality without directly measuring retrieval quality; this equivalence is load-bearing for the main conclusion.
  • domain assumption The R2L tokenization modification faithfully implements the digit-segmentation procedure described in reference [7].
    The paper states it switches the ModernBERT tokenizer to R2L but does not detail the implementation; any mismatch could explain the negative result.

how reviews work

0 comments
Cite this review

Pith. "Pith review of DS@GT at CheckThat! 2025: Evaluating Context and Tokenization Strategies for Numerical Fact Verification." pith.science (2026). https://pith.science/paper/TPIKDZFL

@misc{pith2026250706195,
  author       = {Pith},
  title        = {Pith review of: DS@GT at CheckThat! 2025: Evaluating Context and Tokenization Strategies for Numerical Fact Verification},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/TPIKDZFL}},
  note         = {Machine review of arXiv:2507.06195}
}
read the original abstract

Numerical claims, statements involving quantities, comparisons, and temporal references, pose unique challenges for automated fact-checking systems. In this study, we evaluate modeling strategies for veracity prediction of such claims using the QuanTemp dataset and building our own evidence retrieval pipeline. We investigate three key factors: (1) the impact of more evidences with longer input context windows using ModernBERT, (2) the effect of right-to-left (R2L) tokenization, and (3) their combined influence on classification performance. Contrary to prior findings in arithmetic reasoning tasks, R2L tokenization does not boost natural language inference (NLI) of numerical tasks. A longer context window does also not enhance veracity performance either, highlighting evidence quality as the dominant bottleneck. Our best-performing system achieves competitive macro-average F1 score of 0.57 and places us among the Top-4 submissions in Task 3 of CheckThat! 2025. Our code is available at https://github.com/dsgt-arc/checkthat-2025-numerical.

Figures

Figures reproduced from arXiv: 2507.06195 by the authors.

Figure 1
Figure 1. UMAP visualization of the original claims in the english train dataset, stratified by label True/False/- Conflicting To that end, we apply a Uniform Manifold Approximation and Projection (UMAP) [17] dimensionality reduction to the ModernBERT-large embeddings of the claims in the English training dataset (in their original, non-decomposed form). The resulting scatterplot visualization, shown in [PITH_FULL_IMAGE:figu… view at source ↗
Figure 2
Figure 2. Modeling pipeline outlining the steps: (a) Claim decomposition (b) Evidence retrieval and Reranking and (c) Veracity classifier models for the natural language inference tasks 4.1. Evidence Retrieval At the first step, as done in [6], we also perform with a claim decomposition by using GPT-4o-mini [12], which aims to split the original underlying claim into 3 separate smaller claims, for which we retrieve evidence f… view at source ↗

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Symbolic Augmentation Closes a Canonical-Equivalence Blind Spot in Neural Fact-Checkers

    cs.AI 2026-05 conditional novelty 7.0 of 10

    Typed quantity verification exposes a canonical-equivalence blind spot in neural fact-checkers; Symbolic Augmentation fixes it (36.5%→98.2%) and transfers to SciFact-Open (+0.037 binary macro-F1).

Reference graph

Works this paper leans on

22 extracted references · 9 canonical work pages · cited by 1 Pith paper

  1. [1]

    F. Alam, J. M. Struß, T. Chakraborty, S. Dietze, S. Hafid, K. Korre, A. Muti, P. Nakov, F. Ruggeri, S. Schellhammer, V. Setty, M. Sundriyal, K. Todorov, V. Venktesh, Overview of the CLEF-2025 CheckThat! Lab: Subjectivity, fact-checking, claim normalization, and retrieval, in: J. Carrillo-de Albornoz, J. Gonzalo, L. Plaza, A. García Seco de Herrera, J. Mot...

    2025

  2. [2]

    F. Alam, J. M. Struß, T. Chakraborty, S. Dietze, S. Hafid, K. Korre, A. Muti, P. Nakov, F. Ruggeri, S. Schellhammer, V. Setty, M. Sundriyal, K. Todorov, V. V., The clef-2025 checkthat! lab: Subjectivity, fact-checking, claim normalization, and retrieval, in: C. Hauff, C. Macdonald, D. Jannach, G. Kazai, F. M. Nardini, F. Pinelli, F. Silvestri, N. Tonellot...

    2025

  3. [3]

    Faggioli, N

    G. Faggioli, N. Ferro, P. Rosso, D. Spina (Eds.), Working Notes of CLEF 2025 - Conference and Labs of the Evaluation Forum, CLEF 2025, Madrid, Spain, 2025

    2025

  4. [4]

    Venktesh, V

    V. Venktesh, V. Setty, A. Anand, M. Hasanain, B. Bendou, H. Bouamor, F. Alam, G. Iturra-Bocaz, P. Galuscakova, Overview of the CLEF-2025 CheckThat! lab task 3 on fact-checking numerical claims, in: [3], 2025

    work page 2025

  5. [5]

    Sagara, Consumer Understanding and Use of Numeric Information in Product Claims, Ph.d

    N. Sagara, Consumer Understanding and Use of Numeric Information in Product Claims, Ph.d. dissertation, University of Oregon, Eugene, OR, 2009

    work page 2009

  6. [6]

    Venktesh, A

    V. Venktesh, A. Anand, A. Anand, V. Setty, Quantemp: A real-world open-domain benchmark for fact-checking numerical claims, arXiv preprint arxiv:2403.17169 (2024)

    arXiv 2024

  7. [7]

    G. Lee, G. Penedo, L. von Werra, T. Wolf, From digits to decisions: How tokenization impacts arithmetic in llms, https://huggingface.co/spaces/huggingface/number-tokenization-blog, 2024. Accessed: 2025-07-06

    work page 2024

  8. [8]

    Warner, A

    B. Warner, A. Chaffin, B. Clavié, O. Weller, O. Hallström, S. Taghadouini, A. Gallagher, R. Biswas, F. Ladhak, T. Aarsen, et al., Smarter, better, faster, longer: A modern bidirectional encoder for fast, memory efficient, and long context finetuning and inference, arXiv preprint arXiv:2412.13663 (2024)

    arXiv 2024

Show all 22 references
  1. [9]

    T.-Y. Lin, P. Goyal, R. Girshick, K. He, P. Dollár, Focal loss for dense object detection, in: Proceedings of the IEEE international conference on computer vision, 2017, pp. 2980–2988

    2017

  2. [10]

    E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, W. Chen, et al., Lora: Low-rank adaptation of large language models., ICLR 1 (2022) 3

    2022

  3. [11]

    Wadden, Z

    D. Wadden, Z. Lin, L. Liu, M. Gardner, H. Hajishirzi, L. Zettlemoyer, Generating literal and implied subquestions to fact-check complex claims, in: Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (ACL), 2022

    2022

  4. [12]

    OpenAI, Gpt-4 technical report, ArXiv (2023)

    2023

  5. [13]

    Venktesh, A

    V. Venktesh, A. Anand, A. Anand, V. Setty, Numtemp: A real-world benchmark to verify claims with statistical and temporal expressions, CoRR (2024)

    2024

  6. [14]

    Hassan, G

    N. Hassan, G. Zhang, F. Arslan, J. Caraballo, D. Jimenez, S. Gawsane, S. Hasan, M. Joseph, A. Kulka- rni, A. K. Nayak, et al., Claimbuster: The first-ever end-to-end fact-checking system, Proceedings of the VLDB Endowment 10 (2017) 1945–1948

    2017

  7. [15]

    J. Chen, G. Kim, A. Sriram, G. Durrett, E. Choi, Complex claim verification with evidence retrieved in the wild, arXiv preprint arXiv:2305.11859 (2023)

    2023 arXiv

  8. [16]

    K. A. Hambarde, H. Proenca, Information retrieval: recent advances and beyond, IEEE Access 11 (2023) 76581–76604

    2023

  9. [17]

    McInnes, J

    L. McInnes, J. Healy, J. Melville, Umap: Uniform manifold approximation and projection for dimension reduction, arXiv preprint arXiv:1802.03426 (2018)

    2018 arXiv

  10. [18]

    Bajaj, D

    P. Bajaj, D. Campos, N. Craswell, L. Deng, J. Gao, X. Liu, R. Majumder, A. McNamara, B. Mitra, T. Nguyen, et al., Ms marco: A human generated machine reading comprehension dataset, arXiv preprint arXiv:1611.09268 (2016)

    2016 arXiv

  11. [19]

    Clavié, rerankers: A lightweight python library to unify ranking methods, 2024

    B. Clavié, rerankers: A lightweight python library to unify ranking methods, 2024. arXiv:2408.17344

    2024 arXiv

  12. [20]

    Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, V. Stoyanov, Roberta: A robustly optimized bert pretraining approach, arXiv preprint arXiv:1907.11692 (2019)

    2019 arXiv

  13. [21]

    URL: http://www

    PACE, Partnership for an Advanced Computing Environment (PACE), 2017. URL: http://www. pace.gatech.edu

    2017

  14. [22]

    Amini, S

    A. Amini, S. Gabriel, S. Lin, R. Koncel-Kedziorski, Y. Choi, H. Hajishirzi, MathQA: Towards interpretable math word problem solving with operation-based formalisms, in: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Lingui...

    2019

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.