Pith. sign in

REVIEW 4 major objections 8 minor 29 references

UNH at CheckThat! 2025: Fine-tuning Vs Prompting in Claim Extraction

T0 review · 4 major / 8 minor · reviewed 2026-08-04 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Fine-tuning Flan-T5-Large, a 783M-parameter model, produces the highest METEOR score in this comparison, but prompting methods that score lower often produce claims that human reviewers find more useful for fact-checking.

desk verdict Honest shared-task report whose main takeaway is that METEOR on noisy gold claims rewards mimicry, not quality; authors nearly say this themselves. read the letter →

arxiv 2509.06883 v1 pith:LNB6F5QT submitted 2025-09-08 cs.CL cs.AIcs.IR

Joe Wilder , Nikhil Kadapala , Benji Xu , Mohammed Alsaadi , Aiden Parsons , Mitchell Rogers , Palash Agarwal , Adam Hassick
show 1 more author
Laura Dietz
This is my paper
classification cs.CLcs.AIcs.IR
keywords claimextractionfact-checkingMETEORfine-tuningpromptingself-refinementFlan-T5socialmedia
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper asks whether fine-tuning a smaller model or prompting much larger models produces better extractions of a single check-worthy claim from a social media post. It reports that fine-tuning Flan-T5-Large on the task's training set gives the best METEOR among the tested methods (validation 0.5569; test 0.3737, rank 9 of 17). The larger point is that METEOR is not a reliable proxy for claim quality: several prompting methods scored lower but, on manual review, produced claims that were cleaner, more concise, and better targeted at what a fact-checker would verify. The paper also documents that many gold-standard claims are themselves imperfect, containing multiple facts, missing critical assertions, or referencing images that are not in the dataset. Fine-tuning to match those gold claims therefore inherits their flaws, and the paper concludes that optimizing the metric and optimizing fact-checking utility are different goals.

What carries the argument

The central object is the METEOR score, an n-gram overlap metric that rewards closeness to a reference claim; it is the yardstick the paper uses to select and compare methods. The mechanism that achieves the highest score is fine-tuning Flan-T5-Large on the provided training set with a fixed extraction prompt, which internalizes the phrasing patterns of the gold claims. A secondary mechanism is the regurgitation baseline, which simply returns the original post or a truncated version and sets a surprisingly strong reference point that other methods must beat. The prompting side of the comparison is carried mainly by Self-Refine, a feedback-and-revise loop that the paper finds improves concise

What would settle it

Re-annotate the validation posts with complete gold claims that include the omitted facts and the missing image content, then recompute METEOR for all submitted runs; if fine-tuned Flan-T5-Large no longer ranks first among this paper's methods, the headline result depends on the flawed references.

Watch

Extended reading notes

Core claim

On the paper's own terms, its central finding is that fine-tuning the modest Flan-T5-Large model on the training data yields the highest METEOR score in the team's comparison, and that this model, despite its small size, came closest to reproducing the gold-standard extraction behavior. This result is paired with a qualitative finding: methods that scored lower on METEOR, particularly one iteration of Self-Refine combined with keyword few-shot prompting or a step-by-step Claimify process, often produced claims the authors judged more useful for fact-checking because they captured the post's implied assertion rather than a literal restatement. The paper argues that METEOR's single-reference l

Load-bearing premise

The gold-standard claims used for training and for computing METEOR are reliable enough to measure claim-extraction quality, even though the paper reports many are too verbose, miss key assertions, or refer to images that are not in the dataset.

Editorial extensions

If this is right

  • With tight compute budgets, fine-tuning the smallest feasible instruction-tuned model can be the strongest METEOR strategy, making such models a practical default for claim-normalization tasks.
  • Systems that optimize METEOR will produce claims in the style of the training golds, which the paper shows often omit key facts; the metric therefore rewards mimicry over completeness.
  • A single iteration of self-refinement improves claim conciseness, but repeated iterations introduce hallucinations and verbosity across multiple model families.
  • A trivial baseline that truncates the original post to 100 characters beats several LLM systems, so future systems should be measured against such baselines before claiming gains.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A consequence not stated in the paper: pairing a fine-tuned extractor with a one-shot self-refinement pass could combine METEOR alignment with the human-perceived conciseness the paper observes in prompting methods.
  • Because several gold claims reference images that are absent from the dataset, a re-run of this task would benefit from multimodal inputs or an explicit unverifiable flag; the paper notes the problem but does not propose this remedy.
  • The convergence of most methods around METEOR 0.27 suggests the metric has low discriminative power on this dataset; a testable extension is to score each post against multiple gold references, which would likely reshuffle the method rankings.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 8 minor

Summary. This paper reports the UNH submission to CheckThat! 2025 Task 2 (English claim extraction/normalization). The authors compare fine-tuning and prompting methods across several LLM families, using METEOR as the primary metric on a shared validation set. The best validation METEOR is achieved by fine-tuned FLAN-T5-Large (0.5569), which also yields the team's best official test score (0.3737, rank 9 of 17). The paper also introduces a regurgitation baseline, observes that many methods converge near 0.27 METEOR, and concedes that lower-METEOR methods can sometimes produce more useful claims. The authors are candid about the shortcomings of both the gold standard and the METEOR metric.

Significance. Read as a system-report, this is a useful empirical addition to the CheckThat! working notes: it benchmarks several readily available models, includes a strong baseline that outperforms some official systems, and documents a clear tension between METEOR optimization and human-perceived claim quality. However, the central 'best method' claim is only established for METEOR, not for claim-extraction quality, and the comparison is confounded across model families. The main value is as a data point for evaluating claim-normalization systems, not as a general conclusion about fine-tuning versus prompting. The reported test-set scores are externally grounded and agree with the official leaderboard.

major comments (4)
  1. [Sections 2 and 5 (Tables 1-2)] The paper frames its contribution as 'Fine-tuning Vs Prompting', but the comparison is confounded: fine-tuning is applied to FLAN-T5-Large (2.1), T5-Base (2.2), and DeepSeek-R1-Distill-Llama-8b (2.3), whereas prompting methods use Grok3 (2.4), LLaMA 3.3 70B (2.5-2.6), and LLaMA 3.3 70B (2.7). No method is evaluated in both regimes with the same base model (the only within-model comparison in 2.3 is reported without numbers). Therefore the headline 'fine-tuning beats prompting' cannot be separated from model-family effects. A within-model comparison (e.g., few-shot FLAN-T5-Large) or a matched comparison on at least one model is necessary to support the title's claim.
  2. [Section 6 and Section 5.2] The conclusion that 'with sufficient fine-tuning, the smaller FLAN T5 model was able to approximate the method used to extract gold standard claims' is not supported by the paper's own evidence. In Section 5.2, the fine-tuned FLAN-T5-Large is described as often failing to produce proper claims because it 'focuses solely on optimizing the METEOR score', and Section 6 states that many gold claims 'failed to capture all the critical assertions' and referenced images not in the dataset. Thus a high METEOR score against these gold references may reflect overfitting to a noisy target. The paper should limit its conclusion to 'best METEOR score among our submitted systems' (as the abstract does) and avoid claiming that it approximates the gold extraction method.
  3. [Table 2] The headline validation score for Fine-tuned FLAN-T5-Large (0.5569) is reported without an error bar, whereas several other rows include standard errors (e.g., 0.3310±0.007, 0.2290±0.005). Given that the test score drops to 0.3737 (rank 9 of 17, within a tight cluster from rank 7 to rank 11 in Table 3), the paper should provide variance estimates (e.g., multiple seeds, bootstrap over validation examples) and, ideally, a significance test against the runner-up (0.3310). Without this, the 'best-performing method' claim is not shown to be robust.
  4. [Section 5.2] The secondary claim that 'higher-quality claims can sometimes be extracted using other methods, even when their METEOR scores are lower' rests on 'subjective preference' over a small number of manually inspected validation instances. As stated, this is anecdotal. If it is to remain in the abstract, the authors should either present a rubric-based evaluation with agreement metrics or explicitly mark it as a qualitative observation. In its current form, it conflicts with the leaderboard-based assessment of 'best'.
minor comments (8)
  1. [Section 2.2] The heading reads '2.2. 2.1 LoRA fine-tuning of Flan-T5 Base'; the duplicated section number should be corrected.
  2. [Section 2.2] LoRA hyperparameters (rank, alpha, target modules) are not reported; the statement 'updating only 0.4% of parameters' is insufficient for reproduction.
  3. [Section 2.1] Training hyperparameters (batch size, learning rate, max sequence length, warmup steps) are omitted; please add to an appendix.
  4. [Section 5.2] The BERTScore F1 mean of 0.82 is mentioned without specifying which model computes it, which reference set is used, or whether it covers all 1,170 validation instances.
  5. [Section 2.6] 'extsract' should be 'extract'.
  6. [Table 2] The column header 'Score±Error' implies all rows contain an error; use an em dash or 'n/a' for missing values.
  7. [References] References [6] and [9] are the same Self-Refine paper; keep one to avoid redundancy.
  8. [Throughout] Capitalization of 'Flan-T5' vs 'FLAN-T5' is inconsistent; please standardize.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper is an empirical shared-task evaluation against external validation/test sets, with no load-bearing self-citation or fitted-input-called-prediction step.

full rationale

The paper is a system description for a shared task. Its central claim—that fine-tuned FLAN-T5-Large is the best submitted method—is supported by direct METEOR measurements on an external validation set and an official test leaderboard. The model is not derived from the metric; it is trained on gold claims and then evaluated on held-out data. The statement that fine-tuning let the smaller model 'approximate the method used to extract gold standard claims' is an empirical observation about high METEOR overlap with held-out gold claims, not a definitional equivalence. The 'Max Multi-Prompt' simulated upper bound explicitly selects the highest-METEOR result on the validation set and is not submitted, so it is not presented as a predictive claim. The paper's own criticisms of gold-standard claims (Secs. 1, 5.2, 6) are acknowledged limitations on METEOR's validity as a quality measure, but they do not make the evaluation circular: the evaluation target is external, albeit imperfect. No load-bearing self-citation appears; all cited works are external models, datasets, or methods, and none are by the present authors. Thus the derivation chain is self-contained and empirically grounded.

Assumptions & free parameters 5 free parameters · 3 assumptions · 0 invented entities

The central claim rests on the task's data split and metric, the reliability of gold claims, and the text-only premise. These are standard benchmark assumptions, but the paper itself casts doubt on the gold-claim quality.

free parameters (5)
  • Fine-tuning epochs (FLAN-T5-Large) = 10
    Chosen by the authors; no learning rate or batch size reported (Section 2.1).
  • Subclaim importance threshold = 7
    Subclaims scoring 7 or higher on a self-assigned 1-10 importance score are retained (Section 2.6).
  • Few-shot example count (Claimify) = 4
    Four examples from the training set used in the few-shot prompt (Section 2.4).
  • Oracle selection (Max Multi-Prompt) = highest METEOR
    Simulated upper bound: several prompts applied and the one with highest METEOR is selected per post (Section 2.7).
  • LoRA configuration (T5-Base) = not reported
    Only '0.4% of parameters' updated; rank, alpha, dropout not given (Section 2.2).
assumptions (3)
  • domain assumption The CheckThat! Task 2 data split and METEOR metric are an accepted benchmark for claim extraction.
    The paper uses the task's train/validation/test splits and METEOR as the official evaluation metric (Section 4.1).
  • domain assumption Gold-standard claims are treated as ground truth for metric computation.
    Fine-tuning and METEOR rely on gold claims as targets; the paper later critiques their quality (Sections 1, 5.2, 6).
  • domain assumption Text-only representation is adequate for claim extraction even when posts reference images or videos.
    The authors instruct models to make educated guesses about visual content because images were removed from the dataset (Section 1).

how reviews work

0 comments
Cite this review

Pith. "Pith review of UNH at CheckThat! 2025: Fine-tuning Vs Prompting in Claim Extraction." pith.science (2026). https://pith.science/paper/LNB6F5QT

@misc{pith2026250906883,
  author       = {Pith},
  title        = {Pith review of: UNH at CheckThat! 2025: Fine-tuning Vs Prompting in Claim Extraction},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/LNB6F5QT}},
  note         = {Machine review of arXiv:2509.06883}
}
read the original abstract

We participate in CheckThat! Task 2 English and explore various methods of prompting and in-context learning, including few-shot prompting and fine-tuning with different LLM families, with the goal of extracting check-worthy claims from social media passages. Our best METEOR score is achieved by fine-tuning a FLAN-T5 model. However, we observe that higher-quality claims can sometimes be extracted using other methods, even when their METEOR scores are lower.

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

29 extracted references · 22 canonical work pages

  1. [1]

    Sundriyal, T

    M. Sundriyal, T. Chakraborty, P. Nakov, Overview of the CLEF-2025 CheckThat! lab task 2 on claim normalization, in: G. Faggioli, N. Ferro, P. Rosso, D. Spina (Eds.), Working Notes of CLEF 2025 - Conference and Labs of the Evaluation Forum, CLEF 2025, Madrid, Spain, 2025

    work page 2025

  2. [2]

    Thorne, A

    J. Thorne, A. Vlachos, C. Christodoulopoulos, A. Mittal, FEVER: a large-scale dataset for fact extraction and verification, in: Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics (NAACL), 2018, pp. 809–819. URL: https://aclanthology.org/N18-1079

    work page 2018

  3. [3]

    H. W. Chung, L. Hou, S. Longpre, B. Zoph, Y. Tay, W. Fedus, E. Li, X. Wang, M. Dehghani, S. Brahma, A. Webson, S. S. Gu, Z. Dai, M. Suzgun, X. Chen, A. Chowdhery, S. Narang, G. Mishra, A. Yu, V. Zhao, Y. Huang, A. Dai, H. Yu, S. Petrov, E. H. Chi, J. Dean, J. Devlin, A. Roberts, D. Zhou, Q. V. Le, J. Wei, Scaling instruction-finetuned language models, 202...

    arXiv 2022

  4. [4]

    E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, W. Chen, Lora: Low-rank adaptation of large language models, 2021. URL: https://arxiv.org/abs/2106.09685.arXiv:2106.09685

    arXiv 2021

  5. [5]

    URL: https://huggingface.co/deepseek-ai/ DeepSeek-R1-Distill-Llama-8B

    DeepSeek-AI, Deepseek-r1-distill-llama-8b, 2025. URL: https://huggingface.co/deepseek-ai/ DeepSeek-R1-Distill-Llama-8B

    work page 2025

  6. [7]

    Y. Bai, S. Kadavath, S. Kundu, A. Askell, J. Kernion, A. Jones, A. Chen, A. Goldie, M. Mirhos- seini, C. McKinnon, et al., Training a helpful and harmless assistant with rlhf, arXiv preprint arXiv:2204.05862 (2022)

    arXiv 2022

  7. [8]

    Metropolitansky, J

    D. Metropolitansky, J. Larson, Towards effective extraction and evaluation of factual claims, 2025. URL: https://arxiv.org/abs/2502.10855.arXiv:2502.10855

    arXiv 2025

  8. [9]

    Madaan, N

    A. Madaan, N. Tandon, P. Gupta, S. Hallinan, L. Gao, S. Wiegreffe, U. Alon, N. Dziri, S. Prabhu- moye, Y. Yang, S. Gupta, B. P. Majumder, K. Hermann, S. Welleck, A. Yazdanbakhsh, P. Clark, Self-refine: Iterative refinement with self-feedback, 2023. URL: https://arxiv.org/abs/2303.17651. arXiv:2303.17651

    arXiv 2023

Show all 29 references
  1. [10]

    Kojima, S

    T. Kojima, S. S. Gu, M. Reid, Y. Matsuo, Y. Iwasawa, Large language models are zero-shot reasoners,

  2. [11]

    URL: https://platform.openai.com/docs/models/gpt-4.1-nano

    OpenAI, Gpt-4.1 nano, 2025. URL: https://platform.openai.com/docs/models/gpt-4.1-nano

    2025

  3. [12]

    URL: https://blog.google/technology/google-deepmind/ google-gemini-ai-update-december-2024/

    Google-DeepMind, Gemini 2.0 flash, 2024. URL: https://blog.google/technology/google-deepmind/ google-gemini-ai-update-december-2024/

    2024

  4. [13]

    URL: https://huggingface.co/meta-llama/Llama-3

    Meta, Llama 3.3 70b instruct, 2024. URL: https://huggingface.co/meta-llama/Llama-3. 3-70B-Instruct

    2024

  5. [14]

    URL: https://x.ai/news/grok-3

    xAI, Grok 3 beta — the age of reasoning agents, 2025. URL: https://x.ai/news/grok-3

    2025

  6. [15]

    P. Liu, W. Yuan, G. Neubig, et al., What makes good in-context examples for gpt-3?, in: Findings of EMNLP, 2022

    2022

  7. [16]

    Perez, D

    E. Perez, D. Kiela, K. Cho, True few-shot learning with language models, in: Advances in Neural Information Processing Systems 34 (NeurIPS 2021), 2021, pp. 11054–11070. URL: https: //arxiv.org/abs/2105.11447

    2021 arXiv

  8. [17]

    They will update the policy next year

    H. Daumé III, Frustratingly easy domain adaptation, in: Proceedings of the 45th Annual Meeting of the Association of Computational Linguistics, 2007, pp. 256–263. A. Appendix A.1. Prompts used in Claimifying social media posts with Self-refinement Claim Extraction System Promp...

    2007

  9. [19]

    **Eliminates buttons and buttons**

  10. [20]

    **Stimulates the electrical charges of the cells**

  11. [21]

    **Prevents prostate cancer.**

  12. [22]

    **Improved digestion.**

  13. [23]

    **Very effective against constipation.**

  14. [24]

    **It makes teeth and bones stronger.**

  15. [25]

    example-4

    **Helps the functioning of the kidneys Share men! Everything is for your health! Share it on all social networks.** Let’s think step by step. </user_query> <assistant_response id="example-4"> **Normalized claim:** St. Austin University North Carolina says eating vaginal fluid ...

  16. [26]

    **Verifiability:** To what extent does the response contain claims that can be independently verified using reliable sources? (0 = not verifiable, 10 = fully verifiable)

  17. [27]

    **Likelihood of Being False:** How likely is it that the response contains false or misleading information? (0 = very unlikely, 10 = very likely)

  18. [28]

    **Public Interest:** How likely is the response to be of general public interest or relevance? (0 = not interesting, 10 = highly interesting)

  19. [29]

    **Potential Harm:** How likely is the response to be harmful, offensive, or cause negative consequences? (0 = not harmful, 10 = extremely harmful)

  20. [30]

    Optionally, suggest specific improvements to the response based on your evaluation

    **Check-Worthiness:** How important is it to fact-check this response? (0 = not worth fact-checking, 10 = highly worth fact-checking) For each criterion, provide: - A score (0-10) - Provide a short, precise justification in 1 sentence. Optionally, suggest specific improvements...

  21. [2023]

    URL: https://arxiv.org/abs/2205.11916.arXiv:2205.11916

    arXiv

Pith tools

Reviewed August 4, 2026 · model on record in the stance chip above.