Pith. sign in

REVIEW 3 major objections 5 minor 9 references

From Articles to Premises: Building PrimeFacts, an Extraction Methodology and Resource for Fact-Checking Evidence

T0 review · 3 major / 5 minor · reviewed 2026-08-02 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Fact-checking articles can be converted into stand-alone evidence premises that make automated retrieval and verdict prediction substantially more effective.

desk verdict PrimeFacts is a solid new resource, but the paper's central 'cross-article' retrieval claim is not yet supported because the gold standard is the claim's own article and the extraction prompts feed the claim to the LLM. read the letter →

arxiv 2605.06006 v2 pith:SJCO7FE3 submitted 2026-05-07 cs.CL

Premtim Sahitaj , Jawan Kolanowski , Ariana Sahitaj , Veronika Solopova , Max Upravitelev , Daniel Röder , Iffat Maab , Junichi Yamagishi
show 2 more authors
Sebastian Möller Vera Schmitt
This is my paper · ORCID
classification cs.CL
keywords fact-checkingevidenceextractiondecontextualizationpremiseclaimverificationretrievallargelanguagemodelscorpusconstruction
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Fact-checking articles contain the evidence and reasoning behind a verdict, but this content is buried in unstructured prose. The paper tries to establish that this evidence can be extracted at scale and rewritten into standalone, context-independent premises, and that these premises are a better intermediate representation for automated systems than the original sentences. Using in-article hyperlinks as anchors for evidence, a methodology with three extraction modes is tested: verbatim anchored sentences, LLM-based decontextualized rewrites, and open-ended premises. The paper reports that decontextualized premises improve cross-article evidence retrieval by up to 30 percent in Mean Reciprocal Rank over verbatim sentences, and that using them for verdict prediction raises Macro-F1 by 10-20 points over a majority-class baseline, consistently across model sizes and verdict granularities. A sympathetic reader would care because it suggests a practical path to reuse professional fact-checking work: a structured, portable evidence base that future claims can be checked against.

What carries the argument

The central object is the decontextualized premise: a single, self-contained evidence sentence produced by prompting a large language model to resolve pronouns and implicit references while preserving factual content. The extraction pipeline uses in-article hyperlinks as anchors — Mode A outputs the verbatim anchored sentence, Mode B outputs the decontextualized rewrite of that sentence, and Mode C asks the model to extract premises from the full article without anchor cues. The load-bearing idea is that journalists' hyperlinks identify high-precision evidence, and that rewriting makes that evidence portable. A new measure, the Decontextualization Faithfulness Score, combines forward textual

What would settle it

Run the retrieval evaluation with gold evidence drawn from a different fact-check of the same claim (or a paraphrase), and both with and without removing query-gold shared content words; if the Mode-B gain over Mode-A shrinks or Mode-C's advantage disappears, the reported portability is an artifact of same-article gold and lexical overlap.

Watch

Extended reading notes

Core claim

The central claim is that the decontextualized premise — a single, self-contained evidence sentence derived from a fact-checking article — is a more reusable unit of evidence than the original sentence, because it is both addressable (tied to a specific cited source via the article's hyperlinks) and portable (understandable outside the article). The paper defends this by constructing PrimeFacts, a corpus of more than 13,000 fact-check articles with nearly 50,000 anchor sentences and their rewritten premises, and evaluating the three extraction modes on retrieval and verdict prediction. Across six instruction-tuned large language models, the decontextualized premises consistently outperform v

Load-bearing premise

The load-bearing premise is that the premises extracted from a claim's own fact-check article are the correct gold-standard evidence for retrieval, which means the reported 'cross-article' gains partially measure same-article evidence finding and, for Mode C, evidence generated from the claim itself.

Editorial extensions

If this is right

  • If the central claim is correct, automated fact-checking systems can store prior evidence as decontextualized premises and retrieve relevant past fact-checks substantially more reliably, since ranking gains of up to 30% appear for Mode B over verbatim sentences.
  • Providing these premises to a zero-shot verdict-prediction model raises Macro-F1 by roughly 10-20 points over a majority-class baseline, and by several points over raw anchor sentences, in both binary and five-class settings.
  • The same decontextualization procedure also reduces the number of evidence items a model needs to cite in its justification, suggesting each premise carries more decision-relevant information.
  • The resource itself — over thirteen thousand articles with structured metadata, anchors, and premises — gives researchers a large-scale, addressable evidence base without requiring the copyrighted article text.
  • The hybrid strategy recommended by the authors — use source-grounded Mode B premises as a faithful foundation and add non-redundant Mode C premises for coverage — is a concrete recipe for building evidence stores.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the premise representation proves as portable as claimed, the same methodology could support argument mining by reconstructing the reasoning chain that links premises to verdicts, a step the paper notes is not yet captured.
  • The reported retrieval gains for Mode C may be partly inflated by lexical overlap with the query claim, since Mode C receives the claim as input; a test using paraphrased claims that share no content words with the gold premises would isolate genuine portability.
  • Because Mode B only requires hyperlinks, the methodology is in principle language-agnostic; a multilingual replication on fact-checking outlets in other languages would show whether the decontextualization benefit is tied to English editorial conventions.
  • The Decontextualization Faithfulness Score could be reused outside this paper as a training or filtering signal for any system that rewrites evidence into standalone statements, not just fact-checking.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Request a human review

A listed scientist reviews the paper for a fee and the review publishes here regardless of verdict. See the reviewers or get listed.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper introduces PrimeFacts, a resource and methodology for extracting evidence premises from PolitiFact fact-checking articles. Three extraction modes are proposed: Mode A (verbatim hyperlink-anchored sentences), Mode B (LLM-based decontextualization of those sentences), and Mode C (LLM open extraction of premises from the full article, with the claim provided as input). The authors evaluate the resulting premises on retrieval (BM25) and zero-shot verdict prediction across six LLMs and two verdict granularities, and report that decontextualized premises yield up to 30% relative MRR gains and 10–20 point Macro-F1 gains. A faithfulness score (DFS) and a human annotation study are also presented. The central claim is that extracted, decontextualized premises serve as a portable, addressable intermediate evidence representation for cross-article evidence reuse and automated verification.

Significance. If the portability and verification gains are real, PrimeFacts would be a useful resource for retrieval-augmented fact-checking, and the extraction methodology would be a practical way to convert journalistic sourcing practices into structured evidence. The paper has notable strengths: it releases a large derived dataset (13,106 articles, 49,718 anchors) with provenance metadata; the extraction pipeline is described in enough detail to reproduce; the human study and the proposed DFS score are thoughtful additions. However, the central empirical claim—that decontextualization improves cross-article retrievability—rests on an evaluation in which the gold evidence is taken from the same article as the query claim, and in which Mode C premises are generated with the claim as input. These design choices confound the reported gains with within-document lexical overlap. The significance is therefore conditional on additional controls that separate evidence-reuse from same-article evidence finding.

major comments (3)
  1. [§5.2.1] The retrieval evaluation is not a cross-article reuse test. The gold relevant set is defined as 'the premises from the claim's own fact-check' (Section 5.2.1). Thus, for a claim query, the system is rewarded for retrieving evidence extracted from the same article that contains the claim, not for retrieving evidence from other fact-checks that could verify a rephrased or new claim. High MRR may reflect within-article lexical overlap between the claim and its surrounding evidence sentences, rather than portability. The authors should re-run retrieval with a genuinely cross-article gold standard, for example by annotating relevance between claims and premises from other articles, or by removing the query's own article from the index and judging whether the same claim's evidence is still retrievable from other fact-checks. Without such a control, RQ2 and the '30% relative gain' claim are not
  2. [§A.2 and Table 1] Mode C's retrieval results appear inflated by construction. The Mode C prompt (Appendix A.2) provides the LLM with the claim text ('Claim: {claim}') and asks it to extract premises from the article. Those premises are then indexed and retrieved using the same claim as the query. This creates a self-match: the gold premises are generated from the query itself, so high MRR@10 (0.78–0.88) and Recall@10 (up to 0.57) may reflect lexical overlap between the query and the LLM's claim-conditioned outputs, rather than generalizable evidence reuse. The paper acknowledges the lexical-overlap confound in §5.2.1 ('may also benefit from more direct lexical overlap with the claim wording') but does not control for it. A concrete control is needed, such as measuring the fraction of Mode C premises sharing n-grams with the claim, comparing Mode C retrieval against a variant where the claim is withheld du
  3. [§5.2.2 and Table 3] The verification evaluation shares the same circularity for Mode C. In the verdict-prediction task, the model is given premises generated for that specific claim (Mode C), or decontextualized from the same article (Mode B), and the claim itself. Because Mode C premises were produced with the claim as input, the Macro-F1 gains for Mode C over Mode A/B (e.g., 0.81 vs. 0.68 for Llama-3.3-70B binary) may reflect the extraction model having already seen and incorporated the claim's wording, effectively leaking the query into the evidence. To support the portability claim, the authors should include a control in which evidence is extracted without access to the claim (e.g., from a redacted article), or in which the same evidence set is paired with a different, unrelated claim. This is especially important because the paper's stated goal is to enable verification of new, previously unseen claim
minor comments (5)
  1. [§4.2.2] The term 'cross-article portability' is used to describe the retrieval setup, but since the gold set is from the same article as the query, this is same-article evidence finding. Consider renaming the task or clarifying the distinction.
  2. [Table 1] In the table header and first row, 'BaselineA' is run together without a space, which is likely a typesetting issue. Also, the baseline row is repeated across model blocks; making it a single row with a separator would improve readability.
  3. [§4.2.1, Eq. (1)] The notation t(p) ∩ t(s) uses a multiset intersection, but the text says '(multi)set of tokens' without defining the multiset operator. A brief clarification would avoid ambiguity.
  4. [§5.2.3 and Table 4] The paper's caveat that DFS can underestimate quality for already self-contained source sentences is appreciated, but no reliability analysis is reported for the DeBERTa entailment scores used in DFS. A small human-correlation study for E would strengthen the faithfulness claims.
  5. [Table 3] In the five-class setting, Qwen3-235B Mode B has a notably lower citation coverage (0.46) compared to other models and modes, but this is not discussed. If coverage is diagnostic, a brief explanation would be useful.

Circularity Check

2 steps flagged · score 6.0 of 10

Retrieval evaluation uses same-article, query-conditioned premises as gold, so portability gains are partly self-match.

  1. self definitional [§5.2.1 (Retrieval Performance) with §A.2 (Mode C prompt); cf. §4.2.2]
    ""treating the premises from the claim’s own fact-check as the relevant gold truth set" (§5.2.1). Mode C prompt: "User: 'Claim: {claim}\nArticle(labeled): {segmented_article} \n Return JSON only.'" (§A.2). The paper also concedes: "Mode C ... may also benefit from more direct lexical overlap with the claim wording" (§5.2.1)."

    Mode C premises are generated by an LLM whose prompt includes the very claim that later serves as the retrieval query, and the gold set is defined as the premises of that same fact-check. BM25 is therefore scored on retrieving documents that were generated from the query text itself; near-ceiling Mode C MRR (0.78–0.88) is largely the query matching its own paraphrase. This makes the reported cross-article portability gain (RQ2) partly an artifact of self-match rather than independent evidence reuse.

  2. fitted input called prediction [§4.1.2 / §A.1 with §5.2.1]
    "Mode B prompt: "User: 'Claim: {claim}\nArticle(labeled): {segmented_article} \n Target letter: {letter} \n Target sentence: {target_sentence} \n Return JSON only.'" (§A.1). Retrieval gold: "the premises from the claim’s own fact-check" (§5.2.1)."

    Mode B decontextualization also receives the claim as input, and the rewriting prompt encourages adding entities, numbers, and dates that make the sentence stand alone. When retrieval is then run with the same claim as query and the same article's rewritten premises as the gold set, the reported 30% MRR gain over verbatim Mode A is inflated by query-conditioned rewording: the index contains texts that were constructed from the query. The paper does not control for this by using held-out claims or a fixed external evidence set, so the portability conclusion is not independently established.

full rationale

The paper's central derivation chain for RQ2 is: decontextualized premises are more portable, therefore retrieval with a claim query finds them better, therefore cross-article evidence reuse improves. The load-bearing evaluation, however, defines the gold set as the premises of the claim's own fact-check article (§5.2.1), and both Mode B and Mode C generation prompts explicitly include the claim (§A.1, §A.2). Retrieving those query-conditioned premises with the same claim via BM25 is close to self-retrieval. The paper itself notes the lexical-overlap confound but does not correct it with a held-out external evidence setting. This is a genuine, specific reduction of the headline retrieval claim, though not of the entire resource: the Mode A→B comparison still involves real rewriting, the faithfulness score is an external NLI measure, and the verification results compare representations rather than predicting from fitted parameters. No load-bearing self-citation or imported-uniqueness circularity was found; the one self-citation (Sahitaj et al., 2025) is not load-bearing. Overall, the central portability claim is partially circular, so a score of 6 is appropriate.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The paper fits no numerical parameters; its load-bearing assumptions are the hyperlink-evidence proxy, the mode-specific gold standard, LLM faithfulness, and PolitiFact label reliability.

assumptions (4)
  • domain assumption In-article hyperlinks in fact-check articles serve as a reliable proxy for evidence-bearing sentences.
    Section 4.1 grounds this in journalistic practice (fact-checkers link primary sources); the paper's RQ1 tests this assumption, so it is not externally established.
  • domain assumption The premises from the claim's own fact-check article are the relevant gold set for retrieval evaluation.
    Section 5.2.1 defines gold relevance as the claim's own article premises; this makes the retrieval task same-article evidence finding, not cross-article reuse, and is load-bearing for the 'cross-article' claim.
  • domain assumption LLM-generated decontextualized premises preserve the factual content of the source sentences.
    The paper evaluates this with NLI-based DFS and a human study, but the downstream conclusions (improved verification) assume the rewrites are faithful.
  • domain assumption PolitiFact verdict labels are treated as ground truth for verification.
    Section 2 and Ethical Considerations note verdicts may embed editorial bias; the verification task assumes these labels are correct.

how reviews work

0 comments
Cite this review

Pith. "Pith review of From Articles to Premises: Building PrimeFacts, an Extraction Methodology and Resource for Fact-Checking Evidence." pith.science (2026). https://pith.science/paper/SJCO7FE3

@misc{pith2026260506006,
  author       = {Pith},
  title        = {Pith review of: From Articles to Premises: Building PrimeFacts, an Extraction Methodology and Resource for Fact-Checking Evidence},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/SJCO7FE3}},
  note         = {Machine review of arXiv:2605.06006}
}
read the original abstract

Fact-checking articles encode rich supporting evidence and reasoning, yet this evidence remains largely inaccessible to automated verification systems due to unstructured presentation. We introduce PrimeFacts, a methodology and resource for extracting fine-grained evidence from full fact-checking articles. We compile 13,106 PolitiFact articles with claims, verdicts, and all referenced sources, and we identify 49,718 in-article hyperlinks as natural anchors to pinpoint key evidence. Our framework leverages large language models (LLMs) to rewrite these anchor sentences into stand-alone, context-independent premises and investigates the extraction of additional implicit evidence. In evaluations on cross-article evidence retrieval and claim verification, the extracted premises substantially improve performance. Decontextualized evidence yields higher retrievability, achieving up to a 30 percent relative gain in Mean Reciprocal Rank over verbatim sentences, and using the evidence for verdict prediction raises Macro-F1 by 10-20 points over the baseline. These gains are consistent across different verdict granularities (2-class vs. 5-class) and model architectures. A qualitative analysis indicates that the decontextualized premises remain faithful to the original sources. Our work highlights the promise of reusing fact-checkers' evidence for automation and provides a large-scale resource of structured evidence from real-world fact-checks.

Figures

Figures reproduced from arXiv: 2605.06006 by the authors.

Figure 1
Figure 1. Example fragments from a fact-checking view at source ↗
Figure 2
Figure 2. Extraction pipeline for transforming fact-checks into refined, decontextualized evidence. view at source ↗

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

9 extracted references · 1 linked inside Pith

  1. [3]

    InProceedings of the 60th Annual Meeting of the Association for Com- putational Linguistics (Volume 1: Long Papers), pages 1293–1304, Dublin, Ireland

    WatClaimCheck: A new Dataset for Claim Entailment and Inference. InProceedings of the 60th Annual Meeting of the Association for Com- putational Linguistics (Volume 1: Long Papers), pages 1293–1304, Dublin, Ireland. Association for Computational Linguistics. Martin Klein, Herbert Van de Sompel, Robert Sanderson, Harihar Shankar, Lyudmila Bal- akireva, Ke ...

    2014

  2. [8]

    InProceedings of the 2021 Conference of the North American ChapteroftheAssociationforComputationalLin- guistics: Human Language Technologies, pages 624–643, Online

    Get Your Vitamin C! Robust Fact Verifica- tion with Contrastive Evidence. InProceedings of the 2021 Conference of the North American ChapteroftheAssociationforComputationalLin- guistics: Human Language Technologies, pages 624–643, Online. Association for Computational Linguistics. Christian Stab and Iryna Gurevych. 2017. Parsing argumentation structures i...

    2021

  3. [9]

    Claim: {claim}\nArticle(labeled): {seg- mented_article} \n Target letter: {letter} \n Tar- get sentence: {target_sentence} \n Return JSON only

    Qwen3 technical report.arXiv preprint arXiv:2505.09388. Fengzhu Zeng and Wei Gao. 2024. JustiLM: Few-shotJustificationGenerationforExplainable Fact-Checking of Real-world Claims.Transac- tions of the Association for Computational Lin- guistics, 12:334–354. Ke Zhou, Claire Grover, Martin Klein, and Richard Tobin. 2015. No more 404s: predicting refer- enced...

    arXiv 2024

  4. [2018]

    InProceedings of the 27th International Conference on Computational Lin- guistics, pages 2340–2353, Santa Fe, New Mex- ico, USA

    Stress Test Evaluation for Natural Lan- guage Inference. InProceedings of the 27th International Conference on Computational Lin- guistics, pages 2340–2353, Santa Fe, New Mex- ico, USA. Association for Computational Linguis- tics. Preslav Nakov, Giovanni Da San Martino, Tamer Elsayed,AlbertoBarrón-Cedeño,RubénMíguez, Shaden Shaar, Firoj Alam, Fatima Haoua...

    2021

  5. [2019]

    MultiFC: A Real-World Multi-Domain Dataset for Evidence-Based Fact Checking of Claims. InProceedings of the 2019 Conference on Empirical Methods in Natural Language Pro- cessing and the 9th International Joint Confer- ence on Natural Language Processing (EMNLP- IJCNLP), pages 4685–4697, Hong Kong, China. Association for Computational Linguistics. Samuel R...

    2019

  6. [2021]

    InProceedings of the 2021 Conference of the North American Chap- ter of the Association for Computational Linguis- tics: Human Language Technologies: Demon- strations, pages 84–91

    Improvingevidenceretrievalforautomated explainable fact-checking. InProceedings of the 2021 Conference of the North American Chap- ter of the Association for Computational Linguis- tics: Human Language Technologies: Demon- strations, pages 84–91. Artsiom Sauchuk, James Thorne, Alon Halevy, Nicola Tonellotto, and Fabrizio Silvestri. 2022. On the role of re...

    2021

  7. [2022]

    Fake News

    Missing Counter-Evidence Renders NLP Fact-Checking Unrealistic for Misinformation. In Proceedings of the 2022 Conference on Empir- ical Methods in Natural Language Processing, pages 5916–5936, Abu Dhabi, United Arab Emi- rates. Association for Computational Linguistics. Lucas Graves. 2016.Deciding What’s True: The Rise of Political Fact-Checking in Americ...

    2022

  8. [2024]

    Premtim Sahitaj, Iffat Maab, Junichi Yamagishi, Jawan Kolanowski, Sebastian Möller, and Vera Schmitt

    Claim Detection for Automated Fact- checking: A Survey on Monolingual, Multilingual and Cross-Lingual Research.Natural Language Processing Journal, 7:100066. Premtim Sahitaj, Iffat Maab, Junichi Yamagishi, Jawan Kolanowski, Sebastian Möller, and Vera Schmitt. 2025. Towards Automated Fact- Checking of Real-World Claims: Exploring Task Formulation and Asses...

    2025

Show all 9 references
  1. [2025]

    InFindings of the Association for Computational Linguistics: NAACL 2025, pages 3070–3090

    Language modeling with editable external knowledge. InFindings of the Association for Computational Linguistics: NAACL 2025, pages 3070–3090. Association for Computational Lin- guistics. Xing Han Lù. 2024. BM25S: Orders of magnitude faster lexical search via eager sparse scori...

    2025

Pith tools

Reviewed August 2, 2026 · model on record in the stance chip above.