Pith. sign in

REVIEW 5 major objections 5 minor 21 references

GPE is a new benchmark and evaluation framework that measures how fact-verification methods degrade under controllable GEO-style poisoning, showing that accuracy and token efficiency collapse as poisoned evidence increases and that no metho

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

GPE, a new benchmark with controllable GEO-style poisoning, shows LLM fact verifiers degrade sharply under poisoned evidence, with no single verifier winning across all attack types.

T0 review reviewed 2026-08-01 challenge →

load-bearing objection A useful benchmark resource with a plausible but underspecified evaluation protocol; the fixed three-evidence substitution makes the GEO-poisoning generalization questionable, but the contribution is worth a serious look. the 5 major comments →

arxiv 2607.20730 v1 pith:CGJXLYMI submitted 2026-07-22 cs.CR cs.AI

GPE: Evaluating Robust Evidence Aggregation for Fact Verification under Controllable GEO-Style Poisoning

Zhaoqi Wang , Zijian Zhang , Xiaomei Yuan , Pengtao Kou , Jiamou Liu , Zhen Li , Liehuang Zhu This is my paper
classification cs.CR cs.AI
keywords fact verificationgenerative engine optimizationGEO poisoningadversarial evidenceretrieval-augmented generationrobustness evaluationtoken efficiencybenchmark
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper is trying to establish that fact verification in retrieval-augmented settings needs to be evaluated under adversarial evidence contamination, not just clean retrieval. To show this, it introduces GPE, a benchmark of 638 claims across six categories with human-verified labels and collected evidence, plus an evaluation framework that lets researchers fix the claim and gold label while varying the ratio of poisoned evidence objects from 0% to 100%. Results across four verification methods and four poisoning attacks show that accuracy falls monotonically with poison ratio, that the most damaging attack depends on the method, and that token cost per correct verification rises sharply when accuracy collapses. The central message is that clean accuracy alone gives no reliable signal about robustness under GEO poisoning.

Core claim

GPE demonstrates that adversarial evidence environments reveal robustness and efficiency failures invisible in clean evaluation. Holding claims, evidence selection, and labels fixed while replacing zero, one, two, or all three evidence objects with poisoned documents produces a dose-response: all methods lose accuracy as poison ratio increases, but at different rates and in different attack-specific ways. ATA, which tampers with real documents by altering key facts, is on average the most damaging, while STEEL's staged control flow confers resistance specifically to instruction-like injection but not to content tampering. The paper also introduces token cost per correct verification (TCV), s

What carries the argument

The controllable poisoning protocol: for each claim, the benchmark supplies a fixed set of benign evidence objects and a set of malicious objects generated by one of four attacks (FakeGPT-style generation, PoisonedRAG-style injection, Adaptive Tampering Attack (ATA), and Ignore Injection). Poison ratio α∈{0%,33%,67%,100%} is realized by replacing exactly zero, one, two, or all three selected evidence objects, so performance differences trace to contamination rather than claim difficulty. The efficiency metric TCV (tokens per correct verification) is the companion measurement that converts robustness failure into cost.

Load-bearing premise

The evaluation assumes that replacing exactly three selected evidence objects with poisoned ones at ratio α faithfully represents how real GEO poisoning contaminates a retrieval environment; if real attacks shift retrieval or involve more evidence, the degradation curves may not generalize.

What would settle it

Repeat the GPE protocol using the full per-claim evidence pool instead of exactly three selected objects. If accuracy at 100% poisoning does not drop substantially below clean accuracy, or if the attack ranking changes (e.g., FakeGPT becomes most damaging), then the reported degradation is an artifact of the three-object selection rather than a general property of GEO poisoning.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Fact-verification methods should be evaluated at multiple poison ratios, not only clean evidence; clean accuracy does not predict poisoned accuracy.
  • No single method is robust to all attack types; resistance to one attack (e.g., instruction injection) does not transfer to content tampering.
  • Accuracy-adjusted token cost (TCV) should be reported alongside accuracy, because cheap methods become inefficient when accuracy collapses.
  • The benchmark's fixed evidence environment and on-demand poisoning cache enable reproducible comparisons across methods and attacks.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Real-world GEO poisoning is more likely to alter the retrieval ranking than to replace a fixed evidence list; the benchmark's replacement protocol may underestimate or overestimate real-world impact.
  • The three-object evidence selection is a strong simplification; using the full evidence environment per claim could reveal whether robustness differences are driven by evidence-set size.
  • The benchmark's knowledge graph offers a path to studying whether poisoned evidence propagates across claims through shared entities and sources, which the paper does not explicitly test.
  • Because the attack ordering and method rankings are backbone-dependent, reporting results across multiple base LLMs is necessary; future work might test whether the same patterns hold for open-source verification models.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. GPE is a benchmark and evaluation framework for fact verification under adversarial evidence poisoning. The paper constructs a 638-claim multi-domain dataset with collected raw evidence, knowledge graphs, and four poisoning mechanisms (FakeGPT, PoisonedRAG, ATA, Ignore Injection). The evaluation protocol fixes three evidence objects per claim and defines poisoning ratios α ∈ {0, 1/3, 2/3, 1}. Four verification methods (Direct, RAFTS, SAFE, STEEL) are run on two LLM backbones and compared with exact six-way accuracy, subclaim accuracy, Macro-F1, ordinal mean score, and token efficiency. The central claim is that GPE reveals robustness degradation and efficiency trade-offs that are invisible under clean evaluation, and that no single method is robust across attacks.

Significance. If substantiated, GPE would be a useful resource for an under-served evaluation problem: adversarial GEO-style evidence environments for fact verification. The design is sensible in several respects: claims and labels are held fixed while the evidence mixture is varied; four qualitatively different attacks are included; multiple verifiers are compared under identical contamination; and the paper introduces a cost-corrected metric (TCV) that links accuracy collapse to inference cost. The dose-response curves in Figure 4 and the attack-dependent rankings in Tables I and IV are consistent with the paper's claims. However, the contribution is empirical and the artifacts are not yet released; the reported experiments have no repeated trials, and the evidence-selection protocol is underspecified. These issues do not contradict the internal logic of the paper but they limit the strength of the external-validity claims until addressed.

major comments (5)
  1. [Section V-A, Eq. (6)] The experimental protocol fixes the evidence environment to exactly three 'relevant' evidence objects per claim, with poison ratios α ∈ {0, 33%, 67%, 100%}. The selection criterion is not defined, and the released registry itself reports 11,313 evidence items for 638 claims (~17.7 per claim); thus about 15 of 18 collected objects are discarded. If the selected three are the most decisive or most relevant documents, both clean accuracy and degradation curves in Tables I, IV, V are computed on a non-representative subset. Please specify the selection heuristic, report the distribution of selected items, and include a sensitivity analysis that varies the number of evidence objects or uses random subsets. This is necessary to support the claim that GPE evaluates aggregation over a realistic evidence environment.
  2. [Section V-B, Tables I-IV] Every configuration is evaluated exactly once ('Each fixed configuration is evaluated once on all 638 claims, with seed controlling evidence selection'). With stochastic LLMs, the reported differences among methods at a given attack/ratio (e.g., Table I, DeepSeek, FakeGPT 33%: Direct 46.1 vs. RAFTS 39.2; Table III, several near-ties) may be within random variation. The ranking claims ('no method dominates', ATA strongest on average) require either multiple seeds with variance estimates or a statistical test. Please report at least 3 independent runs per configuration (or a bootstrap over claims) with confidence intervals.
  3. [Section III-C, Eq. (5)] The abstract and framework promise controllable poisoning ratios, but in the experiments α is realized only as replacing 0, 1, 2, or 3 of exactly three objects. This coarse quantization cannot capture fractional contamination of a larger evidence pool, and the replacement-only operation does not model the retrieval-competition component of GEO poisoning described in the Introduction and in [4], where adversarially optimized content becomes more likely to be retrieved and cited. Please add an augmentation condition (adding poisoned objects to the benign pool) and/or evaluate with larger evidence sets, or explicitly justify why the replacement of three fixed objects faithfully operationalizes GEO-style poisoning.
  4. [Section III-B] Human-verified labels and collected evidence are central to the benchmark, but no annotation reliability statistics, label distribution, or category-level counts are reported. Without inter-annotator agreement (or a description of adjudication), readers cannot assess label quality. Please report these statistics in the paper or in the released data card.
  5. [Section V-A vs. Section IV] The experiments use only the dataset-provided evidence setting; the external-search interface described in Section IV is not exercised. Since the title and threat model emphasize GEO poisoning of the retrieval stage, the current experiments demonstrate robustness under direct evidence replacement, not under the retrieval dynamics that GEO attacks exploit. A minimal validation would be to use the external-search mode with a poisoned corpus and measure how often poisoned documents enter the evidence list, or to state clearly that GPE currently evaluates only the evidence-aggregation stage.
minor comments (5)
  1. [Section I] Numerous typos: 'newly informations,' 'appearence,' 'simliar,' 'The verifier have to analysis.' These should be corrected before publication.
  2. [Eq. (10) / Table II] TCV defined as T/C is infinite when C=0; the table presents very large finite numbers but no explicit handling. Clarify units of TCVk and how zero-correct cases are displayed.
  3. [Figures 4 and 5] Figures 4 and 5 lack axis labels and full legends; captions are too sparse to interpret the plots. Please add proper axes and legend information.
  4. [Section V-B] The role of 'seed controlling evidence selection' is ambiguous — does it select the three evidence objects, the LLM decoding, or both? Clarify; if it selects evidence, explain how this interacts with the 'same claim-specific evidence selection for all methods' in V-A.
  5. [Data availability] The paper states 'The implementation will be released after finalizing the codebase.' For a benchmark paper, making the data and attack generation code available at review time (or via an anonymized link) is important; also include licensing and annotation instructions.

Circularity Check

0 steps flagged

No significant circularity: GPE is an empirical benchmark; reported degradation is measured, not derived from fitted inputs or load-bearing self-citations.

full rationale

GPE is an empirical evaluation artifact, not a derivational theory. The formal objects — B_i=(c_i,D_i,G_i,y_i) (Eq. 1), D(α)_i=Mix(D_i,D^m_i,α) (Eq. 5), Acc=C/N, and TCV=T/C (Eq. 10) — are definitions of the evaluation instance, evidence mixing, and metrics; none of them encodes the conclusion that accuracy degrades under poisoning. Clean and poisoned accuracies are measured on fixed claims and evidence with a controlled replacement protocol (§V-A), and no fitted parameter is renamed as a prediction. The only self-citation is [5], used to specify the ATA attack generator in §II and §III-C; it is one of four independently sourced poisoning mechanisms (FakeGPT, PoisonedRAG, ATA, Ignore Injection), and the paper's conclusions do not reduce to ATA's definition. The fixed three-object replacement in §V-A is an external-validity limitation about generalizing to deployed retrieval competition, not a circular step, because the reported numbers are not equal to their inputs by construction. Accordingly, no specific circular step is exhibited.

Axiom & Free-Parameter Ledger

2 free parameters · 3 axioms · 0 invented entities

The paper's central quantitative results rest on three domain assumptions: attack-typology coverage, gold-label correctness, and the 3-object replacement operationalization of poison ratio. The only deliberately chosen numeric settings are the three-object evidence window and the four poison ratios; there are no fitted parameters or invented theoretical entities.

free parameters (2)
  • per-claim evidence selection size = 3
    Section V-A fixes the evidence environment to exactly three 'relevant' evidence objects per claim, so poison ratios {0, .33, .67, 1} map to replacing 0-3 objects. This hand-chosen number determines how attack severity is operationalized.
  • poison ratio set A = {0, 0.33, 0.67, 1.0}
    Section V-A chooses coarse ratios rather than a continuous sweep; the degradation curves depend on this sparse set.
axioms (3)
  • domain assumption The four attack types (FakeGPT-style, PoisonedRAG-style, ATA-style, Ignore Injection) adequately represent GEO poisoning in the wild.
    Section III-C defines the poisoned evidence only through these four mechanisms; if real GEO poisoning is broader, conclusions about robustness may not transfer.
  • domain assumption Gold labels y_i from LLM-as-judge plus human annotators are correct.
    Section III-B states labels are human-verified but reports no inter-annotator agreement or audit; all accuracy numbers inherit this assumption.
  • domain assumption Replacing 0, 1, 2, or 3 of the three supplied evidence objects is a faithful operationalization of poison ratio α, independent of which specific documents were selected.
    Section V-A uses exactly three objects per claim and independent replacement per attack; if the selected objects vary in relevance, accuracy differences could reflect selection rather than poisoning.

reviewed 2026-08-01 · how reviews work

0 comments
Cite this review

Pith. "Pith review of GPE: Evaluating Robust Evidence Aggregation for Fact Verification under Controllable GEO-Style Poisoning." pith.science (2026). https://pith.science/paper/CGJXLYMI

@misc{pith2026260720730,
  author       = {Pith},
  title        = {Pith review of: GPE: Evaluating Robust Evidence Aggregation for Fact Verification under Controllable GEO-Style Poisoning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/CGJXLYMI}},
  note         = {Machine review of arXiv:2607.20730}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Large language models increasingly use search tools to retrieve up-to-date information, introducing a new attack surface in which retrieved documents can be manipulated. This risk is amplified by the development of generative engine optimization, which can make selected content more likely to be retrieved, cited, and adopted by models. Existing fact-verification benchmarks and evaluation frameworks do not provide the controlled evidence environments needed to assess robustness against GEO poisoning. We therefore propose GPE, which consists of a multi-domain fact-verification benchmark and an evaluation framework for controlling evidence sources and poisoning ratios. Experiments across multiple verification methods and poisoning attacks demonstrate that GPE exposes robustness degradation and efficiency trade-offs that cannot be observed through clean evaluation alone, confirming the need to evaluate fact verification under adversarial evidence environments.

Figures

Figures reproduced from arXiv: 2607.20730 by Jiamou Liu, Liehuang Zhu, Pengtao Kou, Xiaomei Yuan, Zhaoqi Wang, Zhen Li, Zijian Zhang.

Figure 1
Figure 1. Figure 1: Benchmark construction pipeline of GPE. graph associated with the instance, and yi is the final human￾verified label. The benchmark interface supplies ci and a selected raw evidence environment to the verification method; it does not prescribe a single analyzed evidence representation. The goal of the verification method is to infer yi while remaining robust when malicious raw evidence is introduced into t… view at source ↗
Figure 2
Figure 2. Figure 2: Evaluation protocol of GPE. The framework supplies a controllable raw evidence environment containing document-based benign and malicious [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: A claim-centered subgraph for 10 claims, not the complete registry. [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Accuracy under ATA on DeepSeek-V4-Flash. [PITH_FULL_IMAGE:figures/full_fig_p007_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Category-level accuracy degradation at 100% poisoning. [PITH_FULL_IMAGE:figures/full_fig_p008_5.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

21 extracted references · 5 linked inside Pith

  1. [1]

    Gpt-4 technical report,

    J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkatet al., “Gpt-4 technical report,”arXiv preprint arXiv:2303.08774, 2023

    Pith/arXiv arXiv 2023

  2. [2]

    Deepseek-v3 technical report,

    A. Liu, B. Feng, B. Xue, B. Wang, B. Wu, C. Lu, C. Zhao, C. Deng, C. Zhang, C. Ruanet al., “Deepseek-v3 technical report,”arXiv preprint arXiv:2412.19437, 2024

    Pith/arXiv arXiv 2024

  3. [3]

    Toolformer: Language models can teach themselves to use tools,

    T. Schick, J. Dwivedi-Yu, R. Dess `ı, R. Raileanu, M. Lomeli, E. Hambro, L. Zettlemoyer, N. Cancedda, and T. Scialom, “Toolformer: Language models can teach themselves to use tools,”Advances in Neural Infor- mation Processing Systems, vol. 36, pp. 68 539–68 551, 2023

    2023

  4. [4]

    Geo: Generative engine optimization,

    P. Aggarwal, V . Murahari, T. Rajpurohit, A. Kalyan, K. Narasimhan, and A. Deshpande, “Geo: Generative engine optimization,” inProceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, 2024, pp. 5–16

    2024

  5. [5]

    Combating knowledge corruption in agent systems: A byzantine- tolerant secure collaborative rag framework,

    Z. Wang, D. He, Z. Zhang, Y . Liu, J. Liu, Z. Zeng, Z. Qin, Z. Li, X. Li, H. Yao, J. An, Y . Liu, Y . Li, Q. Sun, X. Liu, and L. Zhu, “Combating knowledge corruption in agent systems: A byzantine- tolerant secure collaborative rag framework,” inProceedings of the ACM Web Conference 2026, ser. WWW ’26. ACM, 2026. 9 TABLE V OVERALL CLASSIFICATION METRICS(%)...

    2026

  6. [6]

    “liar, liar pants on fire

    W. Y . Wang, ““liar, liar pants on fire”: A new benchmark dataset for fake news detection,” inProceedings of the 55th annual meeting of the association for computational linguistics (volume 2: short papers), 2017, pp. 422–426

    2017

  7. [7]

    Fakenewsnet: A data repository with news content, social context, and spatiotemporal information for studying fake news on social media,

    K. Shu, D. Mahudeswaran, S. Wang, D. Lee, and H. Liu, “Fakenewsnet: A data repository with news content, social context, and spatiotemporal information for studying fake news on social media,”Big data, vol. 8, no. 3, pp. 171–188, 2020

    2020

  8. [8]

    Politifact fact check dataset,

    R. Misra, “Politifact fact check dataset,” 09 2022

    2022

  9. [9]

    Check-covid: Fact-checking covid-19 news claims with scientific evidence,

    G. Wang, K. Harwood, L. Chillrud, A. Ananthram, M. Subbiah, and K. McKeown, “Check-covid: Fact-checking covid-19 news claims with scientific evidence,” inFindings of the Association for Computational Linguistics: ACL 2023, 2023, pp. 14 114–14 127

    2023

  10. [10]

    A survey of fake news: Fundamental the- ories, detection methods, and opportunities,

    X. Zhou and R. Zafarani, “A survey of fake news: Fundamental the- ories, detection methods, and opportunities,”ACM Computing Surveys (CSUR), vol. 53, no. 5, pp. 1–40, 2020

    2020

  11. [11]

    Fakegpt: fake news generation, explanation and detection of large language models,

    Y . Huang and L. Sun, “Fakegpt: fake news generation, explanation and detection of large language models,”arXiv preprint arXiv:2310.05046, 2023

    Pith/arXiv arXiv 2023

  12. [12]

    Poisonedrag: Knowledge corruption attacks to retrieval-augmented generation of large language models,

    W. Zou, R. Geng, B. Wang, and J. Jia, “Poisonedrag: Knowledge corruption attacks to retrieval-augmented generation of large language models,”arXiv preprint arXiv:2402.07867, 2024

    Pith/arXiv arXiv 2024

  13. [13]

    Fighting fire with fire: The dual role of llms in crafting and detecting elusive disinformation,

    J. Lucas, A. Uchendu, M. Yamashita, J. Lee, S. Rohatgi, and D. Lee, “Fighting fire with fire: The dual role of llms in crafting and detecting elusive disinformation,” inProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, 2023, pp. 14 279– 14 305

    2023

  14. [14]

    Teller: A trustworthy framework for explainable, generalizable and controllable fake news detection,

    H. Liu, W. Wang, H. Li, and H. Li, “Teller: A trustworthy framework for explainable, generalizable and controllable fake news detection,” in Findings of the Association for Computational Linguistics: ACL 2024, 2024, pp. 15 556–15 583

    2024

  15. [15]

    Robust fake news detection using large language models under adversarial sentiment attacks,

    S. Tahmasebi, E. M ¨uller-Budack, and R. Ewerth, “Robust fake news detection using large language models under adversarial sentiment attacks,”arXiv preprint arXiv:2601.15277, 2026

    arXiv 2026

  16. [16]

    Re-search for the truth: Multi-round retrieval-augmented large language models are strong fake news detectors,

    G. Li, W. Lu, W. Zhang, D. Lian, K. Lu, R. Mao, K. Shu, and H. Liao, “Re-search for the truth: Multi-round retrieval-augmented large language models are strong fake news detectors,”arXiv preprint arXiv:2403.09747, 2024

    Pith/arXiv arXiv 2024

  17. [17]

    Long-form factuality in large language models,

    J. Wei, C. Yang, X. Song, Y . Lu, N. Hu, J. Huang, D. Tran, D. Peng, R. Liu, D. Huanget al., “Long-form factuality in large language models,”Advances in Neural Information Processing Systems, vol. 37, pp. 80 756–80 827, 2024

    2024

  18. [18]

    Retrieval augmented fact verification by synthesizing contrastive arguments,

    Z. Yue, H. Zeng, L. Shang, Y . Liu, Y . Zhang, and D. Wang, “Retrieval augmented fact verification by synthesizing contrastive arguments,” inProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2024, pp. 10 331– 10 343

    2024

  19. [19]

    FEVER: a large-scale dataset for fact extraction and VERification,

    J. Thorne, A. Vlachos, C. Christodoulopoulos, and A. Mittal, “FEVER: a large-scale dataset for fact extraction and VERification,” inProceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), M. Walker, H. Ji, and A. Stent, Eds. New Orleans, Louisia...

    2018

  20. [20]

    Deepseek-v4: Towards highly efficient million-token context intelligence,

    DeepSeek-AI, “Deepseek-v4: Towards highly efficient million-token context intelligence,” 2026

    2026

  21. [21]

    Introducing GPT-5.4,

    OpenAI, “Introducing GPT-5.4,” https://openai.com/index/ introducing-gpt-5-4/, March 2026, accessed: 2026-03-05. 10

    2026

This paper was first reviewed by deepseek-v4-flash on August 1, 2026.