{"id":"474c8feb-3717-4167-8930-16f38417d4ea","arxiv_id":"2509.08463","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A survey that organizes adversarial attacks on automated fact-checking into claim, evidence, and claim-evidence pair categories, and reports that current defenses address 13 of the 53 surveyed attacks.","lead":"Automated fact-checking systems can be fooled by attacks that change the claim, poison the evidence, or fabricate claim-evidence pairs. This paper maps the published attacks and defenses into one taxonomy and finds that existing defenses cover only a quarter of the known attack surface.","discovery_kind":"review","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The taxonomy's attack-target axis covers only verdict prediction and evidence retrieval, so the claimed 'full-spectrum, full-pipeline' taxonomy omits claim detection and justification production, and the 53-attack/13-defense enumeration is not yet internally consistent.","rationale":"The reader's weakest assumption was corpus representativeness: the hand-curated search may omit attacks outside the selected venues/keywords. That is a legitimate external-completeness concern, but it is hard to settle from the paper alone. The concern I identify is internal and checkable: the paper's own definitions of the AFC pipeline and of its taxonomy axis do not match. Sec. 2 lists four pipeline stages; Sec. 4 restricts attack targets to verdict prediction and evidence retrieval; Fig. 2 has only corrupted-verdict/disrupted-retrieval cells. The three attack families in Sec. 3 are defined by data artifact (claim, evidence, pair), not by pipeline stage, so the claimed 'unified taxonomy that spans attack targets' is not actually a pipeline-spanning taxonomy. This directly bears on the strongest claim of the survey: it is presented as the first comprehensive review of adversarial attacks across the AFC pipeline, with a stress-tested enumeration of the defense gap. If the taxonomy cannot accommodate attacks on claim detection or justification production, or if the enumeration is off by one (Table 4 has 14 evidence attacks while the text says 13), then the survey's organizing contribution and its headline defense-gap statistic are not yet defensible. That said, the concern is reparable: the authors can either narrow the scope statement to claim verification and evidence retrieval, add explicit pipeline-stage coverage with empty cells noted as open problems, or expand the taxonomy. The count mismatch and the Fact Mixing discrepancy reinforce the conditional status but do not by themselves destroy the survey's value. For these reasons I would keep the reader's CONDITIONAL verdict rather than moving to ACCEPT or REJECT: the taxonomy is useful and largely accurate, but the pipeline-completeness framing and the enumeration need correction.","tokens_in":28689,"tokens_out":13333,"duration_ms":120069,"concrete_test":"Using the paper's own Sec. 2 definitions, independently code each attack row in Tables 3-5 to the pipeline stage it compromises (claim detection, evidence retrieval, verdict prediction, justification production) and to its Fig. 2 cell; then reconcile the evidence-attack count among Sec. 5.2 (12 named attacks), Sec. 6.2/Fig. 2 (13), and Table 4 (14 rows), resolving the status of 'Omitting Generate' versus 'Omission Generation' and 'Claim-aligned Re-writingret'. If any attack requires a claim-detection or justification-production cell, or if the reconciled total is 54 rather than 53, the 'full spectrum' and '13 of 53' claims should be revised. If all attacks map only to retrieval/verdict and the count settles at 53, the concern is resolved by an explicit boundary statement.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is a comprehensive review of adversarial attacks across the AFC pipeline plus a unified taxonomy and a quantitative defense gap ('13 of 53 attacks addressed'). The taxonomy as defined does not actually span the pipeline it describes. Sec. 2 defines a four-stage AFC pipeline: claim detection, evidence retrieval, verdict prediction, and justification production. Sec. 4 then defines attack target as 'the specific component of the AFC pipeline being compromised (i.e., verdict prediction or evidence retrieval)', with Fig. 2 using only 'corrupted verdict' and 'disrupted retrieval' cells. Meanwhile, Sec. 3 says the three attack families are 'based on which component of the AFC pipeline they target: claim, evidence, claim-evidence pair'—but claims and evidence are data artifacts, not pipeline stages. So the taxonomy conflates two incompatible axes: data object (claim/evidence/pair) and downstream effect (verdict/retrieval). There is no cell for attacks that compromise claim detection or justification production, and the paper never states that no such attacks exist. In fact, Fact2Fiction is described as exploiting system-generated justifications and compromising sub-claim verification, yet it is labeled only as a verdict-corrupting evidence attack. The enumeration is also not internally stable: Sec. 5.2 names 12 evidence attacks, Sec. 6.2 and Fig. 2 say 13, Table 4 has 14 rows, including 'Omitting Generate' which is never discussed in the text; relatedly, the text claims Fact Mixing yields zero FEVER scores while Table 6 reports 38.07. Because the headline novelty claim is 'full spectrum across the AFC pipeline' and the defense-gap number is a headline result, the taxonomy and the counts must be reconciled before the central claim can be accepted as stated.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This survey reviews adversarial attacks against automated fact-checking (AFC) systems, organizing the literature into three attack families (claim, evidence, and claim-evidence pair attacks), proposing a two-axis taxonomy (attack target by edit granularity), summarizing existing defenses, and quantifying a defense gap: current defenses address only 13 of the 53 surveyed attacks. The paper claims to be the first in-depth, comprehensive review spanning the full AFC pipeline, and it provides structured appendix tables that report attack settings, target datasets, and effectiveness numbers, plus a public resources page.","tokens_in":28813,"tokens_out":4922,"duration_ms":40635,"significance":"If the taxonomy is coherent and the counts are reproducible, this would be a useful organizing reference for a growing but scattered research area, and the 13-of-53 defense-gap figure would be an actionable call for new defenses. The manuscript has notable strengths: the appendix tables (Tables 2–7) are detailed and, on spot-checking, the main-text numbers match the tables (e.g., the Colloquial attack's document recall drop from 90.0 to 72.2 and the Chinese DeBERTa accuracy drop from 86.69 to 57.84); the taxonomy diagram and the attack descriptions cover most of the prominent existing work; and the limitations statement is candid about excluding textual entailment, NLI, and fake-news detection. However, the central claims of \"full-spectrum\" coverage and a consistent enumeration are undermined by the issues below.","major_comments":[{"comment":"The attack-target axis is defined as 'the specific component of the AFC pipeline being compromised (i.e., verdict prediction or evidence retrieval)', but Section 2 defines a four-stage AFC pipeline that also includes claim detection and justification production. Figure 2 only uses 'corrupted verdict' and 'disrupted retrieval' cells. The paper nowhere states that no attacks target claim detection or justification production, and Fact2Fiction (Section 5.2.1) is described as exploiting system-generated justifications and compromising sub-claim verification—yet it is labeled only as a verdict-corruption attack. The claim of a 'full spectrum across the AFC pipeline' is therefore not supported by the taxonomy as defined. Please either explicitly scope the taxonomy to verdict prediction and evidence retrieval with a justification, or extend the attack-target axis to cover all four pipeline stages and classify attacks accordingly.","section":"Section 4 (Taxonomy) vs. Section 2 (Pipeline)"},{"comment":"The count of adversarial evidence attacks is internally inconsistent. Section 5.2 says 'The 13 identified attacks in Fig. 2', Section 6.2 repeats '13', but Table 4 lists 14 rows, including 'Omitting Generate', which is never described in the body text of Section 5.2. The Section 7 claim that 'current defenses address only 13 of the 53 attacks' depends on 38 claim + 13 evidence + 2 pair attacks; with Table 4 showing 14 evidence attacks the total becomes 54, and the defense-coverage fraction changes. Please reconcile the text, figure, and table counts, and state explicitly which items count as distinct attacks (e.g., is 'Omitting Generate' a separate attack or a variant of 'Omitting Paraphrase'?).","section":"Section 5.2, Section 6.2, Table 4, Section 7"},{"comment":"The selection strategy is reported only as reviewing 'the abstract, introduction, conclusion, and limitations of each paper' and selecting '50+ relevant papers'. There is no full-text screening flow, no inclusion/exclusion counts, and no coding rule for assigning each paper to a taxonomy cell. Because the abstract claims to be the first comprehensive review of the full spectrum of adversarial attacks, the absence of a documented and reproducible search protocol makes the completeness claim difficult to verify. Please provide a PRISMA-style flow diagram or a transparent enumeration of searched venues, keyword combinations, screened papers, and exclusion reasons, or soften the claim to describe the survey as representative rather than exhaustive.","section":"Appendix A.2 (Literature Compilation)"}],"minor_comments":[{"comment":"The model name 'Liama3' should be 'Llama 3' (matching the Llama 3 herd of models reference).","section":"Section 5.1.2"},{"comment":"'Word-level manipulations are preformed' should be 'are performed', and 'envidence' should be 'evidence' in the same subsection.","section":"Section 5.2.2"},{"comment":"'critial' should be 'critical'; also 'SubNum' should be 'SubsetNum' for consistency with Section 5.1.","section":"Section 7"},{"comment":"'CorssAug' should be 'CrossAug', and the cell 'Factual-NLI+Quin+' needs formatting clarification (presumably 'Factual-NLI+' with the Quin+ defense).","section":"Table 1"},{"comment":"The phrase 'The A VeriTeC dataset' contains an extra article; the dataset is referred to as 'AVeriTeC' or 'VeriTeC' in the associated literature.","section":"Appendix C.1"},{"comment":"The number of leaf nodes under the evidence-attack branch appears to be fewer than the 13 stated in the text; please verify that the figure, Section 5.2, and Table 4 are mutually consistent.","section":"Figure 2"}],"recommendation":"major_revision","confidential_remarks":"The only author-overlapping reference, Zhang et al. (2020b), is a general NLP adversarial-attack survey coauthored by one of the present authors; it is cited as an excluded NLI-adjacent work rather than as a surveyed AFC attack, so I do not see a substantive self-citation problem. The paper's main risk is the gap between its strong 'first comprehensive review' claims and the actual, currently somewhat under-specified taxonomy scope and counting consistency."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThis survey is the broadest synthesis yet of adversarial attacks on automated fact-checking. The three-way split into claim, evidence, and claim-evidence pair attacks, crossed with attack target and edit granularity, gives the subfield a usable organizing framework. The enumeration of 53 attacks and the observation that only 13 are addressed by current defenses is a concrete, actionable summary. The appendix tables are detailed, my spot-checks matched the main text, and the authors maintain a GitHub resource page, which is a nice extra. If you work on fact-checking robustness or adversarial NLP, this is worth reading.\n\nThe soft spots are real, and two of them matter. First, the stress-test note is on target: the taxonomy does not actually span the four-stage pipeline the paper itself defines. The attack-target axis is limited to verdict prediction and evidence retrieval, so claim detection and justification production have no cell. The paper never says no attacks target those stages, and Fact2Fiction, which exploits agentic justifications, is filed under verdict-corrupting evidence attacks. So the 'full spectrum across the AFC pipeline' claim overreaches. This is fixable by narrowing the scope language and adding a pipeline-stage coverage row.\n\nSecond, the headline numbers are not internally consistent. Section 5.2 says 13 evidence attacks while Table 4 has 14 rows, including Omitting Generate, which is never discussed in the text. The main text also claims Fact Mixing yields zero FEVER scores, but Table 6 reports 38.07. Those discrepancies make the 13-of-53 defense-gap figure shaky until the counts are reconciled.\n\nThe hand-curated corpus selection, based on abstracts and conclusions with no formal audit trail, is a weaker spot than the authors acknowledge. Since the paper's value depends on coverage, a documented search and inclusion procedure would materially strengthen it.\n\nThe 'first' novelty claim is slightly overdone, given Fact-Saboteurs already covered evidence manipulation, but this paper's combined scope is broader. The issues here are fixable. They do not undermine the core usefulness of the taxonomy, but they do mean the paper, as currently written, overstates its coverage.\n\nSend it to peer review. A good referee can push the authors to fix the counts, align the taxonomy with the stated pipeline, and document the selection process. After revision this would be a solid reference for anyone entering the area.","headline":"Useful taxonomy and broadest attack survey to date, but the 'full pipeline' claim and the headline defense-gap count need reconciliation before I'd trust the numbers.","tokens_in":29619,"tokens_out":3874,"would_cite":true,"duration_ms":33067,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This survey claims to be the first systematic review of adversarial attacks on automated fact-checking, organizing 53 attacks into a three-family taxonomy and quantifying that current defenses cover only 13 of them.","keywords":["automated fact-checking","adversarial attacks","taxonomy","claim manipulation","evidence poisoning","claim-evidence pairs","defense coverage","survey"],"falsifier":"Run a systematic search with a broader query set (for example 'fact verification attack', 'claim verification robustness', 'entailment attack', and non-English fact-checking venues), and count how many additional attack techniques do not fit the three-way claim/evidence/claim-evidence split; a substantial number of non-fitting attacks would refute the paper's full-spectrum claim.","tokens_in":28310,"feed_emoji":"🛡️","tokens_out":6436,"duration_ms":53461,"temperature":0.7,"pith_summary":"The paper sets out to give the first in-depth, holistic review of adversarial attacks aimed at automated fact-checking (AFC) systems, an area it argues prior surveys have only touched piecemeal. Its central contribution is a two-axis taxonomy that maps every surveyed attack by which pipeline component it targets (verdict prediction or evidence retrieval) and by the granularity of its edits (from single characters to whole corpora), grouped into three families: claim attacks, evidence attacks, and claim-evidence pair attacks. Across 53 cataloged attacks, the paper finds that published defenses address only 13, which would leave most known attack strategies without a tested mitigation. A sympathetic reader would take the paper's value to be the organizing map itself: a shared coordinate system for comparing attacks, assessing model resilience, and directing future defenses.","feed_headline":"Survey maps 53 attack strategies on automated fact-checking","feed_subtitle":"A two-axis taxonomy spans claim, evidence, and pair attacks; only 13 of 53 have published defenses.","key_machinery":"The organizing device is a two-dimensional taxonomy: the attack target (whether the perturbation attacks verdict prediction or evidence retrieval) crossed with edit granularity (character-level, word-level, sentence-level, article-level, corpus-level, or dataset-level), with attacks first split into three families according to which AFC component they touch — claims, evidence, or claim-evidence pairs. The taxonomy is the paper's main instrument: it converts dozens of ad hoc attack techniques into a single grid, making it possible to count coverage (13 of 53 attacks have published defenses), compare settings, and identify empty cells that future attacks or defenses might occupy.","core_discovery":"The paper's central claim is that adversarial attacks on automated fact-checking have been studied in isolation and that no prior survey has covered the full spectrum across the AFC pipeline. It asserts that a unified taxonomy spanning attack target and edit granularity is needed and delivers one, classifying 53 attacks into adversarial claim attacks (38), adversarial evidence attacks (13), and adversarial claim-evidence pair attacks (2). The empirical synthesis shows that current defenses are matched to only 13 of these 53 attacks, and that most claim attacks operate in black-box settings while attacks such as SubsetNum, Fact Mixing, and Multi-hop Temp. can drive FEVER scores to zero or near zero on affected systems. The paper also documents a growing shift toward LLM-assisted generation (GPT-2, GPT-4, T5, BART, PEGASUS) and identifies open gaps: a universal evaluation benchmark, multimodal and real-time attacks, white-box verification attacks, and robustness testing of LLM-based fact-checkers.","pith_inferences":["A broader literature sweep that includes NLP robustness work on textual entailment, NLI, and fake-news detection would likely uncover attacks that do not fit the three-way claim/evidence/pair split, making the 'full spectrum' label a boundary of the search scope rather than a property of the attack space itself.","The 13-of-53 coverage ratio should not be read as a direct measure of real-world risk: an attack with no named defense may still fail in practice, and a defended attack may return at higher severity; the number measures published attention, not vulnerability.","A concrete extension the paper does not pursue: instantiate each cell of the taxonomy with one representative attack and one representative defense, creating a benchmark covering the whole grid so that robustness gains in one family can be measured against regressions in another."],"forward_implications":["The field gains a common vocabulary: any attack or defense can be located by family, target stage, and edit granularity, which should make robustness claims comparable across studies.","The numeric gap between known attacks (53) and addressed attacks (13) becomes a concrete agenda item: roughly seven of every eight attacks still lack a published, dedicated defense.","The predominance of black-box claim attacks and the scarcity of white-box verification attacks marks white-box settings as an under-explored but increasingly realistic threat as open-weight LLM checkers become common.","The vulnerability of systems to surface edits and heuristic shortcuts (documented for SubsetNum, Fact Mixing, and the Colloquial attack) implies that high benchmark accuracy may overstate reasoning ability; retrieval quality alone is not sufficient for veracity.","The emerging attacks on agentic AFC systems (Fact2Fiction) indicate that the threat model is widening from static pipelines to interactive, decompose-and-verify systems."],"supporting_citations":[{"why":"Provides the reference AFC pipeline (claim detection, evidence retrieval, verdict prediction, justification production) that the survey uses as its frame.","marker":"Guo et al., 2022"},{"why":"Supplies the closest prior taxonomy, focused on evidence manipulation attacks, which the survey extends to the full spectrum.","marker":"Abdelnabi and Fritz, 2023"},{"why":"Defines the FEVER benchmark, the dominant dataset on which most surveyed attacks and defenses are evaluated.","marker":"Thorne et al., 2018a"},{"why":"Introduces the FEVER-adv adversarial claims and the evaluation metrics (potency, resilience, correctness) used across claim attacks.","marker":"Thorne et al., 2019a"},{"why":"Establishes the claim-evidence pair attack family via the Symmetric attack and the FEVER-sym dataset, exposing dataset biases.","marker":"Schuster et al., 2019"},{"why":"Provides the Adv. Trigger attack, an early demonstration of label-cohesive, well-formed adversarial claims via language models.","marker":"Atanasova et al., 2020"},{"why":"Supplies the AdvAdd and AdvMod synthetic disinformation evidence attacks that anchor the evidence poisoning discussion.","marker":"Du et al., 2022"},{"why":"Documents the Colloquial attack, showing retrieval performance drops sharply on informal rephrased claims.","marker":"Kim et al., 2021"},{"why":"Supplies the FactEval benchmark and the systematic evaluation of 17 structured manipulations against LLM-based and traditional fact-checkers.","marker":"Mamta and Cocarascu, 2025"},{"why":"Provides Fact2Fiction, the first corpus-level poisoning attack against agentic fact-checking systems, extending the evidence attack family.","marker":"He et al., 2025"}],"fun_headline_variants":["Survey: 53 attacks on automated fact-checking, only 13 defended","First survey maps 53 fact-check attacks, 40 without defenses","53 ways to fool fact-checkers; 13 defenses exist","First taxonomy of adversarial fact-check attacks: 53 found"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The claim of covering the full spectrum of attacks rests on the assumption that the curated sample of 50-plus papers, selected from a stated list of venues and keywords while explicitly excluding NLI, fake-news, and textual-entailment attacks, is representative of all AFC attack research.","fun_headline_variants_meta":{"raw":{"variants":["Survey: 53 attacks on automated fact-checking, only 13 defended","First survey maps 53 fact-check attacks, 40 without defenses","53 ways to fool fact-checkers; 13 defenses exist","First taxonomy of adversarial fact-check attacks: 53 found"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000651,"raw_usage":{"total_tokens":2967,"prompt_tokens":909,"completion_tokens":2058,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":525,"completion_tokens_details":{"reasoning_tokens":1984}},"tokens_in":525,"tokens_out":2058,"duration_ms":13081,"temperature":1.0,"reasoning_tokens":1984,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T16:09:46.629102+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run a systematic search with a broader query set (for example 'fact verification attack', 'claim verification robustness', 'entailment attack', and non-English fact-checking venues), and count how many additional attack techniques do not fit the three-way claim/evidence/claim-evidence split; a substantial number of non-fitting attacks would refute the paper's full-spectrum claim.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the Adv. Trigger attack, an early demonstration of label-cohesive, well-formed adversarial claims via language models."}],"review_version":2}