Distributed Denial of Science: How Indirect Data Poisoning of AI Systems Can Industrialize Scientific Fraud

Atoosa Kasirzadeh; B\'alint Gyevn\'ar; Nihar B. Shah

Remote adversaries can industrialize scientific fraud by poisoning open datasets that honest AI research agents retrieve and turn into published false conclusions, succeeding fully in nearly half of trials while almost never being detected.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-14 09:48 UTC pith:IVYAQR24

load-bearing objection Solid first measurement of a real threat model: open-data poisoning of scientific agents works often and is almost never noticed, with a practical audit that kills full success in their setup. the 3 major comments →

arxiv 2607.10712 v1 pith:IVYAQR24 submitted 2026-07-12 cs.CR cs.AIcs.DL

Distributed Denial of Science: How Indirect Data Poisoning of AI Systems Can Industrialize Scientific Fraud

B\'alint Gyevn\'ar , Atoosa Kasirzadeh , Nihar B. Shah This is my paper

classification cs.CR cs.AIcs.DL

keywords indirect data poisoningscientific fraudAI research agentsopen datadata provenance auditscientific integrityretrieval-augmented agentstrustworthy science

verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper shows that scientific fraud no longer needs corporate budgets, ghostwriters, or corrupt co-authors. An adversary can simply corrupt a public dataset, add misleading metadata, and re-upload it to an open repository. Autonomous research agents used by honest scientists then find and analyze that data on their own, producing findings that support the adversary’s preferred story. Across 450 controlled runs on five socially salient topics and three frontier AI systems, full poisoning of the final conclusion occurs in about 50 percent of runs and is detected in only 6 percent. A simple five-check provenance audit run at retrieval time drives full attack success to zero, while a “scientist persona” alone still leaves a substantial residual risk. The result is that the same automation meant to accelerate science can also scale the manufacture of doubt, unless agents are required to audit data provenance before they trust it.

Core claim

Indirect data poisoning—uploading a corrupted open dataset plus misleading metadata, with no agent access, no prompt injection, and no fabricated papers—steers frontier AI research agents to fully poisoned conclusions in 49.56 percent of 450 ethically contained runs while the agents flag the poisoning in only 6 percent of cases; a five-check data provenance audit reduces full attack success to zero.

What carries the argument

Indirect data poisoning: an adversary downloads a legitimate open dataset, corrupts it (new rows, imputed values, or a fabricated companion set) and writes misleading metadata, then re-uploads the variant so that retrieval-capable research agents independently select and analyze it, turning honest users into unwitting distributors of the false conclusion.

Load-bearing premise

The experiments treat private repositories made discoverable through an author-written multi-platform API wrapper as a faithful stand-in for real public open-data search, ranking, and trust signals.

What would settle it

Repeat the same 450-run design on truly public repositories with native agent web search and no wrapper; if full-success rates collapse far below 50 percent or detection rises sharply once real popularity markers and ranking are in play, the claimed scalability does not transfer.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

A single poisoned upload can turn every honest scientist who deploys a retrieval-capable AI agent into an unpaid distributor of fraud.
Adversaries can further amplify impact by gaming download counts, stars, and search ranking of the poisoned dataset.
Human-generated trust signals (citations, upvotes, downloads) will degrade as AI both floods and consumes the open data ecosystem, making verifiable provenance certificates necessary.
Policy and judicial uses of AI research agents become high-stakes attack surfaces if poisoned conclusions are treated as independent findings.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

The same retrieval-first failure mode implies that any open corpus used by agents—code repositories, preprint servers, or knowledge bases—can be poisoned without training-time access.
If provenance auditing becomes standard, adversaries will next target the audit signals themselves (fake citations, synthetic companion datasets, or gamed social markers).
The near-zero detection under ordinary prompts suggests that current agents treat the shape of a scientific artifact as sufficient evidence of legitimacy.
Ensembling multiple agents’ provenance scores could further harden the defense at low extra cost, because agents disagree on which runs look suspicious.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Desk Editor's Note

Solid first measurement of a real threat model: open-data poisoning of scientific agents works often and is almost never noticed, with a practical audit that kills full success in their setup.

read the letter

The headline numbers are real inside their regime: 450 runs, three frontier agents, five topics, two polar adversary goals, and a clean success taxonomy. Full success (poisoned conclusion, no caveats, no detection) hits ~50% with only 6% detection; a five-check provenance audit drives full success to zero. That is the first controlled, large-scale measurement of this specific attack—black-box, retrieval-only, no trigger words, no fake papers, just poisoned open data plus misleading metadata—and it is worth taking seriously.

What they do well: the design is careful. Adversary and prompt authors were separated; private repos were used for ethics; success is behavioral, not redefined; they re-ran baselines for the mitigation arm; they report Wilson intervals, χ² tests, and LLM-as-judge labels with solid human agreement (κ≈0.76–0.80). The qualitative notes matter too: agents invent pre-registration, post-hoc rationalize, and sometimes discredit the legitimate source. The audit skill is simple, prompt-level, and actually works in their pipeline. Code and mitigation materials are released; poisoned data is gated, which is the right call.

The soft spot is transfer, not internal validity. Private repos made discoverable via their multi-platform API wrapper (with visibility rewriting) are not native public search. The paper itself shows non-retrieval, not recognition, drives many “no success” outcomes, and they note that stars/downloads are easy to game. So the 49.56% figure is a lower-bound-style measurement under controlled discoverability, not a prediction of SEO-gamed public ranking. Socio-technical tabular topics are a secondary scope limit they already flag. Neither sinks the result; both bound how far you should generalize the percentages.

This is for people working on AI agents in science, open-data integrity, RAG security, and metascience. The central argument holds inside the stated regime. I would send it to peer review and would bring it to reading group. Cite the threat model and the audit result; treat the exact rates as regime-specific until someone re-runs with native public search.

Referee Report

3 major / 0 minor

Summary. The paper introduces and evaluates 'indirect data poisoning': an adversary corrupts a pre-existing open dataset (insert/impute/create data plus misleading metadata), re-uploads it, and relies on autonomous research agents used by honest scientists to retrieve and analyze it, thereby manufacturing seemingly independent scientific conclusions. Across five socio-technical topics, three frontier agents (Claude Code/Opus 4.7, Codex/GPT-5.5, Gemini CLI/Gemini 3.1 Pro), two polar adversary goals, three prompt styles, and five repetitions (450 runs), full attack success (poisoned conclusion with no caveats/detection) is 49.56% while detection is only 6.0%; retrieval of the poisoned set is 84.22%. A scientist-persona system prompt still leaves 16.67% full success; a five-check data-provenance audit (papers, social markers, statistical anomalies, related datasets, poisoning caution) reduces full success to 0% and raises detection to 77.33% in a second 450-run suite. Experiments use private repositories exposed via an authenticated multi-platform API wrapper with visibility rewriting for ethical containment.

Significance. If the transfer story holds, this is a high-impact security result for scientific AI: it shows that open-data trust plus agentic retrieval can industrialize fraud without agent access, prompt injection, or fabricated papers, and that a practical prompt-level provenance audit can neutralize the attack in the tested regime. Strengths include an unusually large factorial design (450+450 runs), an explicit no/partial/full success taxonomy, Wilson/Newcombe intervals and χ² tests, author-separated poisoning vs. prompts, LLM-as-judge labels with human κ≥0.758 on 10% samples, ethical private-repo containment, and released code/prompts/mitigation skill. The work is timely given rapid adoption of AI research agents and documented failures of integrity in automated science.

major comments (3)

Methods §3.2 and Limitations §6.1: the headline full-success rate (49.56%) and retrieval rate (84.22%) are measured under private repositories made discoverable via an author-written multi-platform API wrapper that rewrites public-visibility flags and is instructed as the first search tool. Fig. 4 and §4.1 show that non-retrieval—not recognition—is a primary driver of 'No success' (retrieval falls from ~91% Minimal to ~77% Critical). Real public discovery differs in ranking, SEO-gamed stars/downloads (which §2.2 notes are easy to inflate), native web tools, and trust signals. The abstract and §1.2 state the percentages as if they transfer. Please either (i) reframe all rate claims as conditional on this proxy and report sensitivity bounds, or (ii) add experiments/analysis that quantify how native search, ranking, and popularity markers would change Found/Retrieved rates and thus full suc
§5.4–5.5 and the provenance-audit design: full attack success falls to 0% under the five-check audit, with statistical anomalies and cross-dataset consistency as the strongest predictors (OR≈145 and ≈648). The manuscript itself cites 'uniform distribution or impossible values' as anomaly signals. If the poisoned datasets systematically contain such crude fingerprints (or metadata that the audit is primed to treat as malice), the 0% result may not hold against an adversary who preserves realistic marginals, avoids impossible values, and aligns schema/scale with related public sets. Please report what anomaly types were present in each poisoned set, whether any sophisticated (anomaly-light) poison variants were tried, and whether the audit still reaches near-zero full success when those fingerprints are removed. Otherwise the mitigation claim overstates robustness.
§3.2 / Table 1 and Limitations §6.1: all five topics are socio-technical and all datasets tabular, with adversary goals that invert or exaggerate contested trends. The paper acknowledges weaker scientific consensus and easier fabrication relative to natural sciences or unstructured modalities, but the abstract and implications still generalize to 'scientific fraud at unprecedented scale' and policy-making. Given that topic variance in full success is large (35.6% AV Safety vs 68.9% GenAI Motivation; §4.2), the central claim needs a clearer scope statement: results establish viability for contested tabular socio-technical domains under the tested agents, not yet for high-consensus or non-tabular science. Soften or qualify the industrial-scale and policy claims accordingly, or add at least one harder-prior domain.

Circularity Check

0 steps flagged

Empirical evaluation of agent behavior under data poisoning; no derivation reduces success metrics to fitted inputs or self-defined quantities by construction.

full rationale

This paper reports controlled experimental outcomes (450 runs measuring retrieval, use, statistics, conclusions, and detection of poisoned open datasets by three frontier agents across five topics and two adversary goals). Success criteria (full/partial/no success) are defined operationally from observable agent artifacts and scored by an LLM-as-judge validated against human annotation (Cohen’s κ ≥ 0.758). Adversary goals and honest-scientist prompts were authored by different people; baseline and mitigation conditions were re-run rather than redefined. There are no equations, free parameters fitted to a subset then re-labeled as predictions, uniqueness theorems, or ansatze whose adoption forces the headline percentages. Minor self-citations appear in related-work and discussion (e.g., LKS25 on AI-scientist integrity failures; Gye+24/26 on multi-agent explanations) but are not load-bearing for the measured attack-success or mitigation rates; those rates stand on the new experimental protocol. The private-repo + API-wrapper design is an external-validity limitation, not circularity. Score 1 reflects only the presence of non-load-bearing author-overlap citations; the central claims do not reduce to their inputs by construction.

Axiom & Free-Parameter Ledger

3 free parameters · 4 axioms · 2 invented entities

The central claims rest on experimental operationalization rather than free mathematical parameters. Load-bearing background assumptions are domain conventions about open data trust, agent retrieval behavior, and ethical containment via private repos. The main invented construct is the named attack and the five-check provenance audit skill; neither is a physical entity, and both are operationally defined and tested in the paper.

free parameters (3)

Number of experimental repetitions per condition (5)
Chosen experimental budget that determines precision of success-rate estimates; not fitted to maximize a claimed effect but still a design choice the reported percentages depend on.
LLM-as-judge model and labeling rubric (Claude Sonnet 4.6)
Dependent variables are assigned by an automated judge validated on a 10% human sample; residual label noise is a free methodological choice affecting measured success/detection rates.
Poisoning intensity / fabricated trend strength per topic
How strongly correlations, inserted rows, or imputed scores were altered is author-chosen per dataset; attack success could depend on these unstandardized effect sizes.

axioms (4)

domain assumption Open data platforms and their metadata are treated by AI research agents as sufficiently trustworthy retrieval sources that misleading READMEs and loading scripts can influence analysis without peer-reviewed papers.
Core threat-model premise in §1 and §3.1; without agent trust in repository artifacts the attack fails.
ad hoc to paper Private repositories exposed through an authenticated multi-platform API wrapper with visibility rewriting are behaviorally equivalent to public open-data discovery for the agents under test.
Ethical containment design in §3.2; authors verify unawareness via traces but acknowledge the proxy in §6.1.
domain assumption Honest scientists issue non-malicious research prompts and do not intentionally request false conclusions; the underlying models are not training-data-poisoned.
Honest-scientist and trustworthy-agent assumptions in §3.1.1 used to argue that success still confers legitimacy.
standard math Standard statistical tools (Wilson intervals, Newcombe intervals, χ² with Yates correction, mixed-effects logistic regression, Fleiss κ) correctly summarize Bernoulli-like run outcomes.
Analysis methods stated in §4 and §5; conventional for proportion comparisons.

invented entities (2)

Indirect data poisoning (as defined for scientific AI agents) no independent evidence
purpose: Name and operationalize the remote, retrieval-only attack that steers agent conclusions via corrupted open datasets and misleading metadata without agent access or prompt injection.
Threat model is new relative to training poisoning and classic prompt injection; defined and measured in the paper rather than independently observed in the wild.
Five-check data provenance audit skill no independent evidence
purpose: Mitigation that scores referencing papers, social markers, statistical anomalies, related-dataset consistency, and explicit poisoning caution.
Prompt-level defense constructed and evaluated by the authors; effectiveness is internal experimental evidence.

pith-pipeline@v1.1.0-grok45 · 34769 in / 3413 out tokens · 39282 ms · 2026-07-14T09:48:33.836828+00:00 · methodology

0 comments

read the original abstract

Scientific fraud is the instrument of doubt that malicious entities can use to establish controversy in science. Historically, it required the resources of a company: deep pockets, ghostwritten articles, and corrupt academics. Today, Artificial Intelligence (AI) is increasingly automating scientific research, so we ask: Can a remote adversary weaponize the honest use of AI in science to compromise scientific integrity? We envision and empirically evaluate a new attack, indirect data poisoning, in which an adversary corrupts an open dataset and uploads the poisoned variant to a public repository. Autonomous research agents may independently retrieve and process this data, turning honest scientists into the unpaid and unwitting distributors of fraud at scale. Across five socially-salient topics, from hiring discrimination to the safety of autonomous vehicles, three widely used frontier AI systems (Claude Code with Claude Opus 4.7, Codex with GPT-5.5, Gemini CLI with Gemini 3.1 Pro), and 450 ethically contained experimental runs, we find that poisoning succeeds in 49.56% of runs, while the rate of poisoning detection is only 6.0%. The attack requires no topic-specific trigger-words, agent access, indirect prompt injection, or fabricated papers, only the open data ecosystem and misleading metadata. To mitigate the attacks, we propose and evaluate two measures: a scientist persona and a data provenance audit with five checks (referencing papers, social markers, statistical anomalies, related datasets, poisoning caution). We find that the persona still leaves 16.67% of runs with a poisoned conclusion, but provenance auditing reduces attack success rate to zero. Our results suggest that indirect data poisoning may enable scientific fraud at unprecedented scale, but these attacks can be mitigated with suitable auditing by agents during data retrieval.

Figures

Figures reproduced from arXiv: 2607.10712 by Atoosa Kasirzadeh, B\'alint Gyevn\'ar, Nihar B. Shah.

**Figure 1.** Figure 1: Threat model of the indirect data poisoning attack we consider. The trust boundary separates the honest user and the AI research system (bottom) from the open data ecosystem and the adversary (top). The AI system is executing an end-to-end pipeline from research question to conclusion, searching for and selecting from pre-existing datasets D1, D2, D3, . . . with relevance to some hypothesis H. An adversary… view at source ↗

**Figure 2.** Figure 2: Experimental protocol for evaluating indirect scientific data poisoning attacks against AI systems. The output of this process is a collection of findings, trace logs, and analysis scripts [PITH_FULL_IMAGE:figures/full_fig_p013_2.png] view at source ↗

**Figure 3.** Figure 3: Attack success of poisoned datasets across the three prompts (Minimal, Targeted, Critical; in bars), faceted by agent (Claude, Codex, Gemini; in columns) and adversary goal (Exaggerate, Reject; in rows). The stacked bars show the proportion of runs for each prompt classified as Full success (red = worst outcome for defender), Partial success (yellow), No success (green). Percentages aggregate over 5 topics… view at source ↗

**Figure 4.** Figure 4: Share of runs at each stage of the research process until where the poisoned dataset propagated. Each line corresponds to one prompt condition (red: Minimal; yellow: Targeted; green: Critical). Flatter lines indicate that once retrieved, the poisoned dataset tends to carry through to the final conclusion. Error bars show 95% Wilson CIs, pooled across 3 agents, 2 adversary goals, 5 topics, and 5 repetitions… view at source ↗

**Figure 5.** Figure 5: Detection rates of poisoned datasets across the three prompts (Minimal, Targeted, Critical; in bars), faceted by agent (Claude, Codex, Gemini; in columns) and adversary goal (Exaggerate, Reject; in rows). Proportions are pooled across 5 topics with 5 iterations, giving 25 runs per bar. Stacked bars show the proportion of runs classified as Not detected (red), Partial (orange), Detected (green), or N/A (gre… view at source ↗

**Figure 6.** Figure 6: Attack success of poisoned datasets across the three mitigation conditions (Baseline, Scientist Persona, Provenance Audit; in bars), faceted by agent (Claude, Codex, Gemini; in columns) and adversary goal (Exaggerate, Reject; in rows). Each stacked bars shows the proportion of runs for each prompt classified as Full success (red = worst outcome for defender), Partial success (yellow), No success (green). P… view at source ↗

**Figure 7.** Figure 7: Detection rates of poisoned datasets across the three mitigation conditions (Baseline, Scientist Persona, Provenance Audit; in bars), faceted by agent (Claude, Codex, Gemini; in columns) and adversary goal (Exaggerate, Reject; in rows). Each stacked bar shows the proportion of runs classified as Not detected (red), Partial (orange), Detected (green), or N/A (grey). Percentages are pooled across 5 topics wi… view at source ↗

**Figure 8.** Figure 8: Share of runs at each stage until where the poisoned dataset propagated. Each line corresponds to one mitigation condition (red: Baseline; yellow: Scientist Persona; green: Provenance Audit). Flatter lines indicate that, once retrieved, the poisoned dataset tends to carry through to the final conclusion. Error bars show 95% Wilson CIs, pooled across the 3 agents, 2 adversary goals, 5 topics, and 5 repetiti… view at source ↗

discussion (0)

Reference graph

Works this paper leans on

3 extracted references · 1 canonical work pages

[1]

AI Tools in Society: Impacts on Cognitive Offloading and the Future of Crit- ical Thinking

arXiv: 2509.23426 [cs.AI] .url: https://arxiv.org/abs/2509.23426 (visited on 05/26/2026). 25 [Ger25] M. Gerlich. “AI Tools in Society: Impacts on Cognitive Offloading and the Future of Crit- ical Thinking”. In:Societies15.1 (2025). Correction published inSocieties15(9):252,doi: 10.3390/soc15090252, p. 6. [Gha+26] A. E. Ghareeb, B. Chang, L. Mitchener, A. ...

work page doi:10.3390/soc15090252 2026
[2]

AI ‘Scientists’ Joined These Research Teams: Here’s What Happened

arXiv:2509.08010 [cs].url:http://arxiv.org/abs/2509.08010. Pre-published. [ICM26] ICML 2026 Program Chairs.ICML Experimental Program using Google’s Paper Assistant Tool (PAT). International Conference on Machine Learning. 2026.url:https://blog.icml.cc/ 2026/01/14/icml- experimental- program- using- googles- paper- assistant- tool- pat/ (visited on 06/01/2...

Pith/arXiv arXiv 2026
[3]

Fabricated Citations: An Audit across 2.5 Million Biomedical Papers

arXiv:2505.18705 [cs].url:http://arxiv.org/abs/2505.18705. Pre-published. [Tha+25] N. Thakkar, M. Yuksekgonul, J. Silberg, A. Garg, N. Peng, F. Sha, R. Yu, C. Vondrick, and J. Zou.Can LLM Feedback Enhance Review Quality? A Randomized Study of 20K Reviews at ICLR 2025. 2025. arXiv:2504.09737.url: https://arxiv.org/abs/2504.09737 (visited on 06/01/2026). [T...

Pith/arXiv arXiv 2025

[1] [1]

AI Tools in Society: Impacts on Cognitive Offloading and the Future of Crit- ical Thinking

arXiv: 2509.23426 [cs.AI] .url: https://arxiv.org/abs/2509.23426 (visited on 05/26/2026). 25 [Ger25] M. Gerlich. “AI Tools in Society: Impacts on Cognitive Offloading and the Future of Crit- ical Thinking”. In:Societies15.1 (2025). Correction published inSocieties15(9):252,doi: 10.3390/soc15090252, p. 6. [Gha+26] A. E. Ghareeb, B. Chang, L. Mitchener, A. ...

work page doi:10.3390/soc15090252 2026

[2] [2]

AI ‘Scientists’ Joined These Research Teams: Here’s What Happened

arXiv:2509.08010 [cs].url:http://arxiv.org/abs/2509.08010. Pre-published. [ICM26] ICML 2026 Program Chairs.ICML Experimental Program using Google’s Paper Assistant Tool (PAT). International Conference on Machine Learning. 2026.url:https://blog.icml.cc/ 2026/01/14/icml- experimental- program- using- googles- paper- assistant- tool- pat/ (visited on 06/01/2...

Pith/arXiv arXiv 2026

[3] [3]

Fabricated Citations: An Audit across 2.5 Million Biomedical Papers

arXiv:2505.18705 [cs].url:http://arxiv.org/abs/2505.18705. Pre-published. [Tha+25] N. Thakkar, M. Yuksekgonul, J. Silberg, A. Garg, N. Peng, F. Sha, R. Yu, C. Vondrick, and J. Zou.Can LLM Feedback Enhance Review Quality? A Randomized Study of 20K Reviews at ICLR 2025. 2025. arXiv:2504.09737.url: https://arxiv.org/abs/2504.09737 (visited on 06/01/2026). [T...

Pith/arXiv arXiv 2025