Pith. sign in

REVIEW 3 major objections 5 minor 1 cited by

This paper argues that a retrieval-augmented LLM can be steered on an open question simply by injecting factually correct, one-sided passages into its knowledge base, and that a geometric stance metric makes such manipulation measurable and

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 19:20 UTC pith:PZ37LZZ3

load-bearing objection Plausible new attack class, but the polarization metric is built on 8 passages and never checked against ground truth; the headline defense numbers inherit that. the 3 major comments →

arxiv 2512.00804 v3 pith:PZ37LZZ3 submitted 2025-11-30 cs.CR cs.AIcs.DB

Epistemic Bias Injection: Manipulating LLM Opinion via Selective Context Retrieval

Hao Wu , Prateek Saxena This is my paper
classification cs.CR cs.AIcs.DB
keywords retrieval-augmented generationbias injectionknowledge poisoningpolarization scorePCA embedding axisRAG sanitizationcontext manipulationopinion bias
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper establishes a new attack surface for retrieval-augmented generation (RAG): an adversary who can add passages to the knowledge base can shift an LLM's answer on an open-ended question without inserting a single false statement. The trick is to inject many factually correct passages that consistently favor one side; they crowd out opposing views from the top-k retrieved context, and the model's answer follows. To make this precise, the paper defines a polarization score: projecting passage embeddings onto the axis of greatest variance among a small set of synthetic pro/con passages. On this foundation it builds an attack workflow and a lightweight defense, BiasDef, which filters skewed context and reports cutting adversarial retrieval by 15% and answer-bias shift by more than 6.2x relative to the best baseline. If the claim holds, truthfulness and safety are decoupled in RAG systems: content can be accurate and yet systematically manipulative.

Core claim

The paper's central claim is that stance, not factuality, is the manipulable variable in RAG. It introduces the polarization score (PS): the projection of a passage's embedding onto the first principal component of a small set of LLM-generated passages that span opposing viewpoints. Passages with extreme PS values are used as seeds to generate a batch of factually grounded, stylistically natural variants of the same stance; the paper finds that more than 74% of these injected passages simultaneously satisfy the two conditions that matter—high retrieval relevance to the query and a consistent perspective bias. The defense BiasDef searches similarity thresholds for the point where the KL diver

What carries the argument

The load-bearing object is the polarization axis: the first principal component computed by PCA over embeddings of a small set of synthetic passages written from contrasting viewpoints. Projecting any passage embedding onto this axis gives the scalar polarization score (PS), which turns an abstract ideological disagreement into a one-dimensional coordinate. Both the attack (seed selection by extreme PS) and the defense (detecting a skew in the PS distribution among high-similarity passages) are built on this coordinate. The defense's defining operation is a search over similarity thresholds for the maximum KL divergence between PS distributions of passages above and below the threshold; a th

Load-bearing premise

Everything rests on the assumption that the single direction of greatest variance among eight LLM-generated passages is the same axis along which human readers would sort the issue into opposing camps; if that axis instead captures quirks of the generating model or the embedding space, both the attack and the defense are measuring the wrong thing.

What would settle it

Take open-ended questions with human-annotated stance labels, draw eight synthetic passages from several different LLMs, compute the PCA polarization axis for each draw, and check whether the resulting PS scores sort the human-annotated passages into the human-rated order. If the axis flips sign or the rank correlation with human ratings is near zero across repeated draws, the metric that drives both attack and defense fails.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • A RAG system whose corpus includes unvetted web content can be steered toward a predetermined stance without any false claim being retrieved, so correctness of retrieved facts is not a guarantee of neutrality.
  • Fact-checking is not a sufficient defense: the paper's adversarial passages receive a comparable share of 'not enough info' verdicts and a higher share of 'supports' verdicts than benign passages, meaning a fact-check sieve would keep the biased content.
  • Existing diversity-based rerankers leave a substantial share (about a fifth at high injection rates) of adversarial passages in the top-k context and sacrifice many relevant benign passages; BiasDef reduces the former and improves the latter.
  • A post-retrieval filter based on the polarization metric can be added to an existing RAG pipeline without modifying the LLM, with only modest overhead (under half a second in the reported setup).
  • The same PS-shift metric used to measure attack impact also provides a continuous, quantifiable notion of how one-sided an answer is, which could be reported or monitored even when no attack is suspected.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the polarization axis is as stable as the paper assumes, the same geometric score could be used proactively—for instance, to flag a query whose top-k context is strongly one-sided before generation, rather than only filtering after an attack is mounted.
  • The binary pro/con axis is a deliberate simplification; applying PCA to subsequent principal components (as the paper mentions but does not develop) would be a natural way to catch controversies that split along several orthogonal dimensions at once.
  • Because the attack is built on LLM-generated passages, its long-term potency depends on whether dense retrievers keep favoring LLM-style text; a retriever retrained to penalize machine-written prose could blunt the attack, while better generators could make it harder to distinguish from human writing.
  • The paper measures success mostly on the geometric PS axis; a direct study of downstream human trust—whether users recognize the one-sidedness—remains an open test, and the authors explicitly leave it to future work.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes a new class of RAG poisoning attacks called epistemic bias injection (EBI). The attacker injects factually correct but one-sided passages into a vector database; because dense retrievers favor LLM-written, query-relevant content, these passages crowd out benign passages and shift the LLM's answer toward the attacker's stance. The paper introduces a polarization score (PS), defined as the projection of a passage embedding onto the first principal component of J=8 LLM-generated synthetic passages, and uses PS to select attack seeds, to formalize attack properties, and as the evaluation metric. It also proposes BiasDef, a post-retrieval filter that uses KL divergence over PS distributions and Mahalanobis-distance outlier detection to remove adversarial passages. The attack and defense are evaluated on WIKI-BALANCE, Reddit-Dialogues, and HotpotQA with Llama-3-8B, DeepSeek-R1-Distill-Qwen-14B, and GPT-4.1. The reported results claim that existing retrieval-based defenses leave over 19% adversarial passages in top-5 results, that BiasDef reduces adversarial retrieval by ~15% relative to the best baseline, and that BiasDef reduces answer PS shift by more than 6.2x compared to SMART.

Significance. If the PS metric is a valid measure of ideological stance, the paper identifies a realistic and subtle threat to RAG systems and a practical defense that is orthogonal to model-based defenses. The work is also significant for proposing a geometric, unsupervised operationalization of bias in retrieval, and the evaluation spans multiple models, datasets, and attack intensities. The fact-checking analysis is a useful positive control, and the artifact promise, if fulfilled, would support reproducibility. However, the significance is conditional: every headline number is a change along a PCA axis computed from 8 synthetic passages, and the paper never validates that axis against human stance judgments or an independent external label set. The defense, attack, and evaluation all rely on the same PS family, which creates a circularity risk that must be resolved before the quantitative claims can be accepted.

major comments (3)
  1. [Sec. 2.2, Eq. (2)-(3)] The polarization axis is the load-bearing construct of the paper, yet it is never validated. It is the first principal component of J=8 LLM-generated synthetic passages in a high-dimensional embedding space. With only 8 samples, this component can easily capture generation style or lexical artifacts of the prompting LLM rather than human-perceived viewpoint. The manuscript asserts that PCA 'empirically captures the most ideologically divergent dimension' but reports no correlation with human stance judgments, no use of WIKI-BALANCE's available perspective annotations, and no stability analysis across random draws of synthetic passages, different generation LLMs, or different embedding encoders. Since this axis is used for attack seed selection (Eq. 6), attack property definition (Sec. 2.3), BiasDef filtering (Sec. 5), and every PS-shift number in Tables 4-8, an unstable or invalid axis w
  2. [Secs. 4.2, 4.4, 5.1, Tables 5 and 8] The evaluation does not specify which polarization axis is used to compute the answer PS shift. The attacker computes its axis from J=8 synthetic passages using sentence-t5-xl, while BiasDef is described as recomputing PCA on the candidate passages (Sec. 5.1). If the answer PS shift in Tables 5 and 8 is computed along an axis produced by the same PCA procedure as the attack or the defense, then part of the reported 6.2x reduction is built into the construction: BiasDef removes points that are outliers in a space constructed to maximize separation, and success is then measured as reduced separation along a related axis. The paper must state how answer PS is computed in the evaluation and, ideally, re-measure attack success and defense benefit using an external stance ground truth or human labels. Without this, the reported improvement could be suppression of variance along a procedurally
  3. [Sec. 5.2 and Appendix D] BiasDef has several unreported free parameters that affect the quantitative claims. The Mahalanobis threshold T in Eq. (11) is selected via 'hyperparameter tuning based on validation performance,' but neither the tuning procedure nor the chosen values are given. The KL bin count m, the smoothing epsilon, the local-max window delta in Eq. (10), and the stopping criterion for the false-positive removal loop are also unspecified. Since the paper's headline results are quantitative (15% lower A-Recall, 6.2x PS-shift reduction), these values and a sensitivity analysis are necessary for reproducibility and for assessing whether the defense is robust or overfit to the reported benchmark.
minor comments (5)
  1. [Sec. 6.2] The claim that BiasDef outperforms No Def., MMR, SMART, and BRRA by 8.8x, 7.3x, 6.2x, and 8.2x does not match a simple average ratio computed from the values in Tables 5 and 8. Please clarify how the multipliers are computed.
  2. [Table 6 / Table 8] For HotpotQA the unattacked Avg.|PS| is 0.013 with an 85% relative uncertainty, and BiasDef's unattacked Avg.|PS| is reported as -0.015. Percentage shifts relative to a near-zero baseline are unstable; consider reporting absolute differences or a different normalization for this dataset.
  3. [Appendix C] The proof of Theorem 1 uses derivative sign arguments that are stated to hold 'when t_ss is near t*_ss' and handles log(0) via an epsilon smoothing. As written, the guarantee is asymptotic/approximate; stating the finite-sample conditions more precisely would strengthen the theoretical claim.
  4. [Sec. 4.5] The fact-checking result is a useful control, but the claim that adversarial passages are 'as factual or more' relies on a support-ratio comparison among claims deemed verifiable. Given the high fraction of 'NOT ENOUGH INFO' labels, the comparison is indirect; this is acknowledged, but a brief caveat in the main text would improve precision.
  5. [Throughout] Minor typos and infelicities: 'piratical' should be 'practical' (Sec. 5.2), 'retreival' in Sec. 3, and 'deviates from zero' (Sec. 5.2). Also, the phrase 'first largest maxima' (Sec. 5.2) should be reworded.

Circularity Check

1 steps flagged

Answer-bias metric is self-referential (PCA axis from 8 synthetic passages drives attack, defense, and evaluation), but A-Recall/Recall and fact-checking results are independent, so circularity is partial.

specific steps
  1. self definitional [Sec. 2.2 (Eqs. 2-4), Sec. 3 (steps 3-4), Sec. 4.2 (Eq. 4)]
    "To compute the polarization axis, we first generate J synthetic passages ... We then perform PCA on these embeddings to extract the polarization axis ... Each real passage's embedding is then projected onto this axis to compute a scalar polarization score PS ... The attacker selects a seed passage d_seed by choosing the passage with the highest or lowest PS ... The attacker then repeatedly queries the LLM to generate n adversarial passages ... that share the same perspective as the seed passage."

    PS is the paper's operationalization of 'semantic bias.' The attack chooses a seed with extreme PS along the PCA axis of J=8 LLM-generated synthetic passages and then generates adversarial passages to match that seed, so the injected passages are constructed to lie at one end of this axis. The attack's success and the defense's headline reduction are then measured as PS shift (Eq. 4) along the same (or, for BiasDef, a re-derived) polarization axis. Thus the headline answer-bias numbers partly measure movement along the very direction used to construct the attack, rather than an independently validated ideological stance dimension. The axis itself is never validated against human stance judgments or external labels, so the '6.2x reduction in answer bias' is partly an artifact of measuring a

full rationale

The paper's central threat and defense are largely self-contained: BiasDef re-derives its own PCA axis from candidate passages (Sec. 5.1), the adversarial passages are generated by an LLM rather than directly optimized to the metric, and the A-Recall@5 / Recall@5 results are independent of the PS construction. However, the PS metric itself is fitted to J=8 synthetic passages and then used as the ground-truth stance measure for attack seed selection, defense filtering, and evaluation of answer bias. Because the paper does not validate this axis against any external stance labels, the headline claims about 'perspective shift' and the 6.2x answer-bias reduction are partially self-referential: they demonstrate movement along an axis chosen by the same procedure used to measure success. This is a construct-validity concern as much as a circularity, but it is load-bearing for the answer-bias claims. The paper's own limitations (Sec. 6.3) acknowledge that the PS score captures only one polarizing dimension and does not fully capture human impact, which supports this moderate assessment rather than a higher circularity score.

Axiom & Free-Parameter Ledger

5 free parameters · 4 axioms · 0 invented entities

The paper introduces no new physical or ontological entities. Its novel constructs are mathematical procedures—the PCA-based polarization score and the BiasDef KL/Mahalanobis filter—not postulated objects requiring independent evidence. The free parameters are concentrated in the defense's threshold selection and the PCA sample size.

free parameters (5)
  • J (number of synthetic passages for PCA) = 8
    Sec. 3 step 1: the polarization axis is computed from only 8 LLM-generated passages, making the PCA axis potentially unstable and generator-dependent.
  • Mahalanobis threshold T = not reported (tuned on validation)
    Eq. (11): controls which low-similarity passages are recovered as adversarial false negatives; no value or sensitivity analysis is given, and the defense's headline reductions depend on it.
  • KL bin count m = not reported
    Eq. (8): PS distributions are discretized into m equal-width bins; m must be 'sufficiently large' but no value or stability study is provided.
  • KL smoothing epsilon = not reported
    Appendix D: added to avoid log(0) and log(infinity); changes the KL values and therefore the selected Max-KL boundary.
  • Local-max window delta = not reported
    Eq. (10): t*_ss is defined as a local KL maximum within a delta-neighborhood, but delta and the 'first largest maxima' heuristic are not quantified.
axioms (4)
  • domain assumption The first principal component of passage embeddings is the dominant ideological polarization axis.
    Sec. 2.2 Eq. (2)-(3): no external stance labels are used; the paper asserts the PCA direction 'captures the most ideologically divergent dimension' based on empirical observation.
  • ad hoc to paper Benign and adversarial passages are approximately separable in the SS-PS space under the attack's Property 1 and Property 2.
    Theorem 1 and Sec. 5.2 assume perfect separability; Table 1 shows only 74-78% of generated passages satisfy both properties, so the defense's false-positive/false-negative heuristics are needed to close the gap.
  • domain assumption Dense retrievers rank by cosine similarity, and LLM answers follow the retrieved context despite 'critical thinking' instructions.
    Sec. 2.1 and Sec. 4.4: the attack's success depends on the retriever preferring adversarial passages and the generator absorbing their viewpoint; this is empirically demonstrated but not theoretically grounded.
  • domain assumption LLM-generated biased passages are factually correct and linguistically indistinguishable from benign passages.
    Sec. 3 and Sec. 4.5: the attack requires the passages to pass fact-checking and plausibility; the WikiCheck comparison supports this for the tested generator but it is not guaranteed for other generators.

pith-pipeline@v1.3.0-alltime-deepseek · 24455 in / 11080 out tokens · 117721 ms · 2026-08-03T19:20:31.127959+00:00 · methodology

0 comments
read the original abstract

When answering user queries, LLMs often retrieve knowledge from external sources stored in retrieval-augmented generation (RAG) databases. These are often populated from unvetted sources, e.g. the open web, and can contain maliciously crafted data. This paper studies attacks that can manipulate the context retrieved by LLMs from such RAG databases. Prior work on such context manipulation primarily injects false or toxic content, which can often be detected by fact-checking or linguistic analysis. A more subtle threat, which we call epistemic bias injection (EBI), is where adversaries inject factually correct yet epistemically biased passages that systematically favor one side of an open-ended issue. Although linguistically coherent and truthful, such adversarial passages effectively crowd out alternative viewpoints during retrieval from the RAG and push LLM outputs towards an attack-desired stance. As a core contribution, we propose a novel characterization of the problem: We give a geometric metric that quantifies stance polarity and epistemic bias. This metric can be computed directly on embeddings of text passages. Leveraging it, we construct EBI attacks and develop a lightweight prototype defense called BiasDef for them. We evaluate them both on a comprehensive benchmark constructed from public question answering datasets. Our results show that: (1) the proposed attack induces significant stance polarity shifts, effectively evading existing retrieval-based sanitization defenses, and (2) BiasDef substantially reduces adversarial retrieval and epistemic bias in LLM's answers. Overall, this demonstrates the new threat as well as the ease of employing epistemic bias metrics for filtering in RAG-enabled LLMs.

Figures

Figures reproduced from arXiv: 2512.00804 by Hao Wu, Prateek Saxena.

Figure 1
Figure 1. Figure 1: Attack and defense in a typical RAG system. [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: An example from real LLM responses: When [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Illustration of state-of-the-art retrieval strategies in [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Both answer and retrieval PS values demonstrate [PITH_FULL_IMAGE:figures/full_fig_p008_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Fact-checking results: (a) Adversarial passages contain comparable proportions of sentences labeled as “NOT ENOUGH INFO” (0.69 and 0.68 on average for positive and negative bias injections, respectively), which are similar to that of benign passages (0.71). (b) Among verifiable claims, adversarial passages contain higher pro￾portions of sentences labeled as “SUPPORTS” (0.73 and 0.79 on average for positive… view at source ↗
Figure 6
Figure 6. Figure 6: Workflow of BiasDef. (a) The SS value space is searched to identify the Max-KL boundary [PITH_FULL_IMAGE:figures/full_fig_p010_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Comparison of BiasDef with baselines in A-Recall@5 and Recall@5. Solid and hollow markers [PITH_FULL_IMAGE:figures/full_fig_p012_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: CDF of BiasDef’s A-Recall@5. BiasDef com [PITH_FULL_IMAGE:figures/full_fig_p012_8.png] view at source ↗
Figure 10
Figure 10. Figure 10: In both procedures, the KL divergence are two [PITH_FULL_IMAGE:figures/full_fig_p017_10.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Personalization Meets Safety:Mechanisms,Risks,and Mitigations in Personalized LLMs

    cs.AI 2026-06 unverdicted novelty 5.0

    A survey that maps safety risks in personalized LLMs, introduces a unified taxonomy, and highlights three structural inadequacies in existing research on user-invariant safety, isolated techniques, and short-term evaluations.

Reference graph

Works this paper leans on

60 extracted references · 24 linked inside Pith · cited by 1 Pith paper

  1. [1]

    Meta llama,

    Meta AI, “Meta llama,” https://huggingface.co/meta-llama, 2025, ac- cessed June 2025

    2025

  2. [2]

    Gpt-4 technical report,

    OpenAI, J. Achiam, S. Adler, S. Agarwalet al., “Gpt-4 technical report,” 2024. [Online]. Available: https://arxiv.org/abs/2303.08774

    Pith/arXiv arXiv 2024

  3. [3]

    Deepseek- r1: Incentivizing reasoning capability in llms via reinforcement learning,

    DeepSeek-AI, D. Guo, D. Yang, H. Zhanget al., “Deepseek- r1: Incentivizing reasoning capability in llms via reinforcement learning,” 2025. [Online]. Available: https://arxiv.org/abs/2501.12948

    Pith/arXiv arXiv 2025

  4. [4]

    Towards expert-level medical question answering with large language models,

    K. Singhal, T. Tu, J. Gottweis, R. Sayreset al., “Towards expert-level medical question answering with large language models,” 2023. [Online]. Available: https://arxiv.org/abs/2305.09617

    Pith/arXiv arXiv 2023

  5. [5]

    Evaluating large language models trained on code,

    M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. D. O. Pinto, J. Ka- plan, H. Edwards, Y . Burda, N. Joseph, G. Brockmanet al., “Evaluating large language models trained on code,”arXiv preprint arXiv:2107.03374, 2021

    Pith/arXiv arXiv 2021

  6. [6]

    Github copilot: Your ai pair programmer,

    GitHub, “Github copilot: Your ai pair programmer,” https://github. com/features/copilot, 2021, accessed: July 2025

    2021

  7. [7]

    Cursor: The ai-first code editor,

    Cursor, “Cursor: The ai-first code editor,” https://www.cursor.so/, 2023, accessed: July 2025

    2023

  8. [8]

    Introducing Deep Research,

    OpenAI, “Introducing Deep Research,” https://openai.com/index/ introducing-deep-research/, Feb. 2025, accessed June 2025

    2025

  9. [9]

    Retrieval-augmented generation for knowledge-intensive nlp tasks,

    P. Lewis, E. Perez, A. Piktus, F. Petroniet al., “Retrieval-augmented generation for knowledge-intensive nlp tasks,”Advances in neural information processing systems, vol. 33, pp. 9459–9474, 2020

    2020

  10. [10]

    Poisonedrag: Knowledge corruption attacks to retrieval-augmented generation of large language models,

    W. Zou, R. Geng, B. Wang, and J. Jia, “Poisonedrag: Knowledge corruption attacks to retrieval-augmented generation of large language models,” 2024. [Online]. Available: https://arxiv.org/abs/2402.07867

    Pith/arXiv arXiv 2024

  11. [11]

    Badrag: Identifying vulnerabilities in retrieval augmented generation of large language models,

    J. Xue, M. Zheng, Y . Hu, F. Liu, X. Chen, and Q. Lou, “Badrag: Identifying vulnerabilities in retrieval augmented generation of large language models,” 2024. [Online]. Available: https: //arxiv.org/abs/2406.00083

    Pith/arXiv arXiv 2024

  12. [12]

    The good and the bad: Exploring privacy issues in retrieval-augmented generation (rag),

    S. Zeng, J. Zhang, P. He, Y . Xing, Y . Liu, H. Xu, J. Ren, S. Wang, D. Yin, Y . Chang, and J. Tang, “The good and the bad: Exploring privacy issues in retrieval-augmented generation (rag),”

  13. [13]

    Injecagent: Benchmarking indirect prompt injections in tool-integrated large language model agents,

    Q. Zhan, Z. Liang, Z. Ying, and D. Kang, “Injecagent: Benchmarking indirect prompt injections in tool-integrated large language model agents,” 2024. [Online]. Available: https://arxiv.org/abs/2403.02691

    Pith/arXiv arXiv 2024

  14. [14]

    Agentpoison: Red-teaming llm agents via poisoning memory or knowledge bases,

    Z. Chen, Z. Xiang, C. Xiao, D. Song, and B. Li, “Agentpoison: Red-teaming llm agents via poisoning memory or knowledge bases,”

  15. [15]

    Not what you’ve signed up for: Compromising real- world llm-integrated applications with indirect prompt injection,

    K. Greshake, S. Abdelnabi, S. Mishra, C. Endres, T. Holz, and M. Fritz, “Not what you’ve signed up for: Compromising real- world llm-integrated applications with indirect prompt injection,” in Proceedings of the 16th ACM workshop on artificial intelligence and security, 2023, pp. 79–90

    2023

  16. [16]

    Available: https://arxiv.org/abs/2407.12784

    [Online]. Available: https://arxiv.org/abs/2407.12784

    Pith/arXiv arXiv

  17. [17]

    Formalizing and benchmarking prompt injection attacks and defenses,

    Y . Liu, Y . Jia, R. Geng, J. Jia, and N. Z. Gong, “Formalizing and benchmarking prompt injection attacks and defenses,” in33rd USENIX Security Symposium (USENIX Security 24), 2024, pp. 1831– 1847

    2024

  18. [18]

    Machine-made media: Monitoring the mobilization of machine-generated articles on misinformation and mainstream news websites,

    H. W. A. Hanley and Z. Durumeric, “Machine-made media: Monitoring the mobilization of machine-generated articles on misinformation and mainstream news websites,” 2024. [Online]. Available: https://arxiv.org/abs/2305.09820

    Pith/arXiv arXiv 2024

  19. [19]

    Wikicheck: An end-to- end open source automatic fact-checking api based on wikipedia,

    M. Trokhymovych and D. Saez-Trumper, “Wikicheck: An end-to- end open source automatic fact-checking api based on wikipedia,” inProceedings of the 30th ACM International Conference on Information & Knowledge Management, ser. CIKM ’21. New York, NY , USA: Association for Computing Machinery, 2021, p. 4155–4164. [Online]. Available: https://doi.org/10.1145/3...

    doi:10.1145/3459637 2021

  20. [20]

    Datasentinel: A game-theoretic detection of prompt injection attacks,

    Y . Liu, Y . Jia, J. Jia, D. Song, and N. Z. Gong, “Datasentinel: A game-theoretic detection of prompt injection attacks,” in2025 IEEE Symposium on Security and Privacy (SP). IEEE, 2025, pp. 2190– 2208

    2025

  21. [21]

    Towards more robust retrieval-augmented generation: Evaluating rag under adversarial poisoning attacks,

    J. Su, J. P. Zhou, Z. Zhang, P. Nakov, and C. Cardie, “Towards more robust retrieval-augmented generation: Evaluating rag under adversarial poisoning attacks,” 2024. [Online]. Available: https://arxiv.org/abs/2412.16708

    Pith/arXiv arXiv 2024

  22. [22]

    Real ai agents with fake memories: Fatal context manipulation attacks on web3 agents,

    A. S. Patlan, P. Sheng, S. A. Hebbar, P. Mittal, and P. Viswanath, “Real ai agents with fake memories: Fatal context manipulation attacks on web3 agents,” 2025. [Online]. Available: https://arxiv.org/ abs/2503.16248

    Pith/arXiv arXiv 2025

  23. [23]

    Exploring the escalation of source bias in user, data, and recommender system feedback loop,

    Y . Zhou, S. Dai, L. Pang, G. Wang, Z. Dong, J. Xu, and J.-R. Wen, “Exploring the escalation of source bias in user, data, and recommender system feedback loop,” 2025. [Online]. Available: https://arxiv.org/abs/2405.17998

    Pith/arXiv arXiv 2025

  24. [24]

    Invisible relevance bias: Text-image retrieval models prefer ai-generated images,

    S. Xu, D. Hou, L. Pang, J. Deng, J. Xu, H. Shen, and X. Cheng, “Invisible relevance bias: Text-image retrieval models prefer ai-generated images,” 2024. [Online]. Available: https: //arxiv.org/abs/2311.14084

    Pith/arXiv arXiv 2024

  25. [25]

    The use of mmr, diversity-based reranking for reordering documents and producing summaries,

    J. Carbonell and J. Goldstein, “The use of mmr, diversity-based reranking for reordering documents and producing summaries,” inProceedings of the 21st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, ser. SIGIR ’98. New York, NY , USA: Association for Computing Machinery, 1998, p. 335–336. [Online]. Available...

    arXiv 1998

  26. [26]

    Bias amplification in rag: Poisoning knowledge retrieval to steer llms,

    L. Wang, T. Zhu, L. Qin, L. Gao, and W. Zhou, “Bias amplification in rag: Poisoning knowledge retrieval to steer llms,” 2025. [Online]. Available: https://arxiv.org/abs/2506.11415

    Pith/arXiv arXiv 2025

  27. [27]

    Through the stealth lens: Rethinking attacks and defenses in rag,

    S. Choudhary, N. Palumbo, A. Hooda, K. D. Dvijotham, and S. Jha, “Through the stealth lens: Rethinking attacks and defenses in rag,”

  28. [28]

    Smart-rag: Selection using determinantal matrices for augmented retrieval,

    J. Li, X. Hu, and X. Wan, “Smart-rag: Selection using determinantal matrices for augmented retrieval,” 2024. [Online]. Available: https://arxiv.org/abs/2409.13992

    Pith/arXiv arXiv 2024

  29. [29]

    Hotpotqa: A dataset for diverse, explainable multi-hop question answering,

    Z. Yang, P. Qi, S. Zhang, Y . Bengio, W. W. Cohen, R. Salakhutdinov, and C. D. Manning, “Hotpotqa: A dataset for diverse, explainable multi-hop question answering,”arXiv preprint arXiv:1809.09600, 2018

    Pith/arXiv arXiv 2018

  30. [30]

    A dataset for sentence retrieval for open-ended dialogues,

    I. Harel, H. Taitelbaum, I. Szpektor, and O. Kurland, “A dataset for sentence retrieval for open-ended dialogues,” inProceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval, ser. SIGIR ’22. New York, NY , USA: Association for Computing Machinery, 2022, p. 2960–2969. [Online]. Available: https://doi.o...

    doi:10.1145/3477495 2022

  31. [31]

    Measuring and addressing indexical bias in information retrieval,

    C. Ziems, W. Held, J. Dwivedi-Yu, and D. Yang, “Measuring and addressing indexical bias in information retrieval,” inFindings of the Association for Computational Linguistics ACL 2024, 2024, pp. 12 860–12 877

    2024

  32. [32]

    The faiss library,

    M. Douze, A. Guzhva, C. Deng, J. Johnson, G. Szilvasy, P.-E. Mazar´e, M. Lomeli, L. Hosseini, and H. J ´egou, “The faiss library,”

  33. [33]

    Retrieval-augmented generation for knowledge-intensive nlp tasks,

    P. Lewis, E. Perez, A. Piktus, F. Petroni, V . Karpukhin, N. Goyal, H. K ¨uttler, M. Lewis, W.-t. Yih, T. Rockt ¨aschel, S. Riedel, and D. Kiela, “Retrieval-augmented generation for knowledge-intensive nlp tasks,” inAdvances in Neural Information Processing Systems, H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin, Eds., vol. 33. Curran Associ...

    2020

  34. [34]

    Sentence-bert: Sentence embeddings using siamese bert-networks,

    N. Reimers and I. Gurevych, “Sentence-bert: Sentence embeddings using siamese bert-networks,” 2019. [Online]. Available: https: //arxiv.org/abs/1908.10084

    Pith/arXiv arXiv 2019

  35. [35]

    Bias and unfairness in information retrieval systems: New challenges in the llm era,

    S. Dai, C. Xu, S. Xu, L. Pang, Z. Dong, and J. Xu, “Bias and unfairness in information retrieval systems: New challenges in the llm era,” inProceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, ser. KDD ’24. ACM, Aug. 2024, p. 6437–6447. [Online]. Available: http://dx.doi.org/10.1145/3637528.3671458

    arXiv 2024

  36. [36]

    Available: https://arxiv.org/abs/2401.08281

    [Online]. Available: https://arxiv.org/abs/2401.08281

    Pith/arXiv arXiv

  37. [37]

    Neural retrievers are biased towards llm-generated content,

    S. Dai, Y . Zhou, L. Pang, W. Liu, X. Hu, Y . Liu, X. Zhang, G. Wang, and J. Xu, “Neural retrievers are biased towards llm-generated content,” inProceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, ser. KDD ’24. ACM, Aug. 2024, p. 526–537. [Online]. Available: http://dx.doi.org/10.1145/3637528.3671882

    arXiv 2024

  38. [38]

    Large dual encoders are generalizable retrievers,

    J. Ni, C. Qu, J. Lu, Z. Dai, G. Hernandez Abrego, J. Ma, V . Zhao, Y . Luan, K. Hall, M.-W. Chang, and Y . Yang, “Large dual encoders are generalizable retrievers,” inProceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, Y . Goldberg, Z. Kozareva, and Y . Zhang, Eds. Abu Dhabi, United Arab Emirates: Association for Comput...

    2022

  39. [40]

    Principal component analysis: A natural approach to data exploration,

    F. L. Gewers, G. R. Ferreira, H. F. D. Arruda, F. N. Silva, C. H. Comin, D. R. Amancio, and L. D. F. Costa, “Principal component analysis: A natural approach to data exploration,”ACM Computing Surveys, vol. 54, no. 4, p. 1–34, May 2021. [Online]. Available: http://dx.doi.org/10.1145/3447755

    doi:10.1145/3447755 2021

  40. [41]

    Vershynin,High-Dimensional Probability: An Introduction with Applications in Data Science, ser

    R. Vershynin,High-Dimensional Probability: An Introduction with Applications in Data Science, ser. Cambridge Series in Statistical and Probabilistic Mathematics. Cambridge University Press, 2018

    2018

  41. [42]

    The curious case of neural text degeneration,

    A. Holtzman, J. Buys, L. Du, M. Forbes, and Y . Choi, “The curious case of neural text degeneration,”arXiv preprint arXiv:1904.09751, 2019

    Pith/arXiv arXiv 1904

  42. [43]

    meta-llama/meta-llama-3-8b,

    Meta AI, “meta-llama/meta-llama-3-8b,” https://huggingface.co/ meta-llama/Meta-Llama-3-8B, 2024, accessed: 2025-07-23

    2024

  43. [44]

    Determinantal point processes for machine learning,

    A. Kulesza, “Determinantal point processes for machine learning,” Foundations and Trends® in Machine Learning, vol. 5, no. 2–3, p. 123–286, 2012. [Online]. Available: http://dx.doi.org/10.1561/ 2200000044

    2012

  44. [45]

    Efficient memory management for large language model serving with pagedattention,

    W. Kwon, Z. Li, S. Zhuang, Y . Sheng, L. Zheng, C. H. Yu, J. Gonzalez, H. Zhang, and I. Stoica, “Efficient memory management for large language model serving with pagedattention,” inProceedings of the 29th Symposium on Operating Systems Principles, ser. SOSP ’23. New York, NY , USA: Association for Computing Machinery, 2023, p. 611–626. [Online]. Availabl...

    arXiv 2023

  45. [46]

    sentence-transformers/msmarco- distilbert-base-tas-b,

    Sentence-Transformers, “sentence-transformers/msmarco- distilbert-base-tas-b,” https://huggingface.co/sentence-transformers/ msmarco-distilbert-base-tas-b, 2020, accessed: 2025-07-23

    2020

  46. [47]

    Dense passage retrieval for open- domain question answering,

    V . Karpukhin, B. O ˘guz, S. Min, P. Lewis, L. Wu, S. Edunov, D. Chen, and W. tau Yih, “Dense passage retrieval for open- domain question answering,” 2020. [Online]. Available: https: //arxiv.org/abs/2004.04906

    Pith/arXiv arXiv 2020

  47. [48]

    deepseek-ai/deepseek-r1-distill-qwen-14b,

    DeepSeek AI, “deepseek-ai/deepseek-r1-distill-qwen-14b,” https:// huggingface.co/deepseek-ai/DeepSeek-R1-Distill-Qwen-14B, 2024, accessed: 2025-07-23

    2024

  48. [49]

    Mahalanobis distance,

    G. J. McLachlan, “Mahalanobis distance,”Resonance, vol. 4, no. 6, pp. 20–26, 1999

    1999

  49. [50]

    sentence-transformers/sentence-t5-xl,

    Sentence-Transformers, “sentence-transformers/sentence-t5-xl,” https://huggingface.co/sentence-transformers/sentence-t5-xl, 2024, accessed: 2025-10-23

    2024

  50. [51]

    A practical and secure byzantine robust aggregator,

    D. Z. Lee, A. Kolluri, P. Saxena, and E.-C. Chang, “A practical and secure byzantine robust aggregator,” inProceedings of the ACM SIGSAC conference on computer and communications security (CCS), 2025

    2025

  51. [52]

    On Information and Sufficiency,

    S. Kullback and R. A. Leibler, “On Information and Sufficiency,” The Annals of Mathematical Statistics, vol. 22, no. 1, pp. 79 – 86,

  52. [53]

    Whispers in grammars: Injecting covert backdoors to compromise dense retrieval systems,

    Q. Long, Y . Deng, L. Gan, W. Wang, and S. J. Pan, “Whispers in grammars: Injecting covert backdoors to compromise dense retrieval systems,” 2024. [Online]. Available: https://arxiv.org/abs/2402.13532

    Pith/arXiv arXiv 2024

  53. [54]

    Attacks against abstractive text sum- marization models through lead bias and influence functions,

    P. Thota and S. Nilizadeh, “Attacks against abstractive text sum- marization models through lead bias and influence functions,” in Findings of the Association for Computational Linguistics: EMNLP 2024, 2024, pp. 13 727–13 741

    2024

  54. [55]

    A simple unified framework for detecting out-of-distribution samples and adversarial attacks,

    K. Lee, K. Lee, H. Lee, and J. Shin, “A simple unified framework for detecting out-of-distribution samples and adversarial attacks,” in Proceedings of the 32nd International Conference on Neural Infor- mation Processing Systems, ser. NIPS’18. Red Hook, NY , USA: Curran Associates Inc., 2018, p. 7167–7177

    2018

  55. [56]

    The probabilistic relevance framework: Bm25 and beyond,

    S. Robertson and H. Zaragoza, “The probabilistic relevance framework: Bm25 and beyond,”Foundations and Trends® in Information Retrieval, vol. 3, no. 4, pp. 333–389, 2009. [Online]. Available: http://dx.doi.org/10.1561/1500000019 Appendix A. Examples Table 9 shows examples of bias injection attacks and discusses their associated risks and impacts in practi...

    doi:10.1561/1500000019 2009

  56. [57]

    How not to detect prompt injections with an llm,

    S. Choudhary, D. Anshumaan, N. Palumbo, and S. Jha, “How not to detect prompt injections with an llm,”arXiv preprint arXiv:2507.05630, 2025

    arXiv 2025

  57. [60]

    Beyond relevance: Evaluate and improve retrievers on perspective awareness,

    X. Zhao, T. Chen, S. Chen, H. Zhang, and T. Wu, “Beyond relevance: Evaluate and improve retrievers on perspective awareness,” 2024. [Online]. Available: https://arxiv.org/abs/2405.02714

    Pith/arXiv arXiv 2024

  58. [1951]

    Available: https://doi.org/10.1214/aoms/1177729694

    [Online]. Available: https://doi.org/10.1214/aoms/1177729694

    arXiv

  59. [2024]

    Available: https://arxiv.org/abs/2402.16893

    [Online]. Available: https://arxiv.org/abs/2402.16893

    Pith/arXiv arXiv

  60. [2025]

    Available: https://arxiv.org/abs/2506.04390

    [Online]. Available: https://arxiv.org/abs/2506.04390

    Pith/arXiv arXiv