Pith. sign in

REVIEW 4 major objections 5 minor 94 references

Eliciting Intrinsic Hallucinations in LLMs via Semantically Equivalent Adversarial Attacks

T0 review · 4 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Meaning-preserving rewrites of a question can flip a RAG model's faithful answer into a hallucination.

desk verdict A worthwhile benchmark with a crux problem: the equivalence gate admits meaning-changing rewrites and ASR counts refusals as hallucinations, so the headline 50% figure is inflated even though a real fragility likely remains. read the letter →

arxiv 2608.04286 v1 pith:DUDZVBRZ submitted 2026-08-04 cs.CL cs.LG

Atri Vivek Sharma , Brian Formento , Alessio Lomuscio This is my paper · ORCID
classification cs.CLcs.LG
keywords intrinsichallucinationretrieval-augmentedgenerationadversarialattacksemanticequivalencefaithfulnessevaluationLLMrobustnessqueryperturbationblack-box
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that retrieval-augmented language models are not reliably faithful to their retrieved evidence: simply rewording a user's question into a semantically equivalent form can make the model refuse, fabricate, or contradict the context. It offers this as a lower bound on robustness under meaning-preserving perturbations. To make the point, the authors adapt five attack methods to generate query variants that pass a strict bidirectional-entailment check, then measure how often originally correct answers become unfaithful. Across ten models and three datasets, the attacks succeed on a substantial fraction of cases, including on the strongest closed models. If the finding holds, it shows that faithful grounding is tied to the surface form of the query, not just to the evidence supplied.

What carries the argument

The load-bearing object is a semantic equivalence predicate, $\operatorname{Eq}(q,q') := \operatorname{Entail}(q\Rightarrow q') \land \operatorname{Entail}(q'\Rightarrow q)$, implemented by an LLM judge. It defines the admissible attack set and gates every reported success, so the paper's interpretation of 'meaning-preserving' rests on this predicate. During search, a cheaper sentence-embedding cosine similarity threshold stands in for the judge; at evaluation, only perturbations passing mutual entailment count. The objective being maximized is the expected intrinsic-hallucination score of the generator conditioned on fixed context, and the threat model assumes a perfect retriever so all failures are attributed to the generator.

What would settle it

Have independent human annotators, blind to the intended result, classify whether each reported adversarial query is mutually entailed by the original query, and recompute attack success only on pairs all annotators accept; if the human-accepted subset shrinks the success rates toward zero, the central claim is refuted, while surviving high rates would confirm it.

Watch

Extended reading notes

Core claim

The central claim is that intrinsic hallucinations in RAG can be provoked by perturbations that preserve the query's meaning, with no changes to the retrieved context. The paper formalizes meaning preservation as mutual entailment between original and adversarial queries, computes attacks under white-box, gray-box, and black-box access, and evaluates them on five open and five closed models across three datasets. Under the strongest black-box attack, originally correct answers become unfaithful in over half of cases on some datasets; the headline number is a drop in contextual faithfulness of up to 50% for a state-of-the-art small closed model. The authors read this as establishing a low lower bound on RAG faithfulness: even a fixed, ideal retriever and a fluent, intent-preserving query do not guarantee a grounded answer.

Load-bearing premise

The load-bearing premise is that the LLM judge's mutual-entailment verdict is a correct test of semantic equivalence; if it accepts rewrites that change the question, the reported attacks are not meaning-preserving and the claimed lower bound does not follow.

Editorial extensions

If this is right

  • RAG faithfulness cannot be inferred from clean accuracy: the same context, retriever, and intended query can produce faithful or unfaithful answers depending on phrasing.
  • Token-level optimizers are weak attack vectors because they produce unnatural, high-perplexity queries; LLM-generated rephrasings and short suffix additions are the effective ones.
  • Scaling does not remove the vulnerability: larger open models and frontier closed models show comparable or larger attack success rates.
  • The dominant black-box failure modes differ from white-box ones: reasoning errors replace abstention when the attacker cannot use an 'Unable to answer' target.
  • Any robustness claim for a RAG system should be reported against a paraphrase-sensitive attack set, not a single query form.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: the judge-based equivalence gate is the crux, so the practical strength of the result depends on how lenient or strict the gate is for real human users.
  • Editorial inference: the perfect-retriever assumption isolates generator fragility; real pipelines with noisy retrieval may show different or larger degradation.
  • Testable extension: training models to be invariant to paraphrase, for instance by adversarially rephrasing queries during fine-tuning, could be measured directly with this attack suite.
  • Testable next step: applying the same semantic-equivalence attacks in multi-turn RAG would show whether conversational context anchors faithfulness or adds new failure modes.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes a framework for eliciting intrinsic hallucinations in RAG by adversarially perturbing the user query under a semantic-equivalence constraint. It adapts GCG, AutoDAN, PAIR, SRA, and SECA-GB/SECA-BB to a hallucination-induction objective, evaluates them on 5 open and 5 closed models over FaithEval, ANAH-v2, and FailSafeQA, and reports Attack Success Rates that are often large; the headline example is GPT-5-mini on FaithEval, where SECA-BB reduces Adversarial Accuracy from 0.72 to 0.22 (ASR 0.703). The automated evaluation uses Gemini-2.5-Flash-Lite as both the semantic-equivalence judge and the faithfulness judge, with a 150-example human validation study.

Significance. If the perturbations counted as successful really are semantically equivalent, the paper establishes a practically important lower bound: faithfulness of RAG generators to a fixed context is highly sensitive to benign-looking query reformulation. The paper has real strengths: code is released, the cosine-threshold ablation (Table 6) shows the final gate is doing nontrivial filtering, and the self-preference check (Table 8) addresses one obvious confound. However, the central claim depends on two load-bearing assumptions that the current evidence does not fully support: the reliability of the Eq. (3) mutual-entailment gate on the adversarial distribution, and the interpretation of 'Unable to answer' responses as intrinsic hallucinations.

major comments (4)
  1. [§3, Eq. (3); Appendix A] The semantic-equivalence gate is the load-bearing component of the paper, but its precision on the adversarial distribution is never reported. Appendix A provides concrete gate-passing rewrites that change meaning: A.2 changes 'total number of shares' to 'total number per share', which the paper's own rubric in C.2 explicitly uses as a non-equivalence example (percentage shares vs. ratios), and A.1/A.3 show attacks that append a bare 'A' or '(a' to convert open-ended questions into multiple-choice format, changing the expected answer. The human validation in Table 7 samples 150 random examples and does not condition on the adversarial, gate-passing queries that drive the headline ASR. The authors should report precision of the Eq. (3) judge on the gate-passing adversarial outputs, or re-run the evaluation with a stricter gate and show that the headline numbers are stable.
  2. [Table 3; §C.1; abstract] The failure-mode analysis in Table 3 shows that 22–56% of successful attacks are abstentions ('Unable to answer'). Under the paper's own hallucination rubric (C.1), a response that appropriately says the context is insufficient should receive 'proper uncertainty' = true and should not be counted as an intrinsic hallucination. Counting abstention as a success therefore inflates the ASR and contradicts the abstract's characterization of the failures as 'unfaithful or fabricated information.' The paper should report ASR with abstentions excluded (or at least report the fabrication-only ASR), and the abstract and conclusions should be revised to distinguish evasive responses from fabricated content.
  3. [§3, Eq. (6); §4; §C.1] PAIR and SECA-BB optimize against the same hallucination judge that is later used to compute AA and ASR. This creates a partial circularity: the optimizer can exploit systematic quirks of the judge's rubric rather than produce genuinely unfaithful responses. The human validation does not close this gap because it is a random sample and is not stratified by the adversarial, gate-passing queries selected by the optimizer. The authors should validate the judge on the actual set of successful adversarial outputs (or use a held-out judge) and report the agreement on that subset.
  4. [§3 (SRA paragraph); Table 1; §5] SRA is explicitly described as not strictly comparable to the paraphrase-based attacks because it appends tokens rather than rephrasing the query, yet it is reported in Table 1, highlighted as achieving the highest ASR, and may contribute to the conclusion that ASR exceeds 50% for state-of-the-art models. The paper should either move SRA to the appendix or clearly separate all SRA-based numbers from the headline claims, and the 'over 50%' statement should be recomputed without SRA.
minor comments (5)
  1. [§3, Threat model paragraph] The paragraph contains an exact duplicated sentence: 'Under this threat model, the adversary seeks...' appears twice.
  2. [Appendix A.2] The text refers to a 'K-10 filing' in the adversarial query; the correct term is '10-K filing'.
  3. [Footnote 1 and abstract] The GitHub URL is given as 'https://github.com/atriviveksharma/intrinsic hall' with a space; the URL should be 'intrinsic_hall' (or the correct repository name) and typeset as a link.
  4. [§4, ASR definition] The ASR definition includes the semantic-equivalence indicator in the numerator but not in the denominator; this is presumably intentional, but the notation should be clarified to avoid confusion about the conditioning set.
  5. [Table 1] The perplexity values are unevenly formatted (e.g., 17140, 535, 65.2) and the table is difficult to read; consider a log-scale column or more consistent formatting.

Circularity Check

1 steps flagged · score 4.0 of 10

PAIR and SECA-BB optimize the same hallucination judge later used to compute AA/ASR, so the headline degradation is partly self-measured; the semantic-equivalence gate also admits meaning-changing rewrites in Appendix A.

  1. fitted input called prediction [Section 3 (SECA-BB and PAIR adaptations) and Section 4 (AA/ASR metrics); Eq. (6)-(7); Tables 1-2]
    "In this mode, the zeroth-order optimization algorithm selects the semantically valid candidate that maximizes the discrete integer score provided by the hallucination rubric at each iteration."

    The hallucination rubric referenced here is the same five-criteria rubric (Appendix C.1) that defines the faithfulness judge J used in the AA and ASR metrics in Section 4 ('We use Gemini-2.5-Flash-Lite ... as the judge LLM for both faithfulness and semantic equivalence'). PAIR likewise uses the same S_hall score in its reward R=S_hall−λ(1−S_eq). Thus the black-box attacks that produce the headline numbers (e.g., GPT-5-mini on FaithEval: CA 0.720, AA 0.220, ASR 0.703) are optimized directly against the evaluation metric. The reported 'degradation' is therefore the optimizer's success on its own objective, not an independent measure of hallucination; the human validation in Table 7 samples 150 random examples and does not re-measure the adversarial tail that drives the headline.

full rationale

The paper is a benchmark/evaluation study on external datasets (FaithEval, ANAH-v2, FailSafeQA) with external generator models, and there is no load-bearing self-citation chain. The main circularity concern is that PAIR and SECA-BB are optimized against the same LLM hallucination judge that later computes AA/ASR, which inflates the headline effect; this is a partial evaluation circularity. Independent human validation (Cohen's kappa 0.680 and 0.639 for semantic equivalence; 0.799 and 0.860 for hallucination detection) provides some external grounding, so the central claim is not fully forced. Separately, the load-bearing assumption that Eq. (3)'s mutual-entailment gate reliably implements 'meaning-preserving' is not established on the adversarial distribution: Appendix A shows gate-passing rewrites that change the task (SRA appends a bare 'A', turning an open question into a multiple-choice prompt) and the paper asserts SRA equivalence without applying Eq. (3) ('We treat this as semantically equivalent as the suffix introduces no additional task content or assumptions'). This is a correctness risk rather than a circularity, but it further weakens the 'meaning-preserving' characterization. Since the derivation chain is otherwise self-contained and externally benchmarked, the overall circularity is moderate.

Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

The central claim rests on the fixed-retriever idealization, the reliability of two LLM judges, and a cosine threshold fitted on evaluation data. No new physical or architectural entities are introduced; the judges are components, not invented entities.

free parameters (2)
  • Cosine similarity threshold tau = 0.990
    Search-time proxy for semantic equivalence; chosen in Appendix D.1 because it maximized final gated ASR on the same FaithEval dataset used for evaluation.
  • Equivalence penalty weight lambda in PAIR reward = not reported
    Large scalar hand-chosen to penalize non-equivalent queries; exact value not stated.
assumptions (3)
  • domain assumption Perfect and fixed retriever R(q') = R(q) = c for all admissible perturbations (Eq. 2)
    The paper assumes the adversarial query retrieves the same context; results are conditional on retrieval being invariant, which may not hold in deployed RAG systems.
  • domain assumption The LLM-based mutual entailment judge reliably implements semantic equivalence (Eq. 3)
    The whole method and the reported ASR depend on this judge being accurate; examples in Appendix A suggest it can pass meaning-changing rewrites.
  • ad hoc to paper The hallucination judge's binary rubric correctly identifies intrinsic hallucinations, including 'Unable to answer' responses
    The rubric instructs not to penalize answers that say the context is insufficient, yet abstention responses are counted as attack successes, creating a tension.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Eliciting Intrinsic Hallucinations in LLMs via Semantically Equivalent Adversarial Attacks." pith.science (2026). https://pith.science/paper/DUDZVBRZ

@misc{pith2026260804286,
  author       = {Pith},
  title        = {Pith review of: Eliciting Intrinsic Hallucinations in LLMs via Semantically Equivalent Adversarial Attacks},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/DUDZVBRZ}},
  note         = {Machine review of arXiv:2608.04286}
}
read the original abstract

Large language models (LLMs) are often used in conjunction with external knowledge sources to improve their factual accuracy and decrease hallucinations, through methods such as Retrieval-Augmented Generation (RAG). However, these systems remain susceptible to intrinsic hallucinations, where the model generates unfaithful or fabricated information that is not supported by the retrieved evidence. We propose a novel framework to assess model robustness against this phenomenon by stress-testing using natural, semantically equivalent variations of a user query found via adversarial optimization methods. We apply our framework, which enforces strict semantic equivalence constraints and an intrinsic hallucination objective, to a range of adversarial attack techniques across white-box, gray-box, and black-box adversarial settings. Evaluating these attacks on 5 open-source and 5 closed-source generator models across 3 datasets, we demonstrate that even state-of-the-art models are highly susceptible to meaning-preserving perturbations, which significantly degrade contextual faithfulness (by up to 50% for GPT-5-mini). Our findings indicate that faithful use of in-context evidence remains fragile even in state-of-the-art LLMs, motivating architectures and training objectives that enforce robust grounding independent of surface query form. Code is available at: https://github.com/atriviveksharma/intrinsic_hall

Figures

Figures reproduced from arXiv: 2608.04286 by the authors.

Figure 1
Figure 1. Semantically equivalent adversarial attacks can induce intrinsic hallucinations, [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

94 extracted references · 63 canonical work pages

  1. [1]

    Does safety training of llms generalize to semantically related natural prompts? In Proceedings of the 13th International Conference on Learning Representations (ICLR25)

    Sravanti Addepalli, Yerram Varun, Arun Suggala, Karthikeyan Shanmugam, and Prateek Jain. Does safety training of llms generalize to semantically related natural prompts? In Proceedings of the 13th International Conference on Learning Representations (ICLR25). OpenReview.net, 2025

    2025

  2. [2]

    Business insights using rag--llms: a review and case study

    Muhammad Arslan, Saba Munawar, and Christophe Cruz. Business insights using rag--llms: a review and case study. Journal of Decision Systems, pp.\ 1--30, 2024

    2024

  3. [3]

    Self- RAG : Learning to retrieve, generate, and critique through self-reflection

    Akari Asai, Zeqiu Wu, Yizhong Wang, Avirup Sil, and Hannaneh Hajishirzi. Self- RAG : Learning to retrieve, generate, and critique through self-reflection. In Proceedings of the 12th International Conference on Learning Representations (ICLR24), pp.\ 9112--9141. OpenReview.net, 2024

    2024

  4. [4]

    H allu L ens: LLM hallucination benchmark

    Yejin Bang, Ziwei Ji, Alan Schelten, Anthony Hartshorn, Tara Fowler, Cheng Zhang, Nicola Cancedda, and Pascale Fung. H allu L ens: LLM hallucination benchmark. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (ACL 2025 Volume 1: Long Papers), pp.\ 24128--24156. Association for Computational Linguistics, 2025

    2025

  5. [6]

    Jailbreaking black box large language models in twenty queries

    Patrick Chao, Alexander Robey, Edgar Dobriban, Hamed Hassani, George J Pappas, and Eric Wong. Jailbreaking black box large language models in twenty queries. In Proceedings of the IEEE Conference on Secure and Trustworthy Machine Learning ( SaTML 25) , pp.\ 23--42. IEEE, 2025

    2025

  6. [7]

    Typos that broke the rag's back: Genetic attack on rag pipeline by simulating documents in the wild via low-level perturbations

    Sukmin Cho, Soyeong Jeong, Jeongyeon Seo, Taeho Hwang, and Jong C Park. Typos that broke the rag's back: Genetic attack on rag pipeline by simulating documents in the wild via low-level perturbations. In Findings of the Association for Computational Linguistics: EMNLP 2024, pp.\ 2826--2844. Association for Computational Linguistics, 2024

    2024

  7. [9]

    H ot F lip: White-box adversarial examples for text classification

    Javid Ebrahimi, Anyi Rao, Daniel Lowd, and Dejing Dou. H ot F lip: White-box adversarial examples for text classification. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (ACL 2018 Volume 2: Short Papers), pp.\ 31--36. Association for Computational Linguistics, 2018

    2018

  8. [10]

    a ger, MHI Abdalla, Vincent Cohen-Addad, Johannes Gasteiger, and Stephan G \

    Simon Geisler, Tom Wollschl \"a ger, MHI Abdalla, Vincent Cohen-Addad, Johannes Gasteiger, and Stephan G \"u nnemann. Reinforce adversarial attacks on large language models: An adaptive, distributional, and semantic objective. In Proceedings of the 42nd International Conference on Machine Learning (ICML25), pp.\ 19082--19111. PMLR , 2025

    2025

Show all 94 references
  1. [11]

    \ Topic-FlipRAG \ : \ Topic-Orientated \ adversarial opinion manipulation attacks to \ Retrieval-Augmented \ generation models

    Yuyang Gong, Zhuo Chen, Jiawei Liu, Miaokun Chen, Fengchang Yu, Wei Lu, XiaoFeng Wang, and Xiaozhong Liu. \ Topic-FlipRAG \ : \ Topic-Orientated \ adversarial opinion manipulation attacks to \ Retrieval-Augmented \ generation models. In Proceedings of the 34th USENIX Security ...

    2025

  2. [13]

    Anah-v2: Scaling analytical hallucination annotation of large language models

    Yuzhe Gu, Ziwei Ji, Wenwei Zhang, Chengqi Lyu, Dahua Lin, and Kai Chen. Anah-v2: Scaling analytical hallucination annotation of large language models. In Proceedings of the 38th Conference on Advances in Neural Information Processing Systems (NeurIPS24), pp.\ 60012--60039. Cur...

    2024

  3. [14]

    Parammute: Suppressing knowledge-critical ffns for faithful retrieval-augmented generation

    Pengcheng Huang, Zhenghao Liu, Yukun Yan, Haiyan Zhao, Xiaoyuan Yi, Hao Chen, Zhiyuan Liu, Maosong Sun, Tong Xiao, Ge Yu, et al. Parammute: Suppressing knowledge-critical ffns for faithful retrieval-augmented generation. In Advances in Neural Information Processing Systems (Ne...

    2025

  4. [15]

    Is bert really robust? a strong baseline for natural language attack on text classification and entailment

    Di Jin, Zhijing Jin, Joey Tianyi Zhou, and Peter Szolovits. Is bert really robust? a strong baseline for natural language attack on text classification and entailment. In Proceedings of the 34th AAAI Conference on Artificial Intelligence (AAAI20), pp.\ 8018--8025. AAAI Press, 2020

    2020

  5. [18]

    Summary of a haystack: A challenge to long-context LLM s and RAG systems

    Philippe Laban, Alexander Fabbri, Caiming Xiong, and Chien-Sheng Wu. Summary of a haystack: A challenge to long-context LLM s and RAG systems. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing (EMNLP 2024), pp.\ 9885--9903. Association f...

    2024

  6. [19]

    u ttler, Mike Lewis, Wen-tau Yih, Tim Rockt \

    Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich K \"u ttler, Mike Lewis, Wen-tau Yih, Tim Rockt \"a schel, et al. Retrieval-augmented generation for knowledge-intensive nlp tasks. In Advances in Neural Information Process...

    2020

  7. [20]

    Semantic representation attack against aligned large language models

    Jiawei Lian, Jianhong Pan, Lefan Wang, Yi Wang, Shaohui Mei, and Lap-Pui Chau. Semantic representation attack against aligned large language models. In Proceedings of the 39th Conference on Advances in Neural Information Processing Systems (NeurIPS25). Curran Associates, Inc., 2025

    2025

  8. [21]

    Seca: Semantically equivalent and coherent attacks for eliciting llm hallucinations

    Buyun Liang, Liangzu Peng, Jinqi Luo, Darshan Thaker, Kwan Ho Ryan Chan, and Rene Vidal. Seca: Semantically equivalent and coherent attacks for eliciting llm hallucinations. In Proceedings of the 39th Conference on Advances in Neural Information Processing Systems (NeurIPS25)....

    2025

  9. [22]

    Autodan: Generating stealthy jailbreak prompts on aligned large language models

    Xiaogeng Liu, Nan Xu, Muhao Chen, and Chaowei Xiao. Autodan: Generating stealthy jailbreak prompts on aligned large language models. In Proceedings of the 12th International Conference on Learning Representations (ICLR24). OpenReview.net, 2024

    2024

  10. [23]

    Query rewriting in retrieval-augmented large language models

    Xinbei Ma, Yeyun Gong, Pengcheng He, Hai Zhao, and Nan Duan. Query rewriting in retrieval-augmented large language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP 2023), pp.\ 5303--5315. Association for Computational Lin...

    2023

  11. [24]

    F act E val: Evaluating the robustness of fact verification systems in the era of large language models

    Mamta and Oana Cocarascu. F act E val: Evaluating the robustness of fact verification systems in the era of large language models. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Tech...

    2025

  12. [25]

    S elf C heck GPT : Zero-resource black-box hallucination detection for generative large language models

    Potsawee Manakul, Adian Liusie, and Mark Gales. S elf C heck GPT : Zero-resource black-box hallucination detection for generative large language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP 2023), pp.\ 9004--9017. Ass...

    2023

  13. [26]

    Tree of attacks: Jailbreaking black-box llms automatically

    Anay Mehrotra, Manolis Zampetakis, Paul Kassianik, Blaine Nelson, Hyrum Anderson, Yaron Singer, and Amin Karbasi. Tree of attacks: Jailbreaking black-box llms automatically. In Proceedings of the 38th Conference on Advances in Neural Information Processing Systems (NeurIPS24),...

    2024

  14. [31]

    Morris, Eli Lifland, Jin Yong Yoo, Jake Grigsby, Di Jin, and Yanjun Qi

    John X. Morris, Eli Lifland, Jin Yong Yoo, Jake Grigsby, Di Jin, and Yanjun Qi. Textattack: A framework for adversarial attacks, data augmentation, and adversarial training in NLP . In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: Syst...

    2020

  15. [32]

    Ragtruth: A hallucination corpus for developing trustworthy retrieval-augmented language models

    Cheng Niu, Yuanhao Wu, Juno Zhu, Siliang Xu, KaShun Shum, Randy Zhong, Juntong Song, and Tong Zhang. Ragtruth: A hallucination corpus for developing trustworthy retrieval-augmented language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational ...

    2024

  16. [33]

    Investigating the robustness of retrieval-augmented generation at the query level

    Sezen Per c in, Xin Su, Qutub Sha Syed, Phillip Howard, Aleksei Kuvshinov, Leo Schwinn, and Kay-Ulrich Scholl. Investigating the robustness of retrieval-augmented generation at the query level. In Proceedings of the Fourth Workshop on Generation, Evaluation and Metrics (GEM ^2...

    2025

  17. [34]

    Attacking misinformation detection using adversarial examples generated by language models

    Piotr Przyby a, Euan McGill, and Horacio Saggion. Attacking misinformation detection using adversarial examples generated by language models. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing (EMNLP 2025), pp.\ 27626--27642. Association ...

    2025

  18. [35]

    Fast adversarial attacks on language models in one gpu minute

    Vinu Sankar Sadasivan, Shoumik Saha, Gaurang Sriramanan, Priyatham Kattakinda, Atoosa Chegini, and Soheil Feizi. Fast adversarial attacks on language models in one gpu minute. In Proceedings of the 41st International Conference on Machine Learning (ICML24), pp.\ 42976--42998. ...

    2024

  19. [36]

    Retrieval augmentation reduces hallucination in conversation

    Kurt Shuster, Spencer Poff, Moya Chen, Douwe Kiela, and Jason Weston. Retrieval augmentation reduces hallucination in conversation. In Findings of the Association for Computational Linguistics (EMNLP 2021), pp.\ 3784--3803. Association for Computational Linguistics, 2021

    2021

  20. [38]

    The silent saboteur: Imperceptible adversarial attacks against black-box retrieval-augmented generation systems

    Hongru Song, Yu-An Liu, Ruqing Zhang, Jiafeng Guo, Jianming Lv, Maarten de Rijke, and Xueqi Cheng. The silent saboteur: Imperceptible adversarial attacks against black-box retrieval-augmented generation systems. In Findings of the Association for Computational Linguistics: ACL...

    2025

  21. [39]

    Universal adversarial attacks with natural triggers for text classification

    Liwei Song, Xinwei Yu, Hsuan-Tung Peng, and Karthik Narasimhan. Universal adversarial attacks with natural triggers for text classification. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: (ACL 2021 Human La...

    2021

  22. [42]

    Adversarial GLUE : A multi-task benchmark for robustness evaluation of language models

    Boxin Wang, Chejian Xu, Shuohang Wang, Zhe Gan, Yu Cheng, Jianfeng Gao, Ahmed Hassan Awadallah, and Bo Li. Adversarial GLUE : A multi-task benchmark for robustness evaluation of language models. In Proceedings of the 35th Conference on Advances in Neural Information Processing...

    2021

  23. [43]

    Cbr-rag: case-based reasoning for retrieval augmented generation in llms for legal question answering

    Nirmalie Wiratunga, Ramitha Abeyratne, Lasal Jayawardena, Kyle Martin, Stewart Massie, Ikechukwu Nkisi-Orji, Ruvan Weerasinghe, Anne Liret, and Bruno Fleisch. Cbr-rag: case-based reasoning for retrieval augmented generation in llms for legal question answering. In Proceedings ...

    2024

  24. [44]

    Retrieval head mechanistically explains long-context factuality

    Wenhao Wu, Yizhong Wang, Guangxuan Xiao, Hao Peng, and Yao Fu. Retrieval head mechanistically explains long-context factuality. In Proceedings of the 13th International Conference on Learning Representations (ICLR25). OpenReview.net, 2025

    2025

  25. [45]

    Efficient adversarial training in LLM s with continuous attacks

    Sophie Xhonneux, Alessandro Sordoni, Stephan G \"u nnemann, Gauthier Gidel, and Leo Schwinn. Efficient adversarial training in LLM s with continuous attacks. In Advances in Neural Information Processing Systems (NeurIPS24), pp.\ 1502--1530. Curran Associates, Inc., 2024

    2024

  26. [46]

    On the vulnerability of applying retrieval-augmented generation within knowledge-intensive application domains

    Xun Xian, Ganghua Wang, Xuan Bi, Rui Zhang, Jayanth Srinivasa, Ashish Kundu, Charles Fleming, Mingyi Hong, and Jie Ding. On the vulnerability of applying retrieval-augmented generation within knowledge-intensive application domains. In Proceedings of the 42nd International Con...

    2025

  27. [47]

    An llm can fool itself: A prompt-based adversarial attack

    Xilie Xu, Keyi Kong, Ning Liu, Lizhen Cui, Di Wang, Jingfeng Zhang, and Mohan Kankanhalli. An llm can fool itself: A prompt-based adversarial attack. In Proceedings of the 12th International Conference on Learning Representations (ICLR24). OpenReview.net, 2024

    2024

  28. [49]

    Understanding parametric and contextual knowledge reconciliation within large language models

    Jun Zhao, Yongzhuo Yang, Xiang Hu, Jingqi Tong, Yi Lu, Wei Wu, Tao Gui, Qi Zhang, and Xuanjing Huang. Understanding parametric and contextual knowledge reconciliation within large language models. In Advances in Neural Information Processing Systems (NeurIPS25). Curran Associa...

    2025

  29. [51]

    \ PoisonedRAG \ : Knowledge corruption attacks to \ Retrieval-Augmented \ generation of large language models

    Wei Zou, Runpeng Geng, Binghui Wang, and Jinyuan Jia. \ PoisonedRAG \ : Knowledge corruption attacks to \ Retrieval-Augmented \ generation of large language models. In Proceedings of the 34th USENIX Security Symposium ( USENIX Security 25) , pp.\ 3827--3844. USENIX Association, 2025

    2025

  30. [52]

    Advances in Neural Information Processing Systems (NeurIPS20) , pages =

    Retrieval-augmented generation for knowledge-intensive nlp tasks , author =. Advances in Neural Information Processing Systems (NeurIPS20) , pages =

  31. [53]

    Findings of the Association for Computational Linguistics (EMNLP 2021) , pages =

    Retrieval augmentation reduces hallucination in conversation , author =. Findings of the Association for Computational Linguistics (EMNLP 2021) , pages =

    2021

  32. [54]

    Journal of Decision Systems , pages =

    Business insights using RAG--LLMs: a review and case study , author =. Journal of Decision Systems , pages =

  33. [55]

    Summary of a Haystack: A Challenge to Long-Context

    Laban, Philippe and Fabbri, Alexander and Xiong, Caiming and Wu, Chien-Sheng , booktitle =. Summary of a Haystack: A Challenge to Long-Context

  34. [56]

    Bang, Yejin and Ji, Ziwei and Schelten, Alan and Hartshorn, Anthony and Fowler, Tara and Zhang, Cheng and Cancedda, Nicola and Fung, Pascale , booktitle =

  35. [57]

    Advances in Neural Information Processing Systems (NeurIPS25) , publisher =

    Understanding Parametric and Contextual Knowledge Reconciliation within Large Language Models , author =. Advances in Neural Information Processing Systems (NeurIPS25) , publisher =

  36. [58]

    Advances in Neural Information Processing Systems (NeurIPS25) , publisher =

    ParamMute: Suppressing Knowledge-Critical FFNs for Faithful Retrieval-Augmented Generation , author =. Advances in Neural Information Processing Systems (NeurIPS25) , publisher =

  37. [59]

    Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (ACL 2024 Volume 1: Long Papers) , pages =

    Ragtruth: A hallucination corpus for developing trustworthy retrieval-augmented language models , author =. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (ACL 2024 Volume 1: Long Papers) , pages =

    2024

  38. [60]

    arXiv preprint arXiv:2511.06738 , year =

    Rethinking retrieval-augmented generation for medicine: A large-scale, systematic expert evaluation and practical insights , author =. arXiv preprint arXiv:2511.06738 , year =

  39. [61]

    Proceedings of the 32nd International Conference on Case-Based Reasoning (

    CBR-RAG: case-based reasoning for retrieval augmented generation in LLMs for legal question answering , author =. Proceedings of the 32nd International Conference on Case-Based Reasoning (. 2024 , volume =

    2024

  40. [62]

    Proceedings of the 13th International Conference on Learning Representations (ICLR25) , publisher =

    Retrieval Head Mechanistically Explains Long-Context Factuality , author =. Proceedings of the 13th International Conference on Learning Representations (ICLR25) , publisher =

  41. [63]

    arXiv preprint arXiv:1312.6199 , year =

    Intriguing properties of neural networks , author =. arXiv preprint arXiv:1312.6199 , year =

    arXiv

  42. [64]

    arXiv preprint arXiv:1902.06705 , year =

    On evaluating adversarial robustness , author =. arXiv preprint arXiv:1902.06705 , year =

    1902 arXiv

  43. [65]

    Proceedings of the 34th AAAI Conference on Artificial Intelligence (AAAI20) , pages =

    Is BERT Really Robust? A Strong Baseline for Natural Language Attack on Text Classification and Entailment , author =. Proceedings of the 34th AAAI Conference on Artificial Intelligence (AAAI20) , pages =

  44. [66]

    arXiv preprint arXiv:1511.04599 , year =

    DeepFool: a simple and accurate method to fool deep neural networks , author =. arXiv preprint arXiv:1511.04599 , year =

    arXiv

  45. [67]

    and Lifland, Eli and Yoo, Jin Yong and Grigsby, Jake and Jin, Di and Qi, Yanjun , booktitle =

    Morris, John X. and Lifland, Eli and Yoo, Jin Yong and Grigsby, Jake and Jin, Di and Qi, Yanjun , booktitle =. TextAttack: A Framework for Adversarial Attacks, Data Augmentation, and Adversarial Training in

  46. [68]

    Adversarial

    Wang, Boxin and Xu, Chejian and Wang, Shuohang and Gan, Zhe and Cheng, Yu and Gao, Jianfeng and Awadallah, Ahmed Hassan and Li, Bo , booktitle =. Adversarial

  47. [69]

    arXiv preprint arXiv:2307.15043 , year =

    Universal and transferable adversarial attacks on aligned language models , author =. arXiv preprint arXiv:2307.15043 , year =

    arXiv

  48. [70]

    Proceedings of the 38th Conference on Advances in Neural Information Processing Systems (NeurIPS24) , pages =

    Tree of attacks: Jailbreaking black-box llms automatically , author =. Proceedings of the 38th Conference on Advances in Neural Information Processing Systems (NeurIPS24) , pages =

  49. [71]

    Proceedings of the IEEE Conference on Secure and Trustworthy Machine Learning (

    Jailbreaking black box large language models in twenty queries , author =. Proceedings of the IEEE Conference on Secure and Trustworthy Machine Learning (

  50. [72]

    Proceedings of the 39th Conference on Advances in Neural Information Processing Systems (NeurIPS25) , publisher =

    Semantic Representation Attack against Aligned Large Language Models , author =. Proceedings of the 39th Conference on Advances in Neural Information Processing Systems (NeurIPS25) , publisher =

  51. [73]

    Proceedings of the 41st International Conference on Machine Learning (ICML24) , pages =

    Fast Adversarial Attacks on Language Models In One GPU Minute , author =. Proceedings of the 41st International Conference on Machine Learning (ICML24) , pages =

  52. [74]

    Proceedings of the 12th International Conference on Learning Representations (ICLR24) , publisher =

    AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Models , author =. Proceedings of the 12th International Conference on Learning Representations (ICLR24) , publisher =

  53. [75]

    Proceedings of the 42nd International Conference on Machine Learning (ICML25) , pages =

    REINFORCE Adversarial Attacks on Large Language Models: An Adaptive, Distributional, and Semantic Objective , author =. Proceedings of the 42nd International Conference on Machine Learning (ICML25) , pages =

  54. [76]

    Proceedings of the 39th Conference on Advances in Neural Information Processing Systems (NeurIPS25) , publisher =

    SECA: Semantically Equivalent and Coherent Attacks for Eliciting LLM Hallucinations , author =. Proceedings of the 39th Conference on Advances in Neural Information Processing Systems (NeurIPS25) , publisher =

  55. [77]

    arXiv preprint arXiv:2511.12869 , year =

    On the fundamental limits of LLMs at scale , author =. arXiv preprint arXiv:2511.12869 , year =

  56. [78]

    Proceedings of the Fourth Workshop on Generation, Evaluation and Metrics (GEM ^2 ) , pages =

    Investigating the robustness of retrieval-augmented generation at the query level , author =. Proceedings of the Fourth Workshop on Generation, Evaluation and Metrics (GEM ^2 ) , pages =

  57. [79]

    Proceedings of the 42nd International Conference on Machine Learning (ICML25) , pages =

    On the Vulnerability of Applying Retrieval-Augmented Generation within Knowledge-Intensive Application Domains , author =. Proceedings of the 42nd International Conference on Machine Learning (ICML25) , pages =

  58. [80]

    Findings of the Association for Computational Linguistics: ACL 2025 , pages =

    The silent saboteur: Imperceptible adversarial attacks against black-box retrieval-augmented generation systems , author =. Findings of the Association for Computational Linguistics: ACL 2025 , pages =

    2025

  59. [81]

    Findings of the Association for Computational Linguistics: EMNLP 2024 , pages =

    Typos that broke the rag's back: Genetic attack on rag pipeline by simulating documents in the wild via low-level perturbations , author =. Findings of the Association for Computational Linguistics: EMNLP 2024 , pages =

    2024

  60. [82]

    Proceedings of the 34th USENIX Security Symposium (

    \ PoisonedRAG \ : Knowledge corruption attacks to \ Retrieval-Augmented \ generation of large language models , author =. Proceedings of the 34th USENIX Security Symposium (

  61. [83]

    Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: (ACL 2021 Human Language Technologies) , pages =

    Universal Adversarial Attacks with Natural Triggers for Text Classification , author =. Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: (ACL 2021 Human Language Technologies) , pages =

    2021

  62. [84]

    Ebrahimi, Javid and Rao, Anyi and Lowd, Daniel and Dou, Dejing , booktitle =

  63. [85]

    Proceedings of the 13th International Conference on Learning Representations (ICLR25) , publisher =

    Does safety training of LLMs generalize to semantically related natural prompts? , author =. Proceedings of the 13th International Conference on Learning Representations (ICLR25) , publisher =

  64. [86]

    The Thirty-ninth Annual Conference on Neural Information Processing Systems , year=

    From Noise to Narrative: Tracing the Origins of Hallucinations in Transformers , author=. The Thirty-ninth Annual Conference on Neural Information Processing Systems , year=

  65. [87]

    Proceedings of the 39th Conference on Advances in Neural Information Processing Systems (NeurIPS25) , publisher =

    From Noise to Narrative: Tracing the Origins of Hallucinations in Transformers , author =. Proceedings of the 39th Conference on Advances in Neural Information Processing Systems (NeurIPS25) , publisher =

  66. [88]

    Proceedings of the 34th USENIX Security Symposium (

    \ Topic-FlipRAG \ : \ Topic-Orientated \ Adversarial Opinion Manipulation Attacks to \ Retrieval-Augmented \ Generation Models , author =. Proceedings of the 34th USENIX Security Symposium (

  67. [89]

    Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing (EMNLP 2025) , pages =

    Attacking Misinformation Detection Using Adversarial Examples Generated by Language Models , author =. Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing (EMNLP 2025) , pages =

    2025

  68. [90]

    Proceedings of the 12th International Conference on Learning Representations (ICLR24) , publisher =

    An LLM can Fool Itself: A Prompt-Based Adversarial Attack , author =. Proceedings of the 12th International Conference on Learning Representations (ICLR24) , publisher =

  69. [91]

    Universal Adversarial Triggers for Attacking and Analyzing

    Wallace, Eric and Feng, Shi and Kandpal, Nikhil and Gardner, Matt and Singh, Sameer , booktitle =. Universal Adversarial Triggers for Attacking and Analyzing

  70. [92]

    Proceedings of the 2020 conference on empirical methods in natural language processing (EMNLP 2020) , pages =

    Autoprompt: Eliciting knowledge from language models with automatically generated prompts , author =. Proceedings of the 2020 conference on empirical methods in natural language processing (EMNLP 2020) , pages =

    2020

  71. [93]

    Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (ACl 2025 Volume 1: Long Papers) , pages =

    Faithfulrag: Fact-level conflict modeling for context-faithful retrieval-augmented generation , author =. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (ACl 2025 Volume 1: Long Papers) , pages =

    2025

  72. [94]

    Proceedings of the 38th Conference on Advances in Neural Information Processing Systems (NeurIPS24) , pages =

    Anah-v2: Scaling analytical hallucination annotation of large language models , author =. Proceedings of the 38th Conference on Advances in Neural Information Processing Systems (NeurIPS24) , pages =

  73. [95]

    arXiv preprint arXiv:2502.06329 , year =

    Expect the unexpected: Failsafe long context QA for finance , author =. arXiv preprint arXiv:2502.06329 , year =

    arXiv

  74. [96]

    arXiv preprint arXiv:2407.21783 , year=

    The llama 3 herd of models , author=. arXiv preprint arXiv:2407.21783 , year=

    arXiv

  75. [97]

    arXiv preprint arXiv:2505.09388 , year=

    Qwen3 technical report , author=. arXiv preprint arXiv:2505.09388 , year=

    arXiv

  76. [98]

    arXiv preprint arXiv:2503.19786 , year =

    Gemma 3 Technical Report , author =. arXiv preprint arXiv:2503.19786 , year =

    arXiv

  77. [99]

    arXiv preprint arXiv:2507.06261 , year =

    Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities , author =. arXiv preprint arXiv:2507.06261 , year =

    arXiv

  78. [100]

    The Moon is Made of Marshmallows

    FaithEval: Can Your Language Model Stay Faithful to Context, Even If "The Moon is Made of Marshmallows" , author =. arXiv preprint arXiv:2410.03727 , year =

    arXiv

  79. [101]

    arXiv preprint arXiv:2601.03267 , year =

    OpenAI GPT-5 System Card , author =. arXiv preprint arXiv:2601.03267 , year =

    arXiv

  80. [102]

    arXiv preprint arXiv:2506.13585 , year =

    MiniMax-M1: Scaling Test-Time Compute Efficiently with Lightning Attention , author =. arXiv preprint arXiv:2506.13585 , year =

    arXiv

  81. [103]

    Manakul, Potsawee and Liusie, Adian and Gales, Mark , booktitle =

  82. [104]

    Asai, Akari and Wu, Zeqiu and Wang, Yizhong and Sil, Avirup and Hajishirzi, Hannaneh , booktitle =. Self-

  83. [105]

    Tang, Liyan and Laban, Philippe and Durrett, Greg , booktitle =

  84. [106]

    Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP 2023) , pages =

    Query Rewriting in Retrieval-Augmented Large Language Models , author =. Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP 2023) , pages =

    2023

  85. [107]

    Efficient Adversarial Training in

    Xhonneux, Sophie and Sordoni, Alessandro and G. Efficient Adversarial Training in. Advances in Neural Information Processing Systems (NeurIPS24) , pages =

  86. [108]

    Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL 2025 Volume 1: Long Papers) , pages =

    2025

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.