{"id":"b2ebf0b0-dc22-41a3-b129-55d7fad0b504","arxiv_id":"2608.04286","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Meaning-preserving adversarial rewrites of user queries substantially degrade faithfulness in retrieval-augmented language models, with part of the effect driven by induced refusals.","lead":"This paper tests whether rewriting a user's question in slightly different words with the same meaning can make AI models that answer questions from provided documents give wrong or evasive answers. It finds that state-of-the-art models do fail under such meaning-preserving rewrites, which matters for any real-world system that answers questions using retrieved evidence.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Equivalence gate (Eq. 3) is the crux; Appendix A shows gate-passing attacks that change meaning (e.g., 'total number per share'), so the 'meaning-preserving' degradation may be overstated.","rationale":"Good-faith reading: the paper's contribution is a framework and benchmark for query-level robustness of RAG generators, with a clean threat model (perfect retriever) and a meaningful set of adapted attacks. The strongest, most defensible result is that paraphrase-based attacks (PAIR, SECA-BB) can reduce faithfulness on counterfactual and finance benchmarks, with several Appendix A examples that do appear meaning-preserving (e.g., A.1 'original purpose' → 'initial intent behind the creation'; A.2 subsidiaries paraphrase). The human validation study is a genuine attempt to ground the LLM judge, and the self-preference check for the hallucination judge is a useful control.\n\nThe weakest point is the gate. The central claim is definitionally about meaning-preserving perturbations; if the gate is lenient, the paper measures a different quantity. The appendix examples are the strongest evidence because they are actual attack outputs that passed the gate; they include clear meaning violations. The human validation does not resolve this because it is a random sample, not a targeted audit of the adversarial distribution, and adversarial examples are precisely where a judge is most likely to be fooled. The abstention failure-mode analysis is a secondary but real issue: calling a refusal a 'hallucination' conflates distinct failure types and inflates the abstract's wording.\n\nThe proposed test — a targeted human recheck of the gate-passing queries behind the headline GPT-5-mini number — would settle whether the concern lands. If the gate is sound on those queries, the central claim survives for that result; if not, the 'up to 50%' figure and the 'meaning-preserving' framing need revision. This is exactly the kind of omitted validation the paper's own ablation (Table 6) suggests matters: the search-time proxy rejection rate was high but the final gate rejection rate is never reported in the main tables. We therefore keep the reader's CONDITIONAL verdict.","tokens_in":20803,"tokens_out":6854,"duration_ms":54249,"concrete_test":"Re-run the evaluation for the GPT-5-mini FaithEval SECA-BB row (Table 2) with all gate-passing adversarial queries rechecked by two independent human annotators using the exact C.2 mutual-entailment rubric. Record the proportion of successful attacks that are judged non-equivalent; recompute the ASR and the CA−AA decrease after excluding those queries. If the corrected decrease drops substantially below the reported 0.50, the abstract's central 'meaning-preserving' claim is not supported for that headline result. As a secondary check, recompute ASR counting only responses with fabricated content (excluding abstentions) to separate 'hallucination' from refusal.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The headline result — meaning-preserving perturbations degrade faithfulness by up to 50% (abstract; FaithEval, GPT-5-mini, SECA-BB: CA 0.72 → AA 0.22 in Table 2) — hinges on Eq. (3)'s mutual-entailment gate certifying that every counted adversarial query is semantically equivalent to the original. The gate is a single LLM (Gemini-2.5-Flash-Lite) with no precision reported on the adversarial distribution it is asked to filter. Appendix A provides concrete evidence that the gate admits meaning-changing rewrites: A.2's GCG attack changes 'total number of shares' to 'total number per share', which the paper's own C.2 rubric explicitly treats as non-equivalent ('asking for percentage shares vs. ratios'); A.1 and A.3's SRA/SECA-BB attacks append a bare 'A' or '(a' to turn open-ended questions into multiple-choice formats, changing the expected answer. If even a modest fraction of reported ASR comes from such rewrites, the central 'semantically equivalent' claim is inflated. The human validation (Table 7) samples 150 examples and shows substantial but imperfect agreement (κ ≈ 0.64–0.68); it does not measure precision on the adversarially selected, gate-passing queries that drive the headline. Table 3 further shows 22–56% of 'successes' are abstentions ('Unable to answer'), which are refusals rather than fabricated or unfaithful content — the abstract's 'unfaithful or fabricated information' overstates this failure mode. Thus the load-bearing assumption — that Eq. (3) rejects meaning-changing candidates at high precision — is not established and is contradicted by the paper's own examples.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a framework for eliciting intrinsic hallucinations in RAG by adversarially perturbing the user query under a semantic-equivalence constraint. It adapts GCG, AutoDAN, PAIR, SRA, and SECA-GB/SECA-BB to a hallucination-induction objective, evaluates them on 5 open and 5 closed models over FaithEval, ANAH-v2, and FailSafeQA, and reports Attack Success Rates that are often large; the headline example is GPT-5-mini on FaithEval, where SECA-BB reduces Adversarial Accuracy from 0.72 to 0.22 (ASR 0.703). The automated evaluation uses Gemini-2.5-Flash-Lite as both the semantic-equivalence judge and the faithfulness judge, with a 150-example human validation study.","tokens_in":21139,"tokens_out":5174,"duration_ms":44892,"significance":"If the perturbations counted as successful really are semantically equivalent, the paper establishes a practically important lower bound: faithfulness of RAG generators to a fixed context is highly sensitive to benign-looking query reformulation. The paper has real strengths: code is released, the cosine-threshold ablation (Table 6) shows the final gate is doing nontrivial filtering, and the self-preference check (Table 8) addresses one obvious confound. However, the central claim depends on two load-bearing assumptions that the current evidence does not fully support: the reliability of the Eq. (3) mutual-entailment gate on the adversarial distribution, and the interpretation of 'Unable to answer' responses as intrinsic hallucinations.","major_comments":[{"comment":"The semantic-equivalence gate is the load-bearing component of the paper, but its precision on the adversarial distribution is never reported. Appendix A provides concrete gate-passing rewrites that change meaning: A.2 changes 'total number of shares' to 'total number per share', which the paper's own rubric in C.2 explicitly uses as a non-equivalence example (percentage shares vs. ratios), and A.1/A.3 show attacks that append a bare 'A' or '(a' to convert open-ended questions into multiple-choice format, changing the expected answer. The human validation in Table 7 samples 150 random examples and does not condition on the adversarial, gate-passing queries that drive the headline ASR. The authors should report precision of the Eq. (3) judge on the gate-passing adversarial outputs, or re-run the evaluation with a stricter gate and show that the headline numbers are stable.","section":"§3, Eq. (3); Appendix A"},{"comment":"The failure-mode analysis in Table 3 shows that 22–56% of successful attacks are abstentions ('Unable to answer'). Under the paper's own hallucination rubric (C.1), a response that appropriately says the context is insufficient should receive 'proper uncertainty' = true and should not be counted as an intrinsic hallucination. Counting abstention as a success therefore inflates the ASR and contradicts the abstract's characterization of the failures as 'unfaithful or fabricated information.' The paper should report ASR with abstentions excluded (or at least report the fabrication-only ASR), and the abstract and conclusions should be revised to distinguish evasive responses from fabricated content.","section":"Table 3; §C.1; abstract"},{"comment":"PAIR and SECA-BB optimize against the same hallucination judge that is later used to compute AA and ASR. This creates a partial circularity: the optimizer can exploit systematic quirks of the judge's rubric rather than produce genuinely unfaithful responses. The human validation does not close this gap because it is a random sample and is not stratified by the adversarial, gate-passing queries selected by the optimizer. The authors should validate the judge on the actual set of successful adversarial outputs (or use a held-out judge) and report the agreement on that subset.","section":"§3, Eq. (6); §4; §C.1"},{"comment":"SRA is explicitly described as not strictly comparable to the paraphrase-based attacks because it appends tokens rather than rephrasing the query, yet it is reported in Table 1, highlighted as achieving the highest ASR, and may contribute to the conclusion that ASR exceeds 50% for state-of-the-art models. The paper should either move SRA to the appendix or clearly separate all SRA-based numbers from the headline claims, and the 'over 50%' statement should be recomputed without SRA.","section":"§3 (SRA paragraph); Table 1; §5"}],"minor_comments":[{"comment":"The paragraph contains an exact duplicated sentence: 'Under this threat model, the adversary seeks...' appears twice.","section":"§3, Threat model paragraph"},{"comment":"The text refers to a 'K-10 filing' in the adversarial query; the correct term is '10-K filing'.","section":"Appendix A.2"},{"comment":"The GitHub URL is given as 'https://github.com/atriviveksharma/intrinsic hall' with a space; the URL should be 'intrinsic_hall' (or the correct repository name) and typeset as a link.","section":"Footnote 1 and abstract"},{"comment":"The ASR definition includes the semantic-equivalence indicator in the numerator but not in the denominator; this is presumably intentional, but the notation should be clarified to avoid confusion about the conditioning set.","section":"§4, ASR definition"},{"comment":"The perplexity values are unevenly formatted (e.g., 17140, 535, 65.2) and the table is difficult to read; consider a log-scale column or more consistent formatting.","section":"Table 1"}],"recommendation":"major_revision","confidential_remarks":"The core experimental scaffolding is solid, and the human validation is a genuine plus. The two risks that most affect the paper's headline are the equivalence-gate precision and the counting of abstentions as hallucinations; both are fixable with additional analysis rather than new theory. I would advise the editor that the revision should be judged on whether the authors can demonstrate gate precision on the actual adversarial distribution and report abstention-excluded ASR."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Thanks for the report. I mostly agree with the conditional read, and the stress-test lands. The paper is worth taking seriously, but the headline number overstates what is actually shown.\n\nWhat is new: SECA already showed semantically equivalent attacks can elicit hallucinations, but this paper generalizes the idea to open-ended RAG, adapts five attack families (GCG, AutoDAN, PAIR, SECA-GB/BB, SRA), and runs them across 10 models and 3 datasets including counterfactual and finance contexts. That is a useful benchmark. The human validation is a real plus: Cohen's kappa around 0.8 for hallucination detection and 0.64-0.68 for equivalence is decent, and the self-preference check is thoughtful. Code is out. The authors are transparent that SRA is not strictly comparable and include it as a minimal-perturbation probe; that is fine if it stays separate.\n\nThe soft spots are load-bearing. The equivalence gate (Eq. 3) is the crux, and the paper's own Appendix A shows it passing rewrites that change meaning: \"total number of shares\" becomes \"total number per share\" (A.2), which the paper's rubric C.2 explicitly treats as non-equivalent; appending a bare \"A\" or \"(a\" to turn an open-ended question into a forced-choice format (A.1, A.3) changes the expected answer. If a non-trivial share of the headline ASR comes from such rewrites, the claim about 'meaning-preserving' perturbations is inflated. The gate's precision on the adversarial distribution is never reported. Related, Table 3 shows 22-56% of 'successes' are abstentions ('Unable to answer'), which are refusals, not fabricated content. The abstract's 'unfaithful or fabricated information' overstates this. The circularity concern is real but moderate: PAIR and SECA-BB optimize against the same hallucination judge used for evaluation, and human validation on a random sample does not measure precision on the gate-passing adversarial queries that drive the headline. The cosine threshold tau is chosen on evaluation data via ablation, a minor fitting concern.\n\nIf the authors fix these—report the equivalence gate's precision, exclude or segregate abstentions, and separate SRA—the real effect will likely survive in attenuated form. There is probably a genuine fragility here; it is just not 50% of meaning-preserving perturbations.\n\nThis paper deserves a serious referee. I would send it to review with a request for major revision. I would not cite the numbers as-is, but I would bring it to a reading group to discuss the methodology.","headline":"A worthwhile benchmark with a crux problem: the equivalence gate admits meaning-changing rewrites and ASR counts refusals as hallucinations, so the headline 50% figure is inflated even though a real fragility likely remains.","tokens_in":21668,"tokens_out":2895,"would_cite":false,"duration_ms":26630,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Meaning-preserving rewrites of a question can flip a RAG model's faithful answer into a hallucination.","keywords":["intrinsic hallucination","retrieval-augmented generation","adversarial attack","semantic equivalence","faithfulness evaluation","LLM robustness","query perturbation","black-box attack"],"falsifier":"Have independent human annotators, blind to the intended result, classify whether each reported adversarial query is mutually entailed by the original query, and recompute attack success only on pairs all annotators accept; if the human-accepted subset shrinks the success rates toward zero, the central claim is refuted, while surviving high rates would confirm it.","tokens_in":20585,"feed_emoji":"🎯","tokens_out":6731,"duration_ms":55786,"temperature":0.7,"pith_summary":"This paper tries to establish that retrieval-augmented language models are not reliably faithful to their retrieved evidence: simply rewording a user's question into a semantically equivalent form can make the model refuse, fabricate, or contradict the context. It offers this as a lower bound on robustness under meaning-preserving perturbations. To make the point, the authors adapt five attack methods to generate query variants that pass a strict bidirectional-entailment check, then measure how often originally correct answers become unfaithful. Across ten models and three datasets, the attacks succeed on a substantial fraction of cases, including on the strongest closed models. If the finding holds, it shows that faithful grounding is tied to the surface form of the query, not just to the evidence supplied.","feed_headline":"Rewording a RAG question can halve answer faithfulness","feed_subtitle":"Ten models, from small open weights to closed APIs, lose grounding on rewrites that keep the user's intent identical.","key_machinery":"The load-bearing object is a semantic equivalence predicate, $\\operatorname{Eq}(q,q') := \\operatorname{Entail}(q\\Rightarrow q') \\land \\operatorname{Entail}(q'\\Rightarrow q)$, implemented by an LLM judge. It defines the admissible attack set and gates every reported success, so the paper's interpretation of 'meaning-preserving' rests on this predicate. During search, a cheaper sentence-embedding cosine similarity threshold stands in for the judge; at evaluation, only perturbations passing mutual entailment count. The objective being maximized is the expected intrinsic-hallucination score of the generator conditioned on fixed context, and the threat model assumes a perfect retriever so all failures are attributed to the generator.","core_discovery":"The central claim is that intrinsic hallucinations in RAG can be provoked by perturbations that preserve the query's meaning, with no changes to the retrieved context. The paper formalizes meaning preservation as mutual entailment between original and adversarial queries, computes attacks under white-box, gray-box, and black-box access, and evaluates them on five open and five closed models across three datasets. Under the strongest black-box attack, originally correct answers become unfaithful in over half of cases on some datasets; the headline number is a drop in contextual faithfulness of up to 50% for a state-of-the-art small closed model. The authors read this as establishing a low lower bound on RAG faithfulness: even a fixed, ideal retriever and a fluent, intent-preserving query do not guarantee a grounded answer.","pith_inferences":["Editorial inference: the judge-based equivalence gate is the crux, so the practical strength of the result depends on how lenient or strict the gate is for real human users.","Editorial inference: the perfect-retriever assumption isolates generator fragility; real pipelines with noisy retrieval may show different or larger degradation.","Testable extension: training models to be invariant to paraphrase, for instance by adversarially rephrasing queries during fine-tuning, could be measured directly with this attack suite.","Testable next step: applying the same semantic-equivalence attacks in multi-turn RAG would show whether conversational context anchors faithfulness or adds new failure modes."],"forward_implications":["RAG faithfulness cannot be inferred from clean accuracy: the same context, retriever, and intended query can produce faithful or unfaithful answers depending on phrasing.","Token-level optimizers are weak attack vectors because they produce unnatural, high-perplexity queries; LLM-generated rephrasings and short suffix additions are the effective ones.","Scaling does not remove the vulnerability: larger open models and frontier closed models show comparable or larger attack success rates.","The dominant black-box failure modes differ from white-box ones: reasoning errors replace abstention when the attacker cannot use an 'Unable to answer' target.","Any robustness claim for a RAG system should be reported against a paraphrase-sensitive attack set, not a single query form."],"supporting_citations":[{"why":"Supplies the HalluLens intrinsic-hallucination benchmark from which two of the three evaluation datasets are drawn.","marker":"Bang et al., 2025"},{"why":"Provides FaithEval, whose counterfactual split tests faithfulness when the context contradicts common knowledge.","marker":"Ming et al., 2024"},{"why":"Provides ANAH-v2, the multi-domain analytical hallucination evaluation set used here in free-form generation.","marker":"Gu et al., 2024"},{"why":"Provides FailSafeQA, the long-context finance QA benchmark with structured evidence.","marker":"Kamble et al., 2025"},{"why":"Source of the GCG gradient token-attack method adapted to query-token rephrasing under semantic constraints.","marker":"Zou et al., 2023"},{"why":"Source of AutoDAN, whose synonym-replacement mechanism is adapted with a semantic filter.","marker":"Liu et al., 2024"},{"why":"Source of PAIR, the black-box attacker-judge loop adapted with an additional equivalence judge.","marker":"Chao et al., 2025"},{"why":"Source of SECA, the feasibility-checked semantic attack framework generalized here to open-ended generation.","marker":"Liang et al., 2025"},{"why":"Source of SRA, the suffix-based semantic-representation attack used as a minimal-perturbation probe.","marker":"Lian et al., 2025"},{"why":"Provides the Gemini-2.5-Flash-Lite model used as the LLM judge and as one of the closed-source target models.","marker":"Comanici et al., 2025"}],"fun_headline_variants":["Meaning-preserving query rewrites trigger LLM hallucinations in RAG","Same intent, new wording: LLM grounding drops 50%","Rewording queries to keep meaning halves RAG faithfulness","Reword a query, keep meaning, halve RAG faithfulness","Meaning-preserving query rewrites cut RAG faithfulness by up to 50%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the LLM judge's mutual-entailment verdict is a correct test of semantic equivalence; if it accepts rewrites that change the question, the reported attacks are not meaning-preserving and the claimed lower bound does not follow.","fun_headline_variants_meta":{"raw":{"variants":["Meaning-preserving query rewrites trigger LLM hallucinations in RAG","Same intent, new wording: LLM grounding drops 50%","Rewording queries to keep meaning halves RAG faithfulness","Reword a query, keep meaning, halve RAG faithfulness","Meaning-preserving query rewrites cut RAG faithfulness by up to 50%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00135,"raw_usage":{"total_tokens":5474,"prompt_tokens":931,"completion_tokens":4543,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":547,"completion_tokens_details":{"reasoning_tokens":4452}},"tokens_in":547,"tokens_out":4543,"duration_ms":28712,"temperature":1.0,"reasoning_tokens":4452,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T14:40:54.424542+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Have independent human annotators, blind to the intended result, classify whether each reported adversarial query is mutually entailed by the original query, and recompute attack success only on pairs all annotators accept; if the human-accepted subset shrinks the success rates toward zero, the central claim is refuted, while surviving high rates would confirm it.","supporting_citations":[{"cited_title":"Seca: Semantically equivalent and coherent attacks for eliciting llm hallucinations","cited_arxiv_id":null,"evidence_quote":"Source of SECA, the feasibility-checked semantic attack framework generalized here to open-ended generation."}],"review_version":2}