{"id":"e87e2ed9-2794-40c8-8d24-cce934b7d1c9","arxiv_id":"2411.11407","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Fake authoritative citations matched to the type of harmful request can bypass safety alignment in several commercial and open LLMs.","lead":"This paper introduces DarkCite, a jailbreak attack that inserts fake but authoritative-looking citations into prompts to make large language models produce harmful content. The authors report higher attack success than prior methods and propose a system-prompt defense that raises refusal rates.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The risk-citation matching rules and hand-set weights are not shown to be independent of the AdvBench/HEx-PHI evaluation sets, so the reported ASR advantage may reflect benchmark-specific tuning rather than a generalizable authority-bias attack.","rationale":"The reader identified the same load-bearing assumption: the risk-citation matching rules and hand-set weights may be built from historical vulnerability logs that are not independent of the evaluation sets. My reading of the manuscript confirms this is the weakest point. Section 4.2 defines Φ2 as a 'historical vulnerability' function but gives no description of the logs; the appendix states the citation-type rules were 'crafted based on the distribution patterns of historical vulnerabilities and risk categories'; and the attacker system prompt lists a fixed, manually authored topic-to-citation mapping. Since the evaluation uses the two benchmarks that define the risk categories (AdvBench and HEx-PHI), there is no evidence ruling out that the mapping was tuned to those exact prompts or to models with known responses on those prompts. If the mapping is benchmark-fitted, the headline comparison 'DarkCite 76% versus PAP 68% on Llama-2' overstates the method's general advantage, though the underlying observation that LLMs often comply with authoritative-looking citations could still hold. The paper also lacks a matched-versus-mismatched citation ablation, which would be the natural way to demonstrate that the matching component, not just the citation template, drives the ASR. These are addressable issues, so the conditional verdict is appropriate; my concern does not move the verdict to reject, but it does mean the paper's strongest claim should be treated with caution until the independence of the matching rules is shown. I set verdict_should_be to UNCHANGED because the reader's CONDITIONAL verdict already reflects this risk, and no new concern requires changing the verdict level.","tokens_in":20977,"tokens_out":4487,"duration_ms":46836,"concrete_test":"Download the anonymous DarkCite repository (https://github.com/YancyKahn/DarkCite) and inspect the data or code used to compute the historical vulnerability feature Φ2 and the cited topic-to-citation rules. Check whether any prompt strings from AdvBench harmful_behaviors.csv or HEx-PHI sub10.json appear in that historical data, in the attacker system template, or in the examples used to tune the weights. If overlap exists, the matching rules are directly fitted to the evaluation benchmarks and the Table 1 ASR is over-optimistic; if no overlap exists, the concern is refuted.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is that DarkCite's adaptive risk-citation matching drives its higher attack success rate. That claim rests on the matching classifier in Section 4.2, which combines a risk-type distribution feature with a 'historical vulnerability' feature Φ2(Ri, Cj). The paper never describes the historical logs on which Φ2 is based, their size, or their overlap with the evaluation benchmarks. The appendix's 'System Template for Risk-Citation Type Matching and Citation Content Generate' states directly that 'The specific rules for risk citation are crafted based on the distribution patterns of historical vulnerabilities and risk categories,' and the attacker template contains an explicit, hand-authored topic-to-citation mapping (e.g., bomb-making -> Research Paper, hacking -> Project Repositories, poisoning -> Research Paper) with weights w1=0.1 and w2=0.9. If that mapping or the underlying historical logs were produced by testing which citation types succeed on the same AdvBench and HEx-PHI prompts used in Table 1, then the ASR improvement over baselines is partly a fit to the test set. The paper also reports no quantitative ablation comparing matched versus mismatched citation types, so the contribution of the matching classifier is not isolated. This is not an internal inconsistency, but a real correctness and generalization risk: the authority-bias phenomenon may be real, while DarkCite's specific 'adaptive matching' component may add little beyond any plausible-looking citation in a prompt template.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces DarkCite, a black-box jailbreak attack that constructs fake authoritative citations matched to the risk type of a harmful instruction, and claims higher attack success rates than several prior jailbreak methods on AdvBench and HEx-PHI. It further proposes a system-prompt defense based on authenticity and harm verification, reporting an increase in defense pass rate from 11% to 74%, and argues that citation-driven attacks behave as an implicit form of RAG. The core phenomenon—that aligned LLMs can be induced to comply with harmful instructions when those instructions are framed as coming from authoritative-looking sources—is plausible and the paper provides concrete templates and running examples. However, the quantitative support for the adaptive matching component, the headline ASR comparisons, and the implicit-RAG interpretation is incomplete.","tokens_in":21214,"tokens_out":6970,"duration_ms":64379,"significance":"If the results are substantiated, the paper identifies a practical and largely unmitigated vulnerability: black-box attackers can bypass alignment in several commercial and open models with a handful of queries by generating stylistically authoritative references, and a simple prompt-level defense materially reduces the attack. The attack is efficient and the template-based construction is easy to reproduce, which makes the risk concrete for model deployers. The paper also makes a falsifiable claim that risk-citation matching is the driver of the improvement, and that the effect stems from pretraining distributional biases. At present, however, the reported evidence does not establish these specific claims: the matching classifier's construction is not shown to be independent of the test benchmarks, there is no matched-versus-mismatched citation ablation, the ASR comparisons lack variance and use an optimistic success criterion, and harmfulness is judged by a single automated classifier. These gaps are fixable within the paper's scope.","major_comments":[{"comment":"The matching classifier's independence from the evaluation sets is not established. The historical vulnerability feature Φ2(Ri, Cj) is defined as 'historical success rate based on logs of model responses to similar citations,' but the logs are never described—neither their size, provenance, nor overlap with AdvBench and HEx-PHI. The appendix's attacker template further states that 'the specific rules for risk citation are crafted based on the distribution patterns of historical vulnerabilities and risk categories,' and the template contains a hand-authored topic-to-citation mapping with weights w1=0.1 and w2=0.9. If this mapping or the underlying logs were derived by testing which citation types succeed on the same prompts used in Table 1, the reported ASR advantage of DarkCite is partly a fit to the test set. The authors should either describe the historical logs and demonstrate that they are disjoint from the evaluation data, or construct the matching rules from an explicitly independent development set.","section":"Section 4.2 and Appendix B (System Template for Risk-Citation Type Matching)"},{"comment":"The headline ASR comparisons are presented as point estimates with no variance, and the attack protocol uses an optimistic any-one-of-three success criterion: 'if any one of these three attempts resulted in a successful attack, the attack was deemed successful.' This inflates the per-prompt success probability and compresses differences between methods, while the baseline trials are heterogeneous (e.g., DI at 1 trial, DeepInception at 3, GCG and AutoDAN at 3×100, PAP at 3×40). The 8-percentage-point improvement over PAP on Llama-2 (76% vs 68%) is not accompanied by confidence intervals or repeated independent runs, so it cannot be assessed against sampling noise. Please report the mean and standard deviation over repeated runs, use a consistent trial budget, and also report the all-three-success rate so the reader can see the sensitivity to the success criterion.","section":"Section 5.1 and Table 1"},{"comment":"The central claim that DarkCite's risk-citation matching drives the improvement is not isolated by any matched-versus-mismatched ablation. Figure 6 shows ASR for different citation carriers across risk categories, but it does not compare, for the same set of harmful instructions, the matched citation type chosen by f(Ri, Cj) against a randomly chosen or deliberately mismatched citation type. Without such a comparison, a generic authority-bias effect—any plausible citation improves ASR—remains a competing explanation, and the contribution of the matching score in Section 4.2 is not measured. I would like to see an ablation that keeps the citation content generation fixed and varies only whether the citation type is the matched one, a random one, or a mismatched one.","section":"Section 5.4 and Figure 6"},{"comment":"The only harmfulness labeler is Llama-Guard-3, and the paper reports no human validation or agreement rate for this judge on the attack outputs. Since ASR is the paper's principal metric and the any-of-three criterion is applied on top of the automated judge, a systematic bias in the judge directly changes all headline numbers. Please provide a human-evaluated subset or a second judge with agreement statistics, and clarify whether the judge was applied to each of the three attempts or only to the successful-looking response.","section":"Section 4.4 and Section 5.1 (Harmfulness Assessment)"},{"comment":"The 'implicit RAG attack' interpretation is presented as a finding but is supported only by qualitative KL-divergence and t-SNE plots, with no quantitative values, sample sizes, statistical tests, or a precise definition of the 'with-RAG' condition. The notion of an 'internally retrieved hidden document D' is introduced without direct evidence; at the current level of support this is a hypothesis, not a result. Please either provide quantitative comparisons (e.g., mean KL divergence with confidence intervals and a significance test) or explicitly reframe this section as a conjecture with preliminary evidence.","section":"Section 5.6 and Figures 8-9"}],"minor_comments":[{"comment":"There are several typos, including 'the biases in LLMs is' and 'suchu as'; please copyedit the abstract and the example in Section 3.1.","section":"Abstract and Section 3.1"},{"comment":"The RA-LLM defense is described once as 'retaining only the text with the lowest level of harmfulness' and later as 'if any one of the outputs is harmless, the instruction is classified as harmless'; these two descriptions are inconsistent and should be reconciled.","section":"Section 5.5"},{"comment":"The defense table contains formatting issues ('Default.', 'A VG.') and does not explain why Vicuna appears in the victim list although it is not in the list of victim models in Section 5.1.","section":"Table 3"},{"comment":"The token utilization efficiency metric is not defined in the text; please specify exactly how the number of tokens consumed per unit of attack frequency is computed.","section":"Figure 5"},{"comment":"The HEx-PHI dataset is cited to reference [58], which appears to be a paper on fine-tuning aligned models; please verify and update the citation for HEx-PHI.","section":"Reference [58]"}],"recommendation":"major_revision","confidential_remarks":"The paper's claimed anonymous code repository could not be verified from the manuscript text. Given the evaluation gaps, I would suggest that the editor require the authors to make the matching classifier, the historical vulnerability logs (or a clear statement of their provenance and disjointness), and the evaluation code available as part of the revision, since the central generalization claim depends on these artifacts."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe one thing to know: this paper identifies a real and under-appreciated attack vector. Prefixing harmful instructions with fake authoritative citations can push aligned LLMs into producing harmful content, and the type of citation carrier (research paper, GitHub, Wikipedia, etc.) appears to matter for different risk categories. That core claim is plausible, the paper gives concrete worked examples, and the attack is cheap in query count. The headline numbers (e.g., Llama-2 at 76% ASR vs. 68% for PAP) are solid enough to take seriously.\n\nWhat is actually new: prior work already showed LLMs trust authoritative sources, and RAG-poisoning attacks already exploit retrieved context. DarkCite's contribution is the adaptive risk-citation matching idea plus the observation that different risk types respond to different carriers. The implicit-RAG analysis (KL divergence and t-SNE) is a nice supporting study suggesting citations behave like retrieved documents. The defense is simple—two system-prompt additions asking the model to verify authenticity and potential harm—and shows a jump from 11% to 74% DPR. That is actionable and easy to adopt.\n\nSoft spots, in proportion: they are real but mostly addressable. The evaluation uses an any-of-three success rule over three trials, no error bars or variance, and a single Llama-Guard judge for harmfulness. More substantively, the matching classifier's \"historical vulnerability\" feature is never specified: no description of the logs, their size, or their overlap with AdvBench/HEx-PHI. The appendix then reveals a hand-crafted topic-to-citation mapping with weights w1=0.1 and w2=0.9, and says the specific rules are \"crafted based on the distribution patterns of historical vulnerabilities.\" If that mapping was tuned on the same benchmarks used in Table 1, the ASR advantage over baselines is partly a fit to the test set. There is no ablation comparing matched vs. mismatched citations, so we cannot tell whether matching adds anything beyond \"any plausible-looking citation.\" The defense experiments also do not measure whether the system prompt breaks legitimate citation use or merely causes broad refusal.\n\nThe central authority-bias phenomenon probably survives these issues. The matching component could be overfit, but the attack works even with generic citations in the examples. This is not a paper with a load-bearing flaw in the core phenomenon; it is a paper whose adaptive-matching claim needs tighter evidence.\n\nWho benefits: LLM safety researchers, red-teamers, and people working on RAG security. A serious referee should engage. My recommendation: send it to review, but ask for a matched-vs-mismatched ablation, error bars, and a clear statement of how the historical vulnerability logs were constructed and whether they overlap with the benchmarks.","headline":"A plausible authority-bias jailbreak attack with a simple, useful defense; the adaptive-matching claim is interesting but the current evidence for it is thinner than the headline ASR suggests.","tokens_in":21774,"tokens_out":2106,"would_cite":true,"duration_ms":49861,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Fake authoritative citations jailbreak aligned LLMs at 66 percent average success.","keywords":["jailbreak attacks","authority bias","large language models","black-box attack","citation generation","retrieval-augmented generation","LLM safety","adversarial prompts"],"falsifier":"Construct the risk-citation matching rules and weights from a held-out log of historical attacks with no overlap with the harmful instructions used for evaluation, then replay DarkCite on fresh risk types; if the attack success rate advantage over the strongest persuasive-prompt baseline collapses, the reported gains are a fit to the evaluation set rather than a general authority bias.","tokens_in":20729,"feed_emoji":"🔗","tokens_out":8271,"duration_ms":80843,"temperature":0.7,"pith_summary":"Aligned large language models can be jailbroken in a black-box setting by embedding harmful instructions inside fabricated authoritative citations, as long as the citation type matches the risk category of the request. The paper's method, DarkCite, first classifies a harmful instruction into a risk type, selects the citation carrier the model is most likely to trust for that type (academic papers, code repositories, reports, and the like), then generates a realistic-looking fake reference whose substance tracks the instruction. Across standard harmful-behavior benchmarks it reports the highest average attack success rate among compared methods, with an open seven-billion-parameter chat model at 76% versus 68% for the previous strongest baseline. It also proposes a system-prompt defense that checks citation authenticity and potential harm, lifting the average defense pass rate from 11% to 74%. If the claims hold, the same trust in authoritative sources that improves everyday answers is a low-cost, general bypass of current alignment.","feed_headline":"Fake authoritative citations jailbreak aligned LLMs at 66%","feed_subtitle":"Matching citation type to risk type — papers for bombs, code hosts for malware — bypasses safety filters in black-box attacks.","key_machinery":"The carrying object is the matching score $f(R_i,C_j)=\\sum_{k=1}^n w_k\\Phi_k(R_i,C_j)$, where $\\Phi_1$ measures how often a risk type appears in pretraining data associated with a citation carrier and $\\Phi_2$ measures the historical success rate of that carrier for that risk type, with deployed weights $w_1=0.1$ and $w_2=0.9$. The classifier built on this score maps each harmful instruction to its optimal citation type, and a generator fills a bibliographic template with content that neutrally restates the harmful instruction. An attack template then asks the victim model to summarize the methodology from the supplied reference, and an automated harm classifier scores the output. This machinery turns a diffuse bias into a one-shot black-box prompt that needs no gradient access and no iterative suffix search.","core_discovery":"The paper's central claim is that LLMs exhibit a measurable authority bias that survives alignment: they trust different kinds of authoritative sources to different degrees depending on the risk type of the query, and this trust can be steered toward harmful outputs. DarkCite operationalizes the bias with a three-stage pipeline. A risk-citation matching classifier computes a weighted score over risk-type distribution in pretraining data and historical vulnerability logs to select the optimal citation carrier; a generator then creates a citation whose content is aligned with the harmful instruction but free of obvious safety-triggering keywords; and the crafted prompt asks the model to summarize the cited source. The paper reports an average attack success rate of 66% across open and commercial models, with only a few victim-model calls, and presents distributional evidence that cited prompts behave like retrieval-augmented generation prompts. It closes by showing that a system-prompt requiring authenticity and harm verification raises the average defense pass rate from 11% to 74%.","pith_inferences":["Beyond the paper: if citation-driven jailbreaks are implicit RAG, then any downstream system that trusts retrieved or tool-supplied context inherits the same authority-bias vulnerability, so authenticity checks should be applied to retrieved passages and tool outputs, not only user-supplied citations.","Beyond the paper: the claimed dependence on pretraining risk distribution is directly testable as a mitigation — training or fine-tuning on corpora with balanced risk-topic representation by source should shrink the citation-matching advantage.","Beyond the paper: the same mechanism should transfer to other modalities; a feasible extension is to check whether attaching a fake authoritative source line to a harmful image, audio, or code prompt raises jailbreak success comparably."],"forward_implications":["An attacker with no access to model internals can craft risk-matched fake citations and breach alignment on both open and commercial LLMs, with an average attack success rate of 66 percent.","DarkCite needs only a handful of victim-model calls, in contrast to gradient-based baselines that require hundreds or thousands of iterations, and it consumes fewer tokens per successful attack.","Existing defenses such as moderation filtering, perplexity thresholds, and randomized deletion reduce but do not neutralize the attack; even combined, they leave meaningful success rates on open models.","A system-prompt defense that verifies citation authenticity and checks cited material for harm raises the average defense pass rate from 11 percent to 74 percent.","Cited prompts produce token-level distributions close to retrieval-augmented generation prompts, suggesting the attack is an implicit RAG jailbreak rather than a simple instruction override."],"supporting_citations":[{"why":"Prior evidence that LLMs weight authoritative sources more heavily, providing the behavioral basis for the claimed authority bias.","marker":"[39]"},{"why":"Supplies the harmful-behavior benchmark and the gradient-based attack baseline DarkCite must outperform.","marker":"[40]"},{"why":"Provides the automated harm classifier used to judge whether victim-model outputs are harmful, defining the attack success rate metric.","marker":"[43]"},{"why":"Contributes the ASCII-art jailbreak baseline and the sampling protocol adopted for the second benchmark.","marker":"[47]"},{"why":"The persuasive-prompt baseline whose per-model success rates DarkCite is directly compared against.","marker":"[48]"},{"why":"A commercial moderation API tested as one of the defenses against the attack.","marker":"[52]"},{"why":"The perplexity-filter defense tested and shown to be evaded by DarkCite.","marker":"[54]"},{"why":"The randomized-sampling defense used in the combined defense evaluation.","marker":"[55]"},{"why":"Supplies the second harmful-risk benchmark with eleven risk categories used in the ablation study.","marker":"[58]"},{"why":"The retrieval-augmented generation paradigm used to formulate the claim that citation prompts act as an implicit RAG attack.","marker":"[64]"}],"fun_headline_variants":["LLMs trust authority citations—DarkCite exploits it for jailbreaks","DarkCite jailbreaks LLMs by abusing their trust in authoritative sources","Authority bias: a new LLM jailbreak vector via fake citations","DarkCite turns citation trust into 66% LLM jailbreak success","If it cites authority, LLMs obey: DarkCite's jailbreak exploit"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the risk-citation matching rules and hand-set weights ($w_1=0.1$, $w_2=0.9$) are derived from historical vulnerability logs that are representative of future harmful queries and independent of the evaluation sets, so the reported success rates reflect a generalizable attack rather than overfitting.","fun_headline_variants_meta":{"raw":{"variants":["LLMs trust authority citations—DarkCite exploits it for jailbreaks","DarkCite jailbreaks LLMs by abusing their trust in authoritative sources","Authority bias: a new LLM jailbreak vector via fake citations","DarkCite turns citation trust into 66% LLM jailbreak success","If it cites authority, LLMs obey: DarkCite's jailbreak exploit"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000713,"raw_usage":{"total_tokens":3240,"prompt_tokens":1012,"completion_tokens":2228,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":628,"completion_tokens_details":{"reasoning_tokens":2133}},"tokens_in":628,"tokens_out":2228,"duration_ms":14815,"temperature":1.0,"reasoning_tokens":2133,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T18:33:24.031772+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Construct the risk-citation matching rules and weights from a held-out log of historical attacks with no overlap with the harmful instructions used for evaluation, then replay DarkCite on fresh risk types; if the attack success rate advantage over the strongest persuasive-prompt baseline collapses, the reported gains are a fit to the evaluation set rather than a general authority bias.","supporting_citations":[{"cited_title":"Moderation","cited_arxiv_id":null,"evidence_quote":"A commercial moderation API tested as one of the defenses against the attack."},{"cited_title":"”Retrieval-augmented generation for knowledge- intensive nlp tasks.” Advances in Neural Information Processing Sys- tems 33 (2020): 9459-9474","cited_arxiv_id":null,"evidence_quote":"The retrieval-augmented generation paradigm used to formulate the claim that citation prompts act as an implicit RAG attack."}],"review_version":1}