{"id":"f2a00c54-cceb-45d0-9131-37886bd0cdda","arxiv_id":"2602.06495","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":7,"one_line_summary":"GRASP, a closed-box multi-turn prompt attack, reconstructs type-faithful one-hop subgraphs from defended Graph RAG systems, reaching up to 82.9 F1 where prior attacks fail.","lead":"This paper shows that an attacker who can only send chat queries to a Graph RAG service can reconstruct the one-hop relationships of a target entity, even when the service is protected by prompts that forbid leaking the graph. The authors' attack, GRASP, reaches up to 82.9 F1, and they propose two context-level defenses that reduce it.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"RType F1 may depend on an undocumented REL_TYPE column: vanilla GraphRAG/LightRAG/nano-GraphRAG context has only free-text relationship descriptions, so the headline typed-reconstruction result may not transfer to unmodified deployments.","rationale":"The reader's weakest assumption (context construction) is directionally correct, but the sharper issue is the REL_TYPE column. The paper's attack and evaluation are built around typed triples, yet the public Graph RAG frameworks it claims to test do not expose a relation type field in their default local-search context. The authors may have customized the pipeline without documenting it, which is a reproducibility and scope risk. The paper's own ID Alignment and Decoy defenses show that small changes to the table format can collapse F1 (Table 5), corroborating sensitivity to schema. Nevertheless, the safe-prompt result (baselines near zero, GRASP high) is internally consistent and the attack is well-designed for typed graph schemas; the work is not fatally flawed. The verdict should remain CONDITIONAL, with the explicit condition that the deployed Graph RAG context contains a typed relation field. This is a refinement of, not a disagreement with, the reader's context-construction concern—hence 'partial' agreement.","tokens_in":23975,"tokens_out":13263,"duration_ms":121971,"concrete_test":"Obtain or reproduce the exact graph-construction and context-serving code used for the Enron experiments and check whether the served relation table has a discrete REL_TYPE column distinct from the description. Then rerun GRASP against a vanilla GraphRAG local-search context in which relationships are only (id, source, target, description), keeping the extraction template unchanged, on the same 50 targets and 10-query budget. Compare RType F1 to the reported 65.6/82.9. If RType F1 collapses to the low levels of the naïve baselines, the typed-reconstruction claim is contingent on a nonstandard schema; if it remains high, the concern is refuted. The same check should be repeated for LightRAG and nano-GraphRAG with their default context formats.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The most load-bearing assumption is not merely that relation tables exist in the context, but that they contain an explicit, normalized REL_TYPE field. The extraction template (Fig. 6) instructs the model to emit quadruples (ID, SRC_ENTITY, REL_TYPE, DST_ENTITY) 'exactly as listed in the table' and to use strings verbatim. However, the default local-search context of the evaluated frameworks (Microsoft GraphRAG, LightRAG, nano-GraphRAG) stores relationships as (id, source, target, description); relation semantics live in free-text descriptions, not in a discrete typed column. The paper never shows the graph-construction prompt (§6.1.1 only lists model names; Table 7 omits the prompt). Table 8 reports top-5 'relation types' (compliance, request, ...) that are not part of the vanilla GraphRAG schema, and the Decoy defense's example table (Table 4) shows the relation type embedded in a description field ('Type: call'), suggesting the actual served format differs from standard GraphRAG. Unless the authors modified the graph schema to expose a relation type attribute, the RType metric cannot be satisfied by verbatim extraction, and the headline RType F1 (up to 82.9) may be an artifact of a custom typed schema rather than a property of the cited Graph RAG systems. This is a scope/correctness risk: the paper's central claim is about 'defended Graph RAG deployments' generally, but the evaluation may apply only to schema-enriched variants.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies data extraction from Graph RAG systems under a closed-box threat model in which the provider installs a safe system prompt that forbids verbatim disclosure and graph-structure leakage. It first analyzes six prior extraction prompts and attributes their failure to three causes: explicit exfiltration intent triggering refusals, safety-induced rewriting breaking exact relation fidelity, and fixed-query repetition saturating quickly. The paper then proposes GRASP, a multi-turn attack that (i) reframes extraction as a legitimate relation-extraction task from retrieved context tables, (ii) uses per-instance IDs and a strictly formatted quadruple output to reduce hallucination and preserve typed relations, and (iii) schedules four diversity templates via a momentum-aware policy with a query budget and Good-Turing early stopping. Experiments on Enron and HealthCareMagic knowledge graphs with four LLMs report GRASP reaching up to 82.9 RType F1 and 83.5 Naïve F1 under the safe-prompt defense, with prior baselines collapsing to near-zero RType F1. The paper also evaluates user-prompt, summarization-based, rejection-based, and decoding-time defenses, and proposes two context-construction defenses (ID Alignment and Decoy) that reduce GRASP's F1 to mid-teens while preserving benign QA utility.","tokens_in":24341,"tokens_out":6022,"duration_ms":64758,"significance":"If the findings hold, the paper identifies a real and understudied confidentiality risk: prompt-level safeguards that block verbatim repetition do not prevent a task-framed attacker from recovering typed relational structure from Graph RAG context. The contribution is useful and timely. The strengths are substantive: the paper includes a concrete threat model, a failure analysis of prior attacks with rejection/rewriting measurements, a well-motivated attack design, ablations isolating each design component, a defense evaluation including a decoding-time method, and exact prompts in the appendix. The attack's transfer across three Graph RAG frameworks and four LLMs is a valuable empirical claim. The main limitation is that the central quantitative claim is contingent on the exact schema of the relation tables served by the attacked systems, and this is not documented sufficiently. Reproducibility is also weakened by the absence of code/data and of variance information for the 50-target macro-averages.","major_comments":[{"comment":"The RType metric and the extraction template require an explicit, normalized REL_TYPE field in the served relation table: the template instructs the model to copy 'REL_TYPE' and strings 'exactly as listed in the table'. However, the paper never shows the graph-construction prompt or the actual served context schema. Table 8's typed vocabulary (compliance, request, reporting, meeting, attachment) is not part of the default Microsoft GraphRAG/LightRAG/nano-GraphRAG relationship schema, which stores relation semantics in free-text descriptions; Table 4 itself shows 'Type: call' embedded inside a description field. This raises a load-bearing scope question: either the authors modified the graph schema to add a REL_TYPE attribute, in which case the headline 'up to 82.9 RType F1' applies to schema-enriched variants rather than the cited vanilla Graph RAG frameworks and Figure 10 overstates tra","section":"§5.3, Fig. 6; §6.1.1, Tables 7 and 8; Table 4"},{"comment":"The main quantitative claims rest on macro-averages over 50 randomly sampled target entities, but the paper provides no error bars, confidence intervals, seeds, or per-target distributions. This matters because Algorithm 1 includes stochastic template sampling and the target sets are random; two target samples could plausibly move the headline numbers by several F1 points, especially for the 'baseline collapses to 0.0' comparisons. Please provide variance information, seeds, or a reproducibility artifact (code/data release) so the reader can assess the stability of the central comparison.","section":"§6.1.3, Table 3, Algorithm 1"}],"minor_comments":[{"comment":"The text says the ID 'need not explicitly exist in the retrieved context', but the extraction template in Figure 6 tells the model to 'use ID and strings verbatim'. These statements are in tension and should be clarified: are the IDs assigned by the model or copied from an ID column?","section":"§5.3"},{"comment":"The legend label 'recall of w.o div' appears misplaced or incomplete; it likely refers to the w.o div bar in the recall group.","section":"Fig. 8"},{"comment":"Typographical errors: reference [3] 'Antropic' should be 'Anthropic'; reference [32] 'Micosoft' should be 'Microsoft'.","section":"References"},{"comment":"The 'Na\"ive' heading shows a raw LaTeX escape; please use the proper unicode or formatted text.","section":"Table 3"},{"comment":"Worm and FG are described as 'advanced attacks that iteratively optimize queries', but Table 3 evaluates them with a fixed prompt template plus static formatting (Figure 13), not the full iterative optimization. The text later says they were 'adapted' and 'strengthened', but the caption/description should state clearly that the iterative search components were not exercised for the main comparison, so readers do not infer the baselines include their full published machinery.","section":"§6.1.2, Figure 13"},{"comment":"The Rouge-L comparison compares responses under the original versus the safe prompt, but the safe prompt also changes system behavior beyond enforcing non-disclosure; this is acceptable as a descriptive measure but should not be read as a causal estimate of 'safety-induced paraphrasing' alone.","section":"§4.2"}],"recommendation":"major_revision","confidential_remarks":"The relation-type schema question is the key obstacle. If the authors can provide the actual served context schema and construction prompt, or re-scope the claims to schema-enriched Graph RAG, the paper would be much stronger and potentially acceptable. The empirical design is otherwise thorough. I would also encourage the authors to release code and data; for an attack paper with random target sampling and stochastic scheduling, absence of variance measures is an unusually large gap."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a real contribution with a scope problem in its headline metric. The attack design is clever and the evaluation is the most thorough I've seen for Graph RAG extraction, but the RType numbers almost certainly depend on a relation-type column that the stock frameworks don't provide, and the paper never shows the graph-construction prompt or the exact relation table schema. The Naïve results survive that caveat, so the core finding—defended Graph RAG can be exfiltrated by a query-only adversary—is probably true, but the type-faithful claim needs honest scoping.\n\nWhat the paper does well: it identifies why prior attacks fail (explicit intent, paraphrasing, saturation), reframes extraction as a context-processing task, uses per-record IDs to kill hallucinated relations (precision drops from 64.9 to 34.4 without IDs), and adds a scheduler that materially beats random template choice (59.8 vs 65.6 F1). The defense evaluation is fair—they test prompt-blocking, summarization, PAD, and even their own mitigations, and they honestly report that their best defenses still leave mid-teen F1. That's credible.\n\nSoft spots, in order of importance:\n1. The REL_TYPE issue. Fig 6's extraction template tells the model to emit 'REL_TYPE exactly as listed in the table,' and Table 8 lists relation types (compliance, request, ...) that are not part of vanilla GraphRAG's relationship schema. LightRAG and nano-GraphRAG don't have typed relation columns either. The paper never shows the graph-construction prompt or the relation table format. So I suspect the 82.9 RType F1 is measured against a schema the authors built, not stock frameworks. The stress-test note is right. Interestingly, their own Decoy defense table (Table 4) shows the type embedded in the description ('Type: call ...'), suggesting the attack could still extract types from free text—but that's not tested, and it would likely lower the RType score.\n2. No code/data and no error bars. The numbers are plausible but unverifiable in this form.\n3. Hand-tuned scheduler parameters (α, thresholds, τ, etc.). The ablation helps, but sensitivity analysis is thin.\n\nWeaker concerns: the target sampling only includes degree ≥ 5, so very sparse targets are not tested; the Good-Turing stopping rule is reasonable but the 0.3 threshold is a free parameter.\n\nWho should read it: anyone working on RAG security, and graph-RAG practitioners who assume prompt guards are sufficient. It deserves a serious referee but not as-is. The authors should release the graph schema and prompts, test at least one vanilla configuration without a typed column, and add error bars. If the typed-row concern is resolved, this is a top-conference paper; if it's not, the scope narrows to custom typed graph stores.\n\nMy recommendation: send it to peer review, and make the schema transparency a condition. I'd bring it to a reading group, and I'd cite the Naïve reconstruction result and the two defenses even now.","headline":"A genuinely new Graph RAG extraction attack with a real scope problem: the headline RType F1 almost certainly depends on a typed relation schema that the stock frameworks don't provide, and the paper never shows the schema.","tokens_in":24897,"tokens_out":3978,"would_cite":true,"duration_ms":39597,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A query-only attacker can reconstruct a target entity's typed one-hop subgraph from a defended Graph RAG service by reframing extraction as a routine relation-extraction task, reaching F1 scores in the low 80s where prior attacks fail.","keywords":["subgraph reconstruction","Graph RAG","data extraction attack","relation extraction","prompt-based defense","instance grounding","adaptive query scheduling","privacy leakage"],"falsifier":"Build a Graph RAG service that answers the same anchor query without exposing a normalized relation table — for example, by providing only narrative summaries or by fusing relation records into opaque text — then run GRASP against it. If RType F1 remains high, the attack is not as context-dependent as claimed; if it collapses to the near-zero level of the prior baselines, the paper's core mechanism is confirmed.","tokens_in":23822,"feed_emoji":"🔓","tokens_out":3829,"duration_ms":40389,"temperature":0.7,"pith_summary":"The paper tries to establish that Graph RAG systems remain vulnerable to targeted subgraph reconstruction even when the service explicitly prohibits verbatim disclosure and graph-structure leakage. It introduces GRASP, a closed-box, multi-turn attack that reframes extraction as a legitimate relation-extraction task, and reports type-faithful F1 up to 82.9 while six prior attack prompts collapse to near zero under the same safe-prompt defense. The paper also proposes two lightweight context-construction defenses — ID Alignment and Decoy — that reduce reconstruction fidelity to the teens while preserving benign utility, yet residual leakage persists. If correct, the result means intent-detection style prompt defenses are not enough to protect graph-structured knowledge assets.","feed_headline":"Queries alone reconstruct defended Graph RAG subgraphs","feed_subtitle":"Reframing extraction as a relation-tagging task beats safe prompts, reaching ~83 F1 on typed triples.","key_machinery":"The central mechanism is the per-instance identifier in the quadruple format (ID, src, rtype, dst): it anchors each extracted relation to a specific retrieved record, preventing the model from composing plausible but false relations from pieces of different records. The second mechanism is the adaptive prompt scheduler, which tracks extraction momentum via an exponential moving average, uses a novelty-based stopping rule estimated from the frequency of newly seen relations, and selects among four diversity templates (context-frame drift, type expand, type explore, residual extract) to escape saturation and operate within a strict query budget.","core_discovery":"GRASP recovers typed one-hop subgraphs from a defended, closed-box Graph RAG system by treating extraction as a constrained relation-extraction task rather than as an explicit request to repeat retrieved content. Each attack query asks the model to emit relation quadruples — (ID, src, rtype, dst) — taken verbatim from the retrieved relation tables, and only for relations incident to a target entity. The per-record ID acts as an instance delimiter that suppresses hallucinated cross-record combinations, which the paper identifies as the dominant source of false positives (83.9%). A momentum-aware scheduler then chooses among four diversity templates to keep discovering unseen relations within","pith_inferences":["A natural extension is to use GRASP as a red-team audit tool: providers could run it against their own Graph RAG deployments to measure what fraction of the knowledge graph is query-reconstructible before an attacker exploits the same vector.","The scheduling and stopping dynamics suggest a possible detection signal: a sudden plateau in the novelty of extracted relations across a single user's queries could indicate automated subgraph probing, though the paper does not explore this monitoring angle.","Because the attack succeeds within roughly ten queries, per-account rate limiting or anomaly detection on query diversity might blunt it in practice; the paper does not test such operational countermeasures.","The success of the decoy defense implies that the context-table schema itself is a security parameter, so the arms race may shift toward obfuscating or randomizing the structure of retrieved relation records."],"forward_implications":["Prompt-level safeguards that block explicit exfiltration intent are insufficient: once extraction is framed as a legitimate processing task, the same safe prompt barely reduces reconstruction fidelity.","Stronger safety-aligned LLMs can execute the constrained extraction task more reliably, meaning improved model capability may amplify, not reduce, this attack surface.","Graph RAG deployments that serve undirected relations are more exposed, because the attacker no longer needs to recover edge direction.","Context-construction defenses that disrupt instance identity (ID Alignment) and field attribution (Decoy) reduce reconstruction F1 to the mid-teens while keeping utility intact, but residual leakage remains even when layered with rejection-based blocking."],"fun_headline_variants":["Reframed extraction defeats safe prompts in Graph RAG","Subgraph reconstruction bypasses Graph RAG defenses","Relation-extraction attack recovers defended Graph RAG","Graph RAG leaks subgraphs even with safe prompts","Reframed query task cracks Graph RAG privacy"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The attack works only when the Graph RAG service places the target entity's incident edges into the LLM's context as structured, parseable relation records in response to a generic anchor query; a deployment that hides or omits relation tables would invalidate the central feasibility claim.","fun_headline_variants_meta":{"raw":{"variants":["Reframed extraction defeats safe prompts in Graph RAG","Subgraph reconstruction bypasses Graph RAG defenses","Relation-extraction attack recovers defended Graph RAG","Graph RAG leaks subgraphs even with safe prompts","Reframed query task cracks Graph RAG privacy"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000372,"raw_usage":{"total_tokens":1838,"prompt_tokens":766,"completion_tokens":1072,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":510,"completion_tokens_details":{"reasoning_tokens":996}},"tokens_in":510,"tokens_out":1072,"duration_ms":8259,"temperature":1.0,"reasoning_tokens":996,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-03T03:52:09.174628+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Build a Graph RAG service that answers the same anchor query without exposing a normalized relation table — for example, by providing only narrative summaries or by fusing relation records into opaque text — then run GRASP against it. If RType F1 remains high, the attack is not as context-dependent as claimed; if it collapses to the near-zero level of the prior baselines, the paper's core mechanism is confirmed.","supporting_citations":[],"review_version":1}