{"id":"d4237070-c558-41c4-a043-b1afe1561eeb","arxiv_id":"2606.11198","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"unknown","formal_verification":"none","parameter_count":6,"one_line_summary":"Knowledge graph triples capture 2-3x more attention per token than equivalent natural language due to structural patterns, compressing demonstration attention by up to 42% independent of semantic relevance.","lead":"This paper shows that the format of retrieved knowledge (e.g., knowledge graph triples vs. natural language) independently captures disproportionate attention in LLMs, distorting in-context learning regardless of content relevance. A smart generalist should read it because it identifies a structural bottleneck in RAG systems that is orthogonal to retrieval quality.","discovery_kind":"unclear","skeptic_critique":{"model":"glm-5.2","headline":"C5b verbalization control shows the attention capture persists without KG format in Mistral (C5b/C2 ratio 0.90–1.14), suggesting the effect may be content-driven rather than format-driven for that model.","rationale":"The reader correctly identified the additive separation assumption (Eq. 2) as the weakest theoretical link, and correctly noted the absence of causal intervention. However, the reader framed this as a theoretical concern about an untested assumption. The C5b data for Mistral (Table 21) provide empirical evidence that the separation may actually fail: verbalized triples retain high attention (7.4–9.4%) far above neutral text (~3%), meaning the capture persists without KG format. This is not merely 'unproven' — it is contradicted for one of two models. The reader also listed this under the C5b supporting evidence for S3, but did not flag the implication for the central structural-vs-content claim. That said, the reader's overall CONDITIONAL verdict is appropriate. The attention capture phenomenon (C2≈C3, both > C5) is well-established and interesting. The 'structural' attribution is supported for LLaMA but not Mistral. The statistical power issues, quantization concerns, and lack of causal intervention are all correctly identified. My concern reinforces rather than overturns the CONDITIONAL verdict: the paper has a real observation but an incomplete account of its mechanism, with model-dependent evidence for the structural framing. The reader's confidence of MODERATE is appropriate.","tokens_in":19401,"tokens_out":5293,"duration_ms":170863,"concrete_test":"Directly compare C5b (verbalized triples) attention to C5 (neutral text) attention for both models, reporting the C5b/C5 attention ratio alongside the existing C5b/C2 ratio in Table 21. If C5b/C5 ≈ 1 for a model, format is the driver (verbalized content behaves like neutral text). If C5b/C5 >> 1 (as the reported numbers suggest for Mistral: ~9.4% vs ~3%), the attention capture is content-driven for that model and the 'structural' label is misapplied. Additionally, run a reverse control: extract semantic content from C5 neutral sentences and reformat as pseudo-triples (entity|relation|entity). If pseudo-triple-formatted neutral content shows elevated attention comparable to C2, format is implicated; if not, content is the driver. This would settle whether the effect is structural or semantic for each model.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is that attention capture is 'format-driven' (structural), not 'content-driven' (semantic). The cleanest experimental test of this distinction is C5b (verbalized triples): same semantic content as C2 but in natural-language format. If format drives the capture, C5b attention should drop toward C5 (neutral text) levels. For LLaMA-3-8B, this works: C5b reduces KG-region attention by 17–29% (Table 21, ratios 0.71–0.83). But for Mistral-7B, C5b attention barely changes from C2 (ratios 0.90, 0.90, 1.14 — TriviaQA actually increases). Critically, C5b attention for Mistral (7.4–9.4%) remains far above C5 neutral-text attention (~3%). This means that for Mistral, the attention capture persists even when the KG structural format is removed and the same content is presented as natural language. The paper's framework (Eq. 2) assumes a universal structural mechanism λ·σ(K) that operates independently of content, but the C5b data suggest that for Mistral, the elevated attention is driven by properties of the triple-derived content itself (entity density, vocabulary, semantic specificity) rather than by delimiter patterns or slot structure. The paper acknowledges Mistral shows 'weaker and less consistent effects' but still claims the structural tax as a general phenomenon. If the effect is content-driven in one of two models, the additive separation in Eq. 2 is not just unproven but empirically contradicted for that model, and the 'structural' attribution is misidentified. The reader's concern about additive separation is correct in direction but understates the problem: it is not merely an untested assumption — the C5b data for Mistral actively work against it.","agreement_with_reader":"partial"},"referee_report":{"model":"glm-5.2","summary":"This paper introduces the 'structural attention tax': the claim that knowledge graph (KG) triples, due to their relational delimiters and repeated slot patterns, capture 2–3× more attention per token than semantically equivalent natural-language text, compressing demonstration attention by up to 42% regardless of whether the triples are relevant or noise. The authors formalize this via a decomposition of attention scores into semantic and structural components (Eq. 2), derive a compression bound (Proposition 1), and test four predictions across two models (Mistral-7B, LLaMA-3-8B) and three QA benchmarks using a seven-condition study. They also compare KG retrieval against BM25 Wikipedia passage retrieval, finding a >30 pp source-alignment gap, and propose five mitigation strategies (two empirically tested). The paper is clearly written, the experimental design is thoughtful (particularly the C2/C3/C5 contrast and the C5b verbalization control), and the authors are commendably transparent about limitations. However, two issues bear on the central claim: the C5b control partially contradicts the format-driven (vs. content-driven) attribution for Mistral, and the statistical evidence for performance effects is weak (1/6 Bonferroni-corrected comparisons significant).","tokens_in":20412,"tokens_out":1953,"duration_ms":162505,"significance":"The paper addresses a real and underexplored question: whether the format of retrieved content, independent of its semantic relevance, systematically distorts attention in retrieval-augmented ICL. The C2/C3/C5 contrast is a clean experimental design for isolating format from content effects, and the multi-layer attention analysis (Appendix C.2) strengthens the claim beyond last-layer artifacts. The source-alignment dominance result (>30 pp gap) is a useful empirical anchor. The five-strategy mitigation taxonomy, grounded in the decomposition, provides a structured design space even though only two strategies are tested. The authors provide a full experiment-to-code mapping (Appendix H), which is a positive reproducibility signal. The theoretical framework is acknowledged as heuristic, which is appropriate given the correlational nature of the attention evidence.","major_comments":[{"comment":"§5.1, Table 21 (C5b): The verbalized-triple control is the cleanest test of the paper's central claim that attention capture is format-driven rather than content-driven. For LLaMA-3-8B, C5b reduces KG-region attention by 17–29% (ratios 0.71–0.83), supporting the format-driven attribution. However, for Mistral-7B, C5b attention barely changes from C2 (ratios 0.90, 0.90, 1.14 — TriviaQA actually increases), and C5b attention (7.4–9.4%) remains far above C5 neutral-text attention (~3%). This means that for Mistral, the elevated attention persists even when KG structural format is removed and the same content is presented as natural language. The paper acknowledges Mistral shows 'weaker and less consistent effects' but still claims the structural tax as a general phenomenon across both model families. The authors should either (a) scope the central claim to LLaMA-3-8B (or architectures with ","section":null},{"comment":"§5.2, Table 2: The statistical evidence for the confidence-dependent performance effect (Prediction 2) is weak. Only 1 of 6 Bonferroni-corrected McNemar comparisons is significant (LLaMA HotpotQA, p_Bonf = 0.001); the sign test across all six outcomes yields p = 0.69. The authors acknowledge this in §8 ('the confidence-dependent pattern is a directional trend requiring replication'), but the abstract and conclusion still present the confidence modulation as an established finding ('the semantic term determines whether this helps or hurts'). The framing in the abstract and conclusion should be tempered to match the statistical evidence — e.g., 'directional trend consistent with' rather than 'determines' — so that readers do not over-interpret the performance claims.","section":null},{"comment":"§3.2, Definition 1 and Eq. 2: The structural capture potential σ(K) (Definition 1) is defined in terms of structural pattern tokens and repetitiveness, and the paper then validates the structural tax by showing that high-σ(K) regions capture more attention. This is somewhat definitional: regions with more structural-pattern tokens are defined as having high σ(K), and the validation shows they capture more attention. The paper would benefit from an independent operationalization of σ(K) — e.g., computing σ(K) from surface features alone and then predicting attention capture in held-out data — to demonstrate that the measure has predictive value beyond its definition. Currently, the C2/C3 similarity (both high σ, different relevance) and C5 contrast (low σ) provide circumstantial evidence, but a direct test of σ(K) as a predictor would strengthen the claim.","section":null}],"minor_comments":[{"comment":"§3.3, Eq. 3: The competition coefficient η is introduced with range [0.5, 1.0] but no justification is given for this range. A brief note on why this interval is appropriate would help readers.","section":null},{"comment":"§4.2: The 4-bit NF4 quantization is noted to produce a +10 pp gap on HotpotQA C1 (Appendix A.2). This is a substantial perturbation relative to the 1–3 pp effect sizes in the main results. The paper should more prominently flag that fine-grained accuracy comparisons (e.g., C2 vs. C1 on CSQA: −2.0 pp) are within the quantization noise floor and should be interpreted as directional only.","section":null},{"comment":"§5.3, Table 3: The C7 (BM25) vs. C2 (ConceptNet) comparison conflates retrieval source, retrieval unit (passages vs. triples), token budget, and text coherence. The authors acknowledge this in §8 ('confounded by retrieval-unit differences'), but the >30 pp gap is still presented as the paper's 'strongest finding' in the discussion (§7). The caveat should appear alongside the result where it is first cited.","section":null},{"comment":"§6, Table 4: Three of five mitigation strategies (S2, S4, S5) are untested. The table marks these as 'Untested,' which is transparent, but the abstract's claim of 'five structure-aware mitigation strategies' could be read as implying empirical validation. Consider clarifying in the abstract that only two are empirically evaluated.","section":null},{"comment":"Figure 1: The caption states 'Illustrative values; see Section 5 for measured data.' This is good, but the figure is visually prominent and the values (25%→48%, 35%→20%) are more dramatic than the measured data (e.g., Mistral C2 KG attention ~10%, not 48%). Consider using measured values or adding a note that the figure is schematic and exaggerates the effect for visibility.","section":null},{"comment":"§3.2: The notation λ(l,h) suggests λ varies by layer and head, but the empirical estimates (λ ≈ 0.07–0.10 for Mistral, 0.03–0.06 for LLaMA) appear to be single scalar values. Clarify whether these are averages over layers/heads or from a specific layer.","section":null},{"comment":"Appendix F.1.3: The calibration of the compression bound uses T₀ ≈ 350 and m ≈ 30 without explaining how these values were obtained. A brief note on the derivation would improve transparency.","section":null},{"comment":"Table 1: The C6 column for Mistral TriviaQA (72.1) and LLaMA TriviaQA (73.1) appear to differ from the C6 values in Table 10 (72.1 and 73.1 respectively — consistent on re-reading, but the formatting makes cross-referencing difficult). A footnote or consistent formatting in the table itself would help.","section":null}],"recommendation":"major_revision","confidential_remarks":"The paper's core experimental design is sound and the topic is timely. The main concern is that the central claim ('format-driven, not content-driven') is only cleanly supported for one of two models, and the C5b data for Mistral suggest an alternative (content-driven) explanation that the framework does not accommodate. This is fixable by scoping claims appropriately and discussing the Mistral C5b result as a boundary condition rather than dismissing it as 'weaker effects.' The statistical power issue is acknowledged but the abstract/conclusion overclaim relative to the evidence. I would encourage the authors to reframe rather than retreat: the structural tax is a real and measurable phenomenon for LLaMA-3-8B, and the Mistral C5b data are informative about when format vs. content drives attention capture. The paper fits the journal's scope if the claims are properly scoped."},"author_rebuttal":{"model":"glm-5.2","summary":"We thank the referee for a careful and constructive review. The referee identifies three substantive issues; we agree with all three and will revise accordingly. Two require scoping/tempering of claims, and one suggests an additional experiment that we will conduct. We address each below.","responses":[{"response":"The referee is correct, and we will revise the manuscript to address this. We agree that the C5b results for Mistral-7B do not support the format-driven attribution as cleanly as the LLaMA-3-8B results do. For LLaMA-3-8B, verbalization reduces KG-region attention by 17–29% (ratios 0.71–0.83), which is consistent with the structural tax being format-driven. For Mistral-7B, the ratios (0.90, 0.90, 1.14) show that verbalization does not reliably reduce attention capture, and the residual attention (7.4–9.4%) remains well above the C5 neutral-text baseline (~3%). This means that for Mistral, some component of the elevated attention persists even when the KG structural format is removed and the same content is presented as natural language — which could reflect a content-driven component, a model-specific sensitivity to the verbalized triple content, or an interaction between verbalization style and Mistral's attention patterns that our control does not fully isolate. We do not have a definitive explanation for why Mistral behaves differently, and we will not speculate beyond the data. We will make the following changes: (1) In the abstract, we will scope the format-driven claim to note that the C5b verbalization control confirms the format-driven attribution for LLaMA-3-8B, while Mistral-7B shows the format-invariance pattern (C2≈C3 attention) but weaker evidence from the verbalization control. (2) In §5.1 and §8, we will explicitly state that the structural attention tax is most cleanly demonstrated in LLaMA-3-8B, and that the Mistral results support the format-invariance prediction (P3: noise and relevant triples capture similar attention) but not the format-flattening prediction (S3) as strongly. (3) We will adjust the conclusion to reflect this model-dependent evidence.","revision_made":"yes","referee_comment":"§5.1, Table 21 (C5b): For Mistral-7B, C5b attention barely changes from C2 (ratios 0.90, 0.90, 1.14), and C5b attention (7.4–9.4%) remains far above C5 neutral-text attention (~3%). The paper claims the structural tax as a general phenomenon across both model families but the C5b control partially contradicts the format-driven attribution for Mistral."},{"response":"The referee is correct. The statistical evidence does not support presenting the confidence-dependent performance effect as an established finding. Only LLaMA-3-8B on HotpotQA survives Bonferroni correction (p_Bonf = 0.001); the sign test across all six outcomes is non-significant (p = 0.69). We already acknowledge this in §8 ('the confidence-dependent pattern is a directional trend requiring replication'), but the abstract and conclusion overstate the result. We will revise the abstract to replace 'the semantic term governs whether this helps or hurts' with language such as 'the semantic term is hypothesized to govern whether this helps or hurts, though the performance evidence remains a directional trend (1/6 comparisons significant after Bonferroni correction).' We will make a corresponding change in the conclusion, replacing 'the semantic term determines whether augmentation helps' with 'the semantic term is expected to determine whether augmentation helps, consistent with a directional trend that requires replication.' We will also add a brief caveat in §5.2 noting that the theoretical framework predicts the confidence-dependent pattern but the empirical evidence is underpowered to confirm it.","revision_made":"yes","referee_comment":"§5.2, Table 2: Only 1 of 6 Bonferroni-corrected McNemar comparisons is significant; sign test p = 0.69. The abstract and conclusion still present the confidence modulation as an established finding ('the semantic term determines whether this helps or hurts'). The framing should be tempered."},{"response":"We agree that the current validation is circumstantial rather than a direct predictive test of σ(K), and that this is a legitimate methodological gap. The referee's suggestion — computing σ(K) from surface features alone and predicting attention capture on held-out data — is a clean way to demonstrate that the measure has predictive value beyond its definition. We will conduct this analysis in revision. Specifically, we will: (1) Compute σ(K) for each knowledge region in our existing conditions using only surface features (structural-pattern token density and repetitiveness as defined in Definition 1), without reference to attention values. (2) Additionally generate a set of held-out prompt variants with intermediate σ(K) values — e.g., partially structured formats mixing triples with prose at varying ratios — to provide a continuous range of σ(K) rather than just the binary high/low contrast (C2/C3 vs. C5). (3) Test whether σ(K) predicts KG-region attention capture in these held-out variants using regression or rank correlation. If σ(K) predicts attention capture on held-out data with formats not used to define the measure, this would address the circularity concern. If the predictive relationship is weak or absent, we will report that honestly and discuss what it implies for the framework. We note that even if σ(K) proves to be a strong predictor, this remains correlational evidence; causal validation via attention masking is important future work that we already flag in §8.","revision_made":"yes","referee_comment":"§3.2, Definition 1 and Eq. 2: σ(K) is defined in terms of structural pattern tokens and repetitiveness, and the paper validates the structural tax by showing that high-σ(K) regions capture more attention. This is somewhat definitional. The paper would benefit from an independent operationalization of σ(K) — e.g., computing σ(K) from surface features alone and then predicting attention capture in held-out data."}],"tokens_in":19273,"tokens_out":1705,"duration_ms":53149,"standing_objections":[]},"desk_editor":{"model":"glm-5.2","letter":"The paper identifies a real phenomenon — KG triples capturing 2–3× more attention per token than equivalent natural-language text, independent of relevance — but the 'structural' attribution doesn't hold cleanly for both models, and the performance-effect claims are statistically underpowered. The attention-capture observation itself is the strongest finding; the broader theoretical framework is heuristic and only partially supported.","headline":"The paper identifies a real phenomenon — KG triples capturing 2–3× more attention per token than equivalent natural-language text — but the 'structural' attribution doesn't hold cleanly for both models, and the performance-effect claims are statistically underpowered.","tokens_in":20313,"tokens_out":163,"would_cite":false,"duration_ms":295113,"reading_group":"no","serious_thinker":"no","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"glm-5.2","headline":"KG triples steal 2-3x more attention than equivalent prose, regardless of relevance","keywords":[],"falsifier":"If attention masking or activation patching interventions (acknowledged as future work) fail to show that reducing KG-region attention restores demonstration attention and changes performance accordingly, the causal claim that format-driven attention capture compresses demonstrations would be weakened to a correlational observation.","tokens_in":19336,"feed_emoji":"🔍","tokens_out":1240,"duration_ms":104977,"temperature":0.7,"pith_summary":"The paper identifies and formalises a phenomenon it calls the structural attention tax: when knowledge graph triples are injected into a language model's prompt, their distinctive formatting — relational delimiters, repeated slot patterns, high token regularity — causes them to capture 2–3 times more attention per token than semantically equivalent natural-language text, compressing the attention available for task demonstrations by up to 42%. Crucially, this attention capture is format-driven, not content-driven: irrelevant triples absorb almost exactly as much attention as relevant ones. The authors formalise this by decomposing attention scores into a semantic component (which determines whether diverted attention helps or hurts performance) and a structural component λ·σ(K) (which determines how much attention is taxed away). This decoupling yields two orthogonal axes for improving retrieval-augmented in-context learning: optimising what you retrieve (semantic axis) and reducing format-driven attention capture (structural axis). Empirically, across Mistral-7B and LLaMA-3-8B on three QA benchmarks, the semantic axis dominates — task-matched BM25 retrieval beats mismatched ConceptNet triples by over 30 percentage points on HotpotQA, dwarfing all gating strategies (≤2 pp). Of five framework-derived mitigation strategies, format flattening (converting triples to natural sentences) is validated by both accuracy and attention-level evidence, while structural dispersal (interleaving triples with bridging prose) produces model-dependent effects and can introduce new attention anchors.","feed_headline":"KG triples steal 2-3x more attention than equivalent prose, regardless of relevance","feed_subtitle":"Structured retrieval format independently hijacks LLM attention, compressing demonstration attention by up to 42% — format flattening is a零","key_machinery":"The structural attention tax decomposition (Eq. 2): attention allocated to a knowledge region splits into a semantic relevance component and a structural capture component λ·σ(K). Proposition 1 provides a compression bound showing the structural term enters exponentially, and Definition 1 formalises σ(K) as a function of structured-pattern token density and repetitiveness. The framework generates four testable predictions: source dominance (P1), confidence modulation (P2), format-invariant capture (P3), and compression–performance decoupling (P4).","core_discovery":"The central object is the structural attention tax: a format-driven, content-independent attention bias formalised as λ·σ(K), where σ(K) quantifies the structural intensity of a prompt region (triple density, delimiter frequency, slot repetitiveness) and λ is a model-intrinsic bias coefficient. The decomposition A_K = A_sem + λ·σ(K) separates attention into a semantic term (governing whether captured attention carries useful signal or noise) and a structural term (governing how much attention is captured). The structural term enters the demonstration compression bound exponentially (Proposition 1), meaning even modest format bias produces substantial compression of demonstration attention. A","pith_inferences":[],"forward_implications":["Format normalisation — converting structured data (KG triples, SQL, JSON, code blocks) to natural prose before injection — may be a standard preprocessing step for RAG pipelines, not an optional optimisation.","The structural attention tax likely extends beyond KG triples to any prompt region with distinctive formatting patterns, including SQL queries, JSON objects, and code blocks, though this is untested in the paper.","Retrieval quality (what you retrieve) and retrieval format (how you present it) are independent levers; improving one does not compensate for neglecting the other, and the semantic axis currently dominates by an order of magnitude.","Training-time regularisation that penalises attention to noise-structured regions (Strategy S5) could durably reduce model-intrinsic format bias λ, but this remains untested.","Gating strategies that decide whether to inject knowledge based on confidence (≤2 pp gains) are far less impactful than matching the knowledge source to the task (>30 pp gains), suggesting the field's focus on gating sophistication may be misplaced."],"fun_headline_variants":["Retrieval format steals attention from demonstrations regardless of content","Structured retrieval triples capture 2-3x more attention than prose in LLMs","Format-driven attention tax compresses demonstrations by up to 42 percent","Knowledge graph triples hijack LLM attention independent of semantic relevance","Retrieval format bias competes with semantic quality as orthogonal attention axes"],"cache_read_input_tokens":0,"weakest_assumption_plain":"The framework assumes that attention scores can be cleanly separated into an additive semantic component and an additive structural component, where the structural bias is independent of the semantic content. If structural patterns interact with specific semantic categories — for example, if certain types of facts are more distracting when presented as triples than others — the clean decoupling between 'how much attention is taxed' and 'whether it helps or hurts' would break.","fun_headline_variants_meta":{"raw":{"variants":["Retrieval format steals attention from demonstrations regardless of content","Structured retrieval triples capture 2-3x more attention than prose in LLMs","Format-driven attention tax compresses demonstrations by up to 42 percent","Knowledge graph triples hijack LLM attention independent of semantic relevance","Retrieval format bias competes with semantic quality as orthogonal attention axes"]},"model":"glm-5.2","effort":"high","cost_usd":0.0,"raw_usage":{"total_tokens":816,"prompt_tokens":740,"completion_tokens":76,"prompt_tokens_details":null},"tokens_in":740,"tokens_out":76,"duration_ms":13381,"temperature":1.0,"reasoning_tokens":null,"cache_read_input_tokens":0,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-05T08:55:20.585520+00:00","model_set":{"reader":"glm-5.2"},"falsifier":"If attention masking or activation patching interventions (acknowledged as future work) fail to show that reducing KG-region attention restores demonstration attention and changes performance accordingly, the causal claim that format-driven attention capture compresses demonstrations would be weakened to a correlational observation.","supporting_citations":[],"review_version":1}