{"id":"716b7a7b-00d2-4583-b7c8-0a6e245f7934","arxiv_id":"2505.11995","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"RAG knowledge use in LLMs proceeds through four stages (refinement, elicitation, expression, contestation), and a new entropy metric (KAPE) identifies neurons whose deactivation shifts reliance between internal and external knowledge.","lead":"This paper dissects how large language models use retrieved documents during question answering, identifying four internal stages of knowledge flow. It also introduces a method (KAPE) to find and switch off the specific neurons that drive the model to rely on either its memory or the document.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Key token extraction, the pivot of every information-flow metric, is never operationally defined; the four-stage decomposition is unverifiable until it is specified and tested.","rationale":"The reader's weakest assumption already identifies the same load-bearing gap: the K token group is undefined and the attention sums are unnormalized. My concern sharpens this into a specific falsifiable requirement: the entire macro-level analysis is conditional on an operational K extractor, and the paper provides none. I do not treat this as a rejection because the gap is in principle fixable and the KAPE deactivation experiments provide an independent, though not fully baseline-controlled, source of evidence for neuron-level controllability. A conditional verdict with the requirement to release the K-extraction code and to test robustness to alternative K definitions is therefore appropriate. I also note a related internal inconsistency: Eq. (5) invokes a supervised fine-tuning loss L_sft although all models are described as Base versions and no fine-tuning is reported, so the saliency corroboration in Section 3.3 is not reproducible as written. This reinforces, but does not replace, the main concern about K.","tokens_in":18621,"tokens_out":8216,"duration_ms":90811,"concrete_test":"Reconstruct the pipeline with an explicit K extractor and recompute Figure 1 under three definitions: (a) tokens of the generated answer a_ext appended to the prompt in a second forward pass; (b) the gold answer span located in the passage; (c) a random span of the same token count from the passage. In each variant, also recompute Eq. (4) with per-token-pair normalization (divide by |X|*|Y|). If the layer-wise order and stage boundaries of IF_kc, IF_kq, and IF_ka change materially across these variants, the four-stage decomposition is an artifact of the unspecified K construction rather than a stable model property.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The four-stage decomposition and the relevance-guided streaming result rest entirely on the IF metrics in Eq. (4), all of which are sums of attention (or saliency) weights over token groups C, K, Q, and A. The pivot group K, however, is never defined operationally. Section 3.1 says K is 'the potential answers extracted by the LLM from the context, obtained following Equation (3)', but Eq. (3) merely defines a_ext = LLM(I,q,P); it does not specify whether K is the generated answer string, the gold answer span matched back into the passage, or some other extraction, nor how the corresponding tokens are located and tokenized. Because K appears as the source group in every IF_kc, IF_kq, and IF_ka curve in Figure 1, an ambiguity in K propagates into every stage boundary and every relevance comparison. In addition, Eq. (4) is a raw sum over all token pairs, unnormalized by the sizes of X and Y; as the number of key tokens changes across layers or across passage types, the rise-and-fall shapes could reflect token-count and position artifacts rather than knowledge transfer. Unless K extraction and normalization are pinned down, the attention curves cannot be read as evidence for the four sequential knowledge stages.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper presents a mechanistic study of how LLMs use internal (parametric) and external (retrieved) knowledge in retrieval-augmented open-domain QA. Using attention-based information-flow (IF) metrics defined over four token groups (context C, key K, query Q, and answer prompt A), the authors observe layer-wise trends and propose a four-stage decomposition of knowledge utilization: refinement, elicitation, expression, and contestation (Sec. 3.2). They further report that passage relevance steers the knowledge elicitation stage, corroborate the stages with gradient-based saliency and a causal-mask intervention, and support the relevance-guided account empirically (Secs. 3.3-3.4). At the module level, the paper introduces Knowledge Activation Probability Entropy (KAPE) to identify neurons associated with internal versus external knowledge, and shows that deactivating these neurons shifts QA accuracy on Natural Questions, TriviaQA, and HotpotQA across LLaMA-3-8B, LLaMA-3-70B, and Qwen-2.5-1.5B (Table 2). Finally, unembedding-logit analyses are used to argue that MLP layers verify the accuracy of external knowledge while MHA layers integrate information from multiple sources (Sec. 4.2). The headline claims are that RAG knowledge integration proceeds in identifiable stages and that it can be causally controlled at the neuron level.","tokens_in":18776,"tokens_out":12814,"duration_ms":110208,"significance":"If the results hold, the paper delivers a rare combination of information-flow and causal, neuron-level evidence about RAG, spanning two model families (LLaMA and Qwen), several scales, and three datasets, with KAPE deactivation effects transferring across datasets in directionally consistent ways (deactivating external-knowledge neurons hurts more under gold documents and can improve accuracy under noisy documents). The intervention experiments in Sec. 4.1 go beyond correlational attention analysis, and the cross-model consistency in Sec. 3.2 suggests the four-stage phenomenology is not an artifact of one architecture. The paper is also explicit about its claims, which makes them testable. However, the headline macro-level decomposition rests entirely on the IF definitions in Eq. (4), whose key-token group is not defined operationally and whose aggregation is not normalized; until those definitions are pinned down, the stage boundaries and relevance comparisons cannot be independently verified. The novelty is also partially incremental relative to the cited knowledge-conflict literature (refs. [12, 13] in the text).","major_comments":[{"comment":"The token group K, which is the source for every information-flow metric, is never defined operationally. Section 3.1 states only that K denotes 'the potential answers extracted by the LLM from the context, obtained following Equation (3)', but Eq. (3) merely defines a_ext = LLM(I, q, P) and does not specify how the answer is aligned back into the passage tokens, whether exact string matching is used, how multi-token or paraphrased answers are handled, or which occurrence is selected when the answer appears multiple times. Because K is the source group X in IF^KC_a, IF^KQ_a, and IF^KA_a in Eq. (4), every stage boundary in Figures 1 and 3 and every relevance comparison inherits this ambiguity. Please provide a precise extraction-and-alignment procedure, including worked examples, so that the curves are reproducible and falsifiable.","section":"Sec. 3.1, Eqs. (3)-(4)"},{"comment":"The IF metrics are raw sums of attention weights over all token pairs, with no normalization for the sizes of the groups X and Y. The size of the key group changes across the experimental conditions by construction: the fake passages replace the correct answer with an incorrect string of different length (Sec. 2.2), and the relevance conditions in Sec. 3.4.1 use passages of different lengths and different answer spans. A raw sum can therefore vary in magnitude purely because of token counts, and the claims that gold and fake passage patterns are 'largely consistent' (Sec. 3.2.1) and that decreasing relevance makes the flows 'less noticeable' (Sec. 3.4.2) need to be re-examined with normalized metrics (e.g., mean attention weight per directed token pair) or with length-matched baselines.","section":"Sec. 3.1, Eq. (4)"},{"comment":"The four stage boundaries are identified by visual inspection of the layer-wise curves in Figure 1, and Figure 1 reports only the average over three datasets with no variance or significance information. Since the four-stage decomposition is the paper's central macroscopic claim, the authors should provide a quantitative boundary-detection rule (e.g., extrema or change points of the averaged IF curves), a robustness check across random subsets of questions, and confidence bands. Without such a rule, later claims such as 'relevance primarily affects the knowledge elicitation stage' (Sec. 3.4) are not well-defined, because the elicitation stage's layer span itself is determined by eyeballing the same curves.","section":"Sec. 3.2.2"},{"comment":"The KAPE identification pipeline leaves several load-bearing details unspecified: the threshold used to retain 'neurons with significant activation probabilities' is never given a value, the top-1% selection ratio is a free parameter with no sensitivity analysis, and the number of questions used in the deactivation experiments underlying Table 2 is not reported. Table 2 also reports no error bars or significance tests, so the differences between the 'Deactivate IK' and 'Deactivate EK' rows (e.g., average EM drops of -6.57 versus -12.64 on gold documents for LLaMA-3-8B) cannot be assessed for statistical reliability. Please report the identification settings, the evaluation sample sizes, and bootstrap or standard-error estimates for the main comparisons.","section":"Secs. 4.1.1-4.1.2, Table 2"},{"comment":"Two of the corroborations share the very contrast they are meant to verify. KAPE selects neurons by comparing activation probabilities between the RAG and closed-book settings, and the deactivation experiment then measures the effect of ablating these neurons on exactly that internal-versus-external contrast; the saliency corroboration in Sec. 3.3 uses Eq. (5), which is a gradient-weighted version of the same attention matrices that define the stages. The cross-dataset transfer of the KAPE neurons (identified on NQ, evaluated on TriviaQA and HotpotQA) partially mitigates the first concern, but the paper should still demonstrate that the attribution has content beyond the selection signal, for example by comparing against control neuron sets matched on overall activation frequency or by using an independent localization method such as causal tracing with different interventions.","section":"Secs. 3.3 and 4.1.1"}],"minor_comments":[{"comment":"The caption of Figure 1 says the figure shows 'two versions of LLaMA-3 with different parameter scales', while the text claims consistency across LLaMA-3-8B, LLaMA-3.1-8B, LLaMA-3-70B, Qwen-2.5-1.5B and Qwen-2.5-7B; please state explicitly which models are plotted and which are only discussed verbally.","section":"Fig. 1 and Sec. 3.2.1"},{"comment":"Several figures and one equation contain glyph/font artifacts in the review copy (the '/uni00000013...' sequences in figure labels and the garbled expectation/norm symbols in Eq. (8)); the camera-ready version must ensure that all formula and axis-label text renders with a proper math font.","section":"Figures 1, 3, 4, 7 and Eq. (8)"},{"comment":"The refinement stage is defined in Sec. 3.2 through key-to-context flow (IF^KC_a), but the saliency discussion in Sec. 3.3 describes the model as 'increas[ing] the flows from the context to the key'; the direction of the proposed interaction should be stated consistently for each stage.","section":"Secs. 3.2.1 and 3.3"},{"comment":"The definition of the activation probability p^k_{i,j} is difficult to parse; please rewrite it in standard notation and clarify whether p^IK_{i,j} and p^EK_{i,j} are computed over the same question set in the closed-book and RAG conditions.","section":"Sec. 4.1.1, Eq. (8)"},{"comment":"It is not immediately clear why masking key-to-query attention increases the probability of generating a_ext for higher-relevance passages; a brief explanation of the sign of d in Eq. (6) would make the heatmap in Figure 5 much easier to interpret.","section":"Sec. 3.4.3, Eq. (6)"}],"recommendation":"major_revision","confidential_remarks":"The manuscript fits the journal's interest in retrieval-augmented models and is primarily a mechanistic-interpretability contribution; it is likely to interest the RAG community. Two things to check at production: the review copy had rendering artifacts in several figures and in Eq. (8), and the claim of being 'for the first time' unveiling these mechanisms is stronger than the existing knowledge-conflict literature (refs. [12, 13]) warrants. The authors should soften that claim. My main concern is that the four-stage decomposition is currently unfalsifiable from the manuscript because the key-token group and the normalization in Eq. (4) are unspecified; I therefore recommend major revision rather than rejection, as the intervention experiments in Sec. 4.1 are promising and the missing definitions are fixable."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Key thing to know: this is the first systematic attempt I've seen to trace how RAG models mix parametric and retrieved knowledge through the layers, and the KAPE neuron-targeting result looks real. But the four-stage story is built on a token-classification step that is never actually defined, so the stage boundaries should be treated as suggestive rather than established.\n\nWhat's genuinely new: the four-stage decomposition (refinement, elicitation, expression, contestation), the KAPE metric for finding internal-vs-external knowledge neurons, and the deactivation experiments showing that knocking out those neurons shifts which source dominates. The gold/fake passage contrast and the early-decoding analysis add converging evidence, and the cross-model (LLaMA, Qwen) and cross-dataset (NQ, TriviaQA, HotpotQA) consistency is a real point in its favor.\n\nWhere it gets soft: the whole information-flow analysis depends on a token group called K, \"the potential answers extracted by the LLM from the context,\" and the extraction procedure is never specified. Every IF curve in Figure 1—and therefore every stage boundary—is a sum of attention weights from K tokens to C, Q, or A tokens. If K is defined differently (generated answer string vs. gold span matched back), the shapes change. And Eq. (4) is a raw sum, not normalized by the number of token pairs, so longer or shorter key spans could create rise-and-fall patterns that have nothing to do with knowledge transfer. That is a load-bearing gap.\n\nAlso, KAPE selects neurons using the same RAG-vs-closed-book contrast that the deactivation experiment then measures, so part of the confirmation is baked into the selector. The cross-dataset transfer (neurons chosen on NQ work on TriviaQA and HotpotQA) is a good partial answer to that, but it isn't a full one. No error bars or random-neuron baselines are reported, and the four-stage boundaries are drawn by eye. None of this kills the central claim, but it means the paper is a strong hypothesis-generation result, not a confirmed mechanism.\n\nWho it's for: anyone working on RAG interpretability or knowledge conflicts. It deserves a proper peer review, but the authors should be asked to define K operationally, normalize the IF metrics, and add random-neuron and shuffled-token baselines. As it stands, I'd trust the KAPE deactivation effect more than I'd trust the four-stage boundaries.","headline":"A plausible and useful map of how RAG models mix parametric and retrieved knowledge, with a real neuron-level control result—but the four-stage decomposition rests on an underspecified key-token definition and unnormalized attention sums.","tokens_in":19372,"tokens_out":2216,"would_cite":true,"duration_ms":21327,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims RAG knowledge use in LLMs proceeds through four stages — refinement, elicitation, expression, contestation — steered by passage relevance and rebalanced by deactivating KAPE-identified neurons.","keywords":["Retrieval-Augmented Generation","Knowledge Utilization","Large Language Models","Knowledge Streaming","Knowledge-Specific Neurons","Open-Domain Question Answering","Interpretability","Knowledge Conflict"],"falsifier":"Hold the token layout fixed but destroy the passage's meaning (for example, shuffle its sentences or substitute the answer-bearing words with random tokens) and recompute the three information-flow curves. If the key-to-query peak at the elicitation layers and the four-stage phase pattern persist undiminished, the flows are tracking attention geometry rather than knowledge, and the stage decomposition would be refuted.","tokens_in":18349,"feed_emoji":"🧠","tokens_out":10441,"duration_ms":91628,"temperature":0.7,"pith_summary":"Retrieval-augmented generation is usually treated as plugging retrieved text into a prompt, but this paper asks what the model actually does with that text. By tracing attention and saliency flows between four token groups — the retrieved context, the answer-bearing key tokens inside it, the query, and the answer prompt — the authors find that external knowledge passes through four distinct layer-wise stages: refinement, elicitation, expression, and contestation. The relevance of the retrieved passage to the query steers this stream, and the decisive gate sits at the elicitation stage, where the model decides whether to absorb the external information into the query representation. At the neuron level, the paper introduces KAPE, an entropy measure that identifies neurons dedicated to internal or external knowledge, and shows that deactivating the most knowledge-specific of them shifts the model's reliance between the two sources. If the claims hold, RAG is not a monolithic lookup but a staged, partly steerable pipeline.","feed_headline":"RAG knowledge flow runs in four stages — and can be rebalanced","feed_subtitle":"Layer-wise attention traces define four knowledge stages, giving RAG builders a neuron-level lever over which source the model trusts.","key_machinery":"The key machinery is a set of three attention-based information-flow metrics — $IF^{{kc}}$, $IF^{{kq}}$, $IF^{{ka}}$ — summed over heads and token pairs between the key group K (the answer-bearing tokens extracted from the context) and the context C, query Q, and answer prompt A, respectively, complemented by gradient saliency matrices $IF^{{kc}}$_s, $IF^{{kq}}$_s, $IF^{{ka}}$_s. The phase offsets of these curves across layers define the four stages. On the neuron side, KAPE (knowledge activation probability entropy) computes, for each MLP neuron, an entropy over the L1-normalized activation probabilities for internal versus external knowledge; low KAPE marks a knowledge-specific neuron, and setting its activation to zero is the intervention that shifts reliance between the two sources. Residual-stream and early-decoding unembedding logits are then used to attribute contributions to MHA versus MLP modules, showing that MLP tracks the fine-grained factual correctness of external knowledge.","core_discovery":"The central discovery is that knowledge utilization in LLM-based RAG is a staged process, not a single integration step. Across transformer layers, the flow of attention from answer-bearing key tokens to the context peaks early and then fades (refinement); the flow from key to the query rises next and then falls (elicitation); the flow from key to the answer prompt rises in the middle layers and falls in the final quarter (expression); and in the deepest layers the external-knowledge signal declines while the gradient tendency to strengthen it increases, indicating competition with internal memory (contestation). The paper further shows that passage relevance governs this stream mainly at the elicitation stage: when key-to-query attention is causally masked there, the probability shift for external-knowledge answers is largest for highly relevant passages. Introducing KAPE, the paper identifies the top 1% lowest-entropy neurons as internal-knowledge or external-knowledge specialists and demonstrates that deactivating them measurably changes factual accuracy in the expected direction on Natural Questions, TriviaQA, and HotpotQA, in both LLaMA and Qwen models.","pith_inferences":["Editorial inference: the same four-stage lens could be applied to other context-augmented settings such as in-context learning or long-context reasoning, where the 'key' group would be prompt exemplars or relevant spans; the paper's metrics do not depend on retrieval being the source of external tokens.","Editorial inference: KAPE's entropy criterion suggests a testable recipe for building controllable RAG without fine-tuning — a sparse mask or steering vector over low-entropy neurons could be tuned to balance knowledge sources on new domains, something the paper demonstrates only for the deactivation direction.","Editorial inference: because the saliency analysis shows that the decline in key-to-answer flow in deep layers is accompanied by an increasing gradient tendency to strengthen it, a plausible reading is that internal knowledge actively suppresses external knowledge in the contestation stage; a direct follow-up would test whether removing internal-knowledge neurons in that stage specifically restore"],"forward_implications":["RAG is not a monolithic lookup: knowledge flows from the context into the query and then to the answer in a layer-wise sequence, so interventions that target the elicitation stage should be the most effective way to control whether retrieved evidence is used.","Passage relevance acts mostly at the elicitation stage; a model given irrelevant passages stops integrating external knowledge early in the pipeline, which explains why retrieval quality has an outsized effect on RAG accuracy.","Because low-KAPE neurons are dedicated to one knowledge source, editing the top 1% of them yields targeted shifts in reliance: deactivating external-knowledge neurons hurts factual accuracy with gold documents but helps when documents are noisy, while the reverse holds for internal-knowledge neurons.","MLP layers are the module that verifies fine-grained factual correctness of external knowledge, since gold versus fake passage differences appear there but not in MHA layers, pointing to MLP as the place to inspect for hallucination from noisy context.","The four-stage pattern and the KAPE control both replicate across LLaMA and Qwen families and across single- and multi-hop QA datasets, so the mechanisms are general rather than model-specific."],"supporting_citations":[{"why":"Supplies the information-flow perspective and the methodology used to trace token interactions across layers.","marker":"[37]"},{"why":"Supplies the gradient-based saliency method used to corroborate the four-stage decomposition.","marker":"[1]"},{"why":"Supplies the Taylor-expansion saliency computation underlying the saliency-based flow metrics.","marker":"[22]"},{"why":"Defines the relevance categories (positive, hard, hard-, random) and the factual-knowledge-boundary framing that the relevance-steering analysis builds on.","marker":"[29]"},{"why":"Inspires the entropy-based neuron identification that KAPE adapts to internal versus external knowledge.","marker":"[31]"},{"why":"Provides the knowledge-conflict intervention baseline that the neuron-deactivation experiments build on.","marker":"[13]"},{"why":"Supplies the dense retriever used to construct the external knowledge passages.","marker":"[27]"},{"why":"Provides the Natural Questions dataset on which knowledge-specific neurons are identified and the main deactivation effects are measured.","marker":"[16]"},{"why":"Provides the TriviaQA dataset used to test whether the neuron-deactivation results transfer across single-hop QA datasets.","marker":"[14]"},{"why":"Provides the HotpotQA multi-hop dataset used to test whether the four-stage pattern and deactivation effects transfer beyond single-hop QA.","marker":"[41]"}],"fun_headline_variants":["LLM RAG knowledge flows in four stages, rebalancing possible","Four-stage knowledge flow revealed in retrieval-augmented LLMs","Neuron knob rebalances RAG knowledge sources","Four-stage RAG knowledge flow, tunable via neurons","Deactivate low-entropy neurons to rebalance RAG knowledge"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The analysis assumes that attention and gradient scores between token groups faithfully trace genuine knowledge transfer, and that the answer-bearing words inside the retrieved passage are cleanly extracted; if those scores reflect token counts or positional patterns rather than knowledge, the four-stage decomposition and the relevance-guided streaming results do not follow.","fun_headline_variants_meta":{"raw":{"variants":["LLM RAG knowledge flows in four stages, rebalancing possible","Four-stage knowledge flow revealed in retrieval-augmented LLMs","Neuron knob rebalances RAG knowledge sources","Four-stage RAG knowledge flow, tunable via neurons","Deactivate low-entropy neurons to rebalance RAG knowledge"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001123,"raw_usage":{"total_tokens":4725,"prompt_tokens":1051,"completion_tokens":3674,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":667,"completion_tokens_details":{"reasoning_tokens":3590}},"tokens_in":667,"tokens_out":3674,"duration_ms":26116,"temperature":1.0,"reasoning_tokens":3590,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T20:43:38.475075+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Hold the token layout fixed but destroy the passage's meaning (for example, shuffle its sentences or substitute the answer-bearing words with random tokens) and recompute the three information-flow curves. If the key-to-query peak at the elicitation layers and the four-stage phase pattern persist undiminished, the flows are tracking attention geometry rather than knowledge, and the stage decomposition would be refuted.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Inspires the entropy-based neuron identification that KAPE adapts to internal versus external knowledge."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the Natural Questions dataset on which knowledge-specific neurons are identified and the main deactivation effects are measured."}],"review_version":1}