{"id":"ba117c75-cbd0-4a59-9c7d-8b88cf1e0ceb","arxiv_id":"2502.01563","paper_version":4,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"Disrupting the concentrated massive values in Q and K of RoPE-based LLMs collapses contextual understanding tasks while leaving parametric retrieval mostly intact, and the paper attributes the pattern to RoPE.","lead":"This paper shows that in transformer LLMs with rotary position encoding, the largest values in the attention query and key vectors cluster in a few dimensions, and that deleting these values destroys the model's ability to use the current context while leaving factual recall partially intact.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Massive-value task dissociation is confounded with attention precision; passkey, admitted to be non-contextual in App. A.2, collapses, so the semantic-vs-positional claim is not established.","rationale":"The paper's headline is a functional claim about what massive Q/K values mean, and the behavioral evidence is a double dissociation. Both legs are confounded: contextual-knowledge tasks are multi-token and attention-hungry, whereas parametric-knowledge tasks are mostly single-hop lookup. The passkey benchmark makes this confound explicit because the authors themselves state it does not require contextual understanding, yet they use its collapse as evidence for their dichotomy. Section 4.1's semantic interpretation is also circular because it is inferred from the same disruption results it is meant to explain. The proposed phase-ablation check isolates the positional contribution of the massive dimensions while holding their magnitude and content fixed, directly testing the reader's weakest assumption. If the dissociation persists under phase-only ablation, the paper's semantic-contextual interpretation is wrong in an interesting way; if it disappears, the interpretation is supported. No new training is needed and the released code should make the check feasible. I therefore keep the reader's CONDITIONAL verdict: the empirical phenomenon is credible, but the central interpretive claim needs this decisive control before acceptance.","tokens_in":38356,"tokens_out":10850,"duration_ms":102717,"concrete_test":"On Llama-3-8B, Gemma-2-9B, and Qwen2.5-7B, rerun the Table 1 protocol with one change: instead of replacing massive Q/K coordinates with the global mean, ablate only the RoPE phase of the massive dimensions (set their rotation angle to a constant, e.g., 0, for all positions), preserving magnitudes and token content while removing relative-position information in exactly those dimensions. Compare GSM8K, AQUA, IMDB, and passkey against Cities, Sports, and Celebrity. If phase-only ablation reproduces the CK/PK dissociation, the massive dimensions are load-bearing for positional binding and the semantic-contextual interpretation fails; if contextual tasks survive while only passkey degrades, the semantic interpretation is supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim requires that ablating massive Q/K dimensions selectively destroys contextual-knowledge processing while sparing parametric retrieval. Section 3.3 replaces the highest-L2-norm Q/K coordinates with a global mean during prefill, while the non-massive control replaces the smallest coordinates; this is a magnitude-matched comparison, not a content-matched one. Massive dimensions dominate attention logits, so any task that needs precise multi-token attention (GSM8K, AQUA, IMDB, passkey) should be more sensitive than single-hop fact lookup regardless of whether those dimensions encode contextual knowledge. The dissociation therefore does not yet identify a dedicated semantic channel. Appendix A.2 explicitly states that passkey retrieval 'does not require contextual knowledge understanding ability, only locating the passkey,' yet Table 1 counts passkey as a contextual-knowledge benchmark; its 100% to 0% collapse is exactly what a positional-binding confound predicts. Section 4.1 then infers that massive dimensions 'primarily encode rich semantic content rather than positional information' from the same disruption results, which is circular. The phenomenon itself is real and worth studying, but the central contextual-versus-parametric interpretation is unproven.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper reports that in modern transformer LLMs using RoPE, the query (Q) and key (K) tensors contain a small number of dimensions whose L2 norms are dramatically larger than the rest, that these 'massive values' are concentrated in low-frequency dimensions, and that they are absent in V and in models without RoPE. The authors disrupt these values during the prefill phase and report that accuracy on math reasoning, sentiment analysis, and passkey retrieval collapses, while parametric fact QA degrades only modestly. They additionally compare three quantization methods and conclude that methods protecting massive values better preserve contextual-knowledge performance. The paper attributes the emergence of the concentration to RoPE and provides extensive layer-wise visualizations and open-source code.","tokens_in":38554,"tokens_out":7131,"duration_ms":61064,"significance":"If the central claim were established, the work would provide a practical handle for quantization and a mechanistic distinction between contextual and parametric processing in LLMs. The observational contribution—the systematic documentation of concentrated Q/K outliers across many RoPE-based models and their correlation with RoPE—is valuable and consistent with related work such as Barbero et al. (2025), and the release of code and visualizations is a strength. However, the current evidence does not establish the dedicated 'contextual knowledge' interpretation: the disruption protocol is confounded with attention-logit magnitude, the task taxonomy is internally inconsistent, and parts of the mechanistic argument are circular. The paper's strongest reliable contribution is the empirical phenomenon itself, not yet the mechanistic explanation.","major_comments":[{"comment":"The disruption protocol does not clearly target the 'massive values' defined in Definition 1. Definition 1 (Eq. 7) selects coordinates by thresholding the per-head, per-dimension L2 norm over the sequence, but Eq. (9) replaces the per-token argmax dimension with the global mean, and Eq. (10) replaces per-token minima. These are different sets of coordinates, so the paper does not actually specify that the ablated entries are the massive values of Definition 1. Please clarify the exact selection rule or rewrite Eqs. (9)-(10); as written, the experiment is not reproducible and the reported effects cannot be attributed to the defined massive values.","section":"Section 3.3, Eqs. (9)-(10)"},{"comment":"The non-massive control is not matched in magnitude or functional contribution. Replacing the largest-magnitude Q/K coordinates removes the dominant contributors to attention logits, while replacing the smallest coordinates leaves those logits largely unchanged; hence any task that depends on precise multi-token attention (GSM8K, AQUA, IMDB, passkey) will suffer more than a single-hop fact lookup regardless of whether the coordinates encode 'contextual knowledge.' The passkey result illustrates the confound: Appendix A.2 explicitly states that passkey retrieval 'does not require contextual knowledge understanding ability, only locating the passkey,' yet Table 1 counts it as a contextual-knowledge benchmark, and its 100%-to-0% collapse is exactly what a positional or attention-precision artifact predicts. Moreover, all Table 1 entries are single numbers without error bars or significance tests, so statements such as 'only 15-20% degradation' are not statistically supported. The task dissociation in Table 1 therefore does not by itself establish a dedicated contextual-knowledge channel.","section":"Section 3.4, Table 1; Appendix A.2"},{"comment":"The semantic-versus-positional argument is circular. Evidence 1 concludes that low-frequency massive values 'primarily encode rich semantic content rather than positional information' by citing the same disruption results whose interpretation depends on that assumption, and no independent probe of the encoded information is provided (e.g., decoding positional information from these dimensions, or ablating equally large non-massive low-frequency coordinates). In addition, Section 4.2 and Appendix F state that the patterns are 'largely consistent' before and after applying RoPE and that the concentration is 'gradually formed through training rather than being solely caused by the addition of RoPE,' which conflicts with the abstract's claim that the concentration 'is caused by Rotary Positional Encoding (RoPE).' At minimum, RoPE should be described as a structural condition or contributor, not a direct cause, and the causal claim needs a test that does not rely on the disputed disruption results.","section":"Section 4.1, Evidence 1; Section 4.2; Appendix F"},{"comment":"The paper's treatment of GPT-2 is internally inconsistent: Section 4.1 and Table 3 state that models without RoPE, including GPT-2, show no concentrated massive values, while Appendix F Table 11 and Figure 46 report that GPT-2 does exhibit concentrated massive values (concentrated along heads rather than along depth dimensions). Since the 'RoPE causes concentration' claim is central, this contradiction must be resolved. Either the body-text claim is incorrect, or the appendix uses a different criterion for 'concentrated massive values'; the paper should state the criterion explicitly and apply it uniformly.","section":"Table 3 and Appendix F, Table 11 and Figure 46"},{"comment":"The quantization evidence is reported only as normalized accuracy with no absolute values, no error bars, and no description of the bit-widths, calibration sets, or number of runs for AWQ, SmoothQuant, and GPTQ. With only three methods and no matched control for which weights are altered, Figure 5 does not independently corroborate the contextual-knowledge interpretation; it is also consistent with the confound in my second comment, since methods that preserve attention-dominant coordinates will better preserve any attention-heavy task. Please provide full accuracy tables and either a matched comparison or a clearly stated supporting-role claim.","section":"Section 3.6, Figure 5"}],"minor_comments":[{"comment":"The sentence 'It decreases from 0-(⌈d/2⌉) as the dimensional rotation angle decreases' is unclear and appears to conflate dimension index with rotation frequency; please rewrite with precise definitions.","section":"Section 4.1, Evidence 1"},{"comment":"The perplexity plot in Figure 3b should be checked against Table 7; for example, Llama3-8B IMDB massive-disrupted perplexity is reported as 112.61 in Table 7, but the bar height and axis limits in the figure are not easy to verify, and the legend should state explicitly which condition is shown.","section":"Figure 3b and Table 7"},{"comment":"The tensor notation is inconsistent: X is introduced as R^{lprompt×h×d}, but the equations then use l, h, d without defining the mapping to the per-head dimension D used in Section 3.1; please unify the notation.","section":"Section 2.2, Eqs. (4)-(6)"},{"comment":"Some references are incomplete, such as the Anthropic long-context prompting entry with 'Accessed: date-of-access' as the access date, and a few author lists are left as 'et al.' with no page numbers; these should be completed before publication.","section":"References"},{"comment":"The passkey-retrieval section explicitly says the task 'does not require contextual knowledge understanding ability,' which directly contradicts its inclusion under 'Contextual Knowledge Understanding Task' in Table 1 and Section 3.4; this should be reconciled in the main text.","section":"Appendix A.2"}],"recommendation":"major_revision","confidential_remarks":"The observational part of the paper—the systematic documentation of concentrated Q/K outlier dimensions across RoPE-based models and their absence in V—is likely worth publishing as an empirical phenomenon. The current framing as a dedicated contextual-knowledge mechanism is not supported by the experimental design, and several internal contradictions between the main text and appendix need reconciliation. I would ask the authors to either substantially strengthen the disruption controls and soften the mechanistic claims, or reposition the paper around the outlier phenomenon and its quantization implications. A matched control that disrupts an equal number of non-massive dimensions with comparable logit magnitude, plus a positional-probing experiment, would be the most direct way to test the central claim."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Han, the massive-value phenomenon is real and the quantization angle is useful, but the paper's central dissociation claim does not survive contact with its own appendix. What's new: a systematic survey showing large L2 norms concentrated in specific Q/K dimensions across heads, absent in V, tied to full RoPE architectures; a behavioral split between in-context and parametric tasks under disruption; and a knowledge-conflict experiment where destroying massive values makes models ignore contradictory context and fall back on parametric facts. Those are good leads. The disruption effects are large and consistent across three 7-9B models, and the authors are honest enough to cite prior work (KIVI, KVQuant, Atom, Barbero et al.) and admit they are extending it, so the abstract's 'for the first time' is overstatement, not fraud.\n\nThe soft spots are real. The control replaces only the smallest coordinates, not the same number of mid-range ones, so we can't distinguish magnitude-sensitivity from channel-specificity. Massive dimensions dominate attention logits; any task that needs precise multi-token attention—GSM8K, AQUA, IMDB, passkey—should be more sensitive than single-hop fact lookup regardless of what those dimensions encode. The paper's own Appendix A.2 states passkey retrieval 'does not require contextual knowledge understanding ability,' yet Table 1 counts it as the banner contextual task, and its 100%-to-0% collapse is exactly what a positional-binding confound predicts. Section 4.1 then uses the disruption results to conclude that low-frequency regions 'primarily encode rich semantic content rather than positional information,' which is circular. The RoPE causality claim is also internally inconsistent: the abstract says RoPE causes the concentration, while Section 4.2 and Appendix F say the pattern is identical before and after RoPE and is 'gradually formed through training.' No error bars, no significance tests, no matched count of disrupted values.\n\nIf I were refereeing this, I'd ask for three things: a matched control that disrupts the same number of random or mid-scale coordinates; removal or reclassification of passkey; and a rewrite of the RoPE-causality and semantic-vs-positional language to match the evidence. The practical message for quantization—protect these Q/K channels during prefill, as AWQ and SmoothQuant already do—is likely robust and worth acting on. So: send it to a serious referee, but expect major revisions. Good reading-group material for a discussion on methodological confounds in mechanistic interpretability.","headline":"Real phenomenon, useful quantization insight, but the contextual-vs-parametric dissociation is confounded by attention precision and undercut by the paper's own passkey caveat.","tokens_in":39085,"tokens_out":4654,"would_cite":true,"duration_ms":40571,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Concentrated massive values in Q and K, created by RoPE, are what let LLMs use in-context knowledge.","keywords":["massive values","rotary position encoding","contextual knowledge","parametric knowledge","attention queries and keys","LLM quantization","interpretability"],"falsifier":"Run the same prefill disruption on an RoPE model while holding absolute positions fixed but permuting token identities, or vice versa, and measure whether the massive-value dimensions respond to semantic content or to position; alternatively, ablate only the highest-frequency non-massive RoPE dimensions and check whether long-context passkey retrieval collapses, which would indicate that the massive-value effect is positional rather than semantic. A cleaner test: train or fine-tune an RoPE model with the low-frequency rotation frequencies set to zero so those dimensions cannot encode position at all, and check whether massive values still concentrate there and still carry context-task performance.","tokens_in":38123,"feed_emoji":"🔑","tokens_out":5296,"duration_ms":40988,"temperature":0.7,"pith_summary":"This paper sets out to explain a counterintuitive structure inside transformer LLMs: in the attention queries (Q) and keys (K), a few dimensions carry values far larger than the rest, consistently at the same dimension indices across nearly all attention heads, while the values (V) show nothing similar. The authors argue that these concentrated massive values exist only in models using rotary position encoding (RoPE), appear from the first layers, and are the specific channel through which the model handles knowledge that comes from the current context window rather than from the parameters. If the claim holds, a small, precisely located set of activations is responsible for math reasoning, sentiment reading, and long-context retrieval, whereas memorized factual knowledge lives elsewhere. The practical stake is direct: quantization and compression schemes that protect these dimensions should keep contextual abilities intact, and schemes that flatten or skip them should silently cripple those abilities.","feed_headline":"Massive Q/K values are the key to in-context knowledge","feed_subtitle":"Disrupting a few concentrated dimensions destroys reasoning, sentiment, and passkey tasks but spares memorized facts.","key_machinery":"The load-bearing object is the massive-value coordinate set: dimensions in Q and K whose $\\ell^2$ norm over the sequence is at least five times the head's mean, found to cluster in the low-frequency half of RoPE's paired rotation dimensions. The second object is the prefill-stage disruption operation, which replaces those coordinates by the global mean of the tensor; because only the prefilling phase is altered, generation mechanics stay intact and performance changes are attributable to context processing. RoPE itself is the causal mechanism: its frequency schedule $\\theta_j = 10000^{-2j/d}$ partitions dimensions into high-frequency (position-sensitive) and low-frequency (semantic) regions, and the massive values land in the low-frequency region. That placement is what lets the paper separate contextual knowledge from parametric knowledge, since disrupting the semantic-carrying low-frequency coordinates should selectively impair context tasks.","core_discovery":"The paper defines a massive value as an entry of the per-head, per-dimension norm matrix $M_{h,d} = \\|Q_{:,h,d}\\|_2$ (and likewise for K) that exceeds five times that head's average over dimensions, and calls the pattern concentrated when those entries cluster in the same low-frequency dimension block across heads. The central discovery is that in RoPE-based models, including Llama, Gemma, Qwen, Mistral, Phi, LLaVA, and Qwen2-VL, this concentrated pattern appears in Q and K from the very first layer, is absent from V, and is absent from non-RoPE models such as GPT-2, GPT-Neo, OPT, and Jamba. Replacing these massive-value coordinates with the tensor mean during the prefilling phase drops GSM8K from roughly 77-87% to 4-16%, AQUA from roughly 53-64% to 10-20%, IMDB from roughly 94-97% to 2-11%, and passkey retrieval from 100% to 0-9%, while parametric QA tasks fall only 15-20 percentage points and non-massive-value disruption leaves all tasks within about 1%. Quantization comparisons show that AWQ and SmoothQuant, which preserve or redistribute massive values, retain contextual-task accuracy, while GPTQ, which does not protect them, drops GSM8K and AQUA to about 75% normalized accuracy. The paper attributes the pattern's origin to RoPE's frequency pairing and reports that the pattern appears essentially unchanged before and after the RoPE rotation is applied, indicating that the concentration is formed through training rather than by the rotation itself.","pith_inferences":["If the massive-value coordinates are the semantic channel for context, then steering or patching just those few dimensions during decoding could edit in-context behavior without touching parametric knowledge, a cheap intervention for interpretability and control that the paper does not explore.","The paper's own appendix notes that passkey retrieval does not require contextual understanding, so the observed collapse of passkey performance may indicate that the massive-value dimensions also carry the binding or indexing signal needed to locate tokens; this would blur the clean context-versus-parametric split and is worth testing directly.","Because the massive-value pattern appears identically before and after applying RoPE, the concentration is likely learned into the Q/K projections themselves during training; if so, fine-tuning an RoPE model with scrambled rotation frequencies should make the pattern disappear, offering a direct causal check of RoPE's role."],"forward_implications":["Quantization methods that protect or redistribute massive-value dimensions, such as AWQ and SmoothQuant, should be preferred when serving context-heavy workloads, whereas GPTQ-style uniform quantization will disproportionately hurt in-context reasoning even if factual QA accuracy looks acceptable.","KV-cache and activation compression schemes should allocate more bits to the massive-value coordinates in Q and K to preserve long-context and reasoning abilities.","RoPE-free architectures, including those using absolute position embeddings, do not exhibit the concentrated pattern, so the contextual-knowledge channel described here is tied to the rotary mechanism rather than to attention in general.","Models with partial RoPE, such as GPT-NeoX with rotation on only the first quarter of dimensions, still show concentration but at a different location, consistent with the claim that RoPE's rotation structure determines where massive values land."],"supporting_citations":[{"why":"Defines Rotary Position Embedding, the mechanism the paper identifies as the cause of concentrated massive values in Q and K.","marker":"(Su et al., 2024)"},{"why":"Prior observation that low-frequency RoPE components encode semantics rather than position, which the paper builds on to interpret the massive-value regions.","marker":"(Barbero et al., 2025)"},{"why":"Introduces AWQ, a quantization method that explicitly protects important weights and serves as the comparison point that preserves contextual knowledge.","marker":"(Lin et al., 2024)"},{"why":"Introduces SmoothQuant, a smoothing method that redistributes massive activation values and is shown to preserve contextual-task performance.","marker":"(Xiao et al., 2023)"},{"why":"Introduces GPTQ, the quantization baseline that does not protect massive values and consequently degrades contextual knowledge tasks.","marker":"(Frantar et al., 2022)"},{"why":"Supplies the GSM8K benchmark used to measure contextual mathematical reasoning degradation under massive-value disruption.","marker":"(Cobbe et al., 2021)"},{"why":"Supplies the IMDB sentiment dataset used to show catastrophic context-understanding loss when massive values are disrupted.","marker":"(Maas et al., 2011)"},{"why":"Supplies the Cities parametric knowledge benchmark that remains relatively intact under massive-value disruption.","marker":"(Marks & Tegmark, 2023)"},{"why":"Defines the passkey retrieval task whose collapse from 100% to near zero demonstrates the loss of in-context binding when massive values are disrupted.","marker":"(Mohtashami & Jaggi, 2023)"}],"fun_headline_variants":["Tiny set of Q/K values carries in-context knowledge","RoPE spawns massive Q/K values that drive in-context understanding","Massive Q/K values: the secret to in-context reasoning","Why disrupting few Q/K coordinates kills reasoning, not facts","In-context knowledge lives in a few massive Q/K dimensions"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The argument depends on the premise that the low-frequency dimensions carrying the massive values encode semantic, context-derived content rather than position or ordering information; if those dimensions mostly encode where tokens are, the observed task split would be a positional-information artifact rather than evidence for a dedicated contextual-knowledge channel.","fun_headline_variants_meta":{"raw":{"variants":["Tiny set of Q/K values carries in-context knowledge","RoPE spawns massive Q/K values that drive in-context understanding","Massive Q/K values: the secret to in-context reasoning","Why disrupting few Q/K coordinates kills reasoning, not facts","In-context knowledge lives in a few massive Q/K dimensions"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000555,"raw_usage":{"total_tokens":2721,"prompt_tokens":1103,"completion_tokens":1618,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":719,"completion_tokens_details":{"reasoning_tokens":1530}},"tokens_in":719,"tokens_out":1618,"duration_ms":10323,"temperature":1.0,"reasoning_tokens":1530,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-09T14:57:22.337805+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same prefill disruption on an RoPE model while holding absolute positions fixed but permuting token identities, or vice versa, and measure whether the massive-value dimensions respond to semantic content or to position; alternatively, ablate only the highest-frequency non-massive RoPE dimensions and check whether long-context passkey retrieval collapses, which would indicate that the massive-value effect is positional rather than semantic. A cleaner test: train or fine-tune an RoPE model with the low-frequency rotation frequencies set to zero so those dimensions cannot encode position at all, and check whether massive values still concentrate there and still carry context-task performance.","supporting_citations":[],"review_version":1}