{"id":"0269da78-94e3-4263-8095-fd88120dd8fa","arxiv_id":"2507.11953","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Attention matrices of same-series small and large LLMs are similar enough that replacing up to 50% of a large model's attention layers with the small model's matrices preserves most performance while reducing KV cache and compute.","lead":"This paper proposes replacing up to half of a large language model's attention computations with attention patterns borrowed from a smaller same-family model, selected by cosine similarity during the prompt phase. If the method holds, it could cut KV cache memory by about 22% and speed prefill by 15% with little accuracy loss, and it can be combined with existing cache-reduction tools.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Prefill acceleration claim contradicts Algorithm 1: mapping establishment requires computing all LLM attention matrices, so IAM's prefill adds SLM forward and similarity cost instead of skipping QK^T; the reported TTFT gains are unexplained.","rationale":"The reader's consistency concern is real but secondary. The load-bearing issue is the prefill efficiency accounting. Even if mapping consistency is high, the headline benefit is 'accelerate prefill by 15%'. Algorithm 1 cannot deliver that because the mapping f(i) is unknown until the prefill attention matrices are computed; so mapped layers' QK^T work is performed and then discarded, plus SLM and similarity are added. This is not a matter of external assumptions; it follows from the paper's own equations. The concrete test settles it by profiling. I still find the underlying similarity observation and quality-preservation evidence interesting, and the paper gives some independent support (cross-series generalization, orthogonality to H2O). But as written, the efficiency claim is not derivable. I therefore keep a CONDITIONAL verdict with a stricter condition: provide a component-wise timing breakdown; if the breakdown shows TTFT excluding mapping setup, the claim should be relabeled. The reader's weakest_assumption (consistency) is not the same as this concern, so agreement_with_reader=disagree.","tokens_in":12526,"tokens_out":14947,"duration_ms":173254,"concrete_test":"Profile Algorithm 1 on the two Table 3 workloads (Bsz=64, 512+512 and Bsz=8, 8192+512) using a PyTorch profiler, with careful separation of (a) LLM prefill forward, (b) SLM prefill forward, (c) pairwise similarity computation (Eq. 4), and (d) first decode step. Sum (a)-(d) and compare with Default TTFT in Table 3. If the sum exceeds Default TTFT, the claimed 12-17% TTFT reduction disappears when mapping-setup cost is included; if the paper's IAM numbers only hold when (c) and part of (b) are removed from the clock, the efficiency claim must be restated as decode-side speedup, not prefill acceleration. Report the breakdown per component.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"Section 3.4 and Algorithm 1 establish f(i) during prefill by running the full prompt through the LLM and SLM (lines 7-9) and computing pairwise cosine similarity over attention matrices (Eq. 4-5). This requires executing Q/K projections and QK^T for every LLM layer, including the layers that will later be mapped, before any replacement can happen. IAM therefore has strictly more prefill work than a default LLM forward: it adds the SLM forward pass and the similarity search (for a 512-token prompt with 5120 LLM heads and 360 SLM heads, roughly 1.8M dot products over ~131k-dimensional vectors). Section 4.3 attributes the 12-17% TTFT improvement to not computing Q/K projections and QK^T in mapped layers during prefill, but those computations are needed to produce the Ai used to select the mapping. The only way the reported TTFT can be below the Default TTFT is if the mapping-establishment phase is excluded from the clock or the algorithm differs from Algorithm 1. As written, the central 'accelerate prefill by 15%' claim is not supported. This is an internal inconsistency in the efficiency evaluation, independent of the quality-preservation question.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes IAM, an inference-efficiency method that replaces attention matrices in selected layers of a large language model with attention matrices from a smaller same-family model. The mapping from LLM attention heads to SLM heads is established during prefill by computing pairwise cosine similarity over attention matrices, then reused during decode so that mapped layers skip Q/K projection and QK^T computation. The authors report near-lossless quality at 30% mapping ratio, a 22% reduction in KV-cache usage, and 12–17% TTFT reductions on Qwen2-72B, with additional experiments on LLaMA series models and compatibility with H2O.","tokens_in":12811,"tokens_out":10718,"duration_ms":114862,"significance":"The paper identifies a useful empirical phenomenon—same-series models of different scales exhibit high attention-matrix similarity—and converts it into an optimization that is orthogonal to token-level KV-cache compression. If the reported results survive scrutiny, IAM would be a novel and practical addition to LLM serving toolkits. Strengths include the systematic analysis of similarity metrics, layer selection, and mapping consistency; the evaluation across two model families; a compatibility study with H2O; and public code. However, the current evidence is weakened by selection of hyperparameters on the same benchmarks later reported, an ambiguous Algorithm 1 that does not clearly describe long-context handling, and an efficiency claim that does not account for the cost of mapping establishment.","major_comments":[{"comment":"Algorithm 1 is ambiguous about how mapping establishment interacts with the full prompt for long inputs. Lines 2–3 say that if len(x) > tau_t, the prompt is truncated to tau_t; lines 7–10 then establish the mapping from the truncated prompt, and line 13 generates using M(x, y) where x has been truncated. This would discard all tokens beyond tau_t, which cannot be what was done for the GovReport long-context experiments. The truncation must apply only to the similarity computation, not to the prompt actually processed. Please rewrite Algorithm 1 to make the full-prefill path explicit, including when and how the mapping is applied during the prefill forward pass.","section":"§3.4, Algorithm 1"},{"comment":"The reported TTFT reductions of 12% and 17% are attributed to skipping Q/K projections and QK^T in mapped layers, but mapping establishment (Algorithm 1, lines 7–10) requires exactly those computations on at least a tau_t-token prefix, plus the SLM forward pass and the pairwise similarity search over L·D × l·d attention matrices. The paper does not report a cost breakdown, so it is unclear whether these overheads are included in the TTFT numbers. Please provide an ablation that isolates the mapping-establishment overhead and reports end-to-end prefill latency including the SLM forward and the similarity computation.","section":"§4.3, Table 3"},{"comment":"The similarity metric is selected by evaluating WikiText-v2 perplexity (Table 1), and the layer-mapping schedule is selected using MMLU (Figure 2). These same two benchmarks are then reported as the headline results in Figure 5. This selection-on-test-set procedure inflates the reported quality; the authors should either reserve a held-out set for final evaluation or report results on benchmarks not used in any selection step.","section":"§3.1 and §3.2, Figure 5"},{"comment":"The table appears to contradict the surrounding text. The text states that fine-tuning decreases perplexity, but the row labeled 'w. tuning' has higher perplexity than 'w.o. tuning' at every mapping ratio (e.g., 8.53 vs. 8.47 at 10%). Please correct the labels or the numbers, or explain what 'w.o.' and 'w.' denote.","section":"Appendix A, Table 4"},{"comment":"The mean consistency rate of 91.12% implies that roughly 9% of layer-head mappings change during decoding. The paper asserts that the mapping remains 'stable and reliable' but does not test whether those changes degrade output quality, nor whether consistency holds for longer outputs or out-of-distribution contexts. Since the efficiency gains depend on reusing the prefill mapping, please add an analysis of generation quality as a function of mapping stability, or measure consistency over longer and more diverse prompts.","section":"§3.3, Figure 4"}],"minor_comments":[{"comment":"The abstract reports 'accelerate prefill by 15%', while §4.3 reports 12% and 17% (average 14.5%) and the introduction reports 'average acceleration of 11% in inference speed'; please align these numbers.","section":"Abstract and §4.3"},{"comment":"In Figure 5, the y-axis of the perplexity panel is labeled 'Perplexity', while Table 1 reports log perplexity; please indicate which quantity is plotted to avoid confusion.","section":"Figure 5"},{"comment":"There are duplicated figure references: Section 3.3 refers to 'Figure 3' for the consistency plot, and Section 4.5 also refers to 'Figure 3' for the H2O compatibility plot, but that plot is actually Figure 8.","section":"§3.3 and §4.5"},{"comment":"Algorithm 1 contains a typo: 'lenth' should be 'length'.","section":"Algorithm 1"},{"comment":"Table 3's header 'Bsz Lenth' should be 'Bsz Length'.","section":"Table 3"},{"comment":"The definition of Minkowski distance has a broken summation formatting; please fix the equation.","section":"§3.1, Eq. (2)"}],"recommendation":"major_revision","confidential_remarks":"The manuscript presents a promising idea but is currently under-specified and contains several internal inconsistencies that must be resolved before publication. The benchmark-selection circularity is particularly concerning for a serious journal. The availability of code is a positive signal. I believe the paper can be brought to an acceptable level with major revisions."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First, what's worth keeping: the observation that same-family LLMs of different scales have similar attention matrices is not new—Chen et al. saw it in BERT—but turning it into a layer-level inference optimization is a real step. The analysis of similarity metrics, layer selection, and consistency is systematic, the LLaMA 3.1/Qwen2 generality check is useful, and the orthogonality to H2O is a nice property. The quality-preservation curves (30% mapping near-lossless, 50% still usable) are plausible and worth testing independently.\n\nThe soft spots, though, are not small. The stress-test note on Algorithm 1 lands. The algorithm establishes the mapping during prefill by running the full prompt through both models and computing every LLM attention matrix, plus the SLM forward and pairwise similarity. That means prefill does strictly more work than the default forward, not less. Yet Table 3 reports TTFT drops of 12% and 17% and the abstract claims a 15% prefill acceleration. Nothing in the paper explains how that is possible. The text says mapped layers skip Q/K projections and QK^T \"during the prefill stage,\" but those computations are what produced the attention matrices used to select the mapping. Either the clock excludes mapping establishment, which would be misleading, or the reported numbers come from a different algorithm than the one described. This is internal inconsistency, not a matter of taste.\n\nSecond, the fine-tuning ablation (Table 4) contradicts the claim it supports. The text says fine-tuning \"decreases perplexity significantly,\" but the table shows fine-tuned perplexity is higher at every mapping ratio. That needs a correction or a careful explanation, and it undermines confidence in the Appendix.\n\nThird, smaller issues: no error bars, and the cosine metric and layer mapping strategy are selected on WikiText-v2 and MMLU and then reported on those same benchmarks. That inflates the headline numbers a bit. The consistency rate of 91.12% means about 9% of mappings change during decode, and the paper does not analyze the effect of those changes.\n\nWho is this for? Researchers working on KV cache compression and attention sparsity will find the cross-scale mapping idea interesting. But the paper should not be accepted with the efficiency claims as stated. The quality-preservation result may survive a rewrite; the speedup numbers likely won't.\n\nRecommendation: send it to peer review, but only with a clear accounting of what is included in the prefill clock and a resolution of the Table 4 discrepancy. As written, I wouldn't cite it for the speedup.","headline":"Solid cross-scale attention-mapping idea with useful quality-preservation data, but the headline prefill speedup claim contradicts the paper's own Algorithm 1 and the fine-tuning ablation contradicts its own table.","tokens_in":13319,"tokens_out":3504,"would_cite":false,"duration_ms":38395,"reading_group":"maybe","serious_thinker":"no","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"IAM replaces selected attention computations in a large LLM with attention matrices borrowed from a smaller same-series model, keeping quality near-lossless at 30% mapping and cutting KV cache by about 22% at 50%.","keywords":["attention matrix similarity","cross-scale language models","KV cache reduction","inference acceleration","long-context LLM serving","cosine similarity mapping","layer-wise attention mapping","LLM efficiency"],"falsifier":"Measure IAM's perplexity and downstream accuracy on long prompts with long generation outputs, for example 8k-token prompts with 2k generated tokens, while re-estimating the mapping every 128 tokens versus never; if the never-remapped version degrades noticeably as generation length grows, the consistency premise fails.","tokens_in":12345,"feed_emoji":"⚡","tokens_out":10063,"duration_ms":96212,"temperature":0.7,"pith_summary":"This paper tries to establish that a large language model can skip computing its own attention on selected layers and instead reuse attention matrices from a smaller model of the same family, because same-series models attend to their context in nearly the same way. The proposed framework, called Inference through Attention Mapping (IAM), chooses the most similar small-model attention matrix by cosine similarity during prefill, then reuses that mapping throughout decoding. On Qwen2 models the method keeps performance close to lossless at a 30% mapping ratio and, at 50%, still retains high capability while cutting KV cache by about 22% and prefill time-to-first-token by 12–17%. The attraction of the idea is that it attacks both main bottlenecks of long-context inference at once, and it can be combined with token-level KV eviction methods such as H2O.","feed_headline":"Borrowing small-model attention cuts LLM prefill cost 15% and KV cache 22%","feed_subtitle":"IAM maps a large LLM's attention layers to its smaller sibling, keeping quality near-lossless at 30% mapping.","key_machinery":"The carrying object is the mapping function $f(i) = \\arg\\max_{j} S(A_i, A'_j)$, where $A_i$ is an LLM attention matrix, $A'_j$ is an SLM attention matrix, and $S$ is cosine similarity computed as $\\mathrm{Tr}(A_i^T A'_j)/(\\|A_i\\|_F \\|A'_j\\|_F)$. The paper uses this function to pair each mapped LLM layer with its closest small-model counterpart during prefill, then substitutes $A_i \\leftarrow A'_{f(i)}$ during decoding so the LLM avoids Q/K projection, QK multiplication, and softmax for those layers. Two supporting mechanisms keep the substitution reliable: a delayed establishment threshold that starts mapping only after enough prompt tokens have accumulated, and a truncation threshold that limits similarity computation on very long contexts.","core_discovery":"The paper's central claim is that attention matrices are transferable across models of different scale within the same series: Qwen2-7B and Qwen2-72B attain an average cosine similarity of 0.954 between best-matching attention matrices, and this transfer can be exploited at inference time. IAM replaces the LLM's attention output on a chosen set of layers with the most similar attention matrix from the SLM, using cosine similarity as the matching metric, after instruction-tuning the small model on Alpaca data. The authors show that mapping layers at the end of the network, plus a middle block, is much safer than mapping early layers, and that the prefill-established mapping stays stable during autoregressive generation with a mean consistency rate of 91.12%. With 30% of layers mapped, performance on language modeling, MMLU, HotpotQA, and GovReport stays close to the original LLM; at 50% mapping, KV cache use falls by about 22% and end-to-end throughput rises by about 10–11%.","pith_inferences":["A natural extension is to move from layer-level to head-level mapping: the paper itself notes this limitation, and head-level substitution could recover some of the performance lost at high mapping ratios while saving more cache.","If the 91.12% consistency rate degrades on longer or out-of-distribution contexts, a practical remedy the paper does not explore is periodic re-estimation of the mapping during decoding, or a confidence trigger that recomputes attention when similarity drops.","Because cross-series attention similarity is much lower (0.568 between LLaMA and Qwen2 with token alignment), IAM's practical reach depends on the availability of a well-matched small model in the same family; combining IAM with token-alignment methods might widen its applicability.","The heavy-tailed reuse of certain SLM attention matrices suggests the LLM draws from a small vocabulary of attention patterns; if so, one could precompute a compact set of mapping matrices and skip running the SLM at serve time entirely."],"forward_implications":["At a 30% mapping ratio, performance stays close to lossless across language modeling, MMLU, HotpotQA, and GovReport benchmarks.","At a 50% mapping ratio, the method reduces KV cache usage by about 21.7–22.5% and reduces time-to-first-token by 12–17%, with throughput gains of about 10–11%.","The layer-mapping strategy transfers across tasks: block-level performance on MMLU correlates strongly (Pearson up to 0.95) with perplexity, HotpotQA, and GovReport results.","IAM generalizes to a different model family: LLaMA 3.1-70B with LLaMA 3.2-1B preserves MMLU accuracy under mapping, with a single optimal mapping region at the end of the network.","IAM is orthogonal to token-level KV eviction such as H2O and can be stacked with it without harming language modeling performance beyond the original method's cost."],"supporting_citations":[{"why":"Supplies the starting observation that attention patterns are similar between small and large models, which IAM extends to LLMs.","marker":"Chen et al., 2021"},{"why":"Establishes the shared pretraining corpus and architecture basis the paper invokes to explain cross-scale attention similarity.","marker":"Yang et al., 2024a"},{"why":"Provides the token-level KV eviction baseline and the method IAM combines with to show orthogonality.","marker":"Zhang et al., 2023"},{"why":"Provides the WikiText-2 dataset used for perplexity and consistency-rate measurements that drive metric and layer decisions.","marker":"Merity et al., 2016"},{"why":"Provides the MMLU benchmark used to identify mapable layers and to evaluate mapping-ratio performance.","marker":"Hendrycks et al., 2021"},{"why":"Provides the Alpaca instruction-tuning data used to adapt the small model and mitigate mapping degradation.","marker":"Taori et al., 2023"},{"why":"Provides the GovReport long-context summarization benchmark used in layer-strategy transfer and performance evaluation.","marker":"Huang et al., 2021"},{"why":"Provides the HotpotQA benchmark used to check layer-mapping transfer and overall performance.","marker":"Yang et al., 2018"}],"fun_headline_variants":["Small-model attention maps cut LLM prefill 15% and KV cache 22%","Attention reuse from smaller LLMs speeds prefill by 15% and slims KV cache 22%","Cross-scale attention mapping: 15% faster prefill, 22% less KV cache","Attention mapping between LLM scales: 15% prefill speedup, 22% KV cache cut","Near-lossless attention mapping across LLM scales: 15% faster prefill, 22% less KV cache"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the matching between a large model's attention layers and a small model's attention layers, worked out from the prompt at the start of inference, stays stable enough during the rest of generation that reusing the same matching without recomputing attention does not degrade output quality.","fun_headline_variants_meta":{"raw":{"variants":["Small-model attention maps cut LLM prefill 15% and KV cache 22%","Attention reuse from smaller LLMs speeds prefill by 15% and slims KV cache 22%","Cross-scale attention mapping: 15% faster prefill, 22% less KV cache","Attention mapping between LLM scales: 15% prefill speedup, 22% KV cache cut","Near-lossless attention mapping across LLM scales: 15% faster prefill, 22% less KV cache"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001137,"raw_usage":{"total_tokens":4717,"prompt_tokens":937,"completion_tokens":3780,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":553,"completion_tokens_details":{"reasoning_tokens":3663}},"tokens_in":553,"tokens_out":3780,"duration_ms":29169,"temperature":1.0,"reasoning_tokens":3663,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T16:57:56.248679+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Measure IAM's perplexity and downstream accuracy on long prompts with long generation outputs, for example 8k-token prompts with 2k generated tokens, while re-estimating the mapping every 128 tokens versus never; if the never-remapped version degrades noticeably as generation length grows, the consistency premise fails.","supporting_citations":[{"cited_title":"bert2BERT: Towards Reusable Pretrained Language Models","cited_arxiv_id":"2110.07143","evidence_quote":"Supplies the starting observation that attention patterns are similar between small and large models, which IAM extends to LLMs."}],"review_version":1}