{"id":"109ffa02-1ece-4c1b-9017-d1efa49f8870","arxiv_id":"2505.23520","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"AnchorAttention uses the maximum attention score from initial and local tokens as an anchor to threshold-select important key-value positions at stripe granularity, achieving faster prefill with comparable accuracy.","lead":"A new sparse attention method, AnchorAttention, speeds up the prefill stage of long-context LLMs by computing only the most important attention positions using a threshold against an 'anchor' score. It reports up to 4.6x faster attention than FlashAttention and 1.44x faster than FlexPrefill at 128k tokens, while keeping accuracy close to full attention.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Equation 2's absolute logit cutoff cannot control softmax mass: at θ=12 and 128k context, 100k keys one unit below the threshold sum to ≈18% of attention mass, so the claim of higher sparsity at equal recall is not established for diffuse attention.","rationale":"The reader correctly identified the anchor assumption as the load-bearing premise. I sharpen it: even when the anchor is the true global maximum, Equation 2 is a per-position logit threshold with no control over aggregate pruned mass. The arithmetic example at θ=12 and 128k context shows that a tail of many individually irrelevant keys can sum to a large softmax mass, so the method can lose recall precisely on multi-token aggregation tasks. The paper's benchmarks are mostly retrieval and needle tasks with spike-like attention; LongBench tasks are shorter and mixed, so the tail effect is not isolated. The code is available and the experiments are not called into question; the issue is the scope of the central claim. A concrete test on synthetic logits and on multi-hop QA at 128k would settle whether the concern is realized. If it is, the claim should be qualified (e.g., 'higher sparsity at equal recall on retrieval-style tasks') or Equation 2 should be augmented with a cumulative-mass criterion, which would reintroduce some of the sorting cost the method aims to avoid. The Algorithm 2 variable inconsistency should also be corrected, but it is less fundamental than the mass-control problem, so my recommendation is to keep the reader's CONDITIONAL verdict rather than escalate to REJECT.","tokens_in":13143,"tokens_out":13569,"duration_ms":137999,"concrete_test":"Run two checks. (A) Analytical/synthetic: construct logits with one value 0 and 100k values −13; apply Eq. 2 with θ=12 and compute the selected softmax mass. The expected result is ≈81.6%, i.e., the threshold prunes ≈18% of mass despite every pruned key being only 13 below the anchor. (B) On Llama-3.1-8B-Instruct with 128k-context samples from HotpotQA or Musique, compute exact full-softmax logits, apply Eq. 2 with θ=12 and step=16, and report the distribution of selected softmax mass per query block. If more than 5% of query blocks capture less than 90% of mass, the central 'higher sparsity at same recall' claim is overgeneralized and a cumulative-mass safeguard (or a task-scoped claim) is required.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Equation 2 (Section 3.2) selects every key satisfying x_a − qk ≤ θ and drops everything else. This is a hard per-position logit cutoff, not a cumulative-mass criterion. With the paper's θ=12 and 128k context, take one key at the anchor score x_a and 100k keys at x_a−13; all 100k keys are unselected, yet they contribute 100000·exp(−13) ≈ 0.226 to the softmax denominator, so about 18% of the attention mass is pruned even though no single pruned key is within θ of the anchor. The same arithmetic applies to any long context with a broad tail of moderately important tokens. RULER and Needle-in-a-Haystack are retrieval tasks with spiky attention, so this regime is not exercised; LongBench's shorter sequences reduce tail accumulation but do not remove the structural issue. The headline claim—higher sparsity at the same recall—therefore holds only under a spike-dominated attention prior; it is not a consequence of Equation 2. This is the load-bearing weakness of the central claim, independent of the Algorithm 2 naming inconsistency noted in the reader report.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes AnchorAttention, a sparse attention mechanism for LLM prefill that identifies important key-value positions by comparing attention logits against an ``anchor'' score computed from initial tokens and a local window, using a stripe-level (column-wise) granularity instead of block-level sparsity. The method avoids sorting by applying a fixed threshold θ to the difference between the anchor and each pooled query-key score, then loads discrete KV positions in a custom kernel. The authors report higher sparsity rates at the same recall level compared to block-sparse baselines, a 1.44× speedup over FlexPrefill and 4.6× over FlashAttention at 128k context, and accuracy close to full attention on RULER, LongBench, and Needle-in-a-Haystack for LLaMA-3.1-8B and Qwen2.5-7B. The paper includes kernel-level algorithms, an ablation study, and a code release.","tokens_in":13377,"tokens_out":4802,"duration_ms":51369,"significance":"If the central claims hold, the work is a meaningful step for prefill acceleration: it replaces expensive sorting-based selection with a cheap threshold comparison, moves from block to finer stripe granularity, and demonstrates a real kernel implementation with released code. The idea of exploiting the attention-sink phenomenon to derive a global reference score is interesting and differs from purely local heuristics. However, the significance is currently tempered by unresolved technical inconsistencies in the algorithms and evaluation, and by a mathematical property of the threshold criterion that may undermine the claimed sparsity-recall tradeoff for diffuse attention distributions. The paper's own limitations section appropriately notes the narrow model scope and prefill-only focus.","major_comments":[{"comment":"Algorithm 2 declares the input as ``anchor score Acc'' and computes x_a = avgpool(Acc, bq). In Algorithm 1, however, Acc_i is the accumulated p·V value accumulator (a d-dimensional vector), not the scalar logit maximum. The anchor defined in Eq. (1) is derived from the maximum score, which Algorithm 1 stores in the M buffer. Using avgpool(Acc) as the anchor produces a vector, and subtracting the scalar qk in line 11 is dimensionally inconsistent. The pseudocode should use M (or the max logits) rather than Acc, and the text should state this correspondence explicitly.","section":"Appendix C, Algorithm 2"},{"comment":"The reported sparsity rates do not match the stated Top-K values and block sizes. At a 128k context with N=131072 keys, Block with Top-K=256 would select 32768 keys if Top-K counts 128-element blocks (75% sparsity), or 256 keys if Top-K counts individual keys (99.8% sparsity); neither equals the reported 56.3%. Stripe with Top-K=16384 would give 87.5% sparsity, not 76.6%. Please define exactly what ``Top-K'' counts in each row and recompute the table.","section":"Section 2.1.2, Table 1"},{"comment":"The difference-aware mask is a hard per-key logit cutoff, not a cumulative-mass criterion. The skeptic's arithmetic is correct: with θ=12 at 128k context, 100k keys at logit x_a−13 are all pruned, yet their combined softmax contribution is about 100000·exp(−13) ≈ 0.226, which is roughly 18% of the attention mass relative to a unit anchor term. The claim of higher sparsity at the same recall is therefore not a consequence of Eq. (2) for attention distributions with broad tails. RULER and Needle-in-a-Haystack are retrieval tasks with spiky attention, so the current benchmarks do not exercise this regime. The authors should either provide evidence on tasks with diffuse attention, or modify the selection criterion to bound cumulative pruned mass.","section":"Section 3.2, Eq. (2)"},{"comment":"The recall metric is not precisely defined: ``percentage of attention values that are numerically equal between the current sparse attention and the full attention'' is ambiguous. It is unclear whether recall measures matched logits, matched softmax entries, or captured probability mass, and how ties or near-equal values are counted. Since recall is the primary evaluation metric and the central claim is a sparsity-recall tradeoff, a formal definition is necessary. In addition, the experiments appear to be single runs; headline numbers and recall-sparsity curves should include variance estimates across seeds or heads.","section":"Section 4.2, Figure 4 caption"}],"minor_comments":[{"comment":"The claim that 99% and 90% of maximum attention scores are concentrated in the anchor regions would benefit from a precise definition of ``concentrated'' and from error bars across inputs.","section":"Section 2.2.2, Figure 5"},{"comment":"The meaning of the step parameter in Algorithm 2 is unclear: the loop ``for j=2 to j_end'' appears to skip the initial key block, possibly because it is covered by the anchor, but this should be stated explicitly.","section":"Section 3.4 and Appendix C"},{"comment":"Equation (3) defines the index set I without showing its dependence on the query block i, although the load_discrete operation is query-block dependent; please clarify the notation.","section":"Notation throughout"},{"comment":"The threshold values are inconsistent across figures: Figure 4 uses Difference-Aware(11) while Table 4 ranges over θ=10 to 15; the relationship between these numbers should be explained.","section":"Figure 4 and Table 4"},{"comment":"There are several typographical and formatting issues, including ``AnchorAttention'' appearing as a run-together token in the abstract, inconsistent capitalization of ``MInference'' versus ``Minference'', and the affiliation ``Chipltech''.","section":"Abstract and typing"}],"recommendation":"major_revision","confidential_remarks":"The paper proposes a genuinely interesting mechanism and provides a code release, which I value. However, I cannot endorse the central claim until the Algorithm 2 buffer issue and the Table 1 inconsistencies are fixed, and until the thresholding criterion is either shown to be safe for non-retrieval attention distributions or augmented with a cumulative-mass safeguard. The algorithmic bug is clearly fixable; the threshold-mass concern is more substantive but addressable with additional analysis or targeted experiments. The manuscript's own limitations section is honest, and I would encourage the authors to strengthen the evaluation accordingly."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Xin chào,\n\nThis paper proposes AnchorAttention, a sparse attention method for LLM prefill that uses an anchor score (the max attention score from the initial tokens and local window) as a reference, then prunes keys whose score is more than θ below that anchor. The stripe granularity and discrete KV loading are a real departure from block-based methods like FlexPrefill, and the reported speedups (1.44x over FlexPrefill, 4.6x over FlashAttention at 128k) are plausible given the sparsity levels shown. The code is available, which helps reproducibility.\n\nThe core idea is solid: avoiding sorting via a difference threshold is clever, and the anchor exploits the attention-sink phenomenon. The experiments cover two models and three benchmarks, and accuracy is close to full attention on most tasks. The ablation in Table 4 shows the anchor matters.\n\nThat said, there are real soft spots.\n\nThe stress-test concern is valid. Equation 2 is a hard per-position logit cutoff, not a cumulative-mass criterion. A long tail of 100k keys one unit below the threshold contributes roughly 18% of the attention mass but all are pruned at θ=12. The benchmarks used (RULER, Needle-in-a-Haystack) have spiky attention, so this regime is unseen. The claim that AnchorAttention achieves higher sparsity at the same recall is thus an empirical observation on those benchmarks, not a property of the method. The authors should either tighten the theory or explicitly limit the claim.\n\nAlgorithm 2 has a concrete bug: it computes the anchor average from Acc (the accumulated values) instead of M (the max logits). The text and Equation 1 clearly mean M. That is an easy fix, but it makes the pseudocode look sloppy.\n\nTable 1's sparsity numbers do not align with the stated Top-K values. Block(256) should retain far more than 56% of positions, so either the caption or the numbers are wrong. Needs clarification.\n\nNo error bars are reported, and the recall metric definition is vague: 'percentage of attention values that are numerically equal' is under-specified. A precise definition would help.\n\nOverall, this is an incremental but practical contribution. The flaws are fixable, and the core idea is worth pursuing. I would send it to review and ask for a revision that addresses the softmax-tail issue, fixes the algorithm bug, and clarifies the metrics and table.\n\nRecommendation: accept for peer review.\n\nBest,","headline":"Clever anchor-based sparse attention with real speedups on spiky benchmarks, but the hard logit threshold doesn't guarantee recall on diffuse attention; fixable gaps keep it from being a clean accept.","tokens_in":13911,"tokens_out":4694,"would_cite":false,"duration_ms":43085,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"AnchorAttention prunes attention keys by comparing scores to a cheaply computed anchor, reaching 4.6x faster prefill than FlashAttention at 128k context while keeping recall high.","keywords":["sparse attention","prefill acceleration","stripe granularity","difference-aware threshold","attention sink","large language models","anchor","Triton kernel"],"falsifier":"Run AnchorAttention on a model known to lack a strong attention sink or on inputs where critical information lies in the middle of the context, and measure recall versus full attention at a fixed sparsity level. If any attention head or substantial fraction of inputs shows a large recall drop, such as recall falling below 80% while block-based methods with the same sparsity stay above 90%, the anchor is not a valid global reference for those cases.","tokens_in":1666,"feed_emoji":"⚡","tokens_out":1682,"duration_ms":51389,"temperature":0.7,"pith_summary":"AnchorAttention is a sparse attention mechanism for the prefill phase of large language models that identifies important keys and values without sorting or computing full attention scores. The paper claims that a near-maximum attention score computed from only the initial tokens and a local window reliably anchors the global score distribution, so any key scoring within a threshold of this anchor can be kept and everything else pruned. Operating at stripe granularity, which retains the column dimension of attention blocks but not the full block, it achieves higher sparsity at the same recall as block-based methods. At 128k context, it reports a 1.44x speedup over FlexPrefill and a 4.6x speedup over FlashAttention while maintaining accuracy close to full attention.","feed_headline":"Difference-aware sparse attention: 4.6x faster prefill at 128k","feed_subtitle":"AnchorAttention prunes to stripes using a cheap anchor score, beating FlexPrefill by 1.44x while keeping recall.","key_machinery":"The load-bearing mechanism is the difference-aware threshold against the anchor. The anchor $x_a$ is computed as the maximum attention score over the initial key block and the local key block (Equation 1). Sparsity is then decided by the mask $\\mathrm{mask} = \\mathbb{I}(\\mathrm{avgpool}(x_a) - \\mathrm{avgpool}(Q)K^\\top/\\sqrt{d} \\le \\theta)$ (Equation 2), which avoids the sorting needed by top-k and top-cdf strategies. This anchor-guided comparison is the component that makes the method fast; the stripe granularity and discrete KV loading are the components that make it sparse at fine scale. The kernel implementation caches anchor intermediates and processes multiple query blocks per step to preserve parallelism.","core_discovery":"The paper claims that attention maps in LLMs have a stable structural commonality: the highest attention scores almost always fall in the initial-token region and the local window, and this stable maximum can be used as an anchor for global sparsity decisions. From this anchor, the method computes a difference-aware threshold: pooled queries are dotted with all keys, and any key whose score is within $\\theta$ of the anchor's pooled value is selected, generating a stripe-like sparse mask without sorting operations. This discovery in turn motivates a finer granularity than block sparsity, namely stripe sparsity, where the row dimension of blocks is collapsed to one, and a discrete KV loading scheme that loads only the selected key and value rows. The paper's central claim is that combining these three pieces yields higher sparsity at equal recall and faster prefill than previous sparse attention methods, as demonstrated by a 1.44x speedup over FlexPrefill and a 4.6x speedup over FlashAttention at 128k context.","pith_inferences":["The anchor-based threshold could plausibly extend to decode-phase KV cache eviction, since it provides a parameter-free criterion for which historical keys are likely important, though the paper only evaluates prefill.","The reliability of the anchor depends on the attention-sink phenomenon being strong; models trained without explicit sink tokens or with different positional encodings might produce flatter or differently peaked score distributions, which would weaken the threshold's pruning decisions, but this is untested beyond two models.","The reported 4.6x speedup is measured with a Triton kernel on a single A100; the same algorithm could behave differently on other hardware or with different block sizes, so kernel-level gains are likely implementation-sensitive even if the algorithmic sparsity gains transfer.","Stripe sparsity could be combined with content-based query-aware selection, such as retrieval-style key matching, to catch positions that the anchor misses in inputs where critical information sits mid-context without local peaks."],"forward_implications":["At 128k context, AnchorAttention claims a 1.44x speedup over FlexPrefill and a 4.6x speedup over FlashAttention while achieving a higher recall rate than FlexPrefill.","Stripe granularity attains higher sparsity than block granularity at the same recall level, with 76.6% sparsity versus 56.3% for block selection on LLaMA-3.1-8B at 128k RULER.","The difference-aware strategy achieves recall comparable to top-cdf selection while eliminating the sorting overhead, with similar sparsity heatmaps.","The method preserves accuracy close to full attention on LongBench, RULER, and Needle-in-a-Haystack across context lengths up to 128k, without fine-tuning.","Ablation shows that the anchor is essential: without it, reaching comparable recall requires far higher sparsity and computation time."],"supporting_citations":[{"why":"FlashAttention is the exact-attention baseline and the computational backbone; AnchorAttention's speedup is measured against it and its online-softmax pattern is reused.","marker":"(Dao et al., 2022)"},{"why":"StreamingLLM established the attention-sink observation that initial and local positions dominate attention scores, which motivates the anchor's design.","marker":"(Xiao et al., 2024)"},{"why":"MInference is a static sparse-pattern baseline that AnchorAttention contrasts against, and its recall definition is adopted for evaluation.","marker":"(Jiang et al., 2024)"},{"why":"FlexPrefill is the state-of-the-art dynamic baseline; AnchorAttention claims a 1.44x speedup and higher recall over it.","marker":"(Lai et al., 2025)"},{"why":"RULER is the primary benchmark used for recall, sparsity, and latency comparisons, including the 128k results.","marker":"(Hsieh et al., 2024)"},{"why":"LongBench provides the multi-task accuracy evaluation across diverse input formats.","marker":"(Bai et al., 2024)"}],"fun_headline_variants":["AnchorAttention: 4.6x faster prefill via anchor-based stripe sparsity","Stripe-sparse attention: 1.44x over FlexPrefill, 4.6x over FlashAttention","Anchor-based sparse attention cuts prefill time 4.6x at 128k","Difference-aware stripe sparsity: faster prefill without recall loss","AnchorAttention: fine-grained sparsity gives 1.44x speedup over FlexPrefill"],"cache_read_input_tokens":16128,"weakest_assumption_plain":"The assumption that the highest attention score among the first tokens and the local window is a reliable stand-in for the global maximum, so that any position scoring within $\\theta$ of this anchor is important and anything farther below can be safely discarded.","fun_headline_variants_meta":{"raw":{"variants":["AnchorAttention: 4.6x faster prefill via anchor-based stripe sparsity","Stripe-sparse attention: 1.44x over FlexPrefill, 4.6x over FlashAttention","Anchor-based sparse attention cuts prefill time 4.6x at 128k","Difference-aware stripe sparsity: faster prefill without recall loss","AnchorAttention: fine-grained sparsity gives 1.44x speedup over FlexPrefill"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001035,"raw_usage":{"total_tokens":4406,"prompt_tokens":1043,"completion_tokens":3363,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":659,"completion_tokens_details":{"reasoning_tokens":3248}},"tokens_in":659,"tokens_out":3363,"duration_ms":22697,"temperature":1.0,"reasoning_tokens":3248,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T12:44:42.300107+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run AnchorAttention on a model known to lack a strong attention sink or on inputs where critical information lies in the middle of the context, and measure recall versus full attention at a fixed sparsity level. If any attention head or substantial fraction of inputs shows a large recall drop, such as recall falling below 80% while block-based methods with the same sparsity stay above 90%, the anchor is not a valid global reference for those cases.","supporting_citations":[],"review_version":1}