{"id":"b78bcb76-c403-4723-842d-672962e6376e","arxiv_id":"2607.19358","paper_version":1,"verdict":"REJECT","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":4,"one_line_summary":"LISA replaces standard attention with linear attention plus indexer-selected sparse attention, claiming O(nM) inference cost and accuracy gains, but the complexity analysis ignores the indexer's full-context scoring.","lead":"LISA proposes a hybrid attention that combines a linear-attention long-range stream with an indexer-selected sparse self-attention stream to cut inference cost for long chain-of-thought reasoning. The paper claims O(nM) complexity and accuracy gains, but the described indexer computes scores over the full context at every step, which is O(n^2) overall.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The O(nM) claim omits the Indexer's O(t) full-context scoring and the retained full-attention layers (Sec. 6); total generation cost remains O(n^2).","rationale":"The reader's reject verdict is well-founded. The single most load-bearing flaw is the Indexer's full-context scoring at every generation step, which makes the central O(nM) complexity claim mathematically false. The paper itself describes this Indexer computation in §3.2, so the contradiction is internal, not a matter of external consensus. Section 6 provides additional, self-admitted evidence: the retained standard attention layers every 6 layers contribute ~70% of attention FLOPs, which by itself yields an O(n^2) term. The paper also claims 'constant per-step computation' in the introduction, but the described pipeline has per-step cost that grows linearly with t. No amount of engineering optimization removes this asymptotic growth. The empirical speedup at 16K might be a genuine constant-factor improvement, but the paper's headline complexity reduction and the inference from it (e.g., scalability to much longer contexts) do not follow. Because the reader already identified the Indexer cost as the weakest assumption and rejected on that basis, my stress-test does not change the verdict.","tokens_in":19298,"tokens_out":3828,"duration_ms":36644,"concrete_test":"Recompute the total FLOPs for generating n tokens under the Stage 2 algorithm: per step t, cost = O(M) sparse SA + O(t) Indexer scoring + O(t) for the retained full-attention layers, giving Σ O(t) = O(n^2). To settle empirically, profile LISA on a single A100 at n = 4K, 8K, 16K, 32K (generating new tokens from a fixed prefix) and record per-token latency. Under the paper's O(nM) claim, per-token latency should stay roughly flat in n; under O(n^2), it grows linearly in n. Also run a step where M = 256 and t = 16K and compare measured Indexer FLOPs vs sparse-attention FLOPs; the Indexer should dominate by ~t/M ≈ 64×.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The abstract and §3.2 claim LISA 'reduces inference complexity from O(n^2) to O(nM) (M << n) for generating n tokens' and that per-step cost is 'constant (independent of sequence length)' (Section 1). Both claims are internally contradicted by the Stage 2 description: 'At each generation step t, the Indexer performs an attention computation over the preceding t−1 tokens, selecting M important tokens' (§3.2, Stage 2). That scoring is O(t) per step, so total Indexer cost is Σ_{t=1}^n O(t) = O(n^2). Additionally, Section 6 concedes LISA 'retains the original standard attention layers every 6 layers, which contribute approximately 70% of the total attention FLOPs'; since each such layer attends over all t−1 keys at step t, this alone is also an O(n^2) term. Even if the Indexer were replaced by a constant-time selector, the asymptotic claim as stated would still be false. The reported 50% speedup at 16K may reflect constant-factor savings, but not the advertised complexity reduction.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes LISA, a hybrid attention module that replaces standard self-attention in reasoning LLMs with a linear-attention branch plus an Indexer-guided sparse-attention branch, claiming to reduce inference complexity from O(n^2) to O(nM) for generating n tokens. A two-stage training pipeline first fits a linear+sliding-window hybrid to a frozen teacher via cross-entropy, then trains a K-head Indexer via per-head KL distillation to select top-M tokens. Experiments on AIME, MATH-500, GSM8K, and AMC23 with Qwen2.5-1.5B/7B report lower latency and higher accuracy than the dense baseline and several compression baselines.","tokens_in":19672,"tokens_out":3838,"duration_ms":37724,"significance":"If the asymptotic and empirical claims were correct, LISA would be a practical, plug-and-play attention replacement for long-CoT reasoning: it requires no pretraining from scratch, leaves the original parameters frozen, and proposes a principled distillation objective for token selection. The paper also pursues an interesting two-stage training design and an analysis of linear/sparse synergy. However, the central complexity claim is contradicted by the paper's own description of the Indexer and of the retained full-attention layers, and the empirical gains may be confounded by train/eval overlap. The contribution, as currently presented, does not establish the advertised efficiency result.","major_comments":[{"comment":"The central complexity claim — reducing inference complexity from O(n^2) to O(nM) — is invalid as stated. §3.2 Stage 2 says that at each generation step t, the Indexer “performs an attention computation over the preceding t−1 tokens,” which is O(t) per step and sums to O(n^2). Additionally, §6 concedes that LISA “retains the original standard attention layers every 6 layers, which contribute approximately 70% of the total attention FLOPs”; each such layer attends to all previous keys and adds another O(n^2) term. The per-step “constant” cost claimed in §1 is therefore false unless the Indexer and the retained layers can be implemented subquadratically, which the paper does not show. Since this is the paper's headline efficiency result, the error undermines the main claim.","section":"Abstract, §1, §3.2, §6"},{"comment":"The reported accuracy gains may be confounded by training/evaluation overlap. Stage 1 and Stage 2 train on samples from OpenR1-Math-220K, while evaluation uses MATH-500, AMC23, AIME24, and AIME25. The paper does not report any deduplication or overlap analysis between the training subset and these benchmarks. If benchmark problems or their variants appear in the training set, the +5.6% average improvement over the dense baseline could reflect memorization rather than the LISA architecture. The authors should provide overlap statistics or rerun on contamination-free subsets.","section":"§4.1, §4.3"},{"comment":"The abstract claims a 50% inference speedup under a 16K-token context, but the paper does not describe a 16K-context latency experiment. The Table 1 metric “ReL” is labeled “relative latency” but defined as “average inference time per sample”; its baseline normalization is unclear. For Qwen2.5-7B, LISA's ReL of 226.6 versus the dense baseline's 314.2 is roughly a 28% reduction, not 50%. Without a clearly specified latency protocol and the supporting 16K experiment, the headline speedup cannot be verified.","section":"Table 1, §4.2"},{"comment":"The claim that the Indexer “Recovers Full Self-Attention Dynamics” is supported only by qualitative visualizations of two examples. No quantitative comparison is provided between the Indexer's top-M selection and the teacher's attention distribution. The MASS metric in Eq. (10) is defined with oracle attention, but the paper does not report MASS for the trained Indexer relative to the teacher on a held-out set. This weakens the analysis section, though it is not the central claim.","section":"§4.5"}],"minor_comments":[{"comment":"The “ReL” metric is inconsistently described and the table does not show absolute wall-clock times, standard deviations, or number of runs. Please clarify the normalization and add variance/error bars for accuracy and latency.","section":"§4.3, Table 1"},{"comment":"Equation (3) uses S(l)_{t,i-1} and S(l)_{t,0} but the indices are not clearly defined; in particular, the summation is over j=1..i-1 while the state on the left has index i-1. Please reconcile the indexing.","section":"§3.3"},{"comment":"Typo: “adopt two parallel streams” should be “adopts.”","section":"§5, Conclusion"},{"comment":"The case-study responses appear truncated mid-sentence (e.g., “6 * 5” and “log _{ 1 /”). Please ensure the appendix contains complete outputs.","section":"Appendix A"}],"recommendation":"reject","confidential_remarks":"The paper's main efficiency claim is internally contradicted by the described architecture, and the empirical evaluation lacks contamination control. These are load-bearing issues that cannot be resolved by minor local revisions; the contribution as presented does not meet the bar for publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The stress-test is right. The paper's central claim—that LISA reduces generation complexity from O(n^2) to O(nM)—is contradicted by its own algorithm. The Indexer scores all previous tokens at every step, so that alone is O(n^2) over a full generation. And the paper's Limitations section concedes that standard attention layers are retained every six layers, contributing roughly 70% of attention FLOPs. Those layers also attend over the full context. So the asymptotic claim as stated is false, period. The reported 50% speedup at 16K may be a real constant-factor win, but it is not the advertised complexity reduction.\n\nThat said, there is genuine content here. The two-stage training pipeline—cold-starting linear attention with cross-entropy before introducing the indexer with a per-head KL loss—is a thoughtful way to avoid the overfitting the authors say they observed with direct distillation. The dropout schedule on top-1.2M candidates is a nice trick. The MASS analysis is a reasonable way to measure whether the indexer is actually finding the same tokens as full attention, and the cosine-similarity analysis between the linear and sparse branches is an interesting diagnostic. The authors are also honest in the Limitations section about scope and about the retained layers; the problem is that the abstract and introduction don't carry that honesty through.\n\nThe other soft spots are real but secondary. Training on 100K samples from OpenR1-Math-220K and then evaluating on MATH-500 and AIME is a serious contamination risk, since OpenR1-Math is built from those very benchmarks. There are no error bars anywhere, and the baseline comparisons use a single run. The 'no pretraining from scratch' claim is true in a narrow sense, but the method still requires hundreds of GPU hours and a two-stage training procedure, so 'plug-and-play' undersells the setup cost.\n\nWho is this for? People working on efficient inference for long chain-of-thought models might get a useful recipe and a cautionary example. But the paper as it stands cannot be accepted: the headline claim is wrong, and the accuracy gains are confounded. It deserves a serious referee, not a desk rejection, because the underlying ideas are worth engaging with and the flaws are fixable. I would send it back for major revision and ask for a rewritten complexity analysis, a properly controlled training/evaluation setup, and error bars or a reproducibility note.\n\nRecommendation: send to peer review with heavy revision required.","headline":"The O(nM) complexity claim is not supported by the paper's own architecture; the work is a competent hybrid-attention recipe with an interesting indexer-distillation idea, but the headline efficiency result needs to be restated or it will mislead.","tokens_in":20084,"tokens_out":2282,"would_cite":false,"duration_ms":26610,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"LISA replaces full self-attention with a linear memory stream plus a top-M sparse stream, lowering long-reasoning generation cost from O(n²) to O(nM) and improving math accuracy by 5.6%.","keywords":["linear attention","sparse attention","chain-of-thought reasoning","long-context inference","token selection","knowledge distillation","KV-cache reduction","test-time scaling"],"falsifier":"Instrument a LISA deployment at 16K, 32K, and 64K context and measure per-step Indexer time and FLOPs. If Indexer cost grows roughly linearly with t, or if end-to-end latency grows faster than linearly with n, the asymptotic claim fails. A second check: swap the Indexer's learned selection for random top-M selection; if accuracy barely changes, the claimed benefit of attention-distilled selection is not doing the work.","tokens_in":19240,"feed_emoji":"⚡","tokens_out":6097,"duration_ms":58101,"temperature":0.7,"pith_summary":"The paper is trying to establish that the quadratic cost of self-attention can be lifted out of long chain-of-thought generation without retraining the base model. Its recipe: keep a linear-attention state that accumulates global context in O(1) per step, and let a lightweight indexer pick the M most relevant tokens from the full history so a small sparse self-attention can do precise retrieval. Trained in two stages on top of a frozen reasoning model, the hybrid is reported to cut 16K-context inference latency by 50% and raise average math-reasoning accuracy by 5.6% over the full-attention baseline. If the cost model holds, the practical consequence is that very long reasoning traces become cheap enough to serve at scale instead of being truncated.","feed_headline":"Hybrid attention halves long-reasoning cost","feed_subtitle":"LISA pairs a linear memory stream with top-M sparse attention, cutting 16K-context latency by half while lifting math accuracy.","key_machinery":"The load-bearing object is the Indexer—a K-head module of query/key projections and ReLU dot products that each step scores all t−1 previous tokens and fills the sparse self-attention with the top M. Its selection is aligned with the teacher by a per-head KL divergence loss with dynamic temperature, and it operates alongside the linear-attention state S_t (an accumulating matrix of outer products) whose O(1) recurrent update gives the hybrid its global memory. A learned scalar gate fuses the two streams. The paper also introduces segment-level state correction at inference: it computes the difference between linear-attention states at the start and end of each k-token segment, momentum-avera","core_discovery":"The central claim, on the paper's own terms, is that a hybrid attention module can replace the standard self-attention of a long-CoT model and both compute less and reason better. One branch, linear attention, compresses the entire preceding context into a state matrix via S_t = Σ kᵢᵀ vᵢ, providing O(1) per-step global memory. The other branch is a budgeted sparse self-attention over M tokens chosen by the Lightning Indexer, which scores all previous tokens with ReLU dot products and picks the top M; a learned gate fuses the two outputs. The Indexer is trained by per-head KL divergence against the frozen teacher's full-attention distributions, with dropout masking and z-loss to keep selectio","pith_inferences":["The paper states that the Indexer 'performs an attention computation over the preceding t−1 tokens' at every step, which is O(t) work per step and O(n²) over n steps; the O(nM) complexity claim stands only if that scoring can be implemented sub-quadratically, and the paper does not show how. Timing the Indexer alone at increasing context lengths would settle this.","The headline 5.6% accuracy gain is the 1.5B-model average; at 7B the gain is 4.7% and on short tasks like GSM8K the method only breaks even, suggesting the benefit concentrates in long-horizon reasoning—so the right deployment target is long traces, not short prompts.","A direct test of the Indexer's value would be to replace its learned selection with random top-M or recency-only selection and re-measure accuracy; if the gap is small, the distillation machinery is not load-bearing.","The momentum-based state correction at inference is a test-time-training trick that the paper couples to LISA; without ablating it separately, part of the reported gains could come from that correction rather than from the sparse attention itself."],"forward_implications":["Deploying LISA on an already-trained reasoning model requires no pretraining from scratch: only the linear-attention branch, the Indexer, and the gate are trained, so it can be dropped into existing models as a plug-in replacement.","Long chain-of-thought traces become substantially cheaper to generate—50% faster at 16K context in the paper's measurements—so test-time scaling can be pushed further for the same serving budget.","Accuracy does not simply degrade when attention is sparsified; the Indexer's learned selection recovers teacher-like attention patterns, improving average math accuracy by 5.6% on the 1.5B model and 4.7% on the 7B model.","The reported architecture still keeps standard attention in every 6th layer, which the authors estimate accounts for roughly 70% of attention FLOPs, so the current numbers are for a partial replacement; fully removing those layers is left as future work."],"fun_headline_variants":["Sparse attention halves long-reasoning cost","Linear-indexed attention cuts 16K-context latency 50%","LISA speeds long-CoT reasoning 50% and lifts math 5.6%","Top-M token attention doubles long-reasoning throughput"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"At every generation step the Indexer scores all preceding tokens to pick the top M, and the paper never shows how that scoring can avoid growing with sequence length; if it cannot, the total cost over n steps is O(n²), not O(nM).","fun_headline_variants_meta":{"raw":{"variants":["Sparse attention halves long-reasoning cost","Linear-indexed attention cuts 16K-context latency 50%","LISA speeds long-CoT reasoning 50% and lifts math 5.6%","Top-M token attention doubles long-reasoning throughput"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000609,"raw_usage":{"total_tokens":2739,"prompt_tokens":878,"completion_tokens":1861,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":622,"completion_tokens_details":{"reasoning_tokens":1787}},"tokens_in":622,"tokens_out":1861,"duration_ms":14852,"temperature":1.0,"reasoning_tokens":1787,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-02T12:44:35.702493+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Instrument a LISA deployment at 16K, 32K, and 64K context and measure per-step Indexer time and FLOPs. If Indexer cost grows roughly linearly with t, or if end-to-end latency grows faster than linearly with n, the asymptotic claim fails. A second check: swap the Indexer's learned selection for random top-M selection; if accuracy barely changes, the claimed benefit of attention-distilled selection is not doing the work.","supporting_citations":[],"review_version":1}