Pith. sign in

REVIEW 4 major objections 4 minor 64 references

LISA replaces full self-attention with a linear memory stream plus a top-M sparse stream, lowering long-reasoning generation cost from O(n²) to O(nM) and improving math accuracy by 5.6%.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 12:44 UTC pith:UMLZJUNI

load-bearing objection The O(nM) complexity claim is not supported by the paper's own architecture; the work is a competent hybrid-attention recipe with an interesting indexer-distillation idea, but the headline efficiency result needs to be restated or it will mislead. the 4 major comments →

arxiv 2607.19358 v1 pith:UMLZJUNI submitted 2026-05-29 cs.AI

LISA: Linear-Indexed Sparse Attention for Efficient Long-Context Reasoning

Yu Zhao , Zekun Zhang , Fan Jiang , Bo Zeng , Linlong Xu , Shimin Shan , Yu Liu , Longyue Wang
show 1 more author
Weihua Luo
This is my paper
classification cs.AI
keywords linear attentionsparse attentionchain-of-thought reasoninglong-context inferencetoken selectionknowledge distillationKV-cache reductiontest-time scaling
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper is trying to establish that the quadratic cost of self-attention can be lifted out of long chain-of-thought generation without retraining the base model. Its recipe: keep a linear-attention state that accumulates global context in O(1) per step, and let a lightweight indexer pick the M most relevant tokens from the full history so a small sparse self-attention can do precise retrieval. Trained in two stages on top of a frozen reasoning model, the hybrid is reported to cut 16K-context inference latency by 50% and raise average math-reasoning accuracy by 5.6% over the full-attention baseline. If the cost model holds, the practical consequence is that very long reasoning traces become cheap enough to serve at scale instead of being truncated.

Core claim

The central claim, on the paper's own terms, is that a hybrid attention module can replace the standard self-attention of a long-CoT model and both compute less and reason better. One branch, linear attention, compresses the entire preceding context into a state matrix via S_t = Σ kᵢᵀ vᵢ, providing O(1) per-step global memory. The other branch is a budgeted sparse self-attention over M tokens chosen by the Lightning Indexer, which scores all previous tokens with ReLU dot products and picks the top M; a learned gate fuses the two outputs. The Indexer is trained by per-head KL divergence against the frozen teacher's full-attention distributions, with dropout masking and z-loss to keep selectio

What carries the argument

The load-bearing object is the Indexer—a K-head module of query/key projections and ReLU dot products that each step scores all t−1 previous tokens and fills the sparse self-attention with the top M. Its selection is aligned with the teacher by a per-head KL divergence loss with dynamic temperature, and it operates alongside the linear-attention state S_t (an accumulating matrix of outer products) whose O(1) recurrent update gives the hybrid its global memory. A learned scalar gate fuses the two streams. The paper also introduces segment-level state correction at inference: it computes the difference between linear-attention states at the start and end of each k-token segment, momentum-avera

Load-bearing premise

At every generation step the Indexer scores all preceding tokens to pick the top M, and the paper never shows how that scoring can avoid growing with sequence length; if it cannot, the total cost over n steps is O(n²), not O(nM).

What would settle it

Instrument a LISA deployment at 16K, 32K, and 64K context and measure per-step Indexer time and FLOPs. If Indexer cost grows roughly linearly with t, or if end-to-end latency grows faster than linearly with n, the asymptotic claim fails. A second check: swap the Indexer's learned selection for random top-M selection; if accuracy barely changes, the claimed benefit of attention-distilled selection is not doing the work.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Deploying LISA on an already-trained reasoning model requires no pretraining from scratch: only the linear-attention branch, the Indexer, and the gate are trained, so it can be dropped into existing models as a plug-in replacement.
  • Long chain-of-thought traces become substantially cheaper to generate—50% faster at 16K context in the paper's measurements—so test-time scaling can be pushed further for the same serving budget.
  • Accuracy does not simply degrade when attention is sparsified; the Indexer's learned selection recovers teacher-like attention patterns, improving average math accuracy by 5.6% on the 1.5B model and 4.7% on the 7B model.
  • The reported architecture still keeps standard attention in every 6th layer, which the authors estimate accounts for roughly 70% of attention FLOPs, so the current numbers are for a partial replacement; fully removing those layers is left as future work.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The paper states that the Indexer 'performs an attention computation over the preceding t−1 tokens' at every step, which is O(t) work per step and O(n²) over n steps; the O(nM) complexity claim stands only if that scoring can be implemented sub-quadratically, and the paper does not show how. Timing the Indexer alone at increasing context lengths would settle this.
  • The headline 5.6% accuracy gain is the 1.5B-model average; at 7B the gain is 4.7% and on short tasks like GSM8K the method only breaks even, suggesting the benefit concentrates in long-horizon reasoning—so the right deployment target is long traces, not short prompts.
  • A direct test of the Indexer's value would be to replace its learned selection with random top-M or recency-only selection and re-measure accuracy; if the gap is small, the distillation machinery is not load-bearing.
  • The momentum-based state correction at inference is a test-time-training trick that the paper couples to LISA; without ablating it separately, part of the reported gains could come from that correction rather than from the sparse attention itself.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper proposes LISA, a hybrid attention module that replaces standard self-attention in reasoning LLMs with a linear-attention branch plus an Indexer-guided sparse-attention branch, claiming to reduce inference complexity from O(n^2) to O(nM) for generating n tokens. A two-stage training pipeline first fits a linear+sliding-window hybrid to a frozen teacher via cross-entropy, then trains a K-head Indexer via per-head KL distillation to select top-M tokens. Experiments on AIME, MATH-500, GSM8K, and AMC23 with Qwen2.5-1.5B/7B report lower latency and higher accuracy than the dense baseline and several compression baselines.

Significance. If the asymptotic and empirical claims were correct, LISA would be a practical, plug-and-play attention replacement for long-CoT reasoning: it requires no pretraining from scratch, leaves the original parameters frozen, and proposes a principled distillation objective for token selection. The paper also pursues an interesting two-stage training design and an analysis of linear/sparse synergy. However, the central complexity claim is contradicted by the paper's own description of the Indexer and of the retained full-attention layers, and the empirical gains may be confounded by train/eval overlap. The contribution, as currently presented, does not establish the advertised efficiency result.

major comments (4)
  1. [Abstract, §1, §3.2, §6] The central complexity claim — reducing inference complexity from O(n^2) to O(nM) — is invalid as stated. §3.2 Stage 2 says that at each generation step t, the Indexer “performs an attention computation over the preceding t−1 tokens,” which is O(t) per step and sums to O(n^2). Additionally, §6 concedes that LISA “retains the original standard attention layers every 6 layers, which contribute approximately 70% of the total attention FLOPs”; each such layer attends to all previous keys and adds another O(n^2) term. The per-step “constant” cost claimed in §1 is therefore false unless the Indexer and the retained layers can be implemented subquadratically, which the paper does not show. Since this is the paper's headline efficiency result, the error undermines the main claim.
  2. [§4.1, §4.3] The reported accuracy gains may be confounded by training/evaluation overlap. Stage 1 and Stage 2 train on samples from OpenR1-Math-220K, while evaluation uses MATH-500, AMC23, AIME24, and AIME25. The paper does not report any deduplication or overlap analysis between the training subset and these benchmarks. If benchmark problems or their variants appear in the training set, the +5.6% average improvement over the dense baseline could reflect memorization rather than the LISA architecture. The authors should provide overlap statistics or rerun on contamination-free subsets.
  3. [Table 1, §4.2] The abstract claims a 50% inference speedup under a 16K-token context, but the paper does not describe a 16K-context latency experiment. The Table 1 metric “ReL” is labeled “relative latency” but defined as “average inference time per sample”; its baseline normalization is unclear. For Qwen2.5-7B, LISA's ReL of 226.6 versus the dense baseline's 314.2 is roughly a 28% reduction, not 50%. Without a clearly specified latency protocol and the supporting 16K experiment, the headline speedup cannot be verified.
  4. [§4.5] The claim that the Indexer “Recovers Full Self-Attention Dynamics” is supported only by qualitative visualizations of two examples. No quantitative comparison is provided between the Indexer's top-M selection and the teacher's attention distribution. The MASS metric in Eq. (10) is defined with oracle attention, but the paper does not report MASS for the trained Indexer relative to the teacher on a held-out set. This weakens the analysis section, though it is not the central claim.
minor comments (4)
  1. [§4.3, Table 1] The “ReL” metric is inconsistently described and the table does not show absolute wall-clock times, standard deviations, or number of runs. Please clarify the normalization and add variance/error bars for accuracy and latency.
  2. [§3.3] Equation (3) uses S(l)_{t,i-1} and S(l)_{t,0} but the indices are not clearly defined; in particular, the summation is over j=1..i-1 while the state on the left has index i-1. Please reconcile the indexing.
  3. [§5, Conclusion] Typo: “adopt two parallel streams” should be “adopts.”
  4. [Appendix A] The case-study responses appear truncated mid-sentence (e.g., “6 * 5” and “log _{ 1 /”). Please ensure the appendix contains complete outputs.

Circularity Check

1 steps flagged

Ancillary MASS validation reduces to the Indexer's training target; the efficiency claim is internally contradictory rather than circular.

specific steps
  1. fitted input called prediction [Section 3.2 Eq. (2) and Section 4.5, MASS analysis]
    "At each generation step t, the Indexer performs an attention computation over the preceding t−1 tokens, selecting M important tokens (§3.2). The Indexer loss is defined as LIndexer = 1/KL Σ_j Σ_t KL( T^{(j)}_{t,:} || softmax(S^{(j)}_{t,:}) ) (Eq. 2), where T^{(j)}_{t,:} is the normalized teacher target distribution. Later: 'we introduce a metric termed MASS ... where Attn(q,k) denotes the oracle attention weights. ... Our empirical results demonstrate that at M=256, the MASS reaches 80%, trailing the theoretical upper bound (defined by the oracle's top-M weights) by only 6%.' (§4.5)"

    The Indexer is explicitly trained to match the teacher full-attention distribution through the per-head KL loss in Eq. 2, using stop-gradient targets derived from teacher attention heads. The MASS metric in Eq. 10 then measures how much of the oracle teacher attention mass falls inside the Indexer's selected top-M tokens. Reporting MASS=80% as evidence that the Indexer 'recovers' full self-attention dynamics is therefore a goodness-of-fit check on the training target, not an independent prediction: the measured quantity is close to what the training objective directly optimizes by construction.

full rationale

The main claimed circularity concern is not in the architecture's training procedure: using a frozen teacher's attention as a distillation target is a standard, non-circular setup, and the head-to-head accuracy/latency comparisons against external baselines are independent evidence. The clearest circular element is the MASS validation: the Indexer is fitted to teacher attention with KL loss, and MASS is then reported as if it independently validated the Indexer, though it is essentially a fit diagnostic. This is ancillary rather than the paper's central claim, so the score is kept at 3 rather than 6+. Separately, the advertised O(nM) inference complexity is internally contradicted by the paper's own Stage 2 description (Indexer attention over all preceding t−1 tokens is O(t) per step, summing to O(n^2)) and by the Section 6 concession that retained standard attention layers contribute approximately 70% of attention FLOPs. That is a serious correctness/validity problem, but I do not count it as a circularity reduction: the claim is unsupported and inconsistent with the stated algorithm, not equivalent to its inputs by definition. The training-data overlap issue (OpenR1-Math-220K vs. MATH-500/AIME) is a possible external-validity confound, but the paper does not demonstrate the overlap, so it cannot be treated as an exhibited circular reduction under the hard rules.

Axiom & Free-Parameter Ledger

4 free parameters · 4 axioms · 0 invented entities

The central claim rests on two load-bearing assumptions that are questionable: that the Indexer's full-context scoring is negligible (violated by the described algorithm) and that the training data does not leak into the evaluation sets. The free parameters M, α, and unspecified indexer details further make the method under-determined.

free parameters (4)
  • M (window size / top-M token budget) = 256
    Chosen empirically via the MASS metric (§4.5), balancing coverage and compute. Used throughout as the sparse self-attention budget and sliding-window size.
  • α (state-transfer correction coefficient) = not reported
    Fixed hyperparameter in Eq (8) for the inference-time momentum correction. No value or sensitivity analysis is given.
  • Indexer head count K and teacher-head assignment P/K = not reported
    The paper describes a K-head Indexer where each head fits P/K teacher heads, but K and P are not specified. Necessary for replication.
  • Indexer dropout schedule = 20% initial, gradually reduced
    Stage 2 uses top-1.2M candidates with 20% random masking, then reduces dropout; no schedule or final value is given.
axioms (4)
  • domain assumption Linear attention with identity feature map and no normalization is a viable long-range memory
    Section 3.1 adopts this linear attention form, citing prior work, but the paper itself notes 'no direct empirical evidence that linear attention can enhance model reasoning capability'.
  • ad hoc to paper The Indexer's full-context scoring is cheap enough to preserve sub-quadratic total complexity
    Stage 2 (Section 3.2) describes the Indexer computing scores over all previous tokens each step. The O(nM) claim assumes this cost is negligible, which is not justified.
  • domain assumption Training on OpenR1-Math-220K does not overlap with evaluation benchmarks
    Section 4.1 samples 100K from OpenR1-Math-220K to train LISA, and evaluates on GSM8K, MATH-500, AMC, AIME. OpenR1-Math is derived from math datasets including MATH, so overlap is likely but no deduplication is reported.
  • ad hoc to paper Retaining original standard attention every 6 layers preserves sufficient exact retrieval
    Section 3.2 states this design choice without analysis; the limitations admit these layers contribute ~70% of attention FLOPs, which affects the efficiency claim.

pith-pipeline@v1.3.0-alltime-deepseek · 19060 in / 10595 out tokens · 95100 ms · 2026-08-02T12:44:35.702493+00:00 · methodology

0 comments
read the original abstract

Recent advances in long chain-of-thought reasoning models such as DeepSeek-R1 have led to increasingly longer inference context lengths under the test-time scaling paradigm. However, the O(n^2) computational complexity of standard self-attention causes inference costs to grow sharply with long sequences, limiting the deployment of long-CoT reasoning in production settings. To address this, we propose LISA (Linear-Indexed Sparse Attention), a plug-and-play attention replacement module that requires no pretraining from scratch. LISA integrates two lightweight components in parallel within the original model: (1) a Linear Attention module that provides long-range memory with O(n) time complexity; (2) a Lightning Indexer that selects the top-M important tokens from the full context to feed into a Sparse Self-Attention. The two branches are fused via a gating mechanism, reducing inference complexity from O(n^2) to O(nM) (M << n) for generating n tokens. We design a two-stage training pipeline: Stage 1 initializes the model by integrating the linear attention to capture long-range dependencies, complemented by a sliding-window attention mechanism that is optimized via knowledge distillation to approximate the full self-attention distribution of a frozen teacher model. In Stage 2, we further introduce the Indexer to replace the static sliding-window mechanism, enabling dynamic token selection from broader contexts. The Indexer is trained using a novel per-head KL divergence loss, which aligns its selection behavior with the attention patterns of the teacher model. Experiments on DeepSeek-distilled-Qwen models demonstrate that LISA achieves a 50% inference speedup under 16K-token context, while improving average performance by 5.6% on reasoning benchmarks including AIME and MATH-500.

Figures

Figures reproduced from arXiv: 2607.19358 by Bo Zeng, Fan Jiang, Linlong Xu, Longyue Wang, Shimin Shan, Weihua Luo, Yu Liu, Yu Zhao, Zekun Zhang.

Figure 1
Figure 1. Figure 1: Overview of the LISA architecture. controls the information flow ratio between the two streams. Following related works on linear attention [MiniMax et al., 2025], we replace the standard attention in the teacher model with our linear attention variant while retaining the original standard attention every 6 layers to preserve exact retrieval capability. Specifically, we divide the training into two stages.… view at source ↗
Figure 2
Figure 2. Figure 2: Cosine similarity between linear attention and sparse self-attention outputs across all layers. [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Visualization of the top-256 tokens selected by the Indexer across layers. [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

64 extracted references · 20 linked inside Pith

  1. [1]

    Transformers are

    Katharopoulos, Angelos and Vyas, Apoorv and Pappas, Nikolaos and Fleuret, Fran. Transformers are. Proceedings of ICML , year=

  2. [2]

    arXiv preprint , year=

    The llama 4 herd: The beginning of a new era of natively multimodal AI innovation , author=. arXiv preprint , year=

  3. [3]

    2025 , eprint=

    Qwen3 Technical Report , author=. 2025 , eprint=

    2025

  4. [4]

    Proceedings of NeurIPS , year=

    Attention is all you need , author=. Proceedings of NeurIPS , year=

  5. [5]

    Finetuning pretrained transformers into

    Kasai, Jungo and Peng, Hao and Zhang, Yizhe and Yogatama, Dani and Ilharco, Gabriel and Pappas, Nikolaos and Mao, Yi and Chen, Weizhu and Smith, Noah A , booktitle=. Finetuning pretrained transformers into

  6. [6]

    Proceedings of ICLR , year=

    Random feature attention , author=. Proceedings of ICLR , year=

  7. [7]

    arXiv:2307.08621 , year=

    Retentive network: A successor to transformer for large language models , author=. arXiv:2307.08621 , year=

    Pith/arXiv arXiv

  8. [8]

    Proceedings of ICML , year=

    Gated linear attention transformers with hardware-efficient training , author=. Proceedings of ICML , year=

  9. [9]

    Proceedings of NeurIPS , year=

    Chain-of-thought prompting elicits reasoning in large language models , author=. Proceedings of NeurIPS , year=

  10. [10]

    Proceedings of NeurIPS , year=

    Large language models are zero-shot reasoners , author=. Proceedings of NeurIPS , year=

  11. [11]

    2026 , eprint=

    OpenAI o1 System Card , author=. 2026 , eprint=

    2026

  12. [12]

    arXiv:2501.12948 , year=

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning , author=. arXiv:2501.12948 , year=

    Pith/arXiv arXiv

  13. [13]

    arXiv:2502.15589 , year=

    Lightthinker: Thinking step-by-step compression , author=. arXiv:2502.15589 , year=

    arXiv

  14. [14]

    arXiv:2412.18547 , year=

    Token-budget-aware llm reasoning , author=. arXiv:2412.18547 , year=

    Pith/arXiv arXiv

  15. [15]

    arXiv:2407.19825 , year=

    Concise thoughts: Impact of output length on llm reasoning and cost , author=. arXiv:2407.19825 , year=

    Pith/arXiv arXiv

  16. [16]

    arXiv:2502.12067 , year=

    Tokenskip: Controllable chain-of-thought compression in llms , author=. arXiv:2502.12067 , year=

    arXiv

  17. [17]

    arXiv:2503.06692 , year=

    Inftythink: Breaking the length limits of long-context reasoning in large language models , author=. arXiv:2503.06692 , year=

    arXiv

  18. [18]

    Hu, Edward J and Shen, Yelong and Wallis, Phillip and Allen-Zhu, Zeyuan and Li, Yuanzhi and Wang, Shean and Wang, Lu and Chen, Weizhu and others , booktitle=

  19. [19]

    2025 , eprint=

    Learning to (Learn at Test Time): RNNs with Expressive Hidden States , author=. 2025 , eprint=

    2025

  20. [20]

    arXiv:2505.17827 , year=

    Not all tokens are what you need in thinking , author=. arXiv:2505.17827 , year=

    Pith/arXiv arXiv

  21. [21]

    Proceedings of ICML , year=

    Do not think that much for 2+3=? on the overthinking of o1-like llms , author=. Proceedings of ICML , year=

  22. [22]

    arXiv:2507.06203 , year=

    A survey on latent reasoning , author=. arXiv:2507.06203 , year=

    Pith/arXiv arXiv

  23. [23]

    Proceedings of NeurIPS , year=

    Tree of thoughts: Deliberate problem solving with large language models , author=. Proceedings of NeurIPS , year=

  24. [24]

    Proceedings of ICLR , year=

    Self-consistency improves chain of thought reasoning in language models , author=. Proceedings of ICLR , year=

  25. [25]

    Proceedings of ICLR , year=

    Least-to-most prompting enables complex reasoning in large language models , author=. Proceedings of ICLR , year=

  26. [26]

    https://qwenlm.github.io/blog/qwq-32b-preview/ , year=

    QwQ: Reflect deeply on the boundaries of the unknown , author=. https://qwenlm.github.io/blog/qwq-32b-preview/ , year=

  27. [27]

    Proceedings of NeurIPS , year=

    Can language models learn to skip steps? , author=. Proceedings of NeurIPS , year=

  28. [28]

    Proceedings of ACL , year=

    The impact of reasoning step length on large language models , author=. Proceedings of ACL , year=

  29. [29]

    Proceedings of ICLR , year=

    The expressive power of transformers with chain of thought , author=. Proceedings of ICLR , year=

  30. [30]

    arXiv:2504.09858 , year=

    Reasoning models can be effective without thinking , author=. arXiv:2504.09858 , year=

    Pith/arXiv arXiv

  31. [31]

    arXiv:2502.20122 , year=

    Self-training elicits concise reasoning in large language models , author=. arXiv:2502.20122 , year=

    Pith/arXiv arXiv

  32. [32]

    arXiv:2503.04697 , year=

    L1: Controlling how long a reasoning model thinks with reinforcement learning , author=. arXiv:2503.04697 , year=

    Pith/arXiv arXiv

  33. [33]

    arXiv:2503.04472 , year=

    Dast: Difficulty-adaptive slow-thinking for large reasoning models , author=. arXiv:2503.04472 , year=

    arXiv

  34. [34]

    arXiv:2502.09601 , year=

    Cot-valve: Length-compressible chain-of-thought tuning , author=. arXiv:2502.09601 , year=

    Pith/arXiv arXiv

  35. [35]

    Proceedings of AAAI , year=

    C3ot: Generating shorter chain-of-thought without compromising effectiveness , author=. Proceedings of AAAI , year=

  36. [36]

    arXiv:2505.16838 , year=

    R1-Compress: Long Chain-of-Thought Compression via Chunk Compression and Search , author=. arXiv:2505.16838 , year=

    arXiv

  37. [37]

    2025 , eprint=

    Qwen2.5 Technical Report , author=. 2025 , eprint=

    2025

  38. [38]

    arXiv:2309.17453 , year=

    Efficient streaming language models with attention sinks , author=. arXiv:2309.17453 , year=

    Pith/arXiv arXiv

  39. [39]

    2024 , eprint=

    SnapKV: LLM Knows What You are Looking for Before Generation , author=. 2024 , eprint=

    2024

  40. [40]

    2025 , eprint=

    PyramidKV: Dynamic KV Cache Compression based on Pyramidal Information Funneling , author=. 2025 , eprint=

    2025

  41. [41]

    Proceedings of NeurIPS , year=

    H2O: Heavy-hitter oracle for efficient generative inference of large language models , author=. Proceedings of NeurIPS , year=

  42. [42]

    Proceedings of ICML , year=

    Sepllm: Accelerate large language models by compressing one segment into one separator , author=. Proceedings of ICML , year=

  43. [43]

    arXiv:2110.14168 , year=

    Training verifiers to solve math word problems , author=. arXiv:2110.14168 , year=

    Pith/arXiv arXiv

  44. [44]

    Proceedings of NeurIPS , year=

    Measuring mathematical problem solving with the MATH dataset , author=. Proceedings of NeurIPS , year=

  45. [45]

    Rein, David and Hou, Betty Li and Stickland, Asa Cooper and Petty, Jackson and Pang, Richard Yuanzhe and Dirani, Julien and Michael, Julian and Bowman, Samuel R , booktitle=

  46. [46]

    arXiv:2107.03374 , year=

    Evaluating large language models trained on code , author=. arXiv:2107.03374 , year=

    Pith/arXiv arXiv

  47. [47]

    2023 , eprint=

    FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning , author=. 2023 , eprint=

    2023

  48. [48]

    2019 , eprint=

    A Study of BFLOAT16 for Deep Learning Training , author=. 2019 , eprint=

    2019

  49. [49]

    American Invitational Mathematics Examination (AIME) 2024-I & II , year =

    2024

  50. [50]

    American Invitational Mathematics Examination (AIME) 2025-I & II , year =

    2025

  51. [51]

    American Mathematics Competitions , year =

  52. [52]

    Proceedings of NeurIPS , year=

    Parallelizing linear transformers with the delta rule over sequence length , author=. Proceedings of NeurIPS , year=

  53. [53]

    HGRN2: Gated linear

    Qin, Zhen and Yang, Songlin and Sun, Weixuan and Shen, Xuyang and Li, Dong and Sun, Weigao and Zhong, Yiran , booktitle=. HGRN2: Gated linear

  54. [54]

    Proceedings of ICLR , year=

    Longhorn: State space models are amortized online learners , author=. Proceedings of ICLR , year=

  55. [55]

    Proceedings of ICML , year=

    Liger: Linearizing large language models to gated recurrent structures , author=. Proceedings of ICML , year=

  56. [56]

    arXiv:2310.01427 , year=

    Attention sorting combats recency bias in long context language models , author=. arXiv:2310.01427 , year=

    Pith/arXiv arXiv

  57. [57]

    arXiv:2407.01100 , year=

    Eliminating position bias of language models: A mechanistic approach , author=. arXiv:2407.01100 , year=

    Pith/arXiv arXiv

  58. [58]

    arXiv:2406.16747 , year=

    Sparser is faster and less is more: Efficient sparse attention for long-range transformers , author=. arXiv:2406.16747 , year=

    Pith/arXiv arXiv

  59. [59]

    arXiv:2306.12929 , year=

    Quantizable transformers: Removing outliers by helping attention heads do nothing , author=. arXiv:2306.12929 , year=

    Pith/arXiv arXiv

  60. [60]

    2026 , eprint=

    A State-Transition Framework for Efficient LLM Reasoning , author=. 2026 , eprint=

    2026

  61. [61]

    arXiv:2501.08313 , year=

    MiniMax-01: Scaling foundation models with lightning attention , author=. arXiv:2501.08313 , year=

    Pith/arXiv arXiv

  62. [62]

    arXiv:2512.02556 , year=

    DeepSeek-V3.2: Pushing the frontier of open large language models , author=. arXiv:2512.02556 , year=

    Pith/arXiv arXiv

  63. [63]

    arXiv:2602.15763 , year=

    GLM-5: From vibe coding to agentic engineering , author=. arXiv:2602.15763 , year=

    Pith/arXiv arXiv

  64. [64]

    2024 , eprint=

    Marco-o1: Towards Open Reasoning Models for Open-Ended Solutions , author=. 2024 , eprint=

    2024