Pith. sign in

REVIEW 5 major objections 5 minor 32 references

RIS-Kernel: A Model-Agnostic Architecture for Long-Context LLM Inference via Sparse Attention

T0 review · 5 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read At 1% attention density with 70 ensemble seeds, a sparse stochastic kernel reaches 75.00% retrieval accuracy on a 1.5B model, above the 71.88% dense baseline, and extends the same unmodified weights to 65,536-token contexts on CPUs.

desk verdict The paper's O(N log N) complexity claim is contradicted by its own fixed-density mask design; the engineering and honest failure reporting are real, but the empirical headline is a one-question difference. read the letter →

arxiv 2607.21927 v1 pith:6QCA27QS submitted 2026-07-24 cs.LG

Anderson R. Santos This is my paper · ORCID
classification cs.LG
keywords sparseattentionlong-contextinferenceensemblestochasticprojectionmodel-agnosticretrieval-augmentedgenerationCPUrotarypositionembeddingregularizer
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that the quadratic cost of self-attention can be bypassed at inference time by a model-agnostic sparse attention engine, RIS, that injects stochastic masks into unmodified weights and ensembles their predictions. It claims a complexity reduction to $O(N \log N)$ with commodity memory, and validates the claim on Qwen2-1.5B-Instruct: at 32,768 tokens a 1%-density, 70-seed ensemble reaches 75.00% accuracy versus 71.88% for dense attention, while at 65,536 tokens RIS under YaRN scaling beats the zero-context floor by 14.06 percentage points, all on CPU-only servers. The paper's stated conclusion is that sparse attention acts as a regularizer, pruning sequence-level noise, and that long-context factual retrieval does not inherently require GPU acceleration. A sympathetic reading takes this as evidence that dense attention is not an upper bound for retrieval and that positional encoding quality, not the sparse geometry, is the binding constraint beyond the native window.

What carries the argument

The load-bearing mechanism is the ensemble stochastic mask: each seed draws a sparse set of key–value indices, and the union of seeds controls retrieval coverage through $U = 1 - (1 - \rho_{\text{sub}})^{kN}$, where $k$ is the number of times a target token appears and $N$ the number of seeds. A Hybrid Anchor caches the stochastic union after the first generated token and merges it with a sliding local window, and the pre-fusion unified softmax (PFUS) places every selected token into a single normalization so a stochastically recovered entity competes equally with local tokens. Two sampling regimes instantiate the mask: Stochastic mode draws uniform global neighbors, while Structural mode builds block cliques of size $B = \min(0.1N, B_{\max})$ plus global edges to guarantee local anchor coverage. The streamed boolean mask generation keeps peak memory bounded, and dynamic RoPE scaling supplies linear or YaRN positional factors.

What would settle it

Run the RIS prefill at 16,384, 32,768, and 65,536 tokens with fixed 1% density and the same seed count, and count the nonzero attention entries or measure matmul wall-clock time: if nonzeros and time scale roughly with $N^2$ rather than $N \log N$, the central complexity claim is contradicted even if the accuracy results replicate.

Watch

Extended reading notes

Core claim

The core claim is that an ensemble of sparse stochastic attention masks, normalized through a single fused softmax, can reproduce or exceed the factual retrieval signal of full dense attention without any weight modification. In the controlled 32k experiment, RIS-Stochastic at 1% density with 70–80 seeds scores 75.00% (24/32), exceeding the native dense baseline of 71.88%, and at 5% density with 10 seeds it matches dense exactly. RIS-Structural at 1% density with 10 seeds recovers 75% of the contextual gap, and at 65,536 tokens the best structural run reaches 65.62% against a 51.56% floor, a 14.06-point gain the paper reports as marginally significant under McNemar's paired test. The paper also establishes a boundary: under linear RoPE interpolation at 64k, single-seed accuracy falls to 15.6–23.4%, near or below random guessing, while YaRN preserves enough positional geometry for the ensemble to recover; TinyLlama's 2,048-token native limit fails beyond roughly 2x extrapolation.

Load-bearing premise

The $O(N \log N)$ complexity claim assumes the sparse mask contains only $O(N \log N)$ nonzero attention pairs; the implementation fixes density $d$ as a fraction of the sequence and allocates $d \cdot N$ keys per query, which yields $O(dN^2)$ nonzeros, so doubling the sequence quadruples the arithmetic unless density falls as $(\log N)/N$.

Editorial extensions

If this is right

  • Within the native positional window, a user can match or beat the dense baseline using 1% density and enough seeds, so dense attention is not a strict upper bound for factual retrieval.
  • At 2x the native window, retrieval gains over the zero-context floor are attainable on CPU hardware only when the positional encoding is scaled in a way that preserves geometry, such as YaRN.
  • For small-window models, the sparse kernel inherits the host model's positional ceiling; beyond about 2x the trained window, adding seeds and density does not restore retrieval.
  • Because the injection touches no weights, the architecture attaches to any model that exposes its attention layers, making the same masks portable across architectures without retraining.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The coverage formula implies a tunable optimum: peak accuracy is reported near ensemble union coverage $U \approx 50\%$, which gives a density–seed relation $N \approx \ln(0.5)/\ln(1-d)$; this predicts where to set the seed budget and is testable without new theory.
  • A genuine $O(N \log N)$ implementation would require density to shrink with sequence length, roughly as $d \sim (\log N)/N$; at the fixed 1% density benchmarked here, the number of nonzero attention pairs still grows as $O(N^2)$, so the million-token extrapolation from the companion theory would need a different sampling schedule.
  • The same masks could be applied to GPU-based long-context prefill to cut memory by roughly the density factor, but the paper does not test GPU deployments or larger parameter counts, so that extension remains unverified.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper introduces RIS-Kernel, a model-agnostic inference-time sparse-attention architecture for LLMs. It replaces the dense attention mask with stochastic or structural sparse masks generated from a fixed density budget and an ensemble of seed masks, claiming to reduce self-attention complexity from O(N^2) to O(N log N) without modifying weights. The method is evaluated on Qwen2-1.5B-Instruct at 32,768 and 65,536 tokens on CPU servers, reporting accuracy gains over a zero-context floor and, at 32k, an apparent improvement over the dense baseline. The paper also includes hardware profiling, a comparison of linear versus YaRN RoPE scaling, and a qualitative cross-document synthesis experiment.

Significance. If the central claims held, the paper would be significant: a retraining-free sparse attention kernel that enables 65k-token inference on commodity CPUs would lower the hardware barrier for long-context LLM inference, and the reported regularizing effect of low-density ensembles would be an interesting empirical finding. The paper has some genuine strengths: it provides open code and datasets, gives detailed hardware and memory profiling, and its ensemble coverage formula (Eq. 1) is a correct probability identity. The structural-versus-stochastic comparison, including the sub-1% density sweet-spot analysis, is a useful experimental exploration. However, the main complexity claim is internally contradicted by the method's own construction, and the headline empirical results rest on very small accuracy differences that are not supported by appropriate statistical testing. The significance of the reported results is therefore substantially lower than claimed.

major comments (5)
  1. [Abstract, §4.2, §2.2.3] The central complexity claim that RIS reduces self-attention to O(N log N) is unsupported and internally contradicted. Section 4.2 defines attention density d as a fixed fraction of the sequence and allocates d*N keys per query, which yields O(d N^2) nonzero (query, key) pairs per mask. With the fixed densities used in the experiments (1%, 2%, 5%), doubling N quadruples the number of attended pairs, not doubles it. Equation 1, U = 1 - (1 - d)^N_seeds, strengthens this point: for d = 0.01 and N_seeds = 70, the ensemble union covers about 50.5% of the N^2 pairs, so the ensemble mask has roughly 0.5 N^2 nonzeros. Section 4.1 also explicitly states that peak mask memory is bounded by O(N^2) boolean matrices. To achieve O(N log N) arithmetic, the density would need to shrink as O(log N/N), but all experiments use fixed densities independent of N. The abstract's 'O(N log N)' claim and the scalability conclusions built on it are therefore not supported by the presented method.
  2. [§3.4] The statement that 'Doubling the context length doubles the computational cost' is false under the paper's own fixed-density definition. If each query attends to d*N keys, then for fixed d the total work scales as d N^2, so doubling N quadruples the arithmetic cost. This sentence directly contradicts the paper's stated O(N^2) memory footprint in Section 4.1 and the O(d N^2) nonzero count implied by Section 4.2. The claim must be corrected or the complexity analysis must be revised; as written, it is a load-bearing error in the scalability argument.
  3. [§2.2.1, Table 5] The headline result that RIS-Stochastic 'outperforms' native dense attention is based on 24/32 versus 23/32 correct answers, i.e., a single-question difference on a 32-question set. The paper reports McNemar p = 0.227 for the 15.62-percentage-point improvement over the zero-context floor, but reports no paired significance test for the 75.00% versus 71.88% dense comparison. Such a difference is well within noise. The narrative that low density acts as a regularizer that 'filters sequence-level noise' is also selected post hoc from a grid over densities and seed counts; no correction for multiple comparisons is applied, and the effect is not replicated on an independent benchmark or model at the same scale. This weakens the central empirical claim.
  4. [§2.2.2, §4.4.1] The 65,536-token claim of a 14.06-percentage-point retrieval gain (65.62% vs 51.56%, p = 0.078) is reported for the single best configuration selected from a large grid of densities (1%, 2%, 5%, plus sub-1% densities), seed counts (1 to 200), and two modes. The marginal p-value is already above conventional significance, and no multiple-testing correction is applied across the many configurations examined. The claim that this 'confirms' the retrieval signal of RIS is therefore overstated; at best it is a weak, uncorrected trend.
  5. [§4.2] The statement that both modes operate under 'an identical O(N^2) memory and O(N log N) arithmetic footprint' is internally inconsistent. If the mask has O(N^2) nonzeros, the arithmetic cost of the attention pass is at least proportional to the number of nonzeros; a memory footprint of O(N^2) boolean matrices cannot be reconciled with an O(N log N) arithmetic claim unless the mask has a much smaller effective nonzero count, which is not the case under the fixed-density construction. The paper needs a consistent complexity model that accounts for both the mask construction and the attention arithmetic.
minor comments (5)
  1. [Throughout] There are numerous formatting issues: missing spaces such as 'scales asO(N 2)' in the abstract, 'uni00A0' artifacts in Figure 6, and inconsistent notation for density (d, q, ρ, ρ_sub) across sections and equations. These should be cleaned up.
  2. [§3.4, §4.4.2] The coverage formula is written in different forms: Eq. 1 uses d and N_seeds, while §3.4 and §4.4.2 use ρ_sub and k·N. The relationship between these parameters, including the role of term multiplicity k, should be defined explicitly.
  3. [§2.7, Table 3] The sub-1% sweet-spot search reports mean and best accuracies without any measure of variance or significance; statements about 'the optimal sampling window lies between 100 and 150 seeds' are ad hoc descriptions of the table rather than statistically supported conclusions.
  4. [§2.8] The qualitative synthesis experiment uses Qwen2.5-1.5B-Instruct while the main benchmarks use Qwen2-1.5B-Instruct; the paper does not explain whether the architecture behaves identically across these model versions, which complicates the interpretation of the qualitative results.
  5. [§4.4.1] The two-stage validation design is described as providing statistical confirmation, but the 32-question set is too small for reliable paired tests, and the 64-question set is still modest; the paper overstates the inferential power of these sample sizes.

Circularity Check

2 steps flagged · score 4.0 of 10

The O(N log N) complexity premise is imported from the author's own prior work [3], and the 'low-density regularizer' narrative is retrofitted to the grid-search maximum; the empirical benchmark itself is not circular.

  1. self citation load bearing [Section 1 (Introduction), third paragraph; see also Section 4.2]
    "In the foundational work [3], we established the theoretical foundation of Reduced Interaction Sampling (RIS), proving mathematically that the O(N^2) attention bottleneck can be bypassed via stochastic sparsification [4, 5] while preserving factual retrieval. The present paper validates this mechanism empirically under LLM inference."

    The headline claim that 'RIS reduces self-attention complexity to O(N log N)' is not re-derived in this manuscript. Section 4.2 simply asserts an 'identical O(N^2) memory and O(N log N) arithmetic footprint' after defining density as a fixed budget of d*N keys per query. The load-bearing scaling bound for the CPU-feasibility conclusion is thus inherited from the same author's prior proof in [3], rather than from an independent derivation or from a measured scaling curve in this paper. The empirical accuracy comparisons are independent content, but the central complexity claim reduces to a same-author citation.

  2. other [Abstract; elaborated in Section 2.2.3 using Eq. 1]
    "This demonstrates that sparse attention acts as a regularizer: low density (1%) over multiple seeds filters out sequence-level noise, whereas higher density (5%) reintroduces distractor noise."

    The 1%-density/70-seed configuration used to 'demonstrate' the regularizer is the maximum of the same density/seed grid reported in Section 2.2.1. Equation 1 is then evaluated at this exact d=0.01, N=70 configuration to explain the peak, and at d=0.05 to explain the ceiling at the dense baseline. Because the explanatory coverage value is a deterministic function of the hyperparameters selected by the same grid search, the regularizer narrative is a post-hoc description of the grid maximum rather than an independent prediction: the data are used both to locate and to rationalize the optimum.

full rationale

The empirical benchmark is largely self-contained: RIS accuracy is compared to a native dense baseline and a zero-context floor, so those accuracy comparisons do not reduce to the method's inputs. Equation 1 is a parameter-free probability identity with no fitted constants. However, the paper's central O(N log N) complexity claim is presented as a consequence of the author's own prior work [3] and is not re-proven here; moreover, the 'low density regularizes' conclusion is retrofitted to the winning grid cell. These are moderate, non-fatal circularities. Separately, the fixed-density mask construction actually implies O(d N^2) nonzero query-key pairs per mask (with union coverage growing under Eq. 1), which contradicts the asserted O(N log N) scaling; I treat that as a correctness/scaling defect rather than a circularity, because it is an internal contradiction rather than a reduction of a prediction to its inputs.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

No new physical entities or forces are introduced. The free parameters are all standard algorithmic hyperparameters, but the peak configuration is selected from the same data that is used to justify the regularization narrative, which raises the circularity burden. The axioms are mostly standard assumptions about the benchmark and the model, except the low-density regularizer hypothesis, which is post hoc.

free parameters (4)
  • Attention density d = 1%, 2%, 5% at 32k/64k; 0.1%-0.9% in sub-1% sweet-spot
    Chosen by grid search; the 1% value that produces the headline result is selected post hoc from the same data.
  • Ensemble seed count N = 1 to 100; headline uses 70-80 seeds
    Swept over 1-100 seeds; the peak at 70-80 is selected post hoc, and the ensemble behavior is explained using the fitted coverage formula.
  • Local sliding window size L = 1024 tokens
    Fixed by hand as a syntactic coherence window; not justified by data or theory.
  • Block size cap B_max = 66 to 655 tokens in sweet-spot search
    Introduced in Section 2.7 to define structural block densities; the sweet-spot density is selected from the same grid.
assumptions (5)
  • standard math Union coverage of independent random subsets follows U = 1 - (1-d)^N
    Used in Section 2.2.3 and Section 3.4 to explain ensemble scaling; it is a basic probability identity, not fitted.
  • domain assumption The zero-context floor reflects parametric memory rather than data contamination
    Section 3.2 asserts the sources are post-training-cutoff and that the elevated floor is prior knowledge; this is plausible but unverified.
  • domain assumption Discriminative log-probability scoring on five options is a valid measure of factual retrieval
    Section 4.4.1 states answers are scored via logit comparison; this substitutes for open-ended generation and is a reasonable but unvalidated proxy.
  • domain assumption Qwen2-1.5B is representative of sub-2B models for this retrieval protocol
    The paper generalizes from one Qwen2 model and one TinyLlama model; the failure of TinyLlama at 4x extrapolation weakens this generalization.
  • ad hoc to paper Low density acts as a regularizer that filters sequence-level noise
    This hypothesis (Abstract, Section 3.1) is introduced to explain the observed peak at 1% density after the grid search revealed it; it is not derived independently.

how reviews work

0 comments
Cite this review

Pith. "Pith review of RIS-Kernel: A Model-Agnostic Architecture for Long-Context LLM Inference via Sparse Attention." pith.science (2026). https://pith.science/paper/6QCA27QS

@misc{pith2026260721927,
  author       = {Pith},
  title        = {Pith review of: RIS-Kernel: A Model-Agnostic Architecture for Long-Context LLM Inference via Sparse Attention},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/6QCA27QS}},
  note         = {Machine review of arXiv:2607.21927}
}
read the original abstract

Full self-attention in large language models scales as O(N^2), which limits long-context document analysis to 65,536 tokens and requires costly GPU clusters. The Reduced Interaction Sampling (RIS) inference engine addresses this constraint as a model-agnostic architecture. Without modifying weights, RIS reduces self-attention complexity to O(N log N) using sparse stochastic geometry that fits within commodity memory limits. We validate RIS on Qwen2-1.5B-Instruct across two regimes. In controlled evaluations at 32,768 tokens (where native dense attention serves as the upper bound), RIS-Stochastic at 1% density and 70 ensemble seeds achieves 75.00% accuracy, outperforming the native dense baseline (71.88%), while RIS-Stochastic at 5% density and 10 seeds matches it (71.88%). This demonstrates that sparse attention acts as a regularizer: low density (1%) over multiple seeds filters out sequence-level noise, whereas higher density (5%) reintroduces distractor noise. Under the tightest budget, RIS-Structural reaches 68.75% accuracy at 1% density with just 10 seeds, recovering 75% of the contextual gap relative to the zero-context floor (59.38%). At 65,536 tokens, where dense attention triggers out-of-memory faults, RIS yields retrieval gains of up to 14.06 percentage points over the zero-context floor (51.56%), which is confirmed as marginally significant under McNemar's paired test (p = 0.078 < 0.10). All evaluations run on commodity, unaccelerated CPU servers (16-128 GB of RAM), demonstrating that long-context LLM inference is feasible on standard academic hardware without GPU acceleration.

Figures

Figures reproduced from arXiv: 2607.21927 by the authors.

Figure 1
Figure 1. Performance surface of RIS on the balanced 32k window for (a) RIS-Stochastic and (b) RIS￾Structural modes. Cells highlighted with blue borders match or exceed the native dense baseline of 71.88% (seeds/density configurations that fully recover contextual information). Dashed borders show configurations that outperform the zero-context baseline of 59.38%. Cells marked with an asterisk (*) represent configurations exe… view at source ↗
Figure 2
Figure 2. Ensemble scaling and context recovery rate on the balanced 32k window. The left y-axis shows absolute accuracy; the right y-axis indicates the percentage of the full-attention gap recovered relative to the w = 0 baseline. Under Stochastic 5% (10 seeds), the RIS architecture matches the native dense target (71.88%) exactly. Under Structural 1% (10 seeds), the model recovers 75% of the contextual gap at 100× computati… view at source ↗
Figure 3
Figure 3. Peak accuracy comparison by sampling mode across attention densities (1%, 2%, 5%) on the balanced 32k window. Horizontal reference lines indicate the zero-context floor (59.38%) and the full￾attention native dense target (71.88%). Both modes converge to the dense target as density increases; the Structural mode reaches that convergence at substantially lower seed counts when density is at 1%. 2.2.2 Experiment B: Sca… view at source ↗
Figures from the paper (6 more)
Figure 4
Figure 4. Figure 4: Performance surface comparison for (a) RIS-Stochastic and (b) RIS-Structural modes at 65,536 tokens (Qwen2-1.5B, YaRN). Accuracy is shown as a function of ensemble seed count and sampling density. Zero-context baseline: 51.56%. 2.2.3 Ensemble Coverage Analysis The ense…
Figure 5
Figure 5. Figure 5: Ensemble Scaling Law comparison: Accuracy as a function of ensemble seeds (N) at 64k tokens, YaRN scaling, 5% density. Both Stochastic and Structural modes plateau at 40–60 seeds before noise accumulation triggers saturation. 2.3 RoPE Scaling: Linear versus YaRN How co…
Figure 6
Figure 6. Figure 6: Accuracy as a function of context window size under linear and YaRN scaling, Qwen2-1.5B, at the best-performing density and seed count for each window. Within the native positional limit (≤32k), both methods are equivalent (scaling factor = 1.0). At 65,536 tokens, line…
Figure 7
Figure 7. Figure 7: Performance scaling of RIS modes at 65,536 tokens. (a) Stochastic mode scales monotonically with density. (b) Structural mode peaks at extreme sparsity (1–2%) and plateaus/degrades at higher densities. the block-clique constraint makes such coverage certain. The conseq…
Figure 8
Figure 8. Figure 8: Streaming mask generation. Each seed is injected into the master structure and discarded before the next begins, keeping peak memory O(N 2 ) through boolean allocation. 14 [PITH_FULL_IMAGE:figures/full_fig_p014_8.png]
Figure 9
Figure 9. Figure 9: PFUS index fusion. The stochastic anchor is frozen after the first token; the local window slides. A unique() merge feeds a single softmax over the combined set. First, at the 32,768-token window, we leverage a balanced 32-question set (N = 32) with options (A–E) unifo…

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

32 extracted references · 14 canonical work pages

  1. [1]

    Neural Inf

    Dao, T.et al.FlashAttention: Fast and memory-efficient exact attention with IO-awareness.Adv. Neural Inf. Process. Syst.35, 16344–16359 (2022)

    work page 2022

  2. [2]

    Neural Inf

    Vaswani, A.et al.Attention Is All You Need.Adv. Neural Inf. Process. Syst.30, 5998–6008 (2017)

    work page 2017

  3. [3]

    Santos, A. R. Towards million-token context windows: a topology-preserving framework for adaptive transformer sparsification.Sci. Rep.16, 59160 (2026). https://doi.org/10.1038/ s41598-026-59160-z

    work page 2026

  4. [4]

    Child, R.et al.Generating Long Sequences with Sparse Transformers.arXiv preprint arXiv:1904.10509(2019)

    arXiv 2019

  5. [6]

    Li, Y.et al.Adapters for Efficient Fine-tuning.arXiv preprint arXiv:2303.12345(2023)

    arXiv 2023

  6. [7]

    Qwen2 Technical Report.arXiv preprint arXiv:2407.10671(2024)

    Qwen Team. Qwen2 Technical Report.arXiv preprint arXiv:2407.10671(2024)

    arXiv 2024

  7. [8]

    Zhang, P.et al.TinyLlama: An open-source small language model.arXiv preprint arXiv:2401.02385(2024)

    arXiv 2024

  8. [9]

    Lewis, P.et al.Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks.Advances in Neural Information Processing Systems33, 9459–9474 (2020)

    work page 2020

Show all 32 references
  1. [10]

    Ram, O.et al.In-Context Retrieval-Augmented Language Models.arXiv preprint arXiv:2302.00083(2023)

    2023 arXiv

  2. [11]

    Chen, Z.et al.Efficient LLM Inference on CPUs.arXiv preprint arXiv:2311.00502(2023)

    2023 arXiv

  3. [12]

    Wulf, W. A. & McKee, S. A. Hitting the memory wall: implications of the dynamics of memory and processor evolution.ACM SIGARCH Computer Architecture News23(1), 20–24 (1995)

    1995

  4. [13]

    & Patterson, D

    Williams, S., Waterman, A. & Patterson, D. Roofline: an insightful visual model for floating-point performance and multicore architectures.Communications of the ACM52(4), 65–76 (2009)

    2009

  5. [14]

    Ivanov, A.et al.Data movement is all you need: A case study on optimizing transformers.arXiv preprint arXiv:2102.04803(2021)

    2021 arXiv

  6. [15]

    Intel.Intel®Extension for PyTorch* Optimization Guide. (2023)

    2023

  7. [16]

    Zaheer, M.et al.Big bird: Transformers for longer sequences.Advances in Neural Information Processing Systems33, 17283–17297 (2020)

    2020

  8. [17]

    E., & Cohan, A

    Beltagy, I., Peters, M. E., & Cohan, A. Longformer: The long-document transformer.arXiv preprint arXiv:2004.05150(2020)

    2020 arXiv

  9. [18]

    Press, O.et al.Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation.arXiv preprint arXiv:2108.12409(2021)

    2021 arXiv

  10. [19]

    J.et al.LoRA: Low-Rank Adaptation of Large Language Models.arXiv preprint arXiv:2106.09685(2021)

    Hu, E. J.et al.LoRA: Low-Rank Adaptation of Large Language Models.arXiv preprint arXiv:2106.09685(2021)

    2021 arXiv

  11. [20]

    B., Loukas, A

    Cordonnier, J. B., Loukas, A. & Jaggi, M. On the relationship between self-attention and convolutional layers.arXiv preprint arXiv:1911.03584(2019)

    2019 arXiv

  12. [21]

    Shazeer, N.et al.Outrageously large neural networks: The sparsely-gated mixture-of-experts layer.arXiv preprint arXiv:1701.06538(2017)

    2017 arXiv

  13. [22]

    & Shazeer, N

    Fedus, W., Zoph, B. & Shazeer, N. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity.The Journal of Machine Learning Research23(1), 5232–5270 (2022)

    2022

  14. [23]

    Wang, S.et al.Linformer: Self-attention with linear complexity.arXiv preprint arXiv:2006.04768 (2020)

    2020 arXiv

  15. [24]

    Tay, Y.et al.Sparse Sinkhorn Attention.ICML(2020)

    2020

  16. [25]

    Kazemnejad, S.et al.The Impact of Positional Encoding on Length Generalization in Transformers.arXiv preprint arXiv:2305.19466(2023). 18

    2023 arXiv

  17. [26]

    Su, J.et al.Roformer: Enhanced transformer with rotary position embedding.arXiv preprint arXiv:2104.09864(2021)

    2021 arXiv

  18. [27]

    Peng, B.et al.YaRN: Efficient context window extension of large language models.arXiv preprint arXiv:2309.00071(2023)

    2023 arXiv

  19. [28]

    Rajpurkar, P.et al.SQuAD: 100,000+ Questions for Machine Comprehension of Text.arXiv preprint arXiv:1606.05250(2016)

    2016 arXiv

  20. [29]

    Transactions of the Association for Computational Linguistics7, 453–466 (2019)

    Kwiatkowski, T.et al.Natural Questions: a Benchmark for Question Answering Research. Transactions of the Association for Computational Linguistics7, 453–466 (2019)

    2019

  21. [30]

    Thakur, N.et al.BEIR: A Heterogeneous Benchmark for Information Retrieval.arXiv preprint arXiv:2104.08663(2021)

    2021 arXiv

  22. [31]

    Gao, Y.et al.Retrieval-Augmented Generation for Large Language Models: A Survey.arXiv preprint arXiv:2312.10997(2023)

    2023 arXiv

  23. [32]

    & Upfal, E.Probability and Computing: Randomized Algorithms and Probabilistic Analysis

    Mitzenmacher, M. & Upfal, E.Probability and Computing: Randomized Algorithms and Probabilistic Analysis. (Cambridge University Press, 2005)

    2005

  24. [33]

    & Ullman, J

    Leskovec, J., Rajaraman, A. & Ullman, J. D.Mining of Massive Datasets. (Cambridge University Press, 2014). 19 Supplementary Information T able 5:Retrieval performance and context recovery rate on the balanced 32k window (Qwen2-1.5B, YaRN). The native dense baseline (w = 32,768...

    2014

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.