Pith. sign in

REVIEW 2 major objections 5 minor 54 references

Jet-Long extends LLM context to 128K without training by dynamically aliasing remote positions onto the pretrained RoPE grid while keeping a local faithful window free at inference.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-13 06:46 UTC pith:RWCECVYN

load-bearing objection Solid systems paper: analytic dynamic G + cache-preserving correction + fused inclusion-exclusion kernel deliver real RULER/HELMET/PG-19 gains on Qwen3 with near-zero overhead. the 2 major comments →

arxiv 2607.07740 v2 pith:RWCECVYN submitted 2026-07-08 cs.LG cs.AI

Jet-Long: Efficient Long-Context Extension with Dynamic Bifocal RoPE

Haozhan Tang , Zerui Wang , Yuxian Gu , Song Han , Han Cai This is my paper
classification cs.LG cs.AI
keywords long-context extensionRoPEzero-shotbifocal attentiondynamic position mappingFlashAttentionLLM inference
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Open-weight language models are usually trained at a moderate context window, yet real deployments in retrieval, coding agents, and multi-turn tool use routinely exceed that window. Existing zero-shot fixes either lock in one rescaling factor for every length or rely on fitted schedules, so they either damage short-context behavior or fail at long ranges. Jet-Long claims that a bifocal construction solves the tradeoff: a local window that uses ordinary RoPE exactly as pretrained, paired with a remote window whose integer group size is set analytically to the smallest integer that keeps every remote rotation inside the training range. Because the group size is recomputed from the current length alone, the method recovers the base model exactly inside the native window and continues cleanly past it. An inclusion-exclusion merge of three attention passes plus an on-the-fly correction rotation leave the KV cache untouched and, when fused, keep generation overhead at most four percent while accelerating long prefill. On three Qwen3 sizes the method leads the strongest zero-shot baselines on RULER, HELMET-RAG, and PG-19 perplexity, and the same construction transfers without retraining to hybrid linear-attention models.

Core claim

A parameter-free dynamic bifocal RoPE map—local window of width w0 with classic RoPE, remote positions aliased by floor division with G = max(1, ceil(L / w_pretrained))—extends open-weight LLMs past their pretraining window without any fine-tuning, recovers the base model exactly for L ≤ w_pretrained, and, when realized by inclusion-exclusion attention and on-the-fly correction rotations, incurs negligible generation cost while outperforming fixed-factor and fitted zero-shot baselines on standard long-context benchmarks.

What carries the argument

Dynamic bifocal mapping: the remote factor G is the minimal integer that compresses the current length into the pretrained window, with remote positions f(x) = floor(x/G); an inclusion-exclusion of three FlashAttention calls plus additive RoPE correction rotations realize the map without rewriting the KV cache.

Load-bearing premise

That the right remote map is always the smallest integer that lands every remote relative rotation exactly on a pretrained discrete angle, so maximizing resolution under that hard constraint is optimal at every length.

What would settle it

Measure whether continuous frequency interpolation or a larger-than-minimal group size overtakes the analytic G schedule on the same RULER and HELMET-RAG tasks once context is pushed well past 128K, or whether the inclusion-exclusion merge loses numerical fidelity on a different model family.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • A single released open-weight checkpoint can be deployed at arbitrary context lengths without short-context regression or per-length fine-tuning.
  • Long-context prefill can run faster than standard FlashAttention-2 on the same hardware once the three-pass merge is fused.
  • Hybrid architectures that interleave softmax and linear attention inherit the same length-extension gains without architecture-specific retraining.
  • Practitioners can fix the sole hyperparameter (local window size) once; the method remains within a couple of points of the best choice across a wide sweep.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same analytic schedule may be portable to other rotary or relative-position schemes that also possess a discrete training grid of relative angles.
  • At extreme lengths the discrete-aliasing premise may need a hybrid map that gradually hands off to continuous interpolation, as the paper’s own ablation already hints at 128K.
  • Because the KV cache stays in base coordinates, the method can be dropped into existing serving stacks that already manage streaming generation across length boundaries.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. The paper introduces Jet-Long, a tuning-free zero-shot context-extension method for RoPE LLMs. It pairs a local RoPE-faithful window of width w0 with a remote window whose discrete group size G = max(1, ceil(L / w_pretrained)) is recomputed from the current sequence length, aliasing remote positions onto the pretrained rotation grid via f(x) = floor(x/G). Short-context identity follows by construction when L ≤ w_pretrained. An inclusion–exclusion merge of three FlashAttention passes (Eq. 7) plus on-the-fly correction rotations (Eqs. 5–6) that leave the base-position KV cache untouched make the bifocal construction essentially free; a fused CuTe kernel reaches up to 1.39× FA2 prefill on H100 and ≤4% generation overhead. On Qwen3-1.7B/4B/8B up to 128K, Jet-Long leads RULER by +4.79/+2.18/+2.03 pp over the strongest baseline, is best or tied on HELMET-RAG, and attains the lowest PG-19 perplexity; it also transfers to hybrid Jet-Nemotron without retraining and is resilient to w0.

Significance. If the reported results hold, Jet-Long is a practically useful systems contribution for open-weight deployment: an analytically scheduled, essentially free zero-shot extension that recovers the base model exactly inside the training window while improving long-context metrics. Explicit strengths include the parameter-free analytic G schedule (no per-model fitting), exact short-context identity and cache invariance under standard RoPE composition, a fused CuTe kernel with measured H100 throughput against FA2/FA4, multi-size multi-benchmark evaluation (RULER 13 tasks × 7 lengths, HELMET-RAG, PG-19), a w0 sweep showing hyperparameter resilience, an aliasing-vs-frequency ablation, and transfer to a hybrid architecture. The public code link further supports reproducibility. These are concrete engineering and empirical advances over fixed-factor methods (YaRN, Self-Extend, DCA) and fitted length-aware maps.

major comments (2)
  1. §4.1–4.2 and Tables 1–3: The primary empirical claims rest entirely on the Qwen3 base family (plus Jet-Nemotron hybrids from the same research line). While results are consistent across three sizes and the hybrid transfer is a positive signal, the central claim of a general zero-shot method would be more robust with at least one additional independent RoPE architecture family (e.g., Llama- or Mistral-style). This does not invalidate the Qwen3 results but bounds the scope of the generalization statement in the abstract and conclusion.
  2. §3.1 Eqs. (2)–(3) and Table 6: The paper motivates the minimal integer G as the resolution-maximizing choice that keeps every remote angle exactly on the pretrained discrete grid. Table 6 shows that continuous frequency interpolation already overtakes aliasing on isolated tasks (MK-NIAH-2, QA-2) at 128K even though aliasing still wins the overall average at both 64K and 128K. The manuscript should more carefully bound the optimality claim: the analytic schedule is effective and preferred on the tested regime (≤128K on these models), not necessarily asymptotically optimal. A short clarifying sentence in §3.1 and the conclusion would prevent over-reading.
minor comments (5)
  1. Figure 1 caption and abstract: the ≤4% generation overhead and 1.39× prefill claims are supported by Tables 7–8; consider adding an explicit pointer to those tables in the figure caption for readers who land on the teaser first.
  2. §3.3 Eq. (7): the numerical-stability argument (subtract in linear space after LogSumExp shift, FP32 accumulation) is clear, but a one-sentence note on observed max |W_B − W_C| or a brief check that the denominator stays positive in practice would reassure readers familiar with the “numerically catastrophic” critique cited from [54].
  3. Appendix C / Table 9: baseline hyperparameters are fixed across lengths; a short remark that length-adaptive re-tuning of Self-Extend/DCA group sizes was not explored (and why) would forestall a common reviewer question.
  4. Typographical: abstract and §1 use both “Jet-Long” and “Jet-Long,” consistently; ensure hyphenation is uniform. Also “inclusion–exclusion” vs “inclusion-exclusion” appears in both en-dash and hyphen forms.
  5. §4.3 Table 4: the green absolute-gap tags are helpful; adding the same style of relative or absolute gap annotation to the main RULER length curves (Figure 3) would improve scannability.

Circularity Check

0 steps flagged

No significant circularity: analytic G schedule, exact short-context identity, and external benchmarks are independent of fitted targets.

full rationale

Jet-Long is an empirical systems paper whose central claims are (i) a parameter-free analytic group size G = max(1, ceil(L / w_pretrained)) that keeps remote RoPE angles on the pretrained discrete grid, (ii) exact recovery of the base model for L ≤ w_pretrained by construction of that schedule, (iii) an inclusion-exclusion merge plus on-the-fly correction rotation that leaves the KV cache untouched, and (iv) measured gains on external suites (RULER, HELMET-RAG, PG-19) plus near-FA2 throughput of a fused CuTe kernel. Equation (2) is determined solely by the current length L and the known pretrained window; it is never fitted to RULER/HELMET/PG-19 scores. Short-context identity follows immediately when G collapses to 1, so that f(x) = x and Eq. (1) reduces to ordinary RoPE; this is a mathematical identity, not a circular fit. The three-pass merge (Eq. 7) and the additive RoPE correction (Eqs. 5-6) are exact under standard RoPE composition and do not encode evaluation metrics. Ablations (Tables 5-6) and transfer to Jet-Nemotron are reported as empirical checks, not as uniqueness theorems that force the method. Self-citations to Jet-Nemotron are transfer experiments, not load-bearing premises for the Qwen3 results. No step reduces a claimed prediction to a fitted input or to an unverified self-citation chain. Score 0 is therefore the correct finding.

Axiom & Free-Parameter Ledger

1 free parameters · 4 axioms · 3 invented entities

The central claim rests on standard RoPE algebra, the discrete-grid in-distribution premise, FlashAttention LSE arithmetic, and one hand-chosen local window size. No new physical entities; the 'invented' pieces are algorithmic constructions whose value is measured by external benchmarks rather than postulated forces or particles.

free parameters (1)
  • w0 (local protected window size) = 2048
    Only free hyperparameter of Jet-Long; set to 2048 for main results. Ablation (Table 5) shows resilience in {512…4096}, but the value is still chosen by hand rather than derived.
axioms (4)
  • domain assumption RoPE rotations compose additively: R_a R_b = R_{a+b}, so a correction rotation on cached keys realizes the remapped position without rewriting the KV cache.
    Invoked in §3.2 Eqs. 5–6; standard for pure RoPE but would fail under non-rotational position scalings.
  • domain assumption The set of relative RoPE angles seen in pretraining is a discrete grid; aliasing remote positions onto that grid keeps every remote angle exactly in-distribution.
    Stated in §3.1 as the reason for discrete grouping over continuous frequency interpolation; ablated in §4.5 / Table 6.
  • domain assumption FlashAttention’s per-query LogSumExp statistics allow an exact inclusion–exclusion merge of three attention outputs in linear space without materializing a mask or catastrophic cancellation when stabilized by max-shift and FP32 accumulation.
    §3.3 Eq. 7; authors contrast with prior work that rejected subtractive forms as numerically catastrophic.
  • ad hoc to paper G = max(1, ⌈L / w_pretrained⌉) is the minimal integer compression that keeps the compressed sequence inside the pretrained window and is therefore the resolution-maximizing choice at every length.
    Eq. 2 in §3.1; analytic and parameter-free, but optimality is an assumption, not a theorem, and Table 6 shows continuous maps can win on some tasks at 128K.
invented entities (3)
  • Dynamic bifocal position map f(x)=⌊x/G⌋ with G recomputed from current L no independent evidence
    purpose: Route local keys with classic RoPE and remote keys with length-adaptive grouping so short inputs equal the base model and long inputs stay in-distribution.
    Core algorithmic object of the paper; evaluated only through downstream accuracy/ppl, no independent physical prediction.
  • On-the-fly RoPE correction rotation (Δq, Δk) applied in registers no independent evidence
    purpose: Realize the remote view at decode without rewriting or discarding the base-position KV cache when G changes mid-generation.
    §3.2; engineering construct whose correctness follows from RoPE additivity rather than new evidence.
  • Inclusion–exclusion three-pass attention merge fused in a single CuTe kernel no independent evidence
    purpose: Compute exact distance-based local/remote routing at prefill with near-FA2 (or better) throughput.
    §3.3 and §4.6; performance claims are empirical on H100, not independently verified outside the paper.

pith-pipeline@v1.1.0-grok45 · 24133 in / 3699 out tokens · 42207 ms · 2026-07-13T06:46:26.770257+00:00 · methodology

0 comments
read the original abstract

Modern LLMs are increasingly deployed in long-context applications such as retrieval-augmented generation, repository-level coding, and agentic workflows whose accumulated reasoning and tool traces routinely push the input an order of magnitude past the pretraining window, making zero-shot context extension the dominant deployment path for open-weight checkpoints. The dominant zero-shot methods (YaRN, Self-Extend, DCA) fix a single rescaling factor up front, so an aggressive factor sacrifices short-context fidelity while a conservative one breaks down at long contexts; recent length-aware variants adapt the mapping, but with a fitted or distance-dependent schedule. We propose Jet-Long, a tuning-free zero-shot method that pairs a local RoPE-faithful window with a long-range window whose rescaling factor adapts dynamically to the current sequence length via a parameter-free analytic schedule, recovering the base model exactly at short inputs while extrapolating cleanly at long ones. An inclusion-exclusion attention merge and an on-the-fly RoPE correction rotation make the bifocal construction essentially free at inference; fused into a single CuTe kernel, long-context prefill reaches up to $1.39\times$ FA2 throughput on H100 (approaching the Hopper-only FA4), and single-batch generation incurs $\le 4\%$ overhead at every length. On Qwen3-1.7B/4B/8B up to 128K context, Jet-Long leads RULER by $+4.79$/$+2.18$/$+2.03$ pp over the strongest baseline at 1.7B/4B/8B, achieves the best overall accuracy on HELMET-RAG (a benchmark identified by HELMET as the most efficient predictor of downstream long-context performance) and attains the lowest PG-19 perplexity. Jet-Long also generalizes to hybrid attention architectures such as Jet-Nemotron for further long-context improvement without retraining, and remains hyperparameter-resilient for ease of deployment.

Review history (2 revisions) →

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

54 extracted references · 32 linked inside Pith

  1. [1]

    YaRN: Efficient context window extension of large language models.arXiv preprint arXiv:2309.00071, 2023

    Bowen Peng, Jeffrey Quesnelle, Honglu Fan, and Enrico Shippole. YaRN: Efficient context window extension of large language models.arXiv preprint arXiv:2309.00071, 2023

    Pith/arXiv arXiv 2023

  2. [2]

    LLM maybe LongLM: Self-extend LLM context window without tuning.arXiv preprint arXiv:2401.01325, 2024

    Hongye Jin, Xiaotian Han, Jingfeng Yang, Zhimeng Jiang, Zirui Liu, Chia-Yuan Chang, Huiyuan Chen, and Xia Hu. LLM maybe LongLM: Self-extend LLM context window without tuning.arXiv preprint arXiv:2401.01325, 2024

    Pith/arXiv arXiv 2024

  3. [3]

    Training-free long-context scaling of large language models.arXiv preprint arXiv:2402.17463, 2024

    Chenxin An, Fei Huang, Jun Zhang, Shansan Gong, Xipeng Qiu, Chang Zhou, and Lingpeng Kong. Training-free long-context scaling of large language models.arXiv preprint arXiv:2402.17463, 2024

    Pith/arXiv arXiv 2024

  4. [4]

    Extending LLM context window with adaptive grouped positional encoding: A training-free method

    Xinhao Xu, Jiaxin Li, Hui Chen, Zijia Lin, Jungong Han, and Guiguang Ding. Extending LLM context window with adaptive grouped positional encoding: A training-free method. InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 573–587, 2025

    2025

  5. [5]

    LaMPE: Length-aware multi-grained positional encoding for adaptive long-context scaling without training

    Sikui Zhang, Guangze Gao, Ziyun Gan, Chunfeng Yuan, Zefeng Lin, Houwen Peng, Bing Li, and Weiming Hu. LaMPE: Length-aware multi-grained positional encoding for adaptive long-context scaling without training. In Findings of the Association for Computational Linguistics: ACL 2026, pages 32134–32149, 2026

    2026

  6. [6]

    Qwen3 technical report.arXiv preprint arXiv:2505.09388, 2025

    An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report.arXiv preprint arXiv:2505.09388, 2025

    Pith/arXiv arXiv 2025

  7. [7]

    HELMET: How to evaluate long-context language models effectively and thoroughly.arXiv preprint arXiv:2410.02694, 2024

    Howard Yen, Tianyu Gao, Minmin Hou, Ke Ding, Daniel Fleischer, Peter Izsak, Moshe Wasserblat, and Danqi Chen. HELMET: How to evaluate long-context language models effectively and thoroughly.arXiv preprint arXiv:2410.02694, 2024

    Pith/arXiv arXiv 2024

  8. [8]

    Jet-Nemotron: Efficient language model with post neural architecture search.arXiv preprint arXiv:2508.15884, 2025

    Yuxian Gu, Qinghao Hu, Shang Yang, Haocheng Xi, Junyu Chen, Song Han, and Han Cai. Jet-Nemotron: Efficient language model with post neural architecture search.arXiv preprint arXiv:2508.15884, 2025

    arXiv 2025

  9. [9]

    FlashAttention-2: Faster attention with better parallelism and work partitioning.arXiv preprint arXiv:2307.08691, 2023

    Tri Dao. FlashAttention-2: Faster attention with better parallelism and work partitioning.arXiv preprint arXiv:2307.08691, 2023

    Pith/arXiv arXiv 2023

  10. [10]

    LLaMA 2: Open foundation and fine-tuned chat models

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. LLaMA 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023

    Pith/arXiv arXiv 2023

  11. [11]

    DeepSeek-v3.2: Pushing the frontier of open large language models.arXiv preprint arXiv:2512.02556, 2025

    Aixin Liu, Aoxue Mei, Bangcai Lin, Bing Xue, Bingxuan Wang, Bingzheng Xu, Bochao Wu, Bowei Zhang, Chaofan Lin, Chen Dong, et al. DeepSeek-v3.2: Pushing the frontier of open large language models.arXiv preprint arXiv:2512.02556, 2025

    Pith/arXiv arXiv 2025

  12. [12]

    A comprehensive survey on long context language modeling.arXiv preprint arXiv:2503.17407, 2025

    Jiaheng Liu, Dawei Zhu, Zhiqi Bai, Yancheng He, Huanxuan Liao, Haoran Que, Zekun Wang, Chenchen Zhang, Ge Zhang, Jiebin Zhang, et al. A comprehensive survey on long context language modeling.arXiv preprint arXiv:2503.17407, 2025

    arXiv 2025

  13. [13]

    Thus spake long-context large language model.arXiv preprint arXiv:2502.17129, 2025

    Xiaoran Liu, Ruixiao Li, Mianqiu Huang, Zhigeng Liu, Yuerong Song, Qipeng Guo, Siyang He, Qiqi Wang, Linlin Li, Qun Liu, et al. Thus spake long-context large language model.arXiv preprint arXiv:2502.17129, 2025

    arXiv 2025

  14. [14]

    RULER: What’s the real context size of your long-context language models?arXiv preprint arXiv:2404.06654, 2024

    Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, Yang Zhang, and Boris Ginsburg. RULER: What’s the real context size of your long-context language models?arXiv preprint arXiv:2404.06654, 2024

    Pith/arXiv arXiv 2024

  15. [15]

    SWE-bench: Can language models resolve real-world GitHub issues?arXiv preprint arXiv:2310.06770, 2023

    Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. SWE-bench: Can language models resolve real-world GitHub issues?arXiv preprint arXiv:2310.06770, 2023

    Pith/arXiv arXiv 2023

  16. [16]

    Retrieval-augmented generation for large language models: A survey.arXiv preprint arXiv:2312.10997, 2(1):32, 2023

    Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yixin Dai, Jiawei Sun, Haofen Wang, Haofen Wang, et al. Retrieval-augmented generation for large language models: A survey.arXiv preprint arXiv:2312.10997, 2(1):32, 2023

    Pith/arXiv arXiv 2023

  17. [17]

    In defense of RAG in the era of long-context language models.arXiv preprint arXiv:2409.01666, 2024

    Tan Yu, Anbang Xu, and Rama Akkiraju. In defense of RAG in the era of long-context language models.arXiv preprint arXiv:2409.01666, 2024

    Pith/arXiv arXiv 2024

  18. [18]

    A survey on large language model based autonomous agents.Frontiers of Computer Science, 18(6):186345, 2024

    Lei Wang, Chen Ma, Xueyang Feng, Zeyu Zhang, Hao Yang, Jingsen Zhang, Zhiyuan Chen, Jiakai Tang, Xu Chen, Yankai Lin, et al. A survey on large language model based autonomous agents.Frontiers of Computer Science, 18(6):186345, 2024. 10 Jet-Long: Efficient Long-Context Extension with Dynamic Bifocal RoPE

    2024

  19. [19]

    ReAct: Synergizing reasoning and acting in language models.arXiv preprint arXiv:2210.03629, 2022

    Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. ReAct: Synergizing reasoning and acting in language models.arXiv preprint arXiv:2210.03629, 2022

    Pith/arXiv arXiv 2022

  20. [20]

    Toolformer: Language models can teach themselves to use tools.Advances in neural information processing systems, 36:68539–68551, 2023

    Timo Schick, Jane Dwivedi-Yu, Roberto Dessì, Roberta Raileanu, Maria Lomeli, Eric Hambro, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. Toolformer: Language models can teach themselves to use tools.Advances in neural information processing systems, 36:68539–68551, 2023

    2023

  21. [21]

    AgentBench: Evaluating LLMs as agents.arXiv preprint arXiv:2308.03688, 2023

    Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding, Kaiwen Men, Kejuan Yang, et al. AgentBench: Evaluating LLMs as agents.arXiv preprint arXiv:2308.03688, 2023

    Pith/arXiv arXiv 2023

  22. [22]

    SWE-agent: Agent-computer interfaces enable automated software engineering.Advances in Neural Information Processing Systems, 37:50528–50652, 2024

    John Yang, Carlos E Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press. SWE-agent: Agent-computer interfaces enable automated software engineering.Advances in Neural Information Processing Systems, 37:50528–50652, 2024

    2024

  23. [23]

    MetaGPT: Meta programming for a multi-agent collaborative framework

    Sirui Hong, Mingchen Zhuge, Jonathan Chen, Xiawu Zheng, Yuheng Cheng, Jinlin Wang, Ceyao Zhang, Zili Wang, Steven Ka Shing Yau, Zijuan Lin, et al. MetaGPT: Meta programming for a multi-agent collaborative framework. InThe Twelfth International Conference on Learning Representations, 2024

    2024

  24. [24]

    FlashAttention: Fast and memory-efficient exact attention with IO-awareness.Advances in neural information processing systems, 35:16344–16359, 2022

    Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. FlashAttention: Fast and memory-efficient exact attention with IO-awareness.Advances in neural information processing systems, 35:16344–16359, 2022

    2022

  25. [25]

    Ring attention with blockwise transformers for near-infinite context

    Hao Liu, Matei Zaharia, and Pieter Abbeel. Ring attention with blockwise transformers for near-infinite context. arXiv preprint arXiv:2310.01889, 2023

    Pith/arXiv arXiv 2023

  26. [26]

    Why does the effective context length of LLMs fall short?arXiv preprint arXiv:2410.18745, 2024

    Chenxin An, Jun Zhang, Ming Zhong, Lei Li, Shansan Gong, Yao Luo, Jingjing Xu, and Lingpeng Kong. Why does the effective context length of LLMs fall short?arXiv preprint arXiv:2410.18745, 2024

    Pith/arXiv arXiv 2024

  27. [27]

    Data engineering for scaling language models to 128k context.arXiv preprint arXiv:2402.10171, 2024

    Yao Fu, Rameswar Panda, Xinyao Niu, Xiang Yue, Hannaneh Hajishirzi, Yoon Kim, and Hao Peng. Data engineering for scaling language models to 128k context.arXiv preprint arXiv:2402.10171, 2024

    Pith/arXiv arXiv 2024

  28. [28]

    Train short, test long: Attention with linear biases enables input length extrapolation.arXiv preprint arXiv:2108.12409, 2021

    Ofir Press, Noah A Smith, and Mike Lewis. Train short, test long: Attention with linear biases enables input length extrapolation.arXiv preprint arXiv:2108.12409, 2021

    Pith/arXiv arXiv 2021

  29. [29]

    LongRoPE: Extending LLM context window beyond 2 million tokens.arXiv preprint arXiv:2402.13753, 2024

    Yiran Ding, Li Lyna Zhang, Chengruidong Zhang, Yuanyuan Xu, Ning Shang, Jiahang Xu, Fan Yang, and Mao Yang. LongRoPE: Extending LLM context window beyond 2 million tokens.arXiv preprint arXiv:2402.13753, 2024

    Pith/arXiv arXiv 2024

  30. [30]

    gpt-oss-120b & gpt-oss-20b model card.arXiv preprint arXiv:2508.10925, 2025

    Sandhini Agarwal, Lama Ahmad, Jason Ai, Sam Altman, Andy Applebaum, Edwin Arbus, Rahul K Arora, Yu Bai, Bowen Baker, Haiming Bao, et al. gpt-oss-120b & gpt-oss-20b model card.arXiv preprint arXiv:2508.10925, 2025

    Pith/arXiv arXiv 2025

  31. [31]

    DeepSeek-v3 technical report.arXiv preprint arXiv:2412.19437, 2024

    Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. DeepSeek-v3 technical report.arXiv preprint arXiv:2412.19437, 2024

    Pith/arXiv arXiv 2024

  32. [32]

    Extending context window of large language models via positional interpolation.arXiv preprint arXiv:2306.15595, 2023

    Shouyuan Chen, Sherman Wong, Liangjian Chen, and Yuandong Tian. Extending context window of large language models via positional interpolation.arXiv preprint arXiv:2306.15595, 2023

    Pith/arXiv arXiv 2023

  33. [33]

    Scaling laws of RoPE-based extrapolation.arXiv preprint arXiv:2310.05209, 2023

    Xiaoran Liu, Hang Yan, Shuo Zhang, Chenxin An, Xipeng Qiu, and Dahua Lin. Scaling laws of RoPE-based extrapolation.arXiv preprint arXiv:2310.05209, 2023

    Pith/arXiv arXiv 2023

  34. [34]

    NTK-aware scaled RoPE allows LLaMA models to have extended (8k+) context size without any fine-tuning and minimal perplexity degradation

    bloc97. NTK-aware scaled RoPE allows LLaMA models to have extended (8k+) context size without any fine-tuning and minimal perplexity degradation. Reddit post on r/LocalLLaMA, 2023.https://www.reddit. com/r/LocalLLaMA/comments/14lz7j5/ntkaware_scaled_rope_allows_llama_models_to_have/

    2023

  35. [35]

    Dynamically scaled RoPE further increases performance of long context LLaMA with zero fine- tuning

    emozilla. Dynamically scaled RoPE further increases performance of long context LLaMA with zero fine- tuning. Reddit post on r/LocalLLaMA, 2023. https://www.reddit.com/r/LocalLLaMA/comments/14mrgpr/ dynamically_scaled_rope_further_increases/

    2023

  36. [36]

    FlashAttention-4: Algorithm and kernel pipelining co-design for asymmetric hardware scaling.arXiv preprint arXiv:2603.05451, 2026

    Ted Zadouri, Markus Hoehnerbach, Jay Shah, Timmy Liu, Vijay Thakkar, and Tri Dao. FlashAttention-4: Algorithm and kernel pipelining co-design for asymmetric hardware scaling.arXiv preprint arXiv:2603.05451, 2026

    arXiv 2026

  37. [37]

    Compressive transformers for long-range sequence modelling.arXiv preprint arXiv:1911.05507, 2019

    Jack W Rae, Anna Potapenko, Siddhant M Jayakumar, and Timothy P Lillicrap. Compressive transformers for long-range sequence modelling.arXiv preprint arXiv:1911.05507, 2019

    Pith/arXiv arXiv 1911

  38. [38]

    RoFormer: Enhanced transformer with rotary position embedding.Neurocomputing, 568:127063, 2024

    Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. RoFormer: Enhanced transformer with rotary position embedding.Neurocomputing, 568:127063, 2024. 11 Jet-Long: Efficient Long-Context Extension with Dynamic Bifocal RoPE

    2024

  39. [39]

    Exploring the limits of transfer learning with a unified text-to-text transformer.Journal of machine learning research, 21(140):1–67, 2020

    Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer.Journal of machine learning research, 21(140):1–67, 2020

    2020

  40. [40]

    Rethinking and improving relative position encoding for vision transformer

    Kan Wu, Houwen Peng, Minghao Chen, Jianlong Fu, and Hongyang Chao. Rethinking and improving relative position encoding for vision transformer. InProceedings of the IEEE/CVF international conference on computer vision, pages 10033–10041, 2021

    2021

  41. [41]

    LM-Infinite: Zero-shot extreme length generalization for large language models

    Chi Han, Qifan Wang, Hao Peng, Wenhan Xiong, Yu Chen, Heng Ji, and Sinong Wang. LM-Infinite: Zero-shot extreme length generalization for large language models. InProceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 3991–4008, 2024

    2024

  42. [42]

    Lost in the middle: How language models use long contexts.Transactions of the association for computational linguistics, 12:157–173, 2024

    Nelson F Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. Lost in the middle: How language models use long contexts.Transactions of the association for computational linguistics, 12:157–173, 2024

    2024

  43. [43]

    A training-free length extrapolation approach for LLMs: Greedy attention logit interpolation (GALI).arXiv preprint arXiv:2502.02659, 2025

    Yan Li, Tianyi Zhang, Zechuan Li, and Soyeon Caren Han. A training-free length extrapolation approach for LLMs: Greedy attention logit interpolation (GALI).arXiv preprint arXiv:2502.02659, 2025

    Pith/arXiv arXiv 2025

  44. [44]

    Rectified rotary position embeddings.https://github.com/bojone/rerope, 2023

    Jianlin Su. Rectified rotary position embeddings.https://github.com/bojone/rerope, 2023

    2023

  45. [45]

    SELF: Self-extend the context length with logistic growth function.arXiv preprint arXiv:2505.17296, 2025

    Phat Thanh Dang, Saahil Thoppay, Wang Yang, Qifan Wang, Vipin Chaudhary, and Xiaotian Han. SELF: Self-extend the context length with logistic growth function.arXiv preprint arXiv:2505.17296, 2025

    Pith/arXiv arXiv 2025

  46. [46]

    Longformer: The long-document transformer.arXiv preprint arXiv:2004.05150, 2020

    Iz Beltagy, Matthew E Peters, and Arman Cohan. Longformer: The long-document transformer.arXiv preprint arXiv:2004.05150, 2020

    Pith/arXiv arXiv 2004

  47. [47]

    BigBird: Transformers for longer sequences.Advances in neural information processing systems, 33:17283–17297, 2020

    Manzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontanon, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, et al. BigBird: Transformers for longer sequences.Advances in neural information processing systems, 33:17283–17297, 2020

    2020

  48. [48]

    Transformers are RNNs: Fast autoregressive transformers with linear attention

    Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and François Fleuret. Transformers are RNNs: Fast autoregressive transformers with linear attention. InInternational conference on machine learning, pages 5156–5165. PMLR, 2020

    2020

  49. [49]

    Rethinking attention with performers.arXiv preprint arXiv:2009.14794, 2020

    Krzysztof Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreea Gane, Tamas Sarlos, Peter Hawkins, Jared Davis, Afroz Mohiuddin, Lukasz Kaiser, et al. Rethinking attention with performers.arXiv preprint arXiv:2009.14794, 2020

    Pith/arXiv arXiv 2009

  50. [50]

    Mamba: Linear-time sequence modeling with selective state spaces.arXiv preprint arXiv:2312.00752, 2023

    Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces.arXiv preprint arXiv:2312.00752, 2023

    Pith/arXiv arXiv 2023

  51. [51]

    The impact of positional encoding on length generalization in transformers.Advances in Neural Information Processing Systems, 36:24892–24928, 2023

    Amirhossein Kazemnejad, Inkit Padhi, Karthikeyan Natesan Ramamurthy, Payel Das, and Siva Reddy. The impact of positional encoding on length generalization in transformers.Advances in Neural Information Processing Systems, 36:24892–24928, 2023

    2023

  52. [52]

    Kimi K2: Open agentic intelligence.arXiv preprint arXiv:2507.20534, 2025

    Kimi Team, Yifan Bai, Yiping Bao, Y Charles, Cheng Chen, Guanduo Chen, Haiting Chen, Huarong Chen, Jiahao Chen, Ningxin Chen, et al. Kimi K2: Open agentic intelligence.arXiv preprint arXiv:2507.20534, 2025

    Pith/arXiv arXiv 2025

  53. [53]

    NVIDIA Nemotron Nano 2: An accurate and efficient hybrid Mamba-transformer reasoning model.arXiv preprint arXiv:2508.14444, 2025

    Aarti Basant, Abhijit Khairnar, Abhijit Paithankar, Abhinav Khattar, Adithya Renduchintala, Aditya Malte, Akhiad Bercovich, Akshay Hazare, Alejandra Rico, Aleksander Ficek, et al. NVIDIA Nemotron Nano 2: An accurate and efficient hybrid Mamba-transformer reasoning model.arXiv preprint arXiv:2508.14444, 2025

    Pith/arXiv arXiv 2025

  54. [54]

    Why attend to everything? Focus is the key.arXiv preprint arXiv:2604.03260, 2026

    Hengshuai Yao, Xing Chen, Ahmed Murtadha, Jin Li, Yasin Abbasi Yadkori, Shuai Shao, Changling Liu, Guan Wang, Mingli Yuan, William Chen, and Sen Song. Why attend to everything? Focus is the key.arXiv preprint arXiv:2604.03260, 2026. 12 Jet-Long: Efficient Long-Context Extension with Dynamic Bifocal RoPE A. Metric definitions Percentage points (pp).For two...

    Pith/arXiv arXiv 2026