Pith. sign in

REVIEW 3 major objections 7 minor 4 cited by

When Precision Meets Position: BFloat16 Breaks Down RoPE in Long-Context Training

T0 review · 3 major / 7 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read BFloat16 breaks RoPE's relative-position property in long contexts, and a shared first-token anchor restores it.

desk verdict A real numerical observation about BFloat16 and RoPE, a practical method that works, but the causal story needs a Float32 baseline. read the letter →

arxiv 2411.13476 v2 pith:LFAEHAVT submitted 2024-11-20 cs.CL

Haonan Wang , Qian Liu , Chao Du , Tongyao Zhu , Cunxiao Du , Kenji Kawaguchi , Tianyu Pang This is my paper · ORCID
classification cs.CL
keywords BFloat16RoPErelativepositionalencodinglong-contexttrainingAnchorAttentionnumericalprecisionlargelanguagemodels
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that BFloat16, the standard low-precision format for training large language models, breaks the relative-position property of Rotary Position Embedding (RoPE) during long-context training, even though that property holds in exact arithmetic. The failure is small per token but accumulates as the context grows, and the first token accounts for most of the deviation. The paper argues that this numerical breakdown is not benign: it creates inconsistent positional relationships across documents in long-context training, which shows up as degraded performance on long-context benchmarks. On this diagnosis the paper builds AnchorAttention, which designates the first token as a shared anchor with a fixed position ID, makes it visible to every document, and masks attention across documents. If the argument is right, a small change to the attention pattern should improve long-context performance, preserve general abilities, and cut training time by more than half.

What carries the argument

The carrying mechanism is AnchorAttention, an attention mask combined with a positional scheme. A single anchor token—the beginning-of-sequence token, or the end-of-sequence token for models without one—is assigned position ID 0 and is visible to every document in the context window, while tokens from different documents cannot attend to each other and tokens within a document keep continuous position IDs. This concentrates whatever BFloat16-induced error exists on a semantically empty token, gives every document a consistent positional reference, keeps the full rotational span of RoPE learnable even from sequences shorter than the context window, and reduces the number of attention computations. The diagnostic that motivates it is the shift-invariance test of Eq. 3: under exact RoPE, attention logits are invariant to a constant positional shift, and under BFloat16 they are not.

What would settle it

Train the same base model on the same long-context data with full attention in Float32 (or with RoPE rotations kept in higher precision inside the attention kernel) and compare RULER scores against BFloat16 full attention and against AnchorAttention; if Float32 full attention matches AnchorAttention, the BFloat16 deviation is the operative cause. A second check: recompute the shift-invariance metric D from Eq. 4 with the first token excluded, and see whether the remaining attention difference drops to the Float32 level.

Watch

Extended reading notes

Core claim

The central discovery is that RoPE's relative positional encoding property—attention logits should depend only on the relative distance between two token positions, so that shifting every position by a constant leaves attention unchanged—fails under BFloat16 in pretrained models. With a pretrained LLaMA-2-7B, the paper measures a substantial attention difference between two constant positional shifts in BFloat16, while the same computation in Float32 gives zero difference; random initial weights in BFloat16 show a much smaller effect, indicating that pretraining amplifies the error. The deviation concentrates on the first token and grows with sequence length, and the paper traces it to BFloat16's limited mantissa: RoPE rotations are applied in higher precision, but the query-key inner product inside the attention kernel is computed in BFloat16, so the algebraic cancellation that makes RoPE relative no longer cancels. The paper then shows that resetting position IDs per document improves long-context performance, and proposes AnchorAttention as the mechanism that captures this benefit while keeping continuous position IDs and a shared anchor.

Load-bearing premise

The load-bearing assumption is that the BFloat16 attention deviation the paper measures is the operative cause of the long-context training gap, and not merely a correlated artifact, since AnchorAttention also changes attention masking and introduces an attention-sink-like anchor, and the paper itself flags its absolute-position hypothesis as needing rigorous validation.

Editorial extensions

If this is right

  • AnchorAttention should beat full attention and standard intra-document attention on the RULER benchmark at every tested length from 4K to 128K, with the largest gains at the longest contexts.
  • Training on the same infrastructure should take less than half the wall-clock time of full attention, because document masking removes many attention computations.
  • General abilities measured by HellaSwag, MMLU, and LongBench in-context learning should remain close to, and in several cases better than, the original base model.
  • The gap between training on raw SlimPajama and on a carefully upsampled long-sequence mixture should shrink, so AnchorAttention reduces the need for aggressive long-data upsampling.
  • Adding domain tags does not reliably improve AnchorAttention, and interleaving chunks of documents hurts when combined with cross-document masking.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the BFloat16 diagnosis is correct, other low-precision formats with even fewer mantissa bits, such as FP8, should show a stronger positional breakdown, and the shared-anchor idea may transfer directly.
  • Part of AnchorAttention's measured gain could come from the anchor acting as an attention sink rather than from numerical error removal; training with a shared anchor while keeping RoPE in Float32 inside the attention kernel would separate the two effects.
  • Because AnchorAttention masks cross-document attention, it may sacrifice performance on tasks that genuinely need cross-document reasoning, such as multi-document question answering, even if it wins on synthetic long-context benchmarks.
  • The paper's training budget (2B tokens, models at roughly 10B scale) is too small to show whether the numerical breakdown has the same practical importance in frontier-scale continued pretraining.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 7 minor

Summary. This paper makes two claims. First, it presents an empirical diagnostic: for a frozen pretrained LLaMA-2-7B model, attention logits and scores are not invariant to a constant positional shift when the computation runs in BFloat16, whereas the invariance holds in Float32; the deviation grows with sequence length and is concentrated on the first token (Section 2.2, Figure 1). Second, it proposes AnchorAttention, a training-time attention scheme that repurposes the BOS token as a shared anchor with position ID 0, makes it visible to all documents in a packed long context, and masks attention between different documents. Experiments on LLaMA-2-7B, LLaMA-3-8B, Mistral-7B-v0.3, and Qwen-1.5-1.8B show that AnchorAttention improves RULER scores from 8K to 128K over full attention and standard intra-document attention (Tables 5 and 6), largely preserves MMLU and HellaSwag performance (Table 7), and reduces training time by more than 50% through attention sparsity (Figure 6). The paper interprets these gains as evidence that BFloat16 breaks RoPE's relative positional property and that AnchorAttention mitigates the resulting numerical problem, but the training experiments include no Float32 full-attention control, and the paper itself flags the causal interpretation as a hypothesis in Section 3.2.

Significance. If the causal claim were established, this would be a high-impact result for the field: nearly all large-scale long-context training currently runs in BFloat16, and a mechanism linking numeric format to positional-encoding fidelity would change how practitioners think about RoPE and FlashAttention-style kernels. The paper ships real value even without that causal story. The shift-invariance diagnostic in Section 2.2 is clean, falsifiable, and easy to reuse; the position-reset comparison (Figure 3) is an interesting empirical finding about intra-document attention; the shared-anchor method is simple, effective, and evaluated against external benchmarks, so the validation is not circular; the recommendation to average RULER over several checkpoints is a methodological improvement; and the released AnchorContext implementation, with its measured zero logits difference relative to the non-distributed baseline, is a concrete engineering contribution.

major comments (3)
  1. [§2.2, §3.2, §5.1 (Table 5)] The central causal claim — that BFloat16 breaks RoPE and that AnchorAttention's gains come from alleviating this numerical issue — is not directly tested, because every training run in Section 5 uses BFloat16. The stress-test concern about the missing Float32 full-attention control lands: without it, the RULER differences in Table 5 (e.g., 73.25 vs. 66.40 at 64K on SlimPajama-64K; 66.15 vs. 62.75 at 128K on SlimPajama-128K) can be explained by the architectural changes alone, namely cross-document masking (shared with the Intra-Doc baseline) or the shared anchor acting as an attention sink, a possibility the paper itself raises in Section 8. The paper's own text concedes the interpretive gap: Section 3.2 states that the analysis "is based on the hypothesis that RoPE functions in some way as an absolute positional encoding mechanism under BFloat16 precision," and Section 8 states that "future work is needed to more rigorously investigate" the first-position effect; the Introduction's claim that the reset result "confirms that position ID inconsistencies are a key issue" is stronger than this hypothesis language supports. I request a precision-controlled experiment: train full attention with the identical data, hyperparameters, and sequence-parallelism configuration but with Float32 (or at least FP32 attention logits), at least at a reduced context length, and compare against the BFloat16 full-attention run. A low-cost complement would be to rerun the Section 3.2 position-reset comparison in Float32 at a smaller scale: if the reset advantage disappears in Float32, that directly supports the precision explanation rather than the architectural explanation.
  2. [§3.3, §5.1 (Table 5)] The mechanism attribution is underdetermined because AnchorAttention differs from the Full Attention baseline in three ways at once: cross-document masking, a shared anchor visible to all documents, and continuous position IDs without per-document reset. Table 5 ablates only the intra-document variants (with and without reset), so no experiment isolates the anchor's contribution. Concretely, the first-token numerical-error story predicts that the benefit should largely disappear when the anchor is assigned a nonzero fixed position or when the RoPE computation is kept at higher precision, whereas an attention-sink explanation (Xiao et al., 2023, cited in Section 8) predicts that the benefit is relatively insensitive to the anchor's position. I suggest three ablations: (a) an anchor at position 0 that is masked exactly like other tokens, keeping the shared-position-0 property without the special visibility; (b) a shared anchor with a fixed nonzero position ID; and (c) an otherwise identical run in which only the attention-logit computation is done in FP32. Relatedly, the premise in Section 3.3 that the BOS token "lacks explicit semantic meaning" is asserted, not tested; using a semantically meaningful token as the anchor would indicate whether the gain comes from concentrating error on a meaningless token or from the positional-reference effect itself.
  3. [§2.2 and §5.1] The quantitative bridge between the forward-pass diagnostic and training outcomes is missing. The measurements in Section 2.2 are attention-score and logit differences of a frozen model, and the paper does not report the magnitude of the deviations (the y-axis scale of Figure 1, right, is not given numerically) or show that they affect gradients or training dynamics. A parity-of-scale argument is therefore absent: nothing demonstrates that deviations of these sizes would produce the 64K–128K RULER gaps in Table 5. A cheap experiment would directly test the claimed mechanism while holding the architecture fixed: in a BFloat16 full-attention training run, clamp the first token's position ID to a fixed value (say 0) while leaving all other position IDs unchanged, and compare RULER to the standard run. If clamping the first token's position closes most of the gap between Full Attention and AnchorAttention, the first-token mechanism is supported; if not, the first-token narrative would need to be revised even though the method's empirical value remains.
minor comments (7)
  1. [Entire manuscript] The compiled text contains two overlapping versions of the same paper (the arXiv-style body and an embedded anonymous TMLR draft), with repeated passages such as the Introduction and Figure 2 appearing twice; the published version must be a single clean rendering.
  2. [§2.2] "we shop the ∆1 = 16 to improve visualization" should read "we skip the ∆1 = 16 point"; the current wording appears to be a typo and is confusing in context.
  3. [§2.2, Figure 1 (left)] The color references in the Results Discussion are inconsistent with the Experimental Setup: the setup defines blue = pretrained BFloat16, yellow = pretrained Float32, and green = random BFloat16, but the discussion then says the Float32 comparison is shown by "blue vs. green lines" and the pretrained-vs-random BFloat16 comparison by "blue vs. yellow lines," which swaps the two comparisons; please correct the parenthetical references.
  4. [§5.5] The text on FlexAttention says it "facilitates flexible attention masks needed by interleaved chunks, as illustrated in Figure 6," but Figure 6 shows estimated training time; the intended cross-reference is evidently Figure 5, which illustrates interleaved chunks.
  5. [§5.4, Table 7] The baseline row for LLaMA-2-7B reports LongBench ICL equal to 6.22, which is far below the trained-model range of 48.96–66.02 in the same table and also well below few-shot ICL values reported for this model elsewhere; please verify that the baseline was evaluated with the same ICL protocol and that the row aligns correctly with the column headers.
  6. [§4.2, Tables 5–6] The RULER task sets differ across tables: LLaMA-2-7B is evaluated on 11 of 13 tasks (NIAH-Multikey-3 and CWE excluded), while the other models use all 13 tasks; the exclusion is documented and defensible, but the table captions should state the task counts so that the cross-model comparison is not read as being on a common task set, and the sentence "we exclude two tasks (and NIAH-Multikey 3 and Common Word Extraction)" should drop the stray "and."
  7. [§5.5, Figure 6] The "more than 50%" training-time reduction should be qualified as arising from attention sparsity (masking), which is shared with intra-document attention, and the figure or caption should state the measurement conditions (GPU count, sequence-parallelism configuration, and whether batch size was held fixed across methods).

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the BFloat16-RoPE deviation is an independent measurement, AnchorAttention is evaluated on external benchmarks, and the causal mechanism is explicitly left as a hypothesis.

full rationale

The paper's central observational claim is an empirical measurement, not a construction: Section 2.2 defines attention-difference metrics D and D_logit (Eqs. 4 and 5) and reports that BFloat16, unlike Float32, produces nonzero differences under constant positional shifts, with the first token dominating. This is a direct numerical observation and does not presuppose the conclusion. The design of AnchorAttention is indeed motivated by that observation, but the method's evaluation on RULER, LongBench, MMLU, and HellaSwag is external to the observation and could in principle falsify the method; no fitted parameter is renamed as a prediction. The load-bearing causal interpretation, that BFloat16-induced first-token deviation causes the long-context training gap, is not derived by construction: the authors explicitly flag it as a hypothesis in Section 3.2 ('this analysis is based on the hypothesis that RoPE functions in some way as an absolute positional encoding mechanism under BFloat16 precision') and again in Section 8 ('leading us to hypothesize that the position ID of the first token acts as an absolute position. Future work is needed to more rigorously investigate'). The absence of a Float32 full-attention training control means the mechanism is underdetermined, since AnchorAttention also changes cross-document masking and adds a shared anchor that may act as an attention sink, but this is an experimental-design gap and not a circular reduction. The only same-author citation, Gu et al. (2024) in Section 8, supports a speculative connection to attention sinks and is not load-bearing for any result. The paper is therefore self-contained against external benchmarks, and no step in the derivation chain reduces to its own inputs.

Assumptions & free parameters 1 free parameters · 5 assumptions · 1 invented entities

The central claim rests on an empirical observation about BFloat16 rounding, a hypothesis about the first token's absolute-position role, and the design assumption about anchor token neutrality. The method's causal link to the precision issue is not directly validated: no Float32 training baseline, and no control that removes the anchor while keeping the masking.

free parameters (1)
  • RoPE base theta for each context length = 1e6 (16K), 5e6 (64K), 1e7 (128K)
    Chosen by a sweep in Table 3; the base controls rotation frequencies and therefore the magnitude and pattern of BFloat16 rounding errors in the RoPE computation, so it interacts with the paper's central phenomenon.
assumptions (5)
  • standard math RoPE is theoretically translation-invariant in exact arithmetic (Eq. 3).
    Used as the reference property that is claimed to be broken under BFloat16.
  • domain assumption The query-key inner product in FlashAttention2 is computed in BFloat16, while the RoPE rotation is applied in Float32 and then cast.
    This implementation detail is the premise for why BFloat16 rounding enters the attention logits (Section 2.1).
  • ad hoc to paper The first token's position ID acts like an absolute position reference under BFloat16.
    Stated as a hypothesis in Section 3.2 and Future Work; the design of AnchorAttention depends on this being true.
  • ad hoc to paper The BOS token lacks explicit semantic meaning, so concentrating numerical error on it has minimal impact.
    Assumed in Section 3.3 to justify using BOS/EOS as the anchor; not directly tested.
  • ad hoc to paper Long-context continual pretraining only extends existing abilities, so tasks the base model cannot solve at 4K should be excluded from evaluation.
    Used in Section 4.2 and Appendix D to justify excluding NIAH-Multikey-3 and CWE for LLaMA-2-7B.
invented entities (1)
  • Shared anchor token (repurposed BOS or EOS) independent evidence
    purpose: Provide a permanent position-0 reference visible to all documents and concentrate first-token numerical error on a semantically neutral token.
    Its effect is measurable by comparing AnchorAttention with intra-document attention and reset baselines on RULER, which the paper does; however, the specific claim that it fixes BFloat16 error rather than acting as an attention sink is not isolated, so independent evidence for the mechanism is absent.

how reviews work

0 comments
Cite this review

Pith. "Pith review of When Precision Meets Position: BFloat16 Breaks Down RoPE in Long-Context Training." pith.science (2026). https://pith.science/paper/LFAEHAVT

@misc{pith2026241113476,
  author       = {Pith},
  title        = {Pith review of: When Precision Meets Position: BFloat16 Breaks Down RoPE in Long-Context Training},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/LFAEHAVT}},
  note         = {Machine review of arXiv:2411.13476}
}
read the original abstract

Extending context window sizes allows large language models (LLMs) to process longer sequences and handle more complex tasks. Rotary Positional Embedding (RoPE) has become the de facto standard due to its relative positional encoding properties that benefit long-context training. However, we observe that using RoPE with BFloat16 format results in numerical issues, causing it to deviate from its intended relative positional encoding, especially in long-context scenarios. This issue arises from BFloat16's limited precision and accumulates as context length increases, with the first token contributing significantly to this problem. To address this, we develop AnchorAttention, a plug-and-play attention method that alleviates numerical issues caused by BFloat16, improves long-context capabilities, and speeds up training. AnchorAttention reduces unnecessary attention computations, maintains semantic coherence, and boosts computational efficiency by treating the first token as a shared anchor with a consistent position ID, making it visible to all documents within the training context. Experiments on three types of LLMs demonstrate that AnchorAttention significantly improves long-context performance and reduces training time by over 50\% compared to standard full attention mechanisms, while preserving the original LLM's capabilities on general tasks. Our code is available at https://github.com/haonan3/AnchorContext.

Figures

Figures reproduced from arXiv: 2411.13476 by the authors.

Figure 1
Figure 1. Effects of positional shifts on attention computations under different settings. [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 3
Figure 3. Resetting position IDs im￾proves performance, contradicting the￾oretical predictions of RoPE. Results Discussion. The results are presented in [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figure 4
Figure 4. RULER performance varies during long-context training, we recommend reporting [PITH_FULL_IMAGE:figures/full_fig_p009_4.png] view at source ↗
Figures from the paper (3 more)
Figure 6
Figure 6. Figure 6: Estimated training time required to process 1 billion tokens at various context [PITH_FULL_IMAGE:figures/full_fig_p015_6.png]
Figure 7
Figure 7. Figure 7: Visualization of attention score differences under BFloat16 for individual samples. [PITH_FULL_IMAGE:figures/full_fig_p022_7.png]
Figure 8
Figure 8. Figure 8: Training Data Sequence Length Distribution [PITH_FULL_IMAGE:figures/full_fig_p023_8.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 4 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Addressable Memory for Video World Models

    cs.CV 2026-08 conditional novelty 6.0 of 10

    Video world models can recall revisited scenes far beyond their training horizon by storing compressed memory at fixed in-distribution positions and averaging keys in a rotation-free space.

  2. Studying quantization trade-offs for efficient inference deployment in machine translation

    cs.CL 2026-07 conditional novelty 6.0 of 10

    Quantized Hy-MT2 models stay accurate at long context, but quantized EuroLLM 9B/22B models collapse (up to ~60% chrF++ drop) while W4A8/W8A8 plus 200–400-token chunking improves serving throughput.

  3. Scaling Laws for Forgetting during Finetuning with Pretraining Data Injection

    cs.LG 2025-02 conditional novelty 6.0 of 10

    Finetuning forgetting follows a multiplicative scaling law in model size, finetuning tokens, and injected pretraining fraction, with 1% injection nearly eliminating forgetting.

  4. Attention Entropy is a Key Factor: An Analysis of Parallel Context Encoding with Full-attention-based Pre-trained Language Models

    cs.CL 2024-12 conditional novelty 6.0 of 10

    Parallel context encoding makes decoder-only LLMs attend too broadly; shared attention sinks and block-wise selective attention reduce that broadness and narrow the performance gap.

Reference graph

Works this paper leans on

67 extracted references · 8 canonical work pages · cited by 4 Pith papers

  1. [1]

    Phi-3 technical report: A highly capable language model locally on your phone, 2024

    Marah Abdin, Jyoti Aneja, Hany Awadalla, and et al. Phi-3 technical report: A highly capable language model locally on your phone, 2024. URL https://arxiv.org/abs/2404.14219

    arXiv 2024

  2. [2]

    Many-shot in-context learning

    Rishabh Agarwal, Avi Singh, Lei M Zhang, Bernd Bohnet, Stephanie Chan, Ankesh Anand, Zaheer Abbas, Azade Nova, John D Co-Reyes, Eric Chu, et al. Many-shot in-context learning. arXiv preprint arXiv:2404.11018, 2024

    arXiv 2024

  3. [3]

    Gqa: Training generalized multi-query transformer models from multi-head checkpoints

    Joshua Ainslie, James Lee-Thorp, Michiel de Jong, Yury Zemlyanskiy, Federico Lebr \'o n, and Sumit Sanghai. Gqa: Training generalized multi-query transformer models from multi-head checkpoints. arXiv preprint arXiv:2305.13245, 2023

    arXiv 2023

  4. [4]

    Physics of language models: Part 3.3, knowledge capacity scaling laws

    Zeyuan Allen-Zhu and Yuanzhi Li. Physics of language models: Part 3.3, knowledge capacity scaling laws. arXiv preprint arXiv:2404.05405, 2024

    arXiv 2024

  5. [5]

    Longbench: A bilingual, multitask benchmark for long context understanding

    Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, Yuxiao Dong, Jie Tang, and Juanzi Li. Longbench: A bilingual, multitask benchmark for long context understanding. arXiv preprint arXiv:2308.14508, 2023

    arXiv 2023

  6. [6]

    Unlimiformer: Long-range transformers with unlimited length input

    Amanda Bertsch, Uri Alon, Graham Neubig, and Matthew Gormley. Unlimiformer: Long-range transformers with unlimited length input. Advances in Neural Information Processing Systems, 36, 2024

    2024

  7. [7]

    Clex: Continuous length extrapolation for large language models, 2024

    Guanzheng Chen, Xin Li, Zaiqiao Meng, Shangsong Liang, and Lidong Bing. Clex: Continuous length extrapolation for large language models, 2024. URL https://arxiv.org/abs/2310.16450

    arXiv 2024

  8. [8]

    Extending context window of large language models via positional interpolation

    Shouyuan Chen, Sherman Wong, Liangjian Chen, and Yuandong Tian. Extending context window of large language models via positional interpolation. arXiv preprint arXiv:2306.15595, 2023 a

    arXiv 2023

Show all 67 references
  1. [9]

    Longlora: Efficient fine-tuning of long-context large language models

    Yukang Chen, Shengju Qian, Haotian Tang, Xin Lai, Zhijian Liu, Song Han, and Jiaya Jia. Longlora: Efficient fine-tuning of long-context large language models. ArXiv, abs/2309.12307, 2023 b

    2023 arXiv

  2. [10]

    Flash A ttention-2: Faster attention with better parallelism and work partitioning

    Tri Dao. Flash A ttention-2: Faster attention with better parallelism and work partitioning. In International Conference on Learning Representations (ICLR), 2024

    2024

  3. [11]

    Bert: Pre-training of deep bidirectional transformers for language understanding

    Jacob Devlin. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018

    2018 arXiv

  4. [12]

    The llama 3 herd of models

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, and et al. The llama 3 herd of models. ArXiv, abs/2407.21783, 2024

    2024 arXiv

  5. [13]

    What is wrong with perplexity for long-context language modeling?, 2024

    Lizhe Fang, Yifei Wang, Zhaoyang Liu, Chenheng Zhang, Stefanie Jegelka, Jinyang Gao, Bolin Ding, and Yisen Wang. What is wrong with perplexity for long-context language modeling?, 2024. URL https://arxiv.org/abs/2410.23771

    2024 arXiv

  6. [14]

    Data engineering for scaling language models to 128k context

    Yao Fu, Rameswar Panda, Xinyao Niu, Xiang Yue, Hanna Hajishirzi, Yoon Kim, and Hao Peng. Data engineering for scaling language models to 128k context. ArXiv, abs/2402.10171, 2024

    2024 arXiv

  7. [15]

    How to train long-context language models (effectively)

    Tianyu Gao, Alexander Wettig, Howard Yen, and Danqi Chen. How to train long-context language models (effectively). arXiv preprint arXiv:2410.02660, 2024

    2024

  8. [16]

    A little goes a long way: Efficient long context training and inference with partial contexts

    Suyu Ge, Xihui Lin, Yunan Zhang, Jiawei Han, and Hao Peng. A little goes a long way: Efficient long context training and inference with partial contexts. arXiv preprint arXiv:2410.01485, 2024

    2024 arXiv

  9. [17]

    When attention sink emerges in language models: An empirical view

    Xiangming Gu, Tianyu Pang, Chao Du, Qian Liu, Fengzhuo Zhang, Cunxiao Du, Ye Wang, and Min Lin. When attention sink emerges in language models: An empirical view. arXiv preprint arXiv:2410.10781, 2024

    2024 arXiv

  10. [18]

    Active-dormant attention heads: Mechanistically demystifying extreme-token phenomena in llms

    Tianyu Guo, Druv Pai, Yu Bai, Jiantao Jiao, Michael I Jordan, and Song Mei. Active-dormant attention heads: Mechanistically demystifying extreme-token phenomena in llms. arXiv preprint arXiv:2410.13835, 2024 a

    2024 arXiv

  11. [19]

    Attention score is not all you need for token importance indicator in kv cache reduction: Value also matters, 2024 b

    Zhiyu Guo, Hidetaka Kamigaito, and Taro Watanabe. Attention score is not all you need for token importance indicator in kv cache reduction: Value also matters, 2024 b . URL https://arxiv.org/abs/2406.12335

    2024 arXiv

  12. [20]

    Lm-infinite: Simple on-the-fly length generalization for large language models

    Chi Han, Qifan Wang, Wenhan Xiong, Yu Chen, Heng Ji, and Sinong Wang. Lm-infinite: Simple on-the-fly length generalization for large language models. arXiv preprint arXiv:2308.16137, 2023

    2023 arXiv

  13. [21]

    Aligning ai with shared human values

    Dan Hendrycks, Collin Burns, Steven Basart, Andrew Critch, Jerry Li, Dawn Song, and Jacob Steinhardt. Aligning ai with shared human values. Proceedings of the International Conference on Learning Representations (ICLR), 2021

    2021

  14. [22]

    Ruler: What's the real context size of your long-context language models? arXiv preprint arXiv:2404.06654, 2024

    Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, Yang Zhang, and Boris Ginsburg. Ruler: What's the real context size of your long-context language models? arXiv preprint arXiv:2404.06654, 2024

    2024 arXiv

  15. [23]

    Can perplexity reflect large language model's ability in long text understanding? arXiv preprint arXiv:2405.06105, 2024

    Yutong Hu, Quzhe Huang, Mingxu Tao, Chen Zhang, and Yansong Feng. Can perplexity reflect large language model's ability in long text understanding? arXiv preprint arXiv:2405.06105, 2024

    2024 arXiv

  16. [24]

    Deepspeed ulysses: System optimizations for enabling training of extreme long sequence transformer models, 2023

    Sam Ade Jacobs, Masahiro Tanaka, Chengming Zhang, Minjia Zhang, Shuaiwen Leon Song, Samyam Rajbhandari, and Yuxiong He. Deepspeed ulysses: System optimizations for enabling training of extreme long sequence transformer models, 2023. URL https://arxiv.org/abs/2309.14509

    2023 arXiv

  17. [25]

    Mistral 7b

    Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv preprint arXiv:2310.06825, 2023

    arXiv

  18. [26]

    SWE -bench: Can language models resolve real-world github issues? In The Twelfth International Conference on Learning Representations, 2024

    Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R Narasimhan. SWE -bench: Can language models resolve real-world github issues? In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/f...

    2024

  19. [27]

    Llm maybe longlm: Self-extend llm context window without tuning

    Hongye Jin, Xiaotian Han, Jingfeng Yang, Zhimeng Jiang, Zirui Liu, Chia-Yuan Chang, Huiyuan Chen, and Xia Hu. Llm maybe longlm: Self-extend llm context window without tuning. arXiv preprint arXiv:2401.01325, 2024

    2024 arXiv

  20. [28]

    A study of bfloat16 for deep learning training

    Dhiraj Kalamkar, Dheevatsa Mudigere, Naveen Mellempudi, Dipankar Das, Kunal Banerjee, Sasikanth Avancha, Dharma Teja Vooturi, Nataraj Jammalamadaka, Jianyu Huang, Hector Yuen, et al. A study of bfloat16 for deep learning training. arXiv preprint arXiv:1905.12322, 2019

    1905 arXiv

  21. [29]

    One thousand and one pairs: A" novel" challenge for long-context language models

    Marzena Karpinska, Katherine Thai, Kyle Lo, Tanya Goyal, and Mohit Iyyer. One thousand and one pairs: A" novel" challenge for long-context language models. arXiv preprint arXiv:2406.16264, 2024

    2024 arXiv

  22. [30]

    Rethinking positional encoding in language pre-training

    Guolin Ke, Di He, and Tie-Yan Liu. Rethinking positional encoding in language pre-training. In International Conference on Learning Representations, 2021. URL https://openreview.net/forum?id=09-528y2Fgf

    2021

  23. [31]

    Sequence parallelism: Long sequence training from system perspective

    Shenggui Li, Fuzhao Xue, Chaitanya Baranwal, Yongbin Li, and Yang You. Sequence parallelism: Long sequence training from system perspective. arXiv preprint arXiv:2105.13120, 2021

    2021 arXiv

  24. [32]

    Ring attention with blockwise transformers for near-infinite context, 2023 a

    Hao Liu, Matei Zaharia, and Pieter Abbeel. Ring attention with blockwise transformers for near-infinite context, 2023 a . URL https://arxiv.org/abs/2310.01889

    2023 arXiv

  25. [33]

    Scaling laws of rope-based extrapolation

    Xiaoran Liu, Hang Yan, Shuo Zhang, Chenxin An, Xipeng Qiu, and Dahua Lin. Scaling laws of rope-based extrapolation. arXiv preprint arXiv:2310.05209, 2023 b

    2023 arXiv

  26. [34]

    Ntk-aware scaled rope allows llama models to have extended (8k+) context size without any fine-tuning and minimal perplexity degration, 2023

    LocalLLaMA. Ntk-aware scaled rope allows llama models to have extended (8k+) context size without any fine-tuning and minimal perplexity degration, 2023. URL https://www.reddit.com/r/LocalLLaMA/comments/14lz7j5/ntkaware_scaled_rope_allows_llama_models_to_have/

    2023

  27. [35]

    Sparser is faster and less is more: Efficient sparse attention for long-range transformers

    Chao Lou, Zixia Jia, Zilong Zheng, and Kewei Tu. Sparser is faster and less is more: Efficient sparse attention for long-range transformers. arXiv preprint arXiv:2406.16747, 2024

    2024 arXiv

  28. [36]

    Lifelong and continual learning dialogue systems

    Sahisnu Mazumder and Bing Liu. Lifelong and continual learning dialogue systems. arXiv preprint arXiv:2211.06553, 2022

    2022 arXiv

  29. [37]

    Base of rope bounds context length, 2024

    Xin Men, Mingyu Xu, Bingning Wang, Qingyu Zhang, Hongyu Lin, Xianpei Han, and Weipeng Chen. Base of rope bounds context length, 2024. URL https://arxiv.org/abs/2405.14591

    2024 arXiv

  30. [38]

    Yarn: Efficient context window extension of large language models

    Bowen Peng, Jeffrey Quesnelle, Honglu Fan, and Enrico Shippole. Yarn: Efficient context window extension of large language models. arXiv preprint arXiv:2309.00071, 2023

    2023 arXiv

  31. [39]

    Randomized positional encodings boost length generalization of transformers

    Anian Ruoss, Gr \'e goire Del \'e tang, Tim Genewein, Jordi Grau-Moya, R \'o bert Csord \'a s, Mehdi Bennani, Shane Legg, and Joel Veness. Randomized positional encodings boost length generalization of transformers. arXiv preprint arXiv:2305.16843, 2023

    2023 arXiv

  32. [40]

    Self-attention with relative position representations

    Peter Shaw, Jakob Uszkoreit, and Ashish Vaswani. Self-attention with relative position representations. arXiv preprint arXiv:1803.02155, 2018

    2018 arXiv

  33. [41]

    SlimPajama: A 627B token cleaned and deduplicated version of RedPajama

    Daria Soboleva, Faisal Al-Khateeb, Robert Myers, Jacob R Steeves, Joel Hestness, and Nolan Dey. SlimPajama: A 627B token cleaned and deduplicated version of RedPajama . https://cerebras.ai/blog/slimpajama-a-627b-token-cleaned-and-deduplicated-version-of-redpajama, 2023. URL ht...

    2023

  34. [42]

    Roformer: Enhanced transformer with rotary position embedding

    Jianlin Su, Yu Lu, Shengfeng Pan, Bo Wen, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. ArXiv, abs/2104.09864, 2021

    2021 arXiv

  35. [43]

    Zico Kolter, and Zhuang Liu

    Mingjie Sun, Xinlei Chen, J. Zico Kolter, and Zhuang Liu. Massive activations in large language models, 2024. URL https://arxiv.org/abs/2402.17762

    2024 arXiv

  36. [44]

    Gemma: Open models based on gemini research and technology

    Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivi \`e re, Mihir Sanjay Kale, Juliette Love, et al. Gemma: Open models based on gemini research and technology. arXiv preprint arXiv:2403.08295, 2024

    2024 arXiv

  37. [45]

    Llama 2: Open foundation and fine-tuned chat models, 2023

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, and et al. Llama 2: Open foundation and fine-tuned chat models, 2023

    2023

  38. [46]

    Attention is all you need

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, ukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017

    2017

  39. [47]

    Leave no document behind: Benchmarking long-context llms with extended multi-doc qa

    Minzheng Wang, Longze Chen, Cheng Fu, Shengyi Liao, Xinghua Zhang, Bingli Wu, Haiyang Yu, Nan Xu, Lei Zhang, Run Luo, et al. Leave no document behind: Benchmarking long-context llms with extended multi-doc qa. arXiv preprint arXiv:2406.17419, 2024 a

    2024 arXiv

  40. [48]

    Bfloat16: The secret to high performance on cloud tpus

    Shibo Wang and Pankaj Kanwar. Bfloat16: The secret to high performance on cloud tpus. https://cloud.google.com/blog/products/ai-machine-learning/bfloat16-the-secret-to-high-performance-on-cloud-tpus, August 23 2019. Accessed: August 31, 2024

    2019

  41. [49]

    Resonance rope: Improving context length generalization of large language models

    Suyuchen Wang, Ivan Kobyzev, Peng Lu, Mehdi Rezagholizadeh, and Bang Liu. Resonance rope: Improving context length generalization of large language models. arXiv preprint arXiv:2403.00071, 2024 b

    2024 arXiv

  42. [50]

    Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mari...

    2020

  43. [51]

    Infllm: Unveiling the intrinsic capacity of llms for understanding extremely long sequences with training-free memory

    Chaojun Xiao, Pengle Zhang, Xu Han, Guangxuan Xiao, Yankai Lin, Zhengyan Zhang, Zhiyuan Liu, Song Han, and Maosong Sun. Infllm: Unveiling the intrinsic capacity of llms for understanding extremely long sequences with training-free memory. arXiv preprint arXiv:2402.04617, 2024

    2024 arXiv

  44. [52]

    Efficient streaming language models with attention sinks

    Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient streaming language models with attention sinks. arXiv preprint arXiv:2309.17453, 2023

    2023 arXiv

  45. [53]

    Effective long-context scaling of foundation models

    Wenhan Xiong, Jingyu Liu, Igor Molybog, Hejia Zhang, Prajjwal Bhargava, Rui Hou, Louis Martin, Rashi Rungta, Karthik Abinav Sankararaman, Barlas Oğuz, Madian Khabsa, Han Fang, Yashar Mehdad, Sharan Narang, Kshitiz Malik, Angela Fan, Shruti Bhosale, Sergey Edunov, Mike Lewis, S...

    2023

  46. [54]

    Qwen2 technical report, 2024

    An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, and et al. Qwen2 technical report, 2024. URL https://arxiv.org/abs/2407.10671

    2024 arXiv

  47. [55]

    Helmet: How to evaluate long-context language models effectively and thoroughly

    Howard Yen, Tianyu Gao, Minmin Hou, Ke Ding, Daniel Fleischer, Peter Izasak, Moshe Wasserblat, and Danqi Chen. Helmet: How to evaluate long-context language models effectively and thoroughly. arXiv preprint arXiv:2410.02694, 2024

    arXiv

  48. [56]

    Stablemask: Refining causal masking in decoder-only transformer

    Qingyu Yin, Xuzheng He, Xiang Zhuang, Yu Zhao, Jianhua Yao, Xiaoyu Shen, and Qiang Zhang. Stablemask: Refining causal masking in decoder-only transformer. arXiv preprint arXiv:2402.04779, 2024

    2024 arXiv

  49. [57]

    Hellaswag: Can a machine really finish your sentence? arXiv preprint arXiv:1905.07830, 2019

    Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine really finish your sentence? arXiv preprint arXiv:1905.07830, 2019

    1905 arXiv

  50. [58]

    J. Zhang. Easycontext: Efficient and extensible long context language models. https://github.com/jzhang38/EasyContext, 2023. Accessed: 2024-09-16

    2023

  51. [59]

    Longcite: Enabling llms to generate fine-grained citations in long-context qa

    Jiajie Zhang, Yushi Bai, Xin Lv, Wanjun Gu, Danqing Liu, Minhao Zou, Shulin Cao, Lei Hou, Yuxiao Dong, Ling Feng, and Juanzi Li. Longcite: Enabling llms to generate fine-grained citations in long-context qa. arXiv preprint arXiv:2409.02897, 2024 a

    2024 arXiv

  52. [60]

    Conditional language learning with context

    Xiao Zhang, Miao Li, and Ji Wu. Conditional language learning with context. arXiv preprint arXiv:2406.01976, 2024 b

    2024 arXiv

  53. [61]

    inftybench: Extending long context evaluation beyond 100k tokens

    Xinrong Zhang, Yingfa Chen, Shengding Hu, Zihang Xu, Junhao Chen, Moo Khai Hao, Xu Han, Zhen Leng Thai, Shuo Wang, Zhiyuan Liu, et al. inftybench: Extending long context evaluation beyond 100k tokens. arXiv preprint arXiv:2402.13718, 2024 c

    2024 arXiv

  54. [62]

    Longskywork: A training recipe for efficiently extending context length in large language models

    Liang Zhao, Tianwen Wei, Liang Zeng, Cheng Cheng, Liu Yang, Peng Cheng, Lijie Wang, Chenxia Li, Xuejie Wu, Bo Zhu, et al. Longskywork: A training recipe for efficiently extending context length in large language models. arXiv preprint arXiv:2406.00605, 2024 a

    2024 arXiv

  55. [63]

    Analysing the impact of sequence composition on language model pre-training

    Yu Zhao, Yuanbin Qu, Konrad Staniszewski, Szymon Tworkowski, Wei Liu, Piotr Mi o \'s , Yuxiang Wu, and Pasquale Minervini. Analysing the impact of sequence composition on language model pre-training. arXiv preprint arXiv:2402.13991, 2024 b

    2024 arXiv

  56. [64]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

  57. [65]

    @esa (Ref

    \@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should ...

  58. [66]

    \@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@firs...

  59. [67]

    @open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibset...

    2000

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.