{"id":"aec92929-8d57-4004-915b-d82156ce86c9","arxiv_id":"2411.13476","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"BFloat16 rounding breaks RoPE's positional shift invariance, and AnchorAttention, which fixes the first token as a shared anchor, improves long-context training accuracy and speed.","lead":"This paper shows that BFloat16, the standard low-precision format for training large language models, makes rotary position embeddings behave inconsistently at long sequence lengths, and proposes a simple attention change that improves long-context training. The method, AnchorAttention, uses the first token as a shared position anchor and masks attention between documents, and the authors report improved long-context benchmark scores with over 50% training time savings.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"No Float32 full-attention training baseline means the central causal claim—that BFloat16 is the operative cause of long-context degradation and AnchorAttention fixes it—is untested; gains may come from masking or the anchor-as-sink.","rationale":"The paper makes a credible empirical observation that BFloat16 introduces measurable deviations in RoPE's translation invariance, and AnchorAttention is a pragmatic method with consistent RULER gains and a real speedup. However, the central causal story—that BFloat16 is the reason long-context training underperforms and that AnchorAttention works by fixing that numerical error—is not directly tested. The absence of a Float32 full-attention training baseline means the gains could be attributed to the masking or the anchor-as-sink, and the paper itself labels the mechanism as a hypothesis. The reader's CONDITIONAL verdict is appropriate. My stress-test identifies the same missing control; therefore I do not change the verdict. I would recommend the authors run the Float32 baseline as a condition for acceptance.","tokens_in":30671,"tokens_out":5166,"duration_ms":49750,"concrete_test":"Re-run the LLaMA-2-7B 64K SlimPajama long-context training from Section 5.1 with the same 2B-token budget, batch size, learning rate, and RoPE base (5M) from Table 1, but replace the BFloat16 FlashAttention inner product with a Float32 reference implementation (e.g., a fused kernel with FP32 accumulation or non-fused FP32 attention at 64K). Evaluate on RULER at 4K–64K using the paper's checkpoint-averaging protocol (Section 4.2). If Float32 full-attention scores are statistically indistinguishable from BFloat16 full attention, the BFloat16 deviation is not the operative cause; if Float32 matches AnchorAttention, the gains come from masking or anchoring, not precision.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim (Abstract) is that BFloat16 breaks RoPE's relative positional encoding and that AnchorAttention 'alleviates numerical issues caused by BFloat16,' improving long-context training. The evidence for a BFloat16-specific degradation is only observational (Section 2.2, Fig. 1): a frozen pretrained LLaMA-2-7B shows nonzero attention-logit differences under constant positional shifts in BFloat16 vs Float32. But all training experiments in Sections 4-5 are run in BFloat16, with no Float32 full-attention training control. Consequently, the RULER gains of AnchorAttention over Full Attention (Table 5) could equally be explained by its two other design choices: cross-document masking (shared with Intra-Doc Attention) and a shared anchor token at position 0, which prior work shows can act as an attention sink (Xiao et al., 2023; Section 8 speculates on this). The paper itself flags the causal link as hypothesis in Section 3.2 ('this analysis is based on the hypothesis that RoPE functions in some way as an absolute positional encoding mechanism under BFloat16 precision') and calls for 'future work' in Section 8. Also, the numerical deviations in Fig. 1 are measured on attention scores of a frozen model; nothing shows these small perturbations translate into the large 64K-128K RULER differences (e.g., 73.25 vs 66.40 on SlimPajama-64K). A Float32 full-attention training baseline is the missing control that would separate precision effects from architectural effects.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper makes two claims. First, it presents an empirical diagnostic: for a frozen pretrained LLaMA-2-7B model, attention logits and scores are not invariant to a constant positional shift when the computation runs in BFloat16, whereas the invariance holds in Float32; the deviation grows with sequence length and is concentrated on the first token (Section 2.2, Figure 1). Second, it proposes AnchorAttention, a training-time attention scheme that repurposes the BOS token as a shared anchor with position ID 0, makes it visible to all documents in a packed long context, and masks attention between different documents. Experiments on LLaMA-2-7B, LLaMA-3-8B, Mistral-7B-v0.3, and Qwen-1.5-1.8B show that AnchorAttention improves RULER scores from 8K to 128K over full attention and standard intra-document attention (Tables 5 and 6), largely preserves MMLU and HellaSwag performance (Table 7), and reduces training time by more than 50% through attention sparsity (Figure 6). The paper interprets these gains as evidence that BFloat16 breaks RoPE's relative positional property and that AnchorAttention mitigates the resulting numerical problem, but the training experiments include no Float32 full-attention control, and the paper itself flags the causal interpretation as a hypothesis in Section 3.2.","tokens_in":30997,"tokens_out":17625,"duration_ms":169878,"significance":"If the causal claim were established, this would be a high-impact result for the field: nearly all large-scale long-context training currently runs in BFloat16, and a mechanism linking numeric format to positional-encoding fidelity would change how practitioners think about RoPE and FlashAttention-style kernels. The paper ships real value even without that causal story. The shift-invariance diagnostic in Section 2.2 is clean, falsifiable, and easy to reuse; the position-reset comparison (Figure 3) is an interesting empirical finding about intra-document attention; the shared-anchor method is simple, effective, and evaluated against external benchmarks, so the validation is not circular; the recommendation to average RULER over several checkpoints is a methodological improvement; and the released AnchorContext implementation, with its measured zero logits difference relative to the non-distributed baseline, is a concrete engineering contribution.","major_comments":[{"comment":"The central causal claim — that BFloat16 breaks RoPE and that AnchorAttention's gains come from alleviating this numerical issue — is not directly tested, because every training run in Section 5 uses BFloat16. The stress-test concern about the missing Float32 full-attention control lands: without it, the RULER differences in Table 5 (e.g., 73.25 vs. 66.40 at 64K on SlimPajama-64K; 66.15 vs. 62.75 at 128K on SlimPajama-128K) can be explained by the architectural changes alone, namely cross-document masking (shared with the Intra-Doc baseline) or the shared anchor acting as an attention sink, a possibility the paper itself raises in Section 8. The paper's own text concedes the interpretive gap: Section 3.2 states that the analysis \"is based on the hypothesis that RoPE functions in some way as an absolute positional encoding mechanism under BFloat16 precision,\" and Section 8 states that \"future work is needed to more rigorously investigate\" the first-position effect; the Introduction's claim that the reset result \"confirms that position ID inconsistencies are a key issue\" is stronger than this hypothesis language supports. I request a precision-controlled experiment: train full attention with the identical data, hyperparameters, and sequence-parallelism configuration but with Float32 (or at least FP32 attention logits), at least at a reduced context length, and compare against the BFloat16 full-attention run. A low-cost complement would be to rerun the Section 3.2 position-reset comparison in Float32 at a smaller scale: if the reset advantage disappears in Float32, that directly supports the precision explanation rather than the architectural explanation.","section":"§2.2, §3.2, §5.1 (Table 5)"},{"comment":"The mechanism attribution is underdetermined because AnchorAttention differs from the Full Attention baseline in three ways at once: cross-document masking, a shared anchor visible to all documents, and continuous position IDs without per-document reset. Table 5 ablates only the intra-document variants (with and without reset), so no experiment isolates the anchor's contribution. Concretely, the first-token numerical-error story predicts that the benefit should largely disappear when the anchor is assigned a nonzero fixed position or when the RoPE computation is kept at higher precision, whereas an attention-sink explanation (Xiao et al., 2023, cited in Section 8) predicts that the benefit is relatively insensitive to the anchor's position. I suggest three ablations: (a) an anchor at position 0 that is masked exactly like other tokens, keeping the shared-position-0 property without the special visibility; (b) a shared anchor with a fixed nonzero position ID; and (c) an otherwise identical run in which only the attention-logit computation is done in FP32. Relatedly, the premise in Section 3.3 that the BOS token \"lacks explicit semantic meaning\" is asserted, not tested; using a semantically meaningful token as the anchor would indicate whether the gain comes from concentrating error on a meaningless token or from the positional-reference effect itself.","section":"§3.3, §5.1 (Table 5)"},{"comment":"The quantitative bridge between the forward-pass diagnostic and training outcomes is missing. The measurements in Section 2.2 are attention-score and logit differences of a frozen model, and the paper does not report the magnitude of the deviations (the y-axis scale of Figure 1, right, is not given numerically) or show that they affect gradients or training dynamics. A parity-of-scale argument is therefore absent: nothing demonstrates that deviations of these sizes would produce the 64K–128K RULER gaps in Table 5. A cheap experiment would directly test the claimed mechanism while holding the architecture fixed: in a BFloat16 full-attention training run, clamp the first token's position ID to a fixed value (say 0) while leaving all other position IDs unchanged, and compare RULER to the standard run. If clamping the first token's position closes most of the gap between Full Attention and AnchorAttention, the first-token mechanism is supported; if not, the first-token narrative would need to be revised even though the method's empirical value remains.","section":"§2.2 and §5.1"}],"minor_comments":[{"comment":"The compiled text contains two overlapping versions of the same paper (the arXiv-style body and an embedded anonymous TMLR draft), with repeated passages such as the Introduction and Figure 2 appearing twice; the published version must be a single clean rendering.","section":"Entire manuscript"},{"comment":"\"we shop the ∆1 = 16 to improve visualization\" should read \"we skip the ∆1 = 16 point\"; the current wording appears to be a typo and is confusing in context.","section":"§2.2"},{"comment":"The color references in the Results Discussion are inconsistent with the Experimental Setup: the setup defines blue = pretrained BFloat16, yellow = pretrained Float32, and green = random BFloat16, but the discussion then says the Float32 comparison is shown by \"blue vs. green lines\" and the pretrained-vs-random BFloat16 comparison by \"blue vs. yellow lines,\" which swaps the two comparisons; please correct the parenthetical references.","section":"§2.2, Figure 1 (left)"},{"comment":"The text on FlexAttention says it \"facilitates flexible attention masks needed by interleaved chunks, as illustrated in Figure 6,\" but Figure 6 shows estimated training time; the intended cross-reference is evidently Figure 5, which illustrates interleaved chunks.","section":"§5.5"},{"comment":"The baseline row for LLaMA-2-7B reports LongBench ICL equal to 6.22, which is far below the trained-model range of 48.96–66.02 in the same table and also well below few-shot ICL values reported for this model elsewhere; please verify that the baseline was evaluated with the same ICL protocol and that the row aligns correctly with the column headers.","section":"§5.4, Table 7"},{"comment":"The RULER task sets differ across tables: LLaMA-2-7B is evaluated on 11 of 13 tasks (NIAH-Multikey-3 and CWE excluded), while the other models use all 13 tasks; the exclusion is documented and defensible, but the table captions should state the task counts so that the cross-model comparison is not read as being on a common task set, and the sentence \"we exclude two tasks (and NIAH-Multikey 3 and Common Word Extraction)\" should drop the stray \"and.\"","section":"§4.2, Tables 5–6"},{"comment":"The \"more than 50%\" training-time reduction should be qualified as arising from attention sparsity (masking), which is shared with intra-document attention, and the figure or caption should state the measurement conditions (GPU count, sequence-parallelism configuration, and whether batch size was held fixed across methods).","section":"§5.5, Figure 6"}],"recommendation":"major_revision","confidential_remarks":"Both the reader's conditional verdict and my own reading converge on the same crux: the missing Float32 full-attention training control. The practical method is solid and the empirical diagnostic is publishable, so I see a clear path to acceptance after the causal narrative is either substantiated by a precision-controlled experiment or appropriately weakened in the abstract and conclusion. I also note that the paper is unusually candid about its own hypothesis status (Sections 3.2 and 8), which should count in its favor during revision. The main risk in revision is scope: the authors might be tempted to soften the claims without adding the control; I would prefer the control experiment at a reduced context length over mere rewording."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know before you read this one. The first is genuine: the paper shows experimentally that BFloat16 breaks RoPE's shift invariance, the deviation concentrates on the first token, and it grows with sequence length. That is a real measurement on a frozen pretrained model, with a Float32 control. The second is that the causal bridge from that observation to the headline claim—that BFloat16 is why long-context training underperforms and AnchorAttention fixes it—is not fully built. There is no Float32 full-attention training baseline. All training is BFloat16, so the RULER gains of AnchorAttention could come from its two other design choices: cross-document masking and the shared anchor token acting as an attention sink. The paper itself admits this in Section 3.2 and flags rigorous validation as future work.\n\nWhat is good: AnchorAttention is a simple, practical plug-in that consistently beats full attention and intra-document attention on RULER across LLaMA-2, LLaMA-3, Mistral, and Qwen, with a >50% training time reduction and released code. The authors are honest about limitations, give full appendix results, and the design—fixed anchor at position 0 visible to all documents plus masking—is coherent. The ablation with position-ID reset suggests the anchor token itself adds value beyond just resetting, which is more than many method papers do.\n\nThe weaknesses are real but mostly fixable. A Float32 training baseline is the natural control: if the gap between full attention and AnchorAttention shrinks under Float32, the BF16 story gains support; if not, the method is still useful but the explanation is wrong. Second, an ablation separating the anchor from the masking would clarify attribution. Third, there are no error bars, and the exclusion of two RULER tasks for LLaMA-2 is post hoc, though full results are in the appendix. These are not fatal; they are the difference between a paper about a numerical phenomenon and a paper about a practical recipe.\n\nBottom line: the empirical observation is new and worth citing; the method is effective and reproducible; the causal narrative overreaches. A serious referee should engage, but the revision needs the Float32 baseline and a cleaner attribution before the headline claim is justified.","headline":"A real numerical observation about BFloat16 and RoPE, a practical method that works, but the causal story needs a Float32 baseline.","tokens_in":31529,"tokens_out":2594,"would_cite":true,"duration_ms":27609,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"BFloat16 breaks RoPE's relative-position property in long contexts, and a shared first-token anchor restores it.","keywords":["BFloat16","RoPE","relative positional encoding","long-context training","AnchorAttention","numerical precision","large language models"],"falsifier":"Train the same base model on the same long-context data with full attention in Float32 (or with RoPE rotations kept in higher precision inside the attention kernel) and compare RULER scores against BFloat16 full attention and against AnchorAttention; if Float32 full attention matches AnchorAttention, the BFloat16 deviation is the operative cause. A second check: recompute the shift-invariance metric D from Eq. 4 with the first token excluded, and see whether the remaining attention difference drops to the Float32 level.","tokens_in":30417,"feed_emoji":"⚓","tokens_out":8109,"duration_ms":74483,"temperature":0.7,"pith_summary":"This paper claims that BFloat16, the standard low-precision format for training large language models, breaks the relative-position property of Rotary Position Embedding (RoPE) during long-context training, even though that property holds in exact arithmetic. The failure is small per token but accumulates as the context grows, and the first token accounts for most of the deviation. The paper argues that this numerical breakdown is not benign: it creates inconsistent positional relationships across documents in long-context training, which shows up as degraded performance on long-context benchmarks. On this diagnosis the paper builds AnchorAttention, which designates the first token as a shared anchor with a fixed position ID, makes it visible to every document, and masks attention across documents. If the argument is right, a small change to the attention pattern should improve long-context performance, preserve general abilities, and cut training time by more than half.","feed_headline":"BFloat16 breaks RoPE's position logic in long contexts","feed_subtitle":"A shared first-token anchor restores long-context training and cuts training time by more than half.","key_machinery":"The carrying mechanism is AnchorAttention, an attention mask combined with a positional scheme. A single anchor token—the beginning-of-sequence token, or the end-of-sequence token for models without one—is assigned position ID 0 and is visible to every document in the context window, while tokens from different documents cannot attend to each other and tokens within a document keep continuous position IDs. This concentrates whatever BFloat16-induced error exists on a semantically empty token, gives every document a consistent positional reference, keeps the full rotational span of RoPE learnable even from sequences shorter than the context window, and reduces the number of attention computations. The diagnostic that motivates it is the shift-invariance test of Eq. 3: under exact RoPE, attention logits are invariant to a constant positional shift, and under BFloat16 they are not.","core_discovery":"The central discovery is that RoPE's relative positional encoding property—attention logits should depend only on the relative distance between two token positions, so that shifting every position by a constant leaves attention unchanged—fails under BFloat16 in pretrained models. With a pretrained LLaMA-2-7B, the paper measures a substantial attention difference between two constant positional shifts in BFloat16, while the same computation in Float32 gives zero difference; random initial weights in BFloat16 show a much smaller effect, indicating that pretraining amplifies the error. The deviation concentrates on the first token and grows with sequence length, and the paper traces it to BFloat16's limited mantissa: RoPE rotations are applied in higher precision, but the query-key inner product inside the attention kernel is computed in BFloat16, so the algebraic cancellation that makes RoPE relative no longer cancels. The paper then shows that resetting position IDs per document improves long-context performance, and proposes AnchorAttention as the mechanism that captures this benefit while keeping continuous position IDs and a shared anchor.","pith_inferences":["If the BFloat16 diagnosis is correct, other low-precision formats with even fewer mantissa bits, such as FP8, should show a stronger positional breakdown, and the shared-anchor idea may transfer directly.","Part of AnchorAttention's measured gain could come from the anchor acting as an attention sink rather than from numerical error removal; training with a shared anchor while keeping RoPE in Float32 inside the attention kernel would separate the two effects.","Because AnchorAttention masks cross-document attention, it may sacrifice performance on tasks that genuinely need cross-document reasoning, such as multi-document question answering, even if it wins on synthetic long-context benchmarks.","The paper's training budget (2B tokens, models at roughly 10B scale) is too small to show whether the numerical breakdown has the same practical importance in frontier-scale continued pretraining."],"forward_implications":["AnchorAttention should beat full attention and standard intra-document attention on the RULER benchmark at every tested length from 4K to 128K, with the largest gains at the longest contexts.","Training on the same infrastructure should take less than half the wall-clock time of full attention, because document masking removes many attention computations.","General abilities measured by HellaSwag, MMLU, and LongBench in-context learning should remain close to, and in several cases better than, the original base model.","The gap between training on raw SlimPajama and on a carefully upsampled long-sequence mixture should shrink, so AnchorAttention reduces the need for aggressive long-data upsampling.","Adding domain tags does not reliably improve AnchorAttention, and interleaving chunks of documents hurts when combined with cross-document masking."],"supporting_citations":[{"why":"Defines RoPE and establishes the relative positional encoding property that the paper claims BFloat16 breaks.","marker":"Su et al., 2021"},{"why":"The attention kernel that computes query-key inner products in BFloat16, identified as the source of the numerical deviation.","marker":"Dao, 2024"},{"why":"Introduces BFloat16, the low-precision format whose limited mantissa causes the breakdown.","marker":"Wang & Kanwar, 2019"},{"why":"Supplies the RULER benchmark used to measure long-context performance gains of AnchorAttention.","marker":"Hsieh et al., 2024"},{"why":"Provides the long-context training data-upsampling recipe and the 1B-token sufficiency claim used in experiments.","marker":"Fu et al., 2024"},{"why":"Establishes the effectiveness of intra-document attention for long-context training and the UDS protocol, serving as a baseline.","marker":"Gao et al., 2024"},{"why":"Introduces intra-document attention masking, the starting point AnchorAttention extends.","marker":"Zhao et al., 2024b"},{"why":"Supplies the training configuration and the Zigzag-Ring attention baseline used in the speed comparison.","marker":"Zhang, 2023"}],"fun_headline_variants":["BFloat16 breaks RoPE's relative position trick in long contexts","Shared anchor fixes BFloat16's RoPE drift for long-context LLMs","AnchorAttention: saving RoPE from BFloat16 precision in long training","How BFloat16 undermines RoPE and a shared anchor fixes it"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing assumption is that the BFloat16 attention deviation the paper measures is the operative cause of the long-context training gap, and not merely a correlated artifact, since AnchorAttention also changes attention masking and introduces an attention-sink-like anchor, and the paper itself flags its absolute-position hypothesis as needing rigorous validation.","fun_headline_variants_meta":{"raw":{"variants":["BFloat16 breaks RoPE's relative position trick in long contexts","Shared anchor fixes BFloat16's RoPE drift for long-context LLMs","AnchorAttention: saving RoPE from BFloat16 precision in long training","How BFloat16 undermines RoPE and a shared anchor fixes it"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000168,"raw_usage":{"total_tokens":1284,"prompt_tokens":989,"completion_tokens":295,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":605,"completion_tokens_details":{"reasoning_tokens":215}},"tokens_in":605,"tokens_out":295,"duration_ms":3529,"temperature":1.0,"reasoning_tokens":215,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T16:23:49.087461+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train the same base model on the same long-context data with full attention in Float32 (or with RoPE rotations kept in higher precision inside the attention kernel) and compare RULER scores against BFloat16 full attention and against AnchorAttention; if Float32 full attention matches AnchorAttention, the BFloat16 deviation is the operative cause. A second check: recompute the shift-invariance metric D from Eq. 4 with the first token excluded, and see whether the remaining attention difference drops to the Float32 level.","supporting_citations":[{"cited_title":"Bfloat16: The secret to high performance on cloud tpus","cited_arxiv_id":null,"evidence_quote":"Introduces BFloat16, the low-precision format whose limited mantissa causes the breakdown."}],"review_version":1}