Pith. sign in

REVIEW 4 major objections 5 minor 83 references

Cracks in the Foundation: Seemingly Minor Architectural Choices Impact Long Context Extension

T0 review · 4 major / 5 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Four individually minor architectural choices compound: three or more can cut long-context benchmark scores by up to 47%, and counting them predicts downstream performance ($R^2=0.61$).

desk verdict A well-run 26-model ablation shows architectural choices compound to hurt long-context extension, but the count-feature predictor and 47% headline are fit over a pool that isn't fully controlled—so the abstract oversells causality. read the letter →

arxiv 2608.10296 v1 pith:QGXPIFON submitted 2026-08-10 cs.CL

Amanda Bertsch , Luca Soldaini , Matthew R. Gormley , Graham Neubig , Hannaneh Hajishirzi , Kyle Lo , Dirk Groeneveld This is my paper · ORCID
classification cs.CL
keywords long-contextlanguagemodelsarchitecturalablationsQK-normgrouped-queryattentionslidingwindowpretrainingcontextlengthextensionOlmPool
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that long-context extensibility in dense transformers is shaped far more by small architectural decisions made before pretraining than by data, tokenizer, or extension recipe, and that these decisions compound. Across 26 controlled 7B models trained on identical data with identical optimization, any single one of four features—QK-norm, grouped-query attention, sliding-window attention, or a 4K pretraining context—costs only a point or two on long-context benchmarks, but combining three or more can cut HELMET at 32K by up to 26.5 points. Simply counting how many of these features a model carries predicts downstream long-context performance with leave-one-out $R^2=0.61$, outperforming any single architectural axis. Standard short-context loss, perplexity, and in-loop benchmarks cannot detect the difference, but an early context-extension run can.

What carries the argument

The load-bearing object is a count: how many of four 'long-context-inhibiting' features a model carries. The features are QK-norm (normalizing query and key vectors, either layerwise or per attention head), grouped-query attention (sharing a small number of key-value heads across many query heads), sliding-window attention (three of every four layers attending only to a 4096-token window), and a 4096-token pretraining context. The count acts as a single ordinal predictor of post-extension performance and outperforms a linear regression over the four individual axes. The paper also identifies a mechanism in the attention maps: models without QK-norm develop strong attention sinks, and sink strength correlates with better long-context performance ($R^2=0.38$).

What would settle it

A decisive check would be to train the worst architecture—GQA, sliding window, headwise QK-norm, and 4K pretraining—with an architecture-specific learning-rate schedule and continue extension well beyond 50B tokens; if it then matches the Llama-like architecture on HELMET at 32K, the claim that the features themselves, rather than optimization fit, limit extensibility would fail. Conversely, finding any model in the released pool with three or four features that scores at the top of the HELMET distribution under the same recipe would break the counting predictor.

Watch

Extended reading notes

Core claim

The central claim is that four architectural choices, each individually minor and each justified by training stability, pretraining throughput, or inference efficiency, together determine much of a transformer's ability to adapt to long contexts. Holding data, tokenizer, and extension recipe fixed, models with three or more of QK-norm, grouped-query attention, sliding-window attention, and a 4096-token pretraining context score up to 47% worse on long-context benchmarks than models with fewer such features. A simple count of the features present is the single most predictive variable for downstream HELMET at 32K (in-sample $R^2=0.67$, leave-one-out $R^2=0.61$), and the pattern persists when extension is lengthened to 50B tokens, when pretraining is extended to 2T tokens, and when the extension strategy is changed. The paper releases OlmPool, a suite of 26 data- and recipe-matched 7B models with checkpoints before and after extension, several of which extend better than the Llama 3 architecture.

Load-bearing premise

The conclusion rests on the four chosen architectural axes and their tested values standing in for the full range of real design choices; if another factor such as optimization hyperparameters or a different sliding-window ratio drives the long-context gap, the feature-count story would not transfer.

Editorial extensions

If this is right

  • Model developers who adopt QK-norm, grouped-query attention, sliding windows, or short pretraining contexts for efficiency or stability should expect a long-context penalty that grows with the number of features, not with any one choice.
  • Context-extension recipes tuned on a Llama-like architecture will tend to look best on that architecture; the same recipe applied to Qwen- or Olmo-like designs can understate both the architecture and the recipe.
  • Short-context loss and validation perplexity cannot substitute for a long-context probe; running a context extension early in pretraining is the cheapest reliable signal the paper finds.
  • Models carrying three or more of these features may need extension phases far beyond the 10B-token standard, since the paper observes no washout even at 50B extension tokens.
  • The OlmPool release itself is a resource: 26 data- and recipe-matched models with checkpoints before and after extension, usable for studying early-training phenomena beyond long context.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A testable extension the paper leaves open: if the feature count is the true driver, then removing any single feature from a four-feature model should recover roughly the same increment regardless of which feature is removed; the released checkpoints make this check possible.
  • The attention-sink correlation suggests an untested remedy: architectures that achieve training stability without suppressing attention sinks, for example by capping attention logits, may obtain both stability and long-context extensibility.
  • A further implication the authors do not draw: if short-context metrics are blind to this degradation, then the common practice of selecting architectures on short-context loss could systematically select for poor long-context extensibility; this prediction could be tested on existing released model pools.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. This paper reports a controlled pretraining study, OlmPool, of 26 comparable 7B transformer models that vary four architectural axes: QK-norm, grouped-query attention, sliding-window attention, and pretraining context length. Holding data, tokenizer, and context-extension recipe fixed, the authors find that short-context metrics poorly predict long-context HELMET/RULER/LongPPL performance after extension, that individual features have modest effects when isolated, and that the number of 'long-context-inhibiting' features present predicts downstream HELMET score with leave-one-out R² = 0.61. They also report robustness checks with a 50B-token extension, pretraining up to 2T tokens, an alternative YaRN recipe, and an analysis of attention-sink and retrieval-head behavior. The paper releases checkpoints for all runs.

Significance. If the central claim survives scrutiny, this is a valuable result for the LLM development community: it provides one of the first controlled, apples-to-apples demonstrations that architectural choices made before pretraining can have large, compounding consequences for long-context extensibility, and the released OlmPool suite (26 models, full checkpoints, 170k GPU hours) is a substantial community asset. The finding that standard short-context metrics fail to flag these differences is practically important, and the early-extension diagnostic is actionable. The strengths are the controlled setup, the breadth of the model pool, the use of multiple long-context benchmarks, and the public release of configurations and checkpoints. The main weakness is that the headline quantitative claims are computed over a non-factorial pool in which feature count is entangled with initialization, fp8, norm-order, and depth, so the causal 'compounding' interpretation needs additional support.

major comments (4)
  1. [Section 4; Tables 1–2; Appendix B] The count-of-features predictor (in-sample R²=0.67, LOO R²=0.61) is fit over the entire OlmPool, but the pool is not a factorial design. Post-sublayer norm is only trained with QK-norm, fp8 appears in a single run (and in rows 1, 7, 8, 13 of Table 1), initialization codes A–K vary across runs, and the Qwen-like model uses 36 layers rather than 32 (Appendix C). Appendix B reports that initialization alone can move long-context scores by up to 17% of the observed range, with a mean of 7.7%. Because the count regression and the 47% relative headline use all runs, 'number of features' may be absorbing uncontrolled differences that correlate with feature count. The authors should report the regression coefficients, test whether the count effect survives after residualizing or conditioning on init code, fp8, norm order, and depth, and show which matched subsets drive the effect.
  2. [Section 4; Appendix B] The paper states that it discusses only differences that are 'substantially larger' than the mean initialization swing, but several quantitative claims do not meet this bar. The claimed 1–2 point effects for SWA and 4K pretraining are comparable to the mean initialization swing (7.7% of the 26.5-point HELMET range is about 2.0 points), and the paired comparisons appear to be single runs. The larger QK-norm effect (6 points on the Olmo architecture) is above this floor, but the 'minor individual effects' narrative in the abstract and Section 4 is not uniformly supported by the reported numbers. Provide seed variance or reframe these as directional findings.
  3. [Sections 5.1 and 5.2] The robustness claims that architectural differences persist under longer extension and longer pretraining rest on very little data: Figure 4 uses one trio of runs at 1B/10B/50B extension, and Figure 5 uses one pair of runs at 70B/140B/280B/2T. With no seed replicates or learning-rate-schedule variation, the statement that differences 'remain relatively stable' is not quantitatively supported. These sections are secondary to the main claim, but they are cited as evidence that the effect is not merely a token-efficiency artifact; either add repeated runs or soften the conclusion to preliminary evidence.
  4. [Section 3] The correlations used to argue that short-context metrics are not predictive (e.g., R²=0.29 for training loss, R²=0.32 for pre-extension HELMET 8K) are computed over 26 runs that share data and optimization, so the effective sample size is smaller than 26 and no multiple-comparison correction is applied across the many metrics considered. The direction of the finding is clear and consistent, but the quantitative R² values should be described as descriptive rather than as formal evidence of non-predictiveness.
minor comments (5)
  1. [Abstract and Section 4] The abstract reports 'up to 47%' while the text and Figure 1 report a 26.5-point HELMET gap; please state explicitly that 47% is a relative drop between the best and worst runs, and define the denominator.
  2. [Abstract and Section 4] The claim that 'any one of these choices alone has a minor impact' is hard to reconcile with the 6-point QK-norm effect on the Olmo architecture and the GQA gradient in Figure 3; please define 'minor' relative to the observed range.
  3. [Figures 1, 7; Tables 1–2] The 'number of features' variable is used as a color key and as the main predictor, but it is never formally defined; please state the binarization rule for GQA (e.g., fewer than 32 KV heads) and add a feature-count column to Tables 1 and 2.
  4. [Appendix B] There are typos ('initalization' twice) and the appendix could be tightened; more importantly, the mapping from initialization codes A–K to the four paired comparisons mentioned in the text is not given.
  5. [Section 5.4] The attention-sink analysis reports R²=0.38 between sink attention and long-context performance over 26 models; this should be labeled exploratory, since no correction is made for the many attention statistics tested.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity identified: the core claims are controlled empirical comparisons, the count predictor is a LOO-validated post-hoc fit, and self-citations are not load-bearing.

full rationale

The derivation chain in this paper is empirical rather than analytic: 26 models are pretrained with data, tokenizer, optimizer, and extension recipe held fixed, and the resulting HELMET/RULER/LongPPL scores are compared. The count-of-features predictor is explicitly fit to those scores and validated with leave-one-out cross-validation (LOO R2=0.61), so it is a post-hoc descriptive regression, not a quantity defined in terms of the outcome; no equation in the paper defines a feature count from HELMET or vice versa. The 'up to 47%' figure is presented as the observed range between the best and worst runs in Figure 1 and Tables 1-2, not as a prediction from the model. The attention-sink and retrieval-head analyses are exploratory correlations and are not used to derive the main effect. Self-citations to the Olmo 3 technical report supply the training configuration (learning rate schedule, Longmino data, SWA pattern) and are methodological inputs to the controlled experiments rather than evidence for the causal conclusion; the causal conclusion rests on the newly reported paired runs. The paper's own Appendix B discloses that initialization can move long-context scores by up to 17% of the observed range and that GQA degree lacks an initialization-controlled trial; this is a real confound and validity limitation for the magnitude claims, because the 47% gap is between runs differing in init, fp8, depth, and norm order, but confounding is not circularity, and no fitted parameter is renamed as a prediction. The omission of the coefficient table for the count regression and the four-axis linear regression also hampers auditability, but this concerns evidence completeness, not circularity. Accordingly, no circular step can be quoted with a specific reduction, and the appropriate score is 0.

Assumptions & free parameters 0 free parameters · 0 assumptions · 0 invented entities

This is an empirical study, not a derivation. There are no fitted constants or invented entities. The paper selects four architectural axes and specific values grounded in prior model families; those are experimental design choices, not free parameters in a model. The feature-count regression is a post hoc predictor validated by LOO-CV, and its coefficients (intercept and slope) are not reported in the text, so they cannot be audited as fitted values.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Cracks in the Foundation: Seemingly Minor Architectural Choices Impact Long Context Extension." pith.science (2026). https://pith.science/paper/QGXPIFON

@misc{pith2026260810296,
  author       = {Pith},
  title        = {Pith review of: Cracks in the Foundation: Seemingly Minor Architectural Choices Impact Long Context Extension},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/QGXPIFON}},
  note         = {Machine review of arXiv:2608.10296}
}
read the original abstract

One might imagine that architectural variations within the dense transformer paradigm have a limited effect on accuracy. However, we demonstrate that this is not the case in the long context setting. Specifically, we show that a set of four minor architectural decisions --- all made by at least one of the Olmo, Llama, and Qwen dense model families --- have a compoundingly negative effect on long context extensibility. Any one of these choices alone has a minor impact on long context performance, but combining three or more can drop the performance downstream by up to 47%. Furthermore, these differences are not detectable from short-context loss or validation datasets. We show that much of the variation in long context ability across model families is driven by these architectural features and detectable from applying context extension early in pretraining. We demonstrate this with controlled ablations that hold data, tokenizer, and extension recipe fixed while varying normalization, GQA, pretraining context length, and sliding window attention. After over 170,000 GPU hours of training, we release the resulting set of models as OlmPool, a set of 26 comparable 7B models with checkpoints before and after long-context extension. This pool includes several architectures that outperform the Llama 3 architecture on long context extensibility. In an analysis of our ablation models, we identify patterns in attention sink behavior and attention distributions across context that are attributable to specific architectural differences.

Figures

Figures reproduced from arXiv: 2608.10296 by the authors.

Figure 1
Figure 1. HELMET 32K scores across all OlmPool models with identical data and context [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Benchmark scores pre-extension largely fail to predict long-context benchmark [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. GQA is harmful to long context performance in our setting, even when adjusting [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (8 more)
Figure 4
Figure 4. Figure 4: Performance on HELMET after 1B, 10B, or 50B token extension for three repre￾sentative runs (the worst architecture, the Olmo 3 architecture, and the Llama 3 archi￾tecture). Longer context extension fails to wash out architectural differences. 70B 140B 280B 2T 0 20 40 6…
Figure 6
Figure 6. Figure 6: Solid color indicates the YaRN 2- stage extension performance; shaded is the NTK 1-stage extension performance. To understand if the differences we observe are sensitive to context extension strategy, we consider an alternate long context extension recipe. We perform a…
Figure 7
Figure 7. Figure 7: Attention entropy and the presence of a a strong attention sink both cluster by [PITH_FULL_IMAGE:figures/full_fig_p009_7.png]
Figure 8
Figure 8. Figure 8: QK norm is harmful for long context (as first observed by [PITH_FULL_IMAGE:figures/full_fig_p023_8.png]
Figure 9
Figure 9. Figure 9: Training loss does not correlate well with long context ability, in either the [PITH_FULL_IMAGE:figures/full_fig_p025_9.png]
Figure 10
Figure 10. Figure 10: R 2 between each standard validation perplexity metric and downstream long￾context metrics (HELMET 32K, RULER 32K, LongPPL). 26 [PITH_FULL_IMAGE:figures/full_fig_p026_10.png]
Figure 11
Figure 11. Figure 11: R 2 between each short-context benchmark metric and downstream long-context metrics (HELMET 32K, RULER 32K, LongPPL). 28 [PITH_FULL_IMAGE:figures/full_fig_p028_11.png]
Figure 12
Figure 12. Figure 12: Spike score vs. HELMET 32K across OlmPool. Stability weakly correlates with [PITH_FULL_IMAGE:figures/full_fig_p029_12.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

83 extracted references · 39 canonical work pages

  1. [1]

    Attention is All you Need , url =

    Vaswani, Ashish and Shazeer, Noam and Parmar, Niki and Uszkoreit, Jakob and Jones, Llion and Gomez, Aidan N and Kaiser, ukasz and Polosukhin, Illia , booktitle =. Attention is All you Need , url =

  2. [2]

    2018 , eprint=

    An Empirical Model of Large-Batch Training , author=. 2018 , eprint=

    2018

  3. [3]

    2025 , eprint=

    Critical Batch Size Revisited: A Simple Empirical Approach to Large-Batch Language Model Training , author=. 2025 , eprint=

    2025

  4. [4]

    2025 , eprint=

    How Does Critical Batch Size Scale in Pre-training? , author=. 2025 , eprint=

    work page 2025

  5. [5]

    Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfer , url =

    Yang, Ge and Hu, Edward and Babuschkin, Igor and Sidor, Szymon and Liu, Xiaodong and Farhi, David and Ryder, Nick and Pachocki, Jakub and Chen, Weizhu and Gao, Jianfeng , booktitle =. Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfer , url =

    work page

  6. [6]

    2025 , eprint=

    Why do LLMs attend to the first token? , author=. 2025 , eprint=

    work page 2025

  7. [7]

    2025 , eprint=

    When Attention Sink Emerges in Language Models: An Empirical View , author=. 2025 , eprint=

    2025

  8. [8]

    2026 , eprint=

    YaRN: Efficient Context Window Extension of Large Language Models , author=. 2026 , eprint=

    work page 2026

Show all 83 references
  1. [9]

    2016 , eprint=

    Pointer Sentinel Mixture Models , author=. 2016 , eprint=

    2016

  2. [10]

    2025 , url =

    Marin 32B Retrospective , author =. 2025 , url =

    2025

  3. [11]

    2025 , eprint=

    Chameleon: Mixed-Modal Early-Fusion Foundation Models , author=. 2025 , eprint=

    2025

  4. [12]

    2021 , eprint=

    Evaluating Large Language Models Trained on Code , author=. 2021 , eprint=

    2021

  5. [13]

    Think you have Solved Question Answering?

    Clark, Peter and Cowhey, Isaac and Etzioni, Oren and Khot, Tushar and Sabharwal, Ashish and Schoenick, Carissa and Tafjord, Oyvind , journal=. Think you have Solved Question Answering?

  6. [14]

    2019 , doi=

    Zellers, Rowan and Holtzman, Ari and Bisk, Yonatan and Farhadi, Ali and Choi, Yejin , booktitle=. 2019 , doi=

    2019

  7. [15]

    Proceedings of the International Conference on Learning Representations (ICLR) , year=

    Measuring Massive Multitask Language Understanding , author=. Proceedings of the International Conference on Learning Representations (ICLR) , year=

  8. [16]

    Liu , title =

    Colin Raffel and Noam Shazeer and Adam Roberts and Katherine Lee and Sharan Narang and Michael Matena and Yanqi Zhou and Wei Li and Peter J. Liu , title =. Journal of Machine Learning Research , year =

  9. [17]

    World Englishes , year =

    Greenbaum, Sidney and Nelson, Gerald , title =. World Englishes , year =

  10. [18]

    2022 , doi =

    Reid, Machel and Zhong, Victor and Gururangan, Suchin and Zettlemoyer, Luke , booktitle =. 2022 , doi =

    2022

  11. [19]

    Gao, Leo and Biderman, Stella and Black, Sid and Golding, Laurence and Hoppe, Travis and Foster, Charles and Phang, Jason and He, Horace and Thite, Anish and Nabeshima, Noa and Presser, Shawn and Leahy, Connor , journal =. The

  12. [20]

    Dolma: an Open Corpus of Three Trillion Tokens for Language Model Pretraining Research

    Soldaini, Luca and Kinney, Rodney and Bhagia, Akshita and Schwenk, Dustin and Atkinson, David and Authur, Russell and Bogin, Ben and Chandu, Khyathi and Dumas, Jennifer and Elazar, Yanai and Hofmann, Valentin and Jha, Ananya and Kumar, Sachin and Lucy, Li and Lyu, Xinxi and La...

    2024

  13. [21]

    2025 , eprint=

    Peri-LN: Revisiting Normalization Layer in the Transformer Architecture , author=. 2025 , eprint=

    2025

  14. [22]

    2026 , eprint=

    Not-Just-Scaling Laws: Towards a Better Understanding of the Downstream Impact of Language Model Design Decisions , author=. 2026 , eprint=

    2026

  15. [23]

    2025 , eprint=

    Gemma 3 Technical Report , author=. 2025 , eprint=

    2025

  16. [24]

    2025 , eprint=

    Qwen3 Technical Report , author=. 2025 , eprint=

    2025

  17. [25]

    2023 , eprint=

    GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints , author=. 2023 , eprint=

    2023

  18. [26]

    and Zheng, Lucia and Guha, Neel and Manning, Christopher D

    Henderson*, Peter and Krass*, Mark S. and Zheng, Lucia and Guha, Neel and Manning, Christopher D. and Jurafsky, Dan and Ho, Daniel E. , title =

  19. [27]

    2021 , eprint=

    Efficient Attentions for Long Document Summarization , author=. 2021 , eprint=

    2021

  20. [28]

    2024 , eprint=

    The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale , author=. 2024 , eprint=

    2024

  21. [29]

    2024 , eprint=

    Efficient Streaming Language Models with Attention Sinks , author=. 2024 , eprint=

    2024

  22. [30]

    The Thirty-ninth Annual Conference on Neural Information Processing Systems , year=

    Gated Attention for Large Language Models: Non-linearity, Sparsity, and Attention-Sink-Free , author=. The Thirty-ninth Annual Conference on Neural Information Processing Systems , year=

  23. [31]

    2025 , eprint=

    Differential Transformer , author=. 2025 , eprint=

    2025

  24. [32]

    2026 , eprint=

    A Unified View of Attention and Residual Sinks: Outlier-Driven Rescaling is Essential for Transformer Training , author=. 2026 , eprint=

    2026

  25. [33]

    2023 , eprint=

    Quantizable Transformers: Removing Outliers by Helping Attention Heads Do Nothing , author=. 2023 , eprint=

    2023

  26. [34]

    arXiv preprint arXiv:2108.07732 , year=

    Program Synthesis with Large Language Models , author=. arXiv preprint arXiv:2108.07732 , year=

    arXiv

  27. [35]

    Advances in Neural Information Processing Systems , volume =

    Lewkowycz, Aitor and Andreassen, Anders and Dohan, David and Dyer, Ethan and Michalewski, Henryk and Ramasesh, Vinay and Slone, Ambrose and Anil, Cem and Schlag, Imanol and Gutman-Solo, Theo and Wu, Yuhuai and Neyshabur, Behnam and Gur-Ari, Guy and Misra, Vedant , title =. Adv...

  28. [36]

    2025 , eprint=

    2 OLMo 2 Furious , author=. 2025 , eprint=

    2025

  29. [37]

    ArXiv , year=

    Holistically Evaluating the Environmental Impact of Creating Language Models , author=. ArXiv , year=

  30. [38]

    2025 , eprint=

    Gated Delta Networks: Improving Mamba2 with Delta Rule , author=. 2025 , eprint=

    2025

  31. [39]

    2023 , eprint=

    RWKV: Reinventing RNNs for the Transformer Era , author=. 2023 , eprint=

    2023

  32. [40]

    2024 , eprint=

    Mamba: Linear-Time Sequence Modeling with Selective State Spaces , author=. 2024 , eprint=

    2024

  33. [41]

    2024 , eprint=

    Retrieval Head Mechanistically Explains Long-Context Factuality , author=. 2024 , eprint=

    2024

  34. [42]

    Fine-Tune on the Format: First Improving Multiple-Choice Evaluation for Intermediate LLM Checkpoints

    Bunn, Alec and Wiegreffe, Sarah and Bogin, Ben. Fine-Tune on the Format: First Improving Multiple-Choice Evaluation for Intermediate LLM Checkpoints. Proceedings of the Fourth Workshop on Generation, Evaluation and Metrics (GEM ). 2025

    2025

  35. [43]

    2024 , eprint=

    Data Engineering for Scaling Language Models to 128K Context , author=. 2024 , eprint=

    2024

  36. [44]

    2024 , eprint=

    LongRecipe: Recipe for Efficient Long Context Generalization in Large Language Models , author=. 2024 , eprint=

    2024

  37. [45]

    2024 , eprint=

    LongRoPE: Extending LLM Context Window Beyond 2 Million Tokens , author=. 2024 , eprint=

    2024

  38. [46]

    2024 , eprint=

    LongSkywork: A Training Recipe for Efficiently Extending Context Length in Large Language Models , author=. 2024 , eprint=

    2024

  39. [47]

    2024 , eprint=

    LongLoRA: Efficient Fine-tuning of Long-Context Large Language Models , author=. 2024 , eprint=

    2024

  40. [48]

    2025 , eprint=

    How to Train Long-Context Language Models (Effectively) , author=. 2025 , eprint=

    2025

  41. [49]

    2024 , eprint=

    A Controlled Study on Long Context Extension and Generalization in LLMs , author=. 2024 , eprint=

    2024

  42. [50]

    Effective Long-Context Scaling of Foundation Models

    Xiong, Wenhan and Liu, Jingyu and Molybog, Igor and Zhang, Hejia and Bhargava, Prajjwal and Hou, Rui and Martin, Louis and Rungta, Rashi and Sankararaman, Karthik Abinav and Oguz, Barlas and Khabsa, Madian and Fang, Han and Mehdad, Yashar and Narang, Sharan and Malik, Kshitiz ...

    2024

  43. [51]

    2025 , eprint=

    APTBench: Benchmarking Agentic Potential of Base LLMs During Pre-Training , author=. 2025 , eprint=

    2025

  44. [52]

    2025 , eprint=

    OctoThinker: Mid-training Incentivizes Reinforcement Learning Scaling , author=. 2025 , eprint=

    2025

  45. [53]

    2023 , eprint=

    Training Trajectories of Language Models Across Scales , author=. 2023 , eprint=

    2023

  46. [54]

    2025 , eprint =

    Rope to Nope and Back Again: A New Hybrid Attention Strategy , author =. 2025 , eprint =

    2025

  47. [55]

    Advances in Neural Information Processing Systems , volume =

    The Impact of Positional Encoding on Length Generalization in Transformers , author =. Advances in Neural Information Processing Systems , volume =. 2023 , url =

    2023

  48. [56]

    The Tenth International Conference on Learning Representations , year =

    Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation , author =. The Tenth International Conference on Learning Representations , year =

  49. [57]

    2023 , eprint =

    Extending Context Window of Large Language Models via Positional Interpolation , author =. 2023 , eprint =

    2023

  50. [58]

    Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages =

    A Length-Extrapolatable Transformer , author =. Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages =. 2023 , address =. doi:10.18653/v1/2023.acl-long.816 , url =

    2023 doi

  51. [59]

    2023 , eprint =

    On the Long Range Abilities of Transformers , author =. 2023 , eprint =

    2023

  52. [60]

    Findings of the Association for Computational Linguistics: ACL 2024 , pages =

    Length Generalization of Causal Transformers without Position Encoding , author =. Findings of the Association for Computational Linguistics: ACL 2024 , pages =. 2024 , address =. doi:10.18653/v1/2024.findings-acl.834 , url =

    2024 doi

  53. [61]

    Le and Ruslan Salakhutdinov , booktitle =

    Zihang Dai and Zhilin Yang and Yiming Yang and Jaime Carbonell and Quoc V. Le and Ruslan Salakhutdinov , booktitle =. 2019 , address =. doi:10.18653/v1/P19-1285 , url =

    2019 doi

  54. [62]

    Megalodon: Efficient

    Xuezhe Ma and Xiaomeng Yang and Wenhan Xiong and Beidi Chen and Lili Yu and Hao Zhang and Jonathan May and Luke Zettlemoyer and Omer Levy and Chunting Zhou , booktitle =. Megalodon: Efficient. 2024 , url =

    2024

  55. [63]

    2023 , eprint =

    Advancing Transformer Architecture in Long-Context Large Language Models: A Comprehensive Survey , author =. 2023 , eprint =

    2023

  56. [64]

    Chiu and Siyu Ren and Fei Yuan and Wenting Zhao and Zhiyong Wu and Alexander M

    Yi Lu and Jing Nathan Yan and Songlin Yang and Justin T. Chiu and Siyu Ren and Fei Yuan and Wenting Zhao and Zhiyong Wu and Alexander M. Rush , year =. A Controlled Study on Long Context Extension and Generalization in. 2409.12181 , archivePrefix =

    arXiv

  57. [65]

    2022 , eprint=

    Training Compute-Optimal Large Language Models , author=. 2022 , eprint=

    2022

  58. [66]

    2023 , eprint=

    Effective Long-Context Scaling of Foundation Models , author=. 2023 , eprint=

    2023

  59. [67]

    2025 , author =

    Olmo 3: Charting a Path Through the Model Flow to Lead Open-Source AI , institution =. 2025 , author =

    2025

  60. [68]

    2024 , eprint=

    The Llama 3 Herd of Models , author=. 2024 , eprint=

    2024

  61. [69]

    Proceedings of the 40th International Conference on Machine Learning , pages =

    Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling , author =. Proceedings of the 40th International Conference on Machine Learning , pages =. 2023 , volume =

    2023

  62. [70]

    Hwang and Luca Soldaini and Akshita Bhagia and Jiacheng Liu and Dirk Groeneveld and Oyvind Tafjord and Noah A

    Ian Magnusson and Nguyen Tai and Ben Bogin and David Heineman and Jena D. Hwang and Luca Soldaini and Akshita Bhagia and Jiacheng Liu and Dirk Groeneveld and Oyvind Tafjord and Noah A. Smith and Pang Wei Koh and Jesse Dodge , booktitle =. 2025 , volume =

    2025

  63. [71]

    2025 , url =

    Howard Yen and Tianyu Gao and Minmin Hou and Ke Ding and Daniel Fleischer and Peter Izsak and Moshe Wasserblat and Danqi Chen , booktitle =. 2025 , url =

    2025

  64. [72]

    Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , year =

    How to Train Long-Context Language Models (Effectively) , author =. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , year =. doi:10.18653/v1/2025.acl-long.366 , url =

    2025 doi

  65. [73]

    2024 , url =

    Cheng-Ping Hsieh and Simeng Sun and Samuel Kriman and Shantanu Acharya and Dima Rekesh and Fei Jia and Boris Ginsburg , booktitle =. 2024 , url =

    2024

  66. [74]

    Transactions of the Association for Computational Linguistics , volume =

    Lost in the Middle: How Language Models Use Long Contexts , author =. Transactions of the Association for Computational Linguistics , volume =. 2024 , publisher =. doi:10.1162/tacl_a_00638 , url =

    2024 doi

  67. [75]

    Smith and Kyle Richardson and Jesse Dodge , booktitle =

    Ian Magnusson and Akshita Bhagia and Valentin Hofmann and Luca Soldaini and Ananya Harsh Jha and Oyvind Tafjord and Dustin Schwenk and Evan Pete Walsh and Yanai Elazar and Kyle Lo and Dirk Groeneveld and Iz Beltagy and Hannaneh Hajishirzi and Noah A. Smith and Kyle Richardson ...

    2024

  68. [76]

    The Thirteenth International Conference on Learning Representations , year =

    What is Wrong with Perplexity for Long-context Language Modeling? , author =. The Thirteenth International Conference on Learning Representations , year =

  69. [77]

    Why Has Predicting Downstream Capabilities of Frontier

    Rylan Schaeffer and Hailey Schoelkopf and Brando Miranda and Gabriel Mukobi and Varun Madan and Adam Ibrahim and Herbie Bradley and Stella Biderman and Sanmi Koyejo , booktitle =. Why Has Predicting Downstream Capabilities of Frontier. 2025 , volume =

    2025

  70. [78]

    Smith and Hannaneh Hajishirzi , booktitle =

    Dirk Groeneveld and Iz Beltagy and Evan Walsh and Akshita Bhagia and Rodney Kinney and Oyvind Tafjord and Ananya Jha and Hamish Ivison and Ian Magnusson and Yizhong Wang and Shane Arora and David Atkinson and Russell Authur and Khyathi Chandu and Arman Cohan and Jennifer Dumas...

    2024

  71. [79]

    2312.06550 , archivePrefix =

    Zhengzhong Liu and Aurick Qiao and Willie Neiswanger and Hongyi Wang and Bowen Tan and Tianhua Tao and Junbo Li and Yuqi Wang and Suqi Sun and Omkar Pangarkar and Richard Fan and Yi Gu and Victor Miller and Yonghao Zhuang and Guowei He and Haonan Li and Fajri Koto and Liping T...

    arXiv

  72. [80]

    2503.15450 , archivePrefix =

    Tongyao Zhu and Qian Liu and Haonan Wang and Shiqi Chen and Xiangming Gu and Tianyu Pang and Min-Yen Kan , year =. 2503.15450 , archivePrefix =

  73. [81]

    2024 , publisher =

    Yushi Bai and Xin Lv and Jiajie Zhang and Hongchang Lyu and Jiankai Tang and Zhidian Huang and Zhengxiao Du and Xiao Liu and Aohan Zeng and Lei Hou and Yuxiao Dong and Jie Tang and Juanzi Li , booktitle =. 2024 , publisher =. doi:10.18653/v1/2024.acl-long.172 , url =

    2024 doi

  74. [82]

    2022 , eprint =

    Training Compute-Optimal Large Language Models , author =. 2022 , eprint =

    2022

  75. [83]

    Advances in Neural Information Processing Systems , year =

    Signal and Noise: A Framework for Reducing Uncertainty in Language Model Evaluation , author =. Advances in Neural Information Processing Systems , year =

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.