{"id":"25336c32-a399-46e7-ae5f-dd5ed4a632c2","arxiv_id":"2608.10296","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Architectural details, especially QK-normalization, GQA, sliding windows, and 4K pretraining context, compound to degrade downstream long-context performance in a controlled 26-model pool.","lead":"Four small architectural choices, each made by at least one popular model family, can compound to cut long-context performance by up to 47% even when data, tokenizer, and training recipe are held fixed. The release of 26 controlled 7B models, called OlmPool, gives practitioners a public testbed for how architectural details change context extension behavior.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Count-of-features predictor and 47% headline are computed over a non-factorial pool in which feature count is entangled with norm order, fp8, initialization, and depth; the paper's own Appendix B reports init swings up to 17% of the range.","rationale":"The reader's conditional verdict is appropriate, and I do not think the paper should be rejected. The controlled pairwise comparisons, the released checkpoints, and the honest appendix disclosures are real evidence for the core finding that architecture matters for long-context extensibility. My concern targets the strongest quantitative form of the claim: the count-of-features regression and the 47% headline. These are computed over a pool that is not a full factorial, and the pool includes at least four additional sources of variation the paper itself flags: post-sublayer norm is entangled with QK norm, fp8 appears only in the worst-scoring run, initialization codes vary, and the Qwen-like model has a different depth. Appendix B's own initialization-swing analysis (up to 17% of the range) shows these confounds are not negligible relative to the small individual effects claimed for SWA and pretraining context length. The paper's guardrail about discussing only effects substantially larger than the mean initialization swing is not applied to the count model or the headline extreme, so the strongest quantitative claim is less secure than the qualitative conclusion. The proposed reanalysis of the released checkpoints is feasible and would settle whether the count model survives confound adjustment and whether the 47% gap is reproducible in a matched comparison.","tokens_in":26384,"tokens_out":11685,"duration_ms":109088,"concrete_test":"Using the released OlmPool checkpoints, refit the HELMET-32K count regression with norm order, fp8, initialization code, layer depth, and parameter count as covariates, reporting partial leave-one-out R2 for the count variable. Then locate the closest matched pair for the headline gap (same init, norm, fp8, and depth, differing only in counted features) and recompute the HELMET difference. If the count's partial LOO R2 falls below roughly 0.3 or the matched gap falls substantially below the reported 26.5 points, the compound-effect claim should be reworded as a pool-level correlation rather than a causal effect of feature count.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central quantitative claim is that counting four architectural features gives leave-one-out R2 = 0.61 and that three or more features can cost up to 47% on HELMET. This conclusion is drawn from 26 runs that are not a full factorial design: post-sublayer norm is only trained with QK norm, fp8 appears in a single run, initialization codes vary across runs, and the Qwen-like model uses 36 layers instead of 32 (Appendix C footnote). Appendix B reports that initialization alone can move long-context scores by up to 17% of the observed range, yet the count regression is fit over all runs and the 47% gap is between the best and worst rows, which differ in all of these uncontrolled factors as well as in the four counted axes. The paper states that discussed differences should be substantially larger than the mean initialization swing, but this guardrail is not applied to the count model or to the headline extreme. Without separating these confounds, 'number of features' may be a proxy for how many uncontrolled deviations a run has rather than a causal compound effect. The missing coefficient table makes this hard to audit from the manuscript.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper reports a controlled pretraining study, OlmPool, of 26 comparable 7B transformer models that vary four architectural axes: QK-norm, grouped-query attention, sliding-window attention, and pretraining context length. Holding data, tokenizer, and context-extension recipe fixed, the authors find that short-context metrics poorly predict long-context HELMET/RULER/LongPPL performance after extension, that individual features have modest effects when isolated, and that the number of 'long-context-inhibiting' features present predicts downstream HELMET score with leave-one-out R² = 0.61. They also report robustness checks with a 50B-token extension, pretraining up to 2T tokens, an alternative YaRN recipe, and an analysis of attention-sink and retrieval-head behavior. The paper releases checkpoints for all runs.","tokens_in":26598,"tokens_out":6458,"duration_ms":60392,"significance":"If the central claim survives scrutiny, this is a valuable result for the LLM development community: it provides one of the first controlled, apples-to-apples demonstrations that architectural choices made before pretraining can have large, compounding consequences for long-context extensibility, and the released OlmPool suite (26 models, full checkpoints, 170k GPU hours) is a substantial community asset. The finding that standard short-context metrics fail to flag these differences is practically important, and the early-extension diagnostic is actionable. The strengths are the controlled setup, the breadth of the model pool, the use of multiple long-context benchmarks, and the public release of configurations and checkpoints. The main weakness is that the headline quantitative claims are computed over a non-factorial pool in which feature count is entangled with initialization, fp8, norm-order, and depth, so the causal 'compounding' interpretation needs additional support.","major_comments":[{"comment":"The count-of-features predictor (in-sample R²=0.67, LOO R²=0.61) is fit over the entire OlmPool, but the pool is not a factorial design. Post-sublayer norm is only trained with QK-norm, fp8 appears in a single run (and in rows 1, 7, 8, 13 of Table 1), initialization codes A–K vary across runs, and the Qwen-like model uses 36 layers rather than 32 (Appendix C). Appendix B reports that initialization alone can move long-context scores by up to 17% of the observed range, with a mean of 7.7%. Because the count regression and the 47% relative headline use all runs, 'number of features' may be absorbing uncontrolled differences that correlate with feature count. The authors should report the regression coefficients, test whether the count effect survives after residualizing or conditioning on init code, fp8, norm order, and depth, and show which matched subsets drive the effect.","section":"Section 4; Tables 1–2; Appendix B"},{"comment":"The paper states that it discusses only differences that are 'substantially larger' than the mean initialization swing, but several quantitative claims do not meet this bar. The claimed 1–2 point effects for SWA and 4K pretraining are comparable to the mean initialization swing (7.7% of the 26.5-point HELMET range is about 2.0 points), and the paired comparisons appear to be single runs. The larger QK-norm effect (6 points on the Olmo architecture) is above this floor, but the 'minor individual effects' narrative in the abstract and Section 4 is not uniformly supported by the reported numbers. Provide seed variance or reframe these as directional findings.","section":"Section 4; Appendix B"},{"comment":"The robustness claims that architectural differences persist under longer extension and longer pretraining rest on very little data: Figure 4 uses one trio of runs at 1B/10B/50B extension, and Figure 5 uses one pair of runs at 70B/140B/280B/2T. With no seed replicates or learning-rate-schedule variation, the statement that differences 'remain relatively stable' is not quantitatively supported. These sections are secondary to the main claim, but they are cited as evidence that the effect is not merely a token-efficiency artifact; either add repeated runs or soften the conclusion to preliminary evidence.","section":"Sections 5.1 and 5.2"},{"comment":"The correlations used to argue that short-context metrics are not predictive (e.g., R²=0.29 for training loss, R²=0.32 for pre-extension HELMET 8K) are computed over 26 runs that share data and optimization, so the effective sample size is smaller than 26 and no multiple-comparison correction is applied across the many metrics considered. The direction of the finding is clear and consistent, but the quantitative R² values should be described as descriptive rather than as formal evidence of non-predictiveness.","section":"Section 3"}],"minor_comments":[{"comment":"The abstract reports 'up to 47%' while the text and Figure 1 report a 26.5-point HELMET gap; please state explicitly that 47% is a relative drop between the best and worst runs, and define the denominator.","section":"Abstract and Section 4"},{"comment":"The claim that 'any one of these choices alone has a minor impact' is hard to reconcile with the 6-point QK-norm effect on the Olmo architecture and the GQA gradient in Figure 3; please define 'minor' relative to the observed range.","section":"Abstract and Section 4"},{"comment":"The 'number of features' variable is used as a color key and as the main predictor, but it is never formally defined; please state the binarization rule for GQA (e.g., fewer than 32 KV heads) and add a feature-count column to Tables 1 and 2.","section":"Figures 1, 7; Tables 1–2"},{"comment":"There are typos ('initalization' twice) and the appendix could be tightened; more importantly, the mapping from initialization codes A–K to the four paired comparisons mentioned in the text is not given.","section":"Appendix B"},{"comment":"The attention-sink analysis reports R²=0.38 between sink attention and long-context performance over 26 models; this should be labeled exploratory, since no correction is made for the many attention statistics tested.","section":"Section 5.4"}],"recommendation":"major_revision","confidential_remarks":"The OlmPool release is a strong community resource and the paper is likely to be influential. My main concern is that the headline 'count of features' result needs to be shown to survive controls for the non-factorial confounds; I do not see this as requiring new training runs, only more careful analysis of the existing pool. No concerns about authorship or scope."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First thing to know: this is a solid, useful paper, and its central finding—four individually minor architectural choices compound to hurt long-context extensibility—holds up. The polished headline, though, is an overreach: count of features gives LOO R2=0.61 and three-plus features cost up to 47%, but that statistic comes from a pool that isn't a clean factorial design.\n\nWhat's genuinely new: a controlled 26-model ablation holding data, tokenizer, and optimization fixed, with paired comparisons over normalization, GQA, sliding-window attention, and pretraining context. OlmPool (38 checkpoints per model) is a reusable asset. The paper cleanly replicates Yang et al.'s QK-norm effect and shows the four factors impair long-context performance roughly additively—that compound picture is the contribution. The robustness checks (YaRN, 50B extension, 2T pretraining) are thoughtful, even if each rests on one or two runs.\n\nSoft spots, in proportion. The count predictor is fit over runs that also differ in initialization, fp8, norm order, and depth (the Qwen-like run has 36 layers vs 32). The paper's own Appendix B reports initialization swings up to 17% of the observed range (mean 7.7%), and the stated guardrail—only discuss differences substantially larger than the mean init swing—is sensible but isn't applied to the count model or the headline 47% gap, which is between the best and worst rows differing in all those uncontrolled factors. So the abstract's causal phrasing outruns the design; 'associated with' would be defensible today. One correction to the stress-test note: fp8 appears in four runs, not one, but its effect is reported as noise, and the entanglement remains. Also missing: the regression coefficients behind the count model, and the 16-KV-head comparison model is slightly larger. None of this sinks the main conclusion—the compound effect is large relative to init noise—but it should be fixed before the count model is cited as a causal result.\n\nWho this is for: model developers choosing a dense-transformer variant for long-context work, and researchers studying early-pretraining proxies. It deserves a serious referee. Recommend conditional accept: add a coefficient table, rerun the count analysis on init-controlled pairs, and tone down the abstract.","headline":"A well-run 26-model ablation shows architectural choices compound to hurt long-context extension, but the count-feature predictor and 47% headline are fit over a pool that isn't fully controlled—so the abstract oversells causality.","tokens_in":27146,"tokens_out":5015,"would_cite":true,"duration_ms":46750,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Four individually minor architectural choices compound: three or more can cut long-context benchmark scores by up to 47%, and counting them predicts downstream performance ($R^2=0.61$).","keywords":["long-context language models","architectural ablations","QK-norm","grouped-query attention","sliding window attention","pretraining context length","context extension","OlmPool"],"falsifier":"A decisive check would be to train the worst architecture—GQA, sliding window, headwise QK-norm, and 4K pretraining—with an architecture-specific learning-rate schedule and continue extension well beyond 50B tokens; if it then matches the Llama-like architecture on HELMET at 32K, the claim that the features themselves, rather than optimization fit, limit extensibility would fail. Conversely, finding any model in the released pool with three or four features that scores at the top of the HELMET distribution under the same recipe would break the counting predictor.","tokens_in":26197,"feed_emoji":"🧠","tokens_out":6856,"duration_ms":63074,"temperature":0.7,"pith_summary":"The paper argues that long-context extensibility in dense transformers is shaped far more by small architectural decisions made before pretraining than by data, tokenizer, or extension recipe, and that these decisions compound. Across 26 controlled 7B models trained on identical data with identical optimization, any single one of four features—QK-norm, grouped-query attention, sliding-window attention, or a 4K pretraining context—costs only a point or two on long-context benchmarks, but combining three or more can cut HELMET at 32K by up to 26.5 points. Simply counting how many of these features a model carries predicts downstream long-context performance with leave-one-out $R^2=0.61$, outperforming any single architectural axis. Standard short-context loss, perplexity, and in-loop benchmarks cannot detect the difference, but an early context-extension run can.","feed_headline":"Four 'minor' choices can cut long-context scores by 47%","feed_subtitle":"A 26-model controlled pretraining pool shows these choices stack, and short-context metrics can't see the damage.","key_machinery":"The load-bearing object is a count: how many of four 'long-context-inhibiting' features a model carries. The features are QK-norm (normalizing query and key vectors, either layerwise or per attention head), grouped-query attention (sharing a small number of key-value heads across many query heads), sliding-window attention (three of every four layers attending only to a 4096-token window), and a 4096-token pretraining context. The count acts as a single ordinal predictor of post-extension performance and outperforms a linear regression over the four individual axes. The paper also identifies a mechanism in the attention maps: models without QK-norm develop strong attention sinks, and sink strength correlates with better long-context performance ($R^2=0.38$).","core_discovery":"The central claim is that four architectural choices, each individually minor and each justified by training stability, pretraining throughput, or inference efficiency, together determine much of a transformer's ability to adapt to long contexts. Holding data, tokenizer, and extension recipe fixed, models with three or more of QK-norm, grouped-query attention, sliding-window attention, and a 4096-token pretraining context score up to 47% worse on long-context benchmarks than models with fewer such features. A simple count of the features present is the single most predictive variable for downstream HELMET at 32K (in-sample $R^2=0.67$, leave-one-out $R^2=0.61$), and the pattern persists when extension is lengthened to 50B tokens, when pretraining is extended to 2T tokens, and when the extension strategy is changed. The paper releases OlmPool, a suite of 26 data- and recipe-matched 7B models with checkpoints before and after extension, several of which extend better than the Llama 3 architecture.","pith_inferences":["A testable extension the paper leaves open: if the feature count is the true driver, then removing any single feature from a four-feature model should recover roughly the same increment regardless of which feature is removed; the released checkpoints make this check possible.","The attention-sink correlation suggests an untested remedy: architectures that achieve training stability without suppressing attention sinks, for example by capping attention logits, may obtain both stability and long-context extensibility.","A further implication the authors do not draw: if short-context metrics are blind to this degradation, then the common practice of selecting architectures on short-context loss could systematically select for poor long-context extensibility; this prediction could be tested on existing released model pools."],"forward_implications":["Model developers who adopt QK-norm, grouped-query attention, sliding windows, or short pretraining contexts for efficiency or stability should expect a long-context penalty that grows with the number of features, not with any one choice.","Context-extension recipes tuned on a Llama-like architecture will tend to look best on that architecture; the same recipe applied to Qwen- or Olmo-like designs can understate both the architecture and the recipe.","Short-context loss and validation perplexity cannot substitute for a long-context probe; running a context extension early in pretraining is the cheapest reliable signal the paper finds.","Models carrying three or more of these features may need extension phases far beyond the 10B-token standard, since the paper observes no washout even at 50B extension tokens.","The OlmPool release itself is a resource: 26 data- and recipe-matched models with checkpoints before and after extension, usable for studying early-training phenomena beyond long context."],"supporting_citations":[{"why":"Supplies the original observation that QK-norm can limit long-context performance, which this paper replicates and extends.","marker":"(Yang et al., 2025b)"},{"why":"Defines grouped-query attention, the efficiency-motivated feature ablated here.","marker":"(Ainslie et al., 2023)"},{"why":"Provides the sliding-window configuration, pretraining recipe, data, and learning-rate schedule used across all OlmPool runs.","marker":"(Olmo Team et al., 2025)"},{"why":"Motivates the pretraining-context-length ablation with the observation that longer pretraining contexts adapt better to extension.","marker":"(Zhao et al., 2024)"},{"why":"Supplies the RoPE-theta adjustment recipe used for long-context extension.","marker":"(Xiong et al., 2023)"},{"why":"Defines HELMET, the primary downstream long-context benchmark used to rank the models.","marker":"(Yen et al., 2025)"},{"why":"Defines RULER, the synthetic long-context benchmark that corroborates the HELMET rankings.","marker":"(Hsieh et al., 2024)"},{"why":"Defines LongPPL, the long-range-dependency perplexity metric used as the intrinsic long-context check.","marker":"(Fang et al., 2025)"}],"fun_headline_variants":["Four minor tweaks together cause up to 47% long-context loss","Design choices that seem minor can slash long-context scores by 47%","Short-context tests can't see a 47% long-context performance drop","Three or more 'minor' features trigger a 47% long-context plunge"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The conclusion rests on the four chosen architectural axes and their tested values standing in for the full range of real design choices; if another factor such as optimization hyperparameters or a different sliding-window ratio drives the long-context gap, the feature-count story would not transfer.","fun_headline_variants_meta":{"raw":{"variants":["Four minor tweaks together cause up to 47% long-context loss","Design choices that seem minor can slash long-context scores by 47%","Short-context tests can't see a 47% long-context performance drop","Three or more 'minor' features trigger a 47% long-context plunge"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000667,"raw_usage":{"total_tokens":3084,"prompt_tokens":1027,"completion_tokens":2057,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":643,"completion_tokens_details":{"reasoning_tokens":1972}},"tokens_in":643,"tokens_out":2057,"duration_ms":13908,"temperature":1.0,"reasoning_tokens":1972,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T04:09:01.319162+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A decisive check would be to train the worst architecture—GQA, sliding window, headwise QK-norm, and 4K pretraining—with an architecture-specific learning-rate schedule and continue extension well beyond 50B tokens; if it then matches the Llama-like architecture on HELMET at 32K, the claim that the features themselves, rather than optimization fit, limit extensibility would fail. Conversely, finding any model in the released pool with three or four features that scores at the top of the HELMET distribution under the same recipe would break the counting predictor.","supporting_citations":[{"cited_title":"2024 , url =","cited_arxiv_id":null,"evidence_quote":"Defines RULER, the synthetic long-context benchmark that corroborates the HELMET rankings."}],"review_version":1}