{"id":"6dee3ae1-c7e1-46f0-a992-7e9b1b756124","arxiv_id":"2608.12218","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Long-context training shifts language models from parametric knowledge to context reliance, producing an inverted-U in pretraining performance and context addiction in supervised fine-tuning.","lead":"Training language models on long, information-rich contexts can shift them from storing knowledge in their weights to relying on the text they are given, hurting performance when that context is missing or wrong. The paper demonstrates this 'Information Abundance Paradox' across pretraining, fine-tuning, and mechanistic analyses, with direct implications for how context windows are scaled.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The pretraining sweep's token-matched batch changes sequences per step from ~2048 at 512 to ~32 at 32K; the inverted-U may reflect gradient noise or per-batch document diversity, not reduced parametric pressure.","rationale":"The reader's CONDITIONAL verdict is appropriate. The paper has real strengths: the SFT experiment holds the context budget fixed and varies only the relevance of the prepended documents, the module-restricted fine-tuning in §5.2 provides causal evidence that FFN vs SA updates shift context reliance, and the synthetic task study controls the data-generating process. The weakest load-bearing point is the pretraining sweep in §3.1: the claimed isolation of context length is not achieved because sequence count per batch is inversely proportional to W. The concrete block-diagonal test distinguishes the two explanations without requiring an infeasible decoupling of context length from token batch size. If the test validates the confound, the paper's broadest claim about natural-language pretraining would be unsupported, although the SFT and synthetic pillars would still support a more limited version of the paradox. Since the reader already conditioned the verdict on this issue, no verdict adjustment is needed.","tokens_in":34160,"tokens_out":14988,"duration_ms":140760,"concrete_test":"At the 259M scale, pretrain a W=32K variant in which each physical 32K sequence packs four independent 8K documents with a block-diagonal causal mask and position IDs reset per block, so each optimizer step sees 32 physical sequences but 128 independent 8K contexts per ~1.05M-token batch. Train on the same 10B tokens and evaluate on the §3.1 SuperGLUE and MCQA suites. Compare against the original W=8K run (128 sequences/batch) and the original W=32K run (32 coherent sequences/batch). If the block-diagonal run tracks the W=8K run, the 32K decline is caused by the coherent long context, not by the sequence-count confound; if it tracks the W=32K run, the decline is reproduced without coherent long context, confirming the confound (or another physical-sequence artifact) as the driver.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.1 describes the pretraining comparison as 'token-and-update-matched, isolating context window effects under a fixed token budget,' but with the global batch fixed at ~1.05M tokens, the number of sequences per optimization step varies with W: about 2048 at W=512 and 32 at W=32768 (footnote 1 only flags the FLOP difference). The sweep therefore changes gradient noise, per-batch document diversity, and the number of independent loss blocks per update, not only context length. The inverted-U in Figure 2 and the declining FFN/SA gradient ratio in Figure 5b could be produced by these optimization and coverage differences rather than by the hypothesized 'information abundance' shifting pressure away from parametric storage. Because the pretraining result is the broadest natural-language evidence for the paradox, this confound is load-bearing: if it explains the U-shape, the central claim rests only on the narrower SFT and synthetic results, which use LoRA and small-scale tasks. The SFT design in §3.2, which fixes context length and varies only the relevance of documents, is not affected by this confound and remains the cleanest evidence.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes the Information Abundance Paradox: when task-relevant information is available in the training context, a language model can reduce loss by using that information directly rather than encoding it in its parameters, thereby shifting from parametric internalization toward contextualization and producing \"context addiction\" at test time. The authors support this with three main bodies of evidence: (i) a motivating comparison of short- and long-context public models; (ii) pretraining sweeps over context windows showing inverted-U performance on language modeling, SuperGLUE, and closed-book MCQA; and (iii) supervised fine-tuning experiments that fix context length and vary the fraction of task-relevant documents, showing improved supporting-context accuracy but reduced no-context and conflicting-context robustness. They also present a theoretical monotonicity result, gradient-based mechanistic analyses, module-restricted fine-tuning interventions, and inference-time attention analyses. The manuscript argues that long-context training is not a neutral scaling axis and that scaling context windows can change what models internalize.","tokens_in":34415,"tokens_out":9187,"duration_ms":90758,"significance":"If the central claims hold, the paper makes a useful and timely contribution: it challenges the assumption that longer training context is a purely beneficial scaling axis, and it operationalizes a distinction between parametric internalization and contextualization that is relevant to long-context training and evaluation. A particular strength is the SFT design in Section 3.2, which fixes the context budget and varies only the relevance of the prepended documents; this cleanly isolates information abundance and shows consistent effects across five model scales and four domains. The mechanistic analyses, especially the module-restricted fine-tuning and attention-allocation results, provide convergent and falsifiable evidence for the proposed shift. However, the broadest natural-language evidence, the pretraining sweep in Section 3.1, is currently undermined by a sequence-count confound, and the theoretical account in Section 4 is an achievability statement rather than a derivation of the empirical pattern. With additional control experiments and a more careful framing, the contribution would be significant for the long-context training community.","major_comments":[{"comment":"The pretraining comparison is described as \"token-and-update-matched, isolating context window effects under a fixed token budget,\" but with the global batch fixed at approximately 1.05M tokens, the number of sequences per optimization step falls from about 2048 at W=512 to about 32 at W=32768. The sweep therefore changes gradient noise, per-batch document diversity, and the number of independent loss blocks per update, not only the context length. The inverted-U in Figure 2 and the declining FFN/SA gradient ratio in Figure 5b could in principle be produced by these optimization-level changes rather than by reduced pressure toward parametric storage. Footnote 1 flags only the FLOP difference, not the sequence-count confound. Because this pretraining result is the paper's broadest natural-language evidence for the paradox, the confound is load-bearing; please add an experiment or quantitative analysis that directly addresses it, for example by holding sequences per batch fixed while varying the token budget, or by demonstrating that gradient noise and document coverage cannot explain the U-shape.","section":"§3.1 (Training setup) and Figure 2"},{"comment":"Proposition 4.2 states that I_m(ρ) ≤ I_k(ρ) for nested contexts, and the proof shows that a longer-context predictor can simulate a shorter-context predictor by projection. This is a valid achievability statement, but it does not imply that the trained model actually stores less task information in its weights, and because it is monotone in context size it cannot by itself explain the inverted-U in Figure 2 or the supporting/no-context tradeoff in Figure 3. As written, the theoretical account is close to a formal restatement of the hypothesis rather than a derivation of the empirical results. To make this section load-bearing, the paper would need to connect I_k(ρ) to the training dynamics, for example by showing that the learned weight distribution approaches the infimum or by directly estimating I(W;τ) across context sizes, or it should explicitly reframe the proposition as a motivation rather than an explanation.","section":"§4, Proposition 4.2"},{"comment":"The module-restricted fine-tuning comparison is presented as causal evidence that FFN updates confer parametric robustness and SA updates confer context reliance. However, with LoRA rank fixed at r=64, the FFN and SA adapters contain very different numbers of trainable parameters, so the two conditions also differ in update capacity and optimization geometry. The observed differences in no-context and conflicting-context accuracy could therefore reflect parameter-count or learning-rate effects rather than the functional role of the modules. Please match the trainable parameter budget between the FFN-only and SA-only conditions, for example by adjusting the LoRA rank per module, or otherwise control for update capacity before interpreting Figure 6 as causal evidence.","section":"§5.2, Figure 6"}],"minor_comments":[{"comment":"The main text says training uses 10B tokens from Project Gutenberg, while App. C.1 says the models are trained for 4 epochs of a 2.5B-token corpus; please reconcile the corpus size and epoch count in the main text.","section":"§3.1 and App. C.1"},{"comment":"The legend entries such as \"20M Mean 55M CI95% 259M Inflection Point 750M\" are ambiguous about which curve and shaded band correspond to each model scale; use a conventional per-scale color legend with a separate marker for inflection points.","section":"Figure 2 and related figures"},{"comment":"The full results table is extremely dense; consider reporting the k=0 versus k=8 differences or plotting the deltas so that the domain-level pattern can be verified more easily.","section":"App. H.1, Table 16"},{"comment":"The average gradient norm G_k is compared across context lengths k, but the synthetic batches are matched in number of examples rather than number of tokens; please state whether G_k is normalized for sequence length or show that the within-task differences are not an artifact of longer prompts.","section":"§5.1 and App. C.3"}],"recommendation":"major_revision","confidential_remarks":"The SFT evidence in Section 3.2 is clean and is the paper's strongest contribution; the pretraining confound in Section 3.1 is serious but potentially addressable with additional experiments. If the authors cannot provide a sequence-count-matched control or an equally convincing analysis, the title and abstract claims about pretraining should be softened and the paper reframed around the SFT and synthetic results. The motivating public-model comparison in Figure 1 is also confounded by differences in data, post-training, and evaluation procedures, and should be presented only as motivation. I saw no issues with citation or attribution; the related-work discussion is broad and appropriate."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nYou should know this paper for one result: when the training context is fixed in length and only the relevance of its content changes, models systematically trade parametric robustness for contextual accuracy. That SFT finding is solid, spans five model sizes and four domains, and comes with a clean causal intervention (FFN-only vs SA-only LoRA). The rest—the pretraining sweep and the theory—is weaker than the packaging suggests.\n\nWhat is genuinely new: the SFT setup in §3.2. Holding eight documents and swapping their domain composition isolates information abundance without touching context length, and the effects are large and monotonic (k=0→8 lifts supporting-context accuracy by up to ~20 points while dropping no-context and worsening conflicting-context accuracy). Figure 6's module-restricted fine-tuning is a nice mechanistic step. The synthetic study in §5.1 is also controlled and points in the same direction. Those pieces earn the paper a serious look.\n\nWhere it wobbles: the pretraining sweep in §3.1 claims to isolate context window effects, but token-matching the batch means sequences per step drop from ~2048 at W=512 to ~32 at W=32768. Gradient noise, per-batch document diversity, and the number of independent loss blocks all change with W. The inverted-U in Figure 2 could be the ordinary cost of long-sequence batches, not reduced parametric pressure. The paper flags only the FLOP difference (footnote 1), not this sequence-count confound, and its limitations section doesn't mention it either. Since the pretraining result is the broadest natural-language evidence, this is load-bearing. The effects themselves are small (a few points) and the inflection point moves across testbeds, which is a bit suspicious.\n\nThe theory, Proposition 4.2, is an achievability statement: longer context weakly expands the feasible predictor set, so the minimum required parametric information can only decrease. True, and the paper labels it as such, but it is close to a restatement of the hypothesis and does not derive the U-shape or the training dynamics. Treat it as intuition, not explanation.\n\nThe Phi-3/OLMo comparison is post-hoc across differently trained checkpoints; okay as motivation, not as evidence.\n\nBottom line: the SFT core is good enough to publish; the pretraining sweep needs a control that holds sequence count fixed, and the theory should be repositioned as framing. Send it to peer review with that expectation.","headline":"The SFT evidence for context addiction is strong and worth citing; the pretraining sweep is confounded by sequence count and the theory is thin.","tokens_in":34886,"tokens_out":4505,"would_cite":true,"duration_ms":41587,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper argues that when task-relevant information is abundant in the training context, language models can reduce loss by using it directly instead of encoding it in weights, and this shift from parametric internalization to…","keywords":["information abundance paradox","long-context pretraining","parametric knowledge","contextualization","in-context learning","context addiction","inverted-U scaling","transformer mechanisms"],"falsifier":"Run the same pretraining sweep while matching sequences per optimization step instead of tokens per step: if the inverted-U disappears, the central pretraining claim is refuted. Independently, on a synthetic task where the learned rule is known, estimate $I(W;\\tau)$ directly from weights trained with and without in-context demonstrations; if longer context does not reduce measured task information in the weights, the parametric information mechanism is falsified.","tokens_in":33997,"feed_emoji":"📉","tokens_out":11559,"duration_ms":85209,"temperature":0.7,"pith_summary":"The paper claims that abundant task-relevant information in the training context can reduce a language model's incentive to store that information in its weights, shifting learning from parametric internalization toward contextualization. It names this the Information Abundance Paradox. In pretraining, with the token budget, data, and optimizer held fixed, expanding the context window helps language modeling, SuperGLUE, and closed-book multiple-choice QA only up to an intermediate optimum, after which performance declines. In supervised fine-tuning, more task-relevant context improves accuracy when supporting context is present at test time but lowers accuracy when context is absent or conflicting. The paper adds a theoretical bound showing that longer nested contexts weakly reduce the minimum task information the weights must carry for a given risk, and mechanistic evidence that context-rich training lowers solution complexity, shifts gradient pressure from feed-forward networks to attention, and increases inference-time attention to context.","feed_headline":"Longer training contexts make LLMs rely on prompts, not memory","feed_subtitle":"Longer windows help only to an optimum; context-heavy fine-tuning trades robustness for prompt-following.","key_machinery":"The load-bearing object is the parametric information frontier, $I_k(\\rho)=\\inf_{\\Pi,q_k} I(W;\\tau)$ subject to risk $R_k(\\Pi,q_k)\\le\\rho$, together with the monotonicity result that for nested input windows $X^{(k)}=T_{k,m}(X^{(m)})$, $I_m(\\rho)\\le I_k(\\rho)$. It formalizes the claim that context and weights are substitutable carriers of task information, so longer contexts weakly shrink the parametric storage needed for a target risk. The empirical machinery consists of three aligned measurements: average training gradient norm as a proxy for solution complexity; the FFN-to-SA gradient-norm ratio as a proxy for where optimization pressure lands; and module-restricted fine-tuning (updating only attention heads or only feed-forward layers) plus inference-time attention mass, which turns the gradient shift into a causal and behavioral test.","core_discovery":"The central claim is the Information Abundance Paradox: when training context makes task-relevant information available, the model can reduce next-token loss by using that information directly instead of encoding it parametrically, and the learning dynamics shift accordingly. The empirical signature is an inverted-U: across model scales from 20M to 750M parameters, SuperGLUE and closed-book MCQA accuracy peak at a training window of about 2048 tokens and language modeling loss bottoms out around 8192 tokens, then both degrade as the window grows. In fine-tuning, increasing the number of target-domain documents from $k=0$ to $k=8$ under a fixed eight-document budget improves supporting-context accuracy while reducing no-context accuracy and increasing the supporting–conflicting gap. The theoretical account defines the parametric information frontier $I_k(\\rho)$, the minimum mutual information $I(W;\\tau)$ between weights and task needed to reach risk $\\rho$ with context size $k$, and proves $I_m(\\rho)\\le I_k(\\rho)$ for nested contexts $k