Pith.

Artificial Intelligence · arXiv:2609.37725 v1 · submitted 29 Sep 2026 · 75 references

Context Language Models

Rulin Shao, Shannon Zejiang Shen, Junjie Oscar Yin, Yuetai Li, Minheng Wang, Hamish Ivison, Radha Poovendran, Nathan Lambert

show 5 more authors

Teng Xiao, Mike Lewis, Wen-tau Yih, Luke Zettlemoyer, Pang Wei Koh

The paper says

“We introduce Context Language Models (CLMs), language models that natively manage their own context. We implement this by treating the context as a file and allowing the model to make unrestricted updates to this file.”

more of the abstract

This allows the model to learn what is most important to maintain in context, and naturally extends to multi-agent systems where multiple agent contexts coexist as files. Building CLMs zero-shot with existing models outperforms SOTA context management strategies across a variety of tasks: 11.4% higher accuracy with 21.5% fewer FLOPs on BrowseComp-Plus, 5% higher scores with 59% fewer FLOPs on 12-hour EdgeBench, and 65% greater improvement with the same compute on a 24-hour multi-repository agent-swarm task. Moreover, by shifting context management from external harness control to intrinsic model behavior, CLMs naturally enable both in-context and parametric learning of context-management strategies. We show that CLMs can be steered with natural-language instructions evolved through a standard skill-optimization loop, improving held-out accuracy by up to 35.9 points on a context-management task while reducing compute. We also introduce an online reinforcement learning method for CLMs, improving Qwen3.5-9B performance on BrowseComp-Plus by 47.6% while using 12% fewer FLOPs. Finally, we co-design Suffix Cache Reuse for CLM serving, further reducing server-side compute by 35% relative to standard SGLang at matched performance.

from the abstract

Pith says

Machine review · deepseek-v4-flash · 30 Sep 2026 · what was read was not recorded · not checked by a person

It can be wrong. Challenge this review or reply to it.

Holds up, with conditions. The core claim—models with unrestricted write access to their own context—is real and well-evidenced on accuracy, but the headline efficiency numbers rest on a self-defined metric plus CLM-only evaluation privileges, so they should be treated as conditional until re-run. This line is wrong

3 major objections · 4 minor

How often objections from reviews like this one are right: not yet measured

Are you an author? What you can do.

Not checked: figures, code, what the 75 references say.

The paper says, at Section 5.1.1 and Appendix E (BrowseComp-Plus setup)

The review did not quote this passage.

Pith says: Objection 1 of 3 · major

The headline 21.5% FLOPs advantage (Section 5.1.1) is computed under three CLM-only evaluation privileges described in Appendix E: CLM's editing turns do not count toward the 100-turn cap, a request that would exceed the budget is rolled back and retried up to six times, and CLM receives a context-length reminder 2,048 tokens before the budget (footnote 4).

more of objection 1

The rollback is the most damaging asymmetry: it directly lowers the re-prefill cost that prefix-reuse FLOPs measures by letting CLM un-commit context edits whose re-prefill would otherwise dominate the metric, and Appendix G shows the reminder materially improves the very context-length estimation capability that drives efficient compaction. The authors acknowledge the reminder is a 'temporary augmentation,' but the paper reports the headline efficiency numbers without any decomposition. I request ablations reporting accuracy and prefix-reuse FLOPs for CLM with (a) the hint disabled, (b) rollback retries disabled, and (c) editing turns counted against the turn cap, so the reader can see how much of the 21.5% (and the 59% EdgeBench figure) survives each privilege removed.

The paper says, at Section 4.2, Eq. (6), and Table 2

The review did not quote this passage.

A classifier read this sentence of the paper as acknowledging the concern (probability 0.62):

“Outcome rewards provide only weak supervision for context editing, as successful trajectories can contain inefficient edits and failed trajectories useful ones.”

Pith says: Objection 2 of 3 · major

The RL efficiency advantage defined in Eq. (6) rewards trajectories for minimizing the same prefix-reuse FLOPs quantity used for evaluation and for the reported cost numbers, and the RL checkpoints are selected using that metric on a validation set. The 12% efficiency gain after RL (Table 2: 1.52 to 1.34 PFLOPs per question) therefore partially reflects optimization directly against the evaluation metric rather than against an independent notion of compute.

more of objection 2

The success-gating in Eq. (6) is a reasonable guard against one class of reward hacking and the authors state the weak-supervision caveat, but what remains unresolved is whether the trained CLM is still efficient under the metric-neutral accounting once the hint-dependent budget behavior and rollback retries are removed; an ablation training with the efficiency reward but evaluating without the hinted-rollback privileges would address this.

The paper says, at Table 1 and Appendix E (Math optimization)

The review did not quote this passage.

Pith says: Objection 3 of 3 · major

The paper reports best-of-run scores on the four math optimization problems with 'one run per method,' and the abstract's claims of up to 16.8% (Heilbronn) and 3.0% (circle packing) improvements over OpenEvolve are single-run differences with no variance estimate. Given the stochasticity of evolutionary coding-agent workflows on this timescale, a single run cannot support margins of this size; please add multiple seeds or a variance estimate, or explicitly downgrade these results to anecdotal in the abstract and main text.

4 minor objections

The paper says, at Section 5.3 and Appendix B (SCR)

Pith says: minor objection 1 of 4

The SCR claim of 35% compute reduction at matched performance rests on reusing stale KV cache states (Appendix B, 'the reused states were computed under the pre-edit context'), an approximation the authors flag. The Figure 9 matched-performance point should state the question count it is averaged over, since the K-sensitivity analysis in Figure 11 uses only 64 BrowseComp-Plus questions; an end-to-end accuracy check at the 830-question scale would strengthen the claim.

The paper says, at Appendix E (EdgeBench-10 setup)

Pith says: minor objection 2 of 4

The EdgeBench-10 paragraph says 'when a request would exceed it, the harness rolls back the last turn, up to 50 times' but does not state whether this rollback privilege applies to all methods or only to CLM. Since the BrowseComp-Plus rollback is CLM-only, please specify who receives which privileges in each benchmark to make the comparisons unambiguous.

show 2 more

The paper says, at Throughout

Pith says: minor objection 3 of 4

Many passages have run-together words ('implementthis', 'retaininformation', 'andread–write', 'theagentis'), presumably a typesetting artifact; please fix so the text is readable.

The paper says, at Section 3 and Appendix E (ContextBench)

Pith says: minor objection 4 of 4

The ContextBench pilot uses GPT-5.4 while the main benchmarks use Qwen3.6-27B and Claude 4.6 Sonnet; a sentence stating whether the ContextBench conclusions were also checked on at least one of the main evaluation models would clarify the generality of the diagnostic.

What would settle it

Rerun the BrowseComp-Plus comparison (Qwen3.6-27B, 32K budget, 100-turn cap) under matched conditions: (a) give the Codex-style summary baseline the same 2,048-token context-length reminder CLM receives, (b) count CLM's editing turns toward the 100-turn cap, and (c) disable CLM's rollback retries (budget overflow ends the run, as in TerminalBench). Report the accuracy gap and the prefix-reuse FLOPs gap in this fully matched setting. If the accuracy gap (59.4% vs ~53.3%) collapses below significance or the 21.5% FLOPs advantage flips, the headline efficiency and accuracy-magnitude claims overstate the mechanism's intrinsic benefit; if both hold, the concern is resolved.

from the skeptic's reading

How the review went

4 readings by deepseek-v4-flash · 30 Sep 2026

  1. Reader holds up, with conditions confidence moderate
  2. Skeptic partly agrees 29 s · 31,221 tokens read, 3,778 written
  3. Editor wrote the verdict above 12 s · 32,370 tokens read, 1,708 written
  4. Referee major revision: 3 major objections, 4 minor 40 s · 31,614 tokens read, 5,450 written

Not run on this paper: the simulated authors' defence.

Pith did not check this review's objections against the paper.

The review in full

The review in brief, machine-written

Here's my read on the CLM paper. The core idea is genuinely new and well-executed: treating the live context as an editable file with unrestricted model write access collapses the harness-defined action space into a single primitive, and Eq. 2 formally subsuming the append-only transition is a clean way to state it. The zero-shot results on BrowseComp-Plus, TerminalBench, and the multi-agent swarm are backed by external verifiers, so the qualitative claim—models given direct context control do at least as well as human-designed harnesses—looks solid. The ContextBench diagnostic is carefully designed to isolate context management from reasoning and knowledge, and the skill-evolution and success-gated stepwise-GRPO extensions are reasonable. The SCR serving mechanism is a concrete engineering contribution, and the authors are honest that it approximates re-prefilling (b310, b317, b318).

Where I'd push back is the efficiency accounting. The stress-test note lands. The prefix-reuse FLOPs metric is the authors' own, it also serves as the RL reward, and the headline savings (21.5%, 59%, 12%, 35%) are computed under conditions that favor CLM: editing turns don't count toward the turn cap on BrowseComp-Plus, there are up to six rollback retries, and a 2,048-token context-length hint that Appendix G says substantially improves estimation. All of this is acknowledged in the paper (footnote 4, Appendix E), but the reported numbers are not conditioned on it, so the magnitude of the efficiency gains is genuinely uncertain until someone re-runs without those privileges. The math-optimization table is single-run with no variance—a real weakness, though minor in the sense that it's not the load-bearing claim. Several baselines are re-implementations and some runs use proprietary API models, which is standard for the area but limits independent verification.

None of this breaks the central claim. The accuracy results are externally grounded, and the mechanism is demonstrably more flexible than prior action-constrained tools. But the headline efficiency numbers should be read as preliminary. This deserves a serious referee: the novelty and the quality of the diagnostic work justify real reviewer time, and the efficiency claims need independent re-running before they're taken at face value. I'd cite it, and I'd bring it to reading group—the context-as-file framing is likely to be influential regardless of how the FLOPs arithmetic settles.

The strongest objection

The central claim splits into accuracy supremacy and compute efficiency. The accuracy numbers are verifier-grounded (BrowseComp-Plus judge, EdgeBench judge, verifier-scored math) and are not metric-dependent, so the qualitative claim is reasonably secure. The efficiency numbers are not. They are computed on the authors' own prefix-reuse FLOPs model (Eq. 3, Appendix C), which is also the RL efficiency reward (Eq. 6) and the basis for checkpoint selection, and they are produced under conditions that favor CLM in three ways the baselines do not share: (1) CLM's editing turns do not count toward the 100-turn cap on BrowseComp-Plus (Appendix E), so CLM gets more effective decision steps at no turn-budget penalty; (2) when a request would exceed the budget, CLM rolls back the last turn and retries, up to six times (Appendix E), letting it un-commit costly context edits that would otherwise dominate re-prefill FLOPs and kill accuracy; (3) CLM receives an editing reminder 2,048 tokens before the budget (footnote 4, Appendix E), and Appendix G shows such hints substantially improve context-length estimation in the exact regime where the paper finds models degrade — and estimation of remaining budget is precisely the capability that drives efficient compaction. The paper acknowledges the hint is a 'temporary augmentation' and that outcome rewards give weak supervision for editing (b95), but it still reports the 21.5%, 59%, 12%, and 35% FLOPs figures as headline gains without reporting the sensitivity of those numbers to the hint, the retries, or the turn counting. If the 2,048-token hint alone accounts for a large share of CLM's compaction efficiency — a plausible reading of Appendix G, which shows hints 'help substantially' — then the efficiency contrast (e.g., 179 vs 437 PFLOPs on EdgeBench) partly measures an evaluation affordance rather than the context-management mechanism.

The reader's weakest-assumption call — fairness and completeness of the prefix-reuse FLOPs metric plus evaluation asymmetries favoring CLM — is the right general diagnosis, and I agree it is the load-bearing weakness. My stress-test sharpens it and shifts weight: the accuracy side of the claim is external-verifier grounded and largely robust to the metric worry, so the qualitative conclusion (CLM outperforms harnesses) is likely to survive; the fragile part is the efficiency magnitude. Among the three asymmetries, the 2,048-token hint is the most decisive because the paper's own Appendix G demonstrates that context-length estimation degrades at long contexts and that hints improve it substantially — and that capability is what efficient compaction depends on. The rollback retries are a second, independent lever because they let CLM avoid budget-overflow re-prefill costs that a fixed harness cannot. Both are admissions in the paper's own text (footnote 4; Appendix E), so my concern is not a manufactured flaw but a question of whether the headline FLOPs figures have been controlled for these affordances. The reader and I fully agree the numbers need independent re-running; I differ only in emphasis, placing the hint/rollback asymmetry ahead of the single-run variance in Table 1 and the proprietary-API baselines, which are minor contributors. I do not find an internal inconsistency that would break the central claim: Eq. 9's FLOPs accounting is self-consistent, SCR's approximation is explicitly acknowledged (b310, b317, b318), and the stepwise-GRPO training with success-gated efficiency (Eq. 6) is internally coherent even though it shares the evaluation metric. On that basis the disposition stays CONDITIONAL: the qualitative claim is sound and the magnitude claims are plausible but unverified under symmetric evaluation, so ACCEPT with conditions or a confirmed CONDITIONAL is appropriate; I keep the reader's CONDITIONAL unchanged.

The referee's report

If the claims hold, this is an important and timely contribution: the context-as-a-file mechanism is simple, generalizes to multi-agent and subagent settings, produces qualitative behaviors (trackers, an internal-notes role, reusable compaction functions) consistent with a real capability shift, and ships with a code release. The accuracy results are externally grounded—verifier-graded pass/fail on TerminalBench and EdgeBench and a held-out BrowseComp-Plus judge—so the qualitative claim that CLMs match or exceed state-of-the-art context-management harnesses is not circular. The fragile part is the efficiency story. The headline FLOPs figures (21.5%, 59%, 12%, 35%) are computed on the authors' own prefix-reuse FLOPs metric, which is also the RL efficiency reward, and are produced under evaluation privileges (uncounted editing turns, rollback retries, and a context-length hint) that baselines do not share. The efficiency numbers therefore need quantification before the headline claims can be taken at face value; the accuracy claims, however, are reasonably secure.

Pith's reading of the paper (a machine's summary, not the authors')

This paper argues that context management should be a native language-model capability rather than something imposed from outside. The authors implement this by mirroring the model's live context into an editable file the model can rewrite with ordinary commands, replacing the append-only transition $c_{t+1}=c_t\oplus f^{\mathrm{LM}}_\theta(c_t)$ with a model-controlled $c_{t+1}=f^{\mathrm{CLM}}_\theta(c_t)$. On long-horizon agentic benchmarks, zero-shot CLMs built from existing models outperform summary-based and action-constrained harnesses while using fewer prefix-reuse FLOPs, and the same mechanism opens three learning routes: natural-language steering, in-context skill evolution, and online reinforcement learning. If the results hold, letting the model own and edit its context is a route to both better and cheaper long-horizon agency.

What carries the argument. The mechanism is the context-as-a-file implementation: the model's live context is mirrored to a file whose path appears in the system prompt, and the model edits that file with general commands, with every edit synchronized back into the next turn's context; when the model does not edit, tokens are appended as usual. This subsumes and generalizes prior harnesses whose context-management actions are predefined. Three supporting devices carry the efficiency claims: the prefix-reuse FLOPs metric of Eq. 3, which counts prefill of the unmatched context suffix plus decode of generated tokens, thereby assigning a real cost to in-the-middle edits under prefix caching; the success-gated efficiency advantage of Eq. 6, which re-ranks only the successful trajectories within a GRPO group by inference cost so that failed trajectories never get an efficiency bonus; and Suffix Cache Reuse, which retains the cached KV states of tokens that survive an edit past the first mismatch, re-rotating their positional encodings, so that only newly inserted or appended tokens must be prefilled.

Core claim. The central claim is that a language model given full read-write control over its own live context can discover context-management strategies that outperform both harness-scheduled summarization and tool-constrained approaches like AutoCompact, Self-Compact, ACM, and recursive language models. Concretely, a zero-shot CLM scores 59.4% on BrowseComp-Plus with 21.5% fewer prefix-reuse FLOPs than the strongest baseline, matches Codex-style summarization on TerminalBench 2.1 with 29.5% fewer FLOPs, and exceeds it on TBLite (73.7% against 67.0%). On 12-hour EdgeBench it scores 5% higher while using 59% fewer FLOPs, and on a 24-hour six-repository swarm it achieves 65% greater downstream speedup at equal compute. The same mechanism opens three learning routes: a one-sentence instruction measurably changes the model's compaction policy; an in-context skill-evolution loop raises held-out ContextBench accuracy by up to 35.9 points at lower cost; and a success-gated efficiency reward inside stepwise GRPO lifts Qwen3.5-9B from 28.8% to 42.5% on BrowseComp-Plus, a 47.6% relative gain, while using 38.8% fewer FLOPs than a summary harness trained identically. A co-designed serving scheme, Suffix Cache Reuse, cuts server-side re-prefilling by 35% at matched performance and also helps ordinary chat serving where reasoning tokens are stripped.

The premise it rests on. The claim stands on the fairness of the prefix-reuse FLOPs metric for comparing compute and on evaluation choices that favor the CLM condition — editing turns are not counted against the 100-turn budget, CLM may roll back and retry a failed turn up to six times, and CLM receives a 2,048-token context-length hint shortly before the budget limit, an augmentation the authors concede (footnote 4) meaningfully improves length estimation; if these asymmetries bias the comparison, the headline efficiency and accuracy gains overstate the improvement.

If the paper is right (Pith's inferences, not the authors')

Zero-shot CLMs beat both harness-scheduled and action-constrained context managers across deep research, terminal coding, and single- and multi-repository optimization, at 32K budgets and, on EdgeBench, at 128K.

Context management becomes controllable in natural language: a single sentence in the prompt measurably shifts compaction threshold, semantic-boundary timing, and backup behavior with no harness or weight change.

Context-management strategies can be searched and evolved: an in-context skill-evolution loop improves held-out ContextBench accuracy by up to 35.9 points while cutting cost, and evolved skills can strictly dominate the starting point on cost-accuracy frontiers.

Context management can be internalized in model weights: stepwise GRPO with the success-gated efficiency advantage improves Qwen3.5-9B from 28.8% to 42.5% on BrowseComp-Plus while lowering inference FLOPs below a summary harness trained with the same recipe.

Suffix Cache Reuse reduces server-side re-prefilling by 35% at matched accuracy relative to standard SGLang, and the same reuse helps ordinary chat serving where reasoning tokens are stripped even when the agent never edits its context.

If context-length awareness is the binding skill, the gains are mobile: the authors' own probe (Appendix G) shows existing models estimate their context length poorly at long lengths, so models trained with length-awareness signals could shrink or eliminate the need for the environmental hint that evaluation currently supplies.

The editable-context interface is a new prompt-injection persistence surface: a model that can rewrite its own context can carry unauthorized instructions across turns, an attack channel the paper flags and that would need defenses before broad deployment.

Because every harness context operation is expressible as a context transformation, existing harnesses act as procedural skills that could be distilled into CLM actions and then absorbed into weights, suggesting a general harness-to-CLM distillation route beyond the RL demonstrated here.

The benefit appears to scale with the model's ability to plan edits: Qwen3.5-9B edits less often and gains less than Qwen3.6-27B, which predicts that larger or more context-aware models will reap more from the CLM interface.

Check this review

Pith stored no record of this review's passes and took no fingerprint of it. Reviews made by code from 13 Nov 2026 on carry both.

How often reviews like this one are right

Reviews like this one: written by deepseek-v4-flash, where what was read was not recorded.

How often objections from reviews like this one are right: not yet measured

Every value above, in the review's public record (JSON)

Reviews by people

This review was written by a machine. No person has reviewed this paper on Pith yet. A scientist listed on Pith can review it for a fee, and their review is published here whatever it concludes.

Request a human review

How Pith checks papers · The full audit of this paper

The paper

Abstract

We introduce Context Language Models (CLMs), language models that natively manage their own context. We implement this by treating the context as a file and allowing the model to make unrestricted updates to this file. This allows the model to learn what is most important to maintain in context, and naturally extends to multi-agent systems where multiple agent contexts coexist as files. Building CLMs zero-shot with existing models outperforms SOTA context management strategies across a variety of tasks: 11.4% higher accuracy with 21.5% fewer FLOPs on BrowseComp-Plus, 5% higher scores with 59% fewer FLOPs on 12-hour EdgeBench, and 65% greater improvement with the same compute on a 24-hour multi-repository agent-swarm task. Moreover, by shifting context management from external harness control to intrinsic model behavior, CLMs naturally enable both in-context and parametric learning of context-management strategies. We show that CLMs can be steered with natural-language instructions evolved through a standard skill-optimization loop, improving held-out accuracy by up to 35.9 points on a context-management task while reducing compute. We also introduce an online reinforcement learning method for CLMs, improving Qwen3.5-9B performance on BrowseComp-Plus by 47.6% while using 12% fewer FLOPs. Finally, we co-design Suffix Cache Reuse for CLM serving, further reducing server-side compute by 35% relative to standard SGLang at matched performance.

The paper's own sentences, arranged, kept beside the review and separate from it. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. No sentence in this record was written by a model.

abstract compiled · 2026-09-30

The authors' main claim, as the classifier picked it out 0.99

“Building CLMs zero-shot with existing models outperforms SOTA context management strategies across a variety of tasks: 11.4% higher accuracy with 21.5% fewer FLOPs on BrowseComp-Plus, 5% higher scores with 59% fewer FLOPs on 12-hour EdgeBench, and 65% greater improvement with the same compute on a 24-hour multi-repository agent-swarm task.”

How the main claim is stated: asserted 1.00

What the abstract itself states, as probabilities: claims first 0.10 · states limitation 0.06 · human participants 0.05 · controlled comparison 0.90 · randomized assignment 0.02 · states quantitative result 0.99 · compares with prior results 0.97 · mentions code or data availability 0.04

The abstract, sentence by sentence

This table labels the abstract only (arXiv listing abstract v1). The number is the probability the classifier gave the label.

SentenceRoleThe authors' own workStates the main conclusionStrength
We introduce Context Language Models (CLMs), language models that natively manage their own context. contribution 1.00 0.83 0.11 asserted 0.99
We implement this by treating the context as a file and allowing the model to make unrestricted updates to this file. method 0.99 0.91 0.04 asserted 1.00
This allows the model to learn what is most important to maintain in context, and naturally extends to multi-agent systems where multiple agent contexts coexist as files. interpretation 0.46 0.45 0.08 asserted 0.84
Building CLMs zero-shot with existing models outperforms SOTA context management strategies across a variety of tasks: 11.4% higher accuracy with 21.5% fewer FLOPs on BrowseComp-Plus, 5% higher scores with 59% fewer FLOPs on 12-hour EdgeBench, and 65% greater improvement with the same compute on a 24-hour multi-repository agent-swarm task. result 1.00 0.95 0.51 asserted 1.00
Moreover, by shifting context management from external harness control to intrinsic model behavior, CLMs naturally enable both in-context and parametric learning of context-management strategies. interpretation 0.92 0.57 0.15 asserted 0.95
We show that CLMs can be steered with natural-language instructions evolved through a standard skill-optimization loop, improving held-out accuracy by up to 35.9 points on a context-management task while reducing compute. result 0.57 0.96 0.09 asserted 1.00
We also introduce an online reinforcement learning method for CLMs, improving Qwen3.5-9B performance on BrowseComp-Plus by 47.6% while using 12% fewer FLOPs. result 0.63 0.96 0.08 asserted 1.00
Finally, we co-design Suffix Cache Reuse for CLM serving, further reducing server-side compute by 35% relative to standard SGLang at matched performance. contribution 0.72 0.96 0.07 asserted 1.00

What kind of paper the classifier reports this is: method or tool 1.00 · evidence from observation or experiment 0.96

What was examined, and what was not

Sources read

  • arXiv listing abstract v1

Not examined

  • full text
  • figures and tables
  • references and citing sentences
  • relations to other papers

Levels run

  • P abstract

Not run

  • S sections
  • T body sentences
  • C citations
  • R cross-paper relations

How the labels were made

Compiled by classifier-1.13.0 via TypeSafe API, question bank abstract-p1-20260923, 2026-09-30: 1 request, 6,470 input tokens.

Every quoted sentence on this record is the authors'. Every label is from Pith's closed vocabulary, with the probability shown beside it. This record as JSON

Figures (22)
Figure 2 of the paper
Figure 2. Illustration of the four tasks in ContextBench and a performance comparison of CLM against baselines using GPT-5.4 with a 32K context limit. 4 Context Language Models (CLMs) 4.1 Formal Definition and Implementation with Context as a File Context Language Models (CLMs) generalize the append-only context transition of a standard LM to a model-controlled context transition. Standard LMs append model output to the current context: 𝑐𝑡+1 = 𝑐𝑡 ⊕ 𝑓 LM 𝜃 (𝑐𝑡), (1) where ⊕ denotes concatenation. In contrast, CLMs delegate full responsibility for maintaining the context to the CLM itself, dire
Figure 14 of the paper
Figure 14. illustrates the computation saved by prefix caching within one turn. The lengths are chosen for illustration and do not come from a particular run: a prompt of 𝑃𝑡 = 20,000 tokens and a response of 𝐺𝑡 = 500 tokens, with the reusable prefix 𝑅𝑡 set to 18,000, 10,000 or 0 tokens. When the turn only appends to its context, prefix caching avoids 87% of the computation the turn would need without a cache, and the turn costs 1.41 × 1014 FLOPs. An edit in the middle of the context leaves 10,000 reusable tokens and raises the cost to 5.74 × 1014 FLOPs. An edit at the start, equivalent here…
Figure 15 of the paper
Figure 15. Context pressure of every ContextBench level. Each row varies one generator setting (unit at right); the label on each point is its value. Total input is every message the environment pushes over the episode, including the task instruction and, for Sudoku, the board once per move, counted in o200k tokens; pressure is total input divided by 32,768. turn protocol, answer format, and grading procedure, without giving context-management advice. The method skill explains how to manage context using the tools exposed by that harness, including the relevant commands or tool calls, when…
Figure 16 of the paper
Figure 16. Operations as the agent receives them, from the smallest level of each task, shortened where marked. Blue text is the output the agent is expected to produce in response; it is graded from the agent’s context. Needle Retention asks for no output: its needles are graded in the final context. 20
Figure 22 of the paper
Figure 22. Context length awareness. Each point compares a model’s estimated context length with the provider-reported prompt length for the same 50 inputs. Panels vary the available hint: none or a token-count anchor at 25%, 50%, or 75% of the input. The dashed line indicates perfect calibration; legend values report mean absolute error (MAE) in tokens. 27

Showing 5 of the paper's 22 figures. Figures are the authors', reproduced from the paper.

References (75)
  1. [1] Lakshya A Agrawal, Shangyin Tan, Dilara Soylu, Noah Ziems, Rishi Khare, Krista Opsahl-Ong, Arnav Singhvi, Herumb Shandilya, Michael J Ryan, Meng Jiang, et al. Gepa: Reflective prompt evolution can outperform reinforcement learning. In International Conference on Learning Repre... on Pith
  2. [2] Daman Arora and Andrea Zanette. Training language models to reason efficiently. Advances in Neural Information Processing Systems, 38: 0 60770--60808, 2026 on Pith
  3. [3] Federico Cassano and Sasha Rush. Training Composer for longer horizons. Cursor Research Blog, March 2026. https://cursor.com/blog/self-summarization. Published March 17, 2026; accessed July 29, 2026 on Pith
  4. [4] Aaron Chan, Ahmed Shalaby, Alexander Wettig, Aman Sanger, Andrew Zhai, Anurag Ajay, Ashvin Nair, Charlie Snell, Chen Lu, Chen Shen, et al. Composer 2 technical report. arXiv e-prints, pages arXiv--2603, 2026 on Pith
  5. [6] In Gim, Guojun Chen, Seung-seob Lee, Nikhil Sarda, Anurag Khandelwal, and Lin Zhong. Prompt cache: Modular attention reuse for low-latency inference. In Proceedings of Machine Learning and Systems, 2024 on Pith
  6. [7] Zhenyu He, Jun Zhang, Shengjie Luo, Jingjing Xu, Zhi Zhang, and Di He. Let the code llm edit itself when you edit the code. In International Conference on Learning Representations, volume 2025, pages 59637--59653, 2025 on Pith
  7. [8] Junhao Hu, Wenrui Huang, Weidong Wang, Haoyi Wang, Tiancheng Hu, Qin Zhang, Hao Feng, Xusheng Chen, Yizhou Shan, and Tao Xie. EPIC : Efficient position-independent caching for serving large language models. In Proceedings of the 42nd International Conference on Machine Learnin... on Pith
  8. [11] Mo Li, LH Xu, Qitai Tan, Long Ma, Hongyong Song, Ting Cao, and Yunxin Liu. Sculptor: Empowering llms with cognitive agency via active context management. In International Conference on Learning Representations, volume 2026, pages 153411--153440, 2026 a on Pith
  9. [14] Yujiang Li, Zhenyu Hou, Yi Jing, Jie Tang, and Yuxiao Dong. CompactionRL : Reinforcement learning with context compaction for long-horizon agents. arXiv preprint arXiv:2607.05378, 2026 d on PithPith's reviewarXiv
  10. [15] Shukai Liu, Bo Jiang, Jian Yang, Yizhi Li, Jinyang Guo, Xianglong Liu, and Bryan Dai. Context as a tool: Context management for long-horizon swe-agents. In Findings of the Association for Computational Linguistics: ACL 2026, pages 20604--20617, 2026 on Pith
  11. [18] OpenAI . Codex CLI : A coding agent for the terminal. https://github.com/openai/codex, 2026 a . Software, version 0.146.0, accessed July 29, 2026 on Pith
  12. [19] OpenAI . Self-generated prompt injections in compaction summaries, September 2026 b . https://alignment.openai.com/misalignment-reports/self-generated-prompt-injections-in-compaction-summaries/ on Pith
  13. [20] Bespoke Labs OpenThoughts-Agent team, Snorkel AI. OpenThoughts-TBLite: A High-Signal Benchmark for Iterating on Terminal Agents . https://www.openthoughts.ai/blog/openthoughts-tblite, February 2026 on Pith
  14. [24] Asankhaya Sharma. Openevolve: an open-source evolutionary coding agent, 2025. https://github.com/algorithmicsuperintelligence/openevolve on Pith
  15. [26] Richard S. Sutton. The bitter lesson. http://www.incompleteideas.net/IncIdeas/BitterLesson.html, 2019 on Pith
  16. [29] Sikuan Yan, Xiufeng Yang, Zuchao Huang, Ercong Nie, Zifeng Ding, Zonggen Li, Xiaowen Ma, Jinhe Bi, Kristian Kersting, Jeff Z Pan, et al. Memory-r1: Enhancing large language model agents to manage and utilize memories via reinforcement learning. In Proceedings of the 64th Annua... on Pith
  17. [30] John Yang, Carlos E Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik R Narasimhan, and Ofir Press. SWE -agent: Agent-computer interfaces enable automated software engineering. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. h... on PithPith's reviewarXiv
  18. [31] Jiayi Yao, Hanchen Li, Yuhan Liu, Siddhant Ray, Yihua Cheng, Qizheng Zhang, Kuntai Du, Shan Lu, and Junchen Jiang. CacheBlend : Fast large language model serving for RAG with cached knowledge fusion. In Proceedings of the Twentieth European Conference on Computer Systems, page... on Pith
  19. [38] Xuan Zhang, Longtao Zheng, Cunxiao Du, Bo An, and Xin Dong. Autocompact: Learning when to compact context in long-horizon coding agents, 2026 b . https://autocompact.github.io/ on Pith
  20. [39] Lianmin Zheng, Liangsheng Yin, Zhiqiang Xie, Chuyue Sun, Jeff Huang, Cody H Yu, Shiyi Cao, Christos Kozyrakis, Ion Stoica, Joseph E Gonzalez, et al. Sglang: Efficient execution of structured language model programs. Advances in neural information processing systems, 37: 0 6255... on Pith
show 55 more
  1. [40] Zijian Zhou, Ao Qu, Zhaoxuan Wu, Sunghwan Kim, Alok Prakash, Daniela Rus, Jinhua Zhao, Bryan Kian Hsiang Low, and Paul Pu Liang. MEM1 : Learning to synergize memory and reasoning for efficient long-horizon agents. In International Conference on Learning Representations, volume... on Pith
  2. [42] Are Performance-Optimization Benchmarks Reliably Measuring Coding Agents? , author=. arXiv preprint arXiv:2607.01211 , year= on PithPith's reviewarXiv
  3. [43] Recursive language models , author=. arXiv preprint arXiv:2512.24601 , year= on PithPith's reviewarXiv
  4. [44] AutoMem: Automated Learning of Memory as a Cognitive Skill , author=. arXiv preprint arXiv:2607.01224 , year= on PithPith's reviewarXiv
  5. [45] Memory-r1: Enhancing large language model agents to manage and utilize memories via reinforcement learning , author=. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages= on Pith
  6. [46] Agentic memory: Learning unified long-term and short-term memory management for large language model agents , author=. arXiv preprint arXiv:2601.01885 , year= on PithPith's reviewarXiv
  7. [47] Meta-harness: End-to-end optimization of model harnesses , author=. arXiv preprint arXiv:2603.28052 , year= on PithPith's reviewarXiv
  8. [48] Richard S. Sutton , title =. 2019 , howpublished = on Pith
  9. [49] Darwin Godel Machine: Open-Ended Evolution of Self-Improving Agents , author=. arXiv preprint arXiv:2505.22954 , year= on PithPith's reviewarXiv
  10. [50] Hyperagents , author=. arXiv preprint arXiv:2603.19461 , year= on PithPith's reviewarXiv
  11. [51] Tmax: A simple recipe for terminal agents , author=. 2026 , eprint= on Pith
  12. [52] Cassano, Federico and Rush, Sasha , year =. Training on Pith
  13. [53] Terminal-bench: Benchmarking agents on hard, realistic tasks in command line interfaces , author=. arXiv preprint arXiv:2601.11868 , year= on PithPith's reviewarXiv
  14. [54] 2026 , howpublished = on Pith
  15. [55] Laguna M. 1/XS. 2 Technical Report , author=. arXiv preprint arXiv:2605.27605 , year= on PithPith's reviewarXiv
  16. [56] OpenForgeRL: Train Harness-native Agents in Any Environment , author=. arXiv preprint arXiv:2607.21557 , year= on PithPith's reviewarXiv
  17. [57] Co-Harness: Co-Evolving Harnesses and Model Weights for LLM Agents , author=. arXiv preprint arXiv:2607.22688 , year= on PithPith's reviewarXiv
  18. [58] AutoCompact: Learning When to Compact Context in Long-Horizon Coding Agents , author =. 2026 , url = on Pith
  19. [59] Context as a tool: Context management for long-horizon swe-agents , author=. Findings of the Association for Computational Linguistics: ACL 2026 , pages= on Pith
  20. [60] ACM: Agentic Context Management for Long Horizon Tasks , author=. arXiv preprint arXiv:2607.23809 , year= on PithPith's reviewarXiv
  21. [61] Scaling long-horizon llm agent via context-folding , author=. arXiv preprint arXiv:2510.11967 , year= on PitharXiv
  22. [62] METR , howpublished =. Brief independent investigation of agents’ behavior, reasoning and collaboration in the. 2026 , month = on Pith
  23. [63] Browsecomp-plus: A more fair and transparent evaluation benchmark of deep-research agent , author=. arXiv preprint arXiv:2508.06600 , year= on PithPith's reviewarXiv
  24. [64] OpenThoughts-Agent team, Snorkel AI, Bespoke Labs , month = Feb, title = on Pith
  25. [65] React: Synergizing reasoning and acting in language models , author=. arXiv preprint arXiv:2210.03629 , year= on PithPith's reviewarXiv
  26. [66] Zhou, Zijian and Qu, Ao and Wu, Zhaoxuan and Kim, Sunghwan and Prakash, Alok and Rus, Daniela and Zhao, Jinhua and Low, Bryan Kian Hsiang and Liang, Paul Pu , booktitle= on Pith
  27. [67] Self-Compacting Language Model Agents , author=. arXiv preprint arXiv:2606.23525 , year= on PithPith's reviewarXiv
  28. [68] Sculptor: Empowering llms with cognitive agency via active context management , author=. International Conference on Learning Representations , volume= on Pith
  29. [69] John Yang and Carlos E Jimenez and Alexander Wettig and Kilian Lieret and Shunyu Yao and Karthik R Narasimhan and Ofir Press , booktitle=. 2024 , url= on Pith
  30. [70] EdgeBench: Unveiling Scaling Laws of Learning from Real-World Environments , author=. arXiv preprint arXiv:2607.05155 , year= on PithPith's reviewarXiv
  31. [71] Alphaevolve: A coding agent for scientific and algorithmic discovery , author=. arXiv preprint arXiv:2506.13131 , year= on PithPith's reviewarXiv
  32. [72] OpenEvolve: an open-source evolutionary coding agent , author =. 2025 , publisher = on Pith
  33. [73] Composer 2 technical report , author=. arXiv e-prints , pages= on Pith
  34. [74] Openresearcher: A fully open pipeline for long-horizon deep research trajectory synthesis , author=. arXiv preprint arXiv:2603.20278 , year= on PithPith's reviewarXiv
  35. [75] Deepseekmath: Pushing the limits of mathematical reasoning in open language models , author=. arXiv preprint arXiv:2402.03300 , year= on PithPith's reviewarXiv
  36. [76] NeedleBench: Can LLMs Do Retrieval and Reasoning in Information-Dense Context? , author=. 2025 , eprint= on Pith
  37. [77] How long can context length of open-source llms truly promise? , author=. NeurIPS 2023 Workshop on Instruction Tuning and Instruction Following , year= on Pith
  38. [78] Gepa: Reflective prompt evolution can outperform reinforcement learning , author=. International Conference on Learning Representations , volume= on Pith
  39. [79] Li, Yujiang and Hou, Zhenyu and Jing, Yi and Tang, Jie and Dong, Yuxiao , journal= on Pith
  40. [80] Kwon, Woosuk and Li, Zhuohan and Zhuang, Siyuan and Sheng, Ying and Zheng, Lianmin and Yu, Cody Hao and Gonzalez, Joseph E. and Zhang, Hao and Stoica, Ion , booktitle=. Efficient Memory Management for Large Language Model Serving with on Pith
  41. [81] Sglang: Efficient execution of structured language model programs , author=. Advances in neural information processing systems , volume= on Pith
  42. [82] Prompt Cache: Modular Attention Reuse for Low-Latency Inference , author=. Proceedings of Machine Learning and Systems , year= on Pith
  43. [83] Yao, Jiayi and Li, Hanchen and Liu, Yuhan and Ray, Siddhant and Cheng, Yihua and Zhang, Qizheng and Du, Kuntai and Lu, Shan and Jiang, Junchen , booktitle= on Pith
  44. [84] Hu, Junhao and Huang, Wenrui and Wang, Weidong and Wang, Haoyi and Hu, Tiancheng and Zhang, Qin and Feng, Hao and Chen, Xusheng and Shan, Yizhou and Xie, Tao , booktitle= on Pith
  45. [85] Yang, Xinyu and Chen, Tianqi and Chen, Beidi , booktitle= on Pith
  46. [86] Efficient Streaming Language Models with Attention Sinks , author=. International Conference on Learning Representations , year= on Pith
  47. [87] Agentfold: Long-horizon web agents with proactive context management , author=. arXiv preprint arXiv:2510.24699 , year= on PitharXiv
  48. [88] Meta context engineering via agentic skill evolution , author=. arXiv preprint arXiv:2601.21557 , year= on PitharXiv
  49. [89] Memgpt: Towards llms as operating systems , author=. arXiv preprint arXiv:2310.08560 , year= on PithPith's reviewarXiv
  50. [90] Let the code llm edit itself when you edit the code , author=. International Conference on Learning Representations , volume= on Pith
  51. [91] Resum: Unlocking long-horizon search intelligence via context summarization , author=. arXiv preprint arXiv:2509.13313 , year= on PitharXiv
  52. [92] Self-generated prompt injections in compaction summaries , year = on Pith
  53. [93] Memento: Teaching llms to manage their own context , author=. arXiv preprint arXiv:2604.09852 , year= on PithPith's reviewarXiv
  54. [94] Training language models to reason efficiently , author=. Advances in Neural Information Processing Systems , volume= on Pith
  55. [95] Think dense, not long: Dynamic decoupled conditional advantage for efficient reasoning , author=. arXiv preprint arXiv:2602.02099 , year= on PitharXiv
Cited by 11 Pith papers

Challenges and discussion

Disagree with the review, or know something it missed? Leave a comment, or challenge this review formally: a challenge becomes a permanent entry in this paper's record, and its decision is published here.

No comments yet.

Sign in to comment