Pith. sign in

REVIEW 3 major objections 5 minor 1 cited by

When Does Continual Learning Require Learning

T0 review · 3 major / 5 minor · reviewed 2026-07-10 · grok-4.5

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Continual learning is not one skill: different kinds of world change demand different kinds of model updates.

desk verdict Useful regime map for LLM continual learning; the framing and protocol are the real product, not a settled rule about when weights are required. read the letter →

arxiv 2607.07847 v1 pith:VKNIHOHJ submitted 2026-07-08 cs.LG

Anne Harrington , Nayan Saxena , Michael Murphy , Anastasia Borovykh , Zeyu Yun , Sridhar Kamath , Ara Eindra Kyi , Trevor Darrell
show 2 more authors
Jitendra Malik Yutong Bai
This is my paper · ORCID
classification cs.LG
keywords continuallearninglargelanguagemodelsdomainshifttemporaldriftcatastrophicforgettingpromptoptimizationself-distillationreinforcement
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The field has treated continual learning mainly as context management and avoiding forgetting. This paper reframes it as the harder problem of raising competence while the world itself changes. Change is split into space (new domains) and time (drift under a fixed task), with time further covering discrete fact revisions, slow noisy trends, and state that accumulates from an agent’s own actions. Under one shared protocol that scores prompt, weight, and compression methods the same way, eight techniques show consistent trade-offs: prompts lock onto the current stage and then forget; distillation accumulates steadily but resists rewriting outdated beliefs; compression saves tokens without teaching new tasks; online reinforcement rewrites facts best but collapses when rewards are noisy. The practical message is that designers must match the update mechanism to the pattern of environmental change rather than search for a single universal continual learner.

What carries the argument

A mechanism-agnostic sequential protocol: stages arrive in fixed order, each method applies an unrestricted update operator under a shared compute budget, and performance is scored with a forgetting matrix that yields backward and forward transfer. This single yardstick lets prompt, weight, and architectural methods be compared on equal ground.

What would settle it

Repeat the same four sequential suites on a substantially larger reasoning model (or with systematically permuted stage orders) and check whether the reported method-by-regime ranking reverses.

Watch

Extended reading notes

Core claim

Continual learning is not a single capability. Different patterns of environmental change—domain shift, discrete fact updates, noisy temporal drift, and agent-driven state accumulation—require fundamentally different update behaviors, which in turn decide when adaptation must live inside model weights and when external scaffolding is enough.

Load-bearing premise

That results on one mid-size non-reasoning backbone under a fixed per-stage budget and the authors’ sequential adaptations of each method will hold for larger or reasoning models and for other task orderings.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper reframes continual learning for LLMs as increasing competence under environmental change, disentangled along space (domain shift) and time (discrete fact updates, continuous temporal drift, and agentic state accumulation). It introduces a mechanism-agnostic staged protocol (θ_k = U_k(θ_{k-1}, D_tr_k) with fixed per-stage budget C) and evaluates eight methods from four families—prompt optimization (GEPA, ACE), offline supervised updates (SFT, SDFT), online RL (GRPO, SDPO), and context compression (Cartridges, In-place TTT)—on four sequential recastings of standard LLM benchmarks (domain chain, TempWiki, 10-K sentiment, WebArena-style agentic chains). Empirically, prompt methods fit stages quickly but degrade on future tasks; distillation accumulates more stably but resists fact rewrite; compression improves efficiency without clear task acquisition; and online RL adapts best to knowledge updates but is sensitive to noisy rewards. The central claim is that continual learning is not a single capability and that different patterns of change require different update behaviors, determining when adaptation must occur in weights versus external scaffolding.

Significance. If the comparative map is reliable, this is a useful organizing contribution for LLM continual learning. The space/time framing separates regimes the literature often conflates; the protocol lets prompt, weight, and architectural updates be compared on equal ground; and the four sequential settings (especially TempWiki with a stable-fact probe, 10-K forward/backward transfer under weak signal, and agentic chains whose order is generated by the agent) are carefully constructed and more realistic than static CL task sequences. Strengths include an explicit sequential adaptation for each method (Table 1), standard transfer metrics (BWT/FWT), released code, and an honest limitations section. The work is diagnostic rather than algorithmic: its value is in guiding evaluation practice and method design. The main risk is over-generalizing family-level trade-offs under one backbone and author-defined sequential lifts into a general rule about when weights are required.

major comments (3)
  1. Abstract and §5.1 claim that different environmental-change patterns determine when adaptation must be learned inside model weights versus external scaffolding. The evidence is a qualitative map of eight operators on four regimes with a single backbone (Qwen3-8B non-thinking). §5.2 already notes relative behavior may change for larger or reasoning models. As written, the leap from “these operators trade off under our protocol” to a general “when weights are required” boundary is stronger than the multi-model support. Either soften the claim to a protocol-level empirical map, or add at least one additional backbone / reasoning-mode check on a subset of regimes so the boundary is not backbone-contingent.
  2. §3.1–3.2 and Table 1: fairness of the mechanism-agnostic protocol hinges on author-defined sequential lifts (θ_{k-1} as teacher for SDFT; θ_{k-1} preference generator for SDPO; carried prompt/playbook for GEPA/ACE; new adapter per stage for Cartridges; resume from θ_{k-1} for SFT/GRPO). These choices are explicit and reasonable, but they are not unique, and they can drive rankings as much as the space/time axes. The manuscript should state more clearly which conclusions are robust to alternative sequential adaptations (e.g., fixed teacher, replay, or shared reference for DPO-style methods) and which are specific to the chosen lifts; a short sensitivity check on one regime would substantially strengthen the “fair comparison” claim.
  3. Figures 2–4 (and the corresponding TempWiki/10-K curves): all primary comparisons are single-run accuracy/F1 trajectories without error bars, seeds, or confidence intervals. Appendix A.1 reports one epoch over 500 examples and method-specific outer learning rates chosen to avoid collapse (Table 2). Under a fixed nominal budget C but different LRs and loss geometries, small differences (e.g., GRPO’s +1.6 drift points, Cartridges near-baseline) are hard to interpret. Report multi-seed means and variability for at least the headline transfer quantities (final-stage scores, BWT/FWT, drift vs. stable F1), or justify single-run reporting with a stability argument.
minor comments (5)
  1. §3.1 defines BWT and FWT but the main figures emphasize per-stage accuracy curves; a compact table of BWT/FWT per method per benchmark would make the transfer claims easier to audit.
  2. Figure 1 caption lists four axes (domain shift, fact update, temporal drift, agentic state) while the text organizes change as two axes (space and time) with time sub-regimes; align the figure language with §1/§4 to avoid a three- vs. four-axis reading.
  3. Appendix A.2.3 (TempWiki-Easy) is important for the catastrophic-memorizing claim; a one-sentence pointer in §4.2 main text would help readers who skip the appendix.
  4. Agentic results (§4.4) use Qwen-32B for ACE and Qwen-8B for SFT, unlike the 8B backbone elsewhere; state this more prominently when comparing to other sections so cross-regime conclusions are not over-read.
  5. Typos/clarity: “abroadrange” (p.1), “and and agentic” (§4), “less sensitive to than prompt methods” (§4.1); fix for camera-ready.

Circularity Check

0 steps flagged · score 0.0 of 10

Empirical comparative study with no derivation chain that reduces predictions to fitted inputs or self-definitional claims.

full rationale

This paper is a mechanism-agnostic empirical comparison of eight continual-learning operators (prompt, SFT/distillation, RL, context compression) under a shared staged protocol on recast LLM benchmarks. Its central claim—that different space/time change regimes favor different update behaviors—is an interpretation of measured accuracy, F1, BWT/FWT, and chain-success curves on held-out eval sets, not a quantity derived from a fitted parameter and then re-labeled as a prediction. BWT and FWT are the standard Lopez-Paz–Ranzato transfer summaries; sequential adaptations (θ_{k−1} as teacher for SDFT/SDPO, carried prompt/playbook for GEPA/ACE, new adapter per stage for Cartridges) are explicit design choices for lifting single-stage methods, not hidden tautologies. Method citations are to external prior work used as baselines, not load-bearing uniqueness theorems by the same authors. There is no self-definitional loop, no fitted-input-called-prediction, and no renaming of a known closed-form result as a first-principles derivation. Circularity burden is therefore zero; residual concerns about backbone choice and fairness of adaptations belong to external validity, not circularity.

Assumptions & free parameters 4 free parameters · 5 assumptions · 3 invented entities

The central claim is an empirical generalization from four constructed regimes under one protocol. It rests on a definitional reframing of continual learning, on the sufficiency of the space/time axes, on the fairness of the sequential adaptations and fixed budget C, and on a large set of method- and benchmark-specific hyper-parameters. No new physical entities are postulated; the invented constructs are conceptual (axes, protocol, “catastrophic memorizing”). Free parameters are the usual training knobs that affect absolute numbers but not the qualitative ranking narrative.

free parameters (4)
  • per-method outer learning rates = 1e-6 / 1e-5 / 3e-6 depending on method-benchmark pair
    Table 2 sets distinct LRs (1e-6 to 1e-5, with warmups) per method and benchmark to avoid collapse; absolute accuracies and some transfer numbers depend on these hand-chosen values.
  • per-stage example count and compute budget C = 500 examples, ~16 steps
    500 training examples and ~16 optimizer steps per phase are fixed by design; changing them would alter acquisition speed and forgetting curves.
  • Cartridges KV length and synthesis size = 2048 / 8192
    2048-token KV and 8192 synthesized Q/A pairs per phase are chosen hyper-parameters that control capacity of the compression baseline.
  • In-place TTT outer steps, layer set, inner LR = 50 steps, ttt_lr=3
    50 outer steps, sparse layers [0,6,...,36], ttt_lr=3 are paper-chosen; they define the locality reference.
assumptions (5)
  • ad hoc to paper Continual learning is defined as increasing model competence as the world changes, rather than solely as mitigating catastrophic forgetting or managing context.
    Stated in abstract and §1; this reframing is the paper’s starting premise, not a theorem derived from prior CL literature.
  • domain assumption Environmental change for LLMs is adequately captured by the two axes of space (domain shift) and time (fact update, temporal drift, agentic state accumulation).
    Introduced in §1 and Figure 1; the four experimental settings instantiate these axes. Limitations acknowledge that real deployments are more open-ended.
  • domain assumption BWT and FWT computed from the staged accuracy matrix are the right transfer quantities for comparing any update operator.
    §3.1 adopts Lopez-Paz & Ranzato-style BWT plus a forward-transfer definition; fairness of the protocol rests on these metrics.
  • ad hoc to paper Sequential adaptations (resume from θ_{k-1}, previous model as teacher/preference generator, carried prompt/playbook, new adapter per stage) preserve the identity of each method family while making them comparable.
    §3.2 and Table 1; different adaptations could change relative rankings.
  • domain assumption Standard evaluation metrics (task accuracy, word-level F1 ≥ 0.5, binary up/down, programmatic environment verifiers) correctly measure competence.
    Used throughout §4; format-fidelity is intentionally entangled with knowledge.
invented entities (3)
  • mechanism-agnostic staged protocol (θ_k = U_k(θ_{k-1}, D_tr_k) with fixed C)
    purpose: Allows prompt, weight, and architectural updates to be compared on equal ground via a common forgetting matrix.
    Defined in §3.1; the protocol itself is the paper’s main methodological invention.
  • catastrophic memorizing
    purpose: Names the failure mode of overwriting stable knowledge while acquiring drifted facts (mirror image of catastrophic forgetting).
    Introduced in §4.2; useful diagnostic label but not independently measured outside this paper’s TempWiki construction.
  • space/time axes of change for LLM continual learning
    purpose: Organizes domain shift versus fact update, temporal drift, and agentic accumulation under one taxonomy.
    Figure 1 and §1; conceptual scaffold rather than an empirically validated ontology of all possible change.

how reviews work

0 comments
Cite this review

Pith. "Pith review of When Does Continual Learning Require Learning." pith.science (2026). https://pith.science/paper/VKNIHOHJ

@misc{pith2026260707847,
  author       = {Pith},
  title        = {Pith review of: When Does Continual Learning Require Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/VKNIHOHJ}},
  note         = {Machine review of arXiv:2607.07847}
}
read the original abstract

As large language models (LLMs) become increasingly capable, the next question is how can we enable models to continually learn? Today, the field largely frames this as a problem of context management and mitigating forgetting. We argue this framing is incomplete: continual learning is fundamentally about increasing model competence as the world changes. We disentangle this change along two axes -- space, where the model encounters new domains, and time, where the underlying data drifts under a fixed task. This framing lets us study continual learning under realistic conditions: new domains arrive over time, facts drift past their training cutoff, and agentic interactions accumulate state across episodes. To evaluate methods under this setting, we recast widely used LLM benchmarks as sequential problems and introduce a single mechanism-agnostic protocol that compares prompt-based methods (GEPA, ACE), supervised learning (SFT, SDFT), reinforcement learning (GRPO, SDPO), and context compression (Cartridges, In-place TTT). Prompt-based methods fit each new stage quickly but degrade on future tasks. Distillation-based methods accumulate knowledge stably but struggle to update outdated facts. Context compression improves efficiency without substantially improving the ability to learn new tasks. Online reinforcement learning adapts most effectively to knowledge updates but remains sensitive to noisy reward signals. Overall, our results suggest that continual learning is not a single capability: different patterns of environmental change require fundamentally different update behaviors, determining when adaptation must be learned inside model weights and when it can be achieved through external scaffolding. We hope that understanding where each method succeeds and fails will guide the design of stronger continual learning systems.

Figures

Figures reproduced from arXiv: 2607.07847 by the authors.

Figure 1
Figure 1. A unified continual learning framework spanning axes of change and update mech￾anisms. We identify four axes along which tasks evolve: domain shift, fact update, temporal drift, and agentic state. All methods are cast within a common framework where an experience stream triggers updates to one of three internal components: context, parameters, or memory/state. Existing methods are unified accordingly: prompting meth… view at source ↗
Figure 2
Figure 2. Per-method accuracy across the three-stage domain chain (ToolUse [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. F1 score across the three TempWiki slices for each method (Qwen3-8B, no thinking). [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (6 more)
Figure 4
Figure 4. Figure 4: Backward and forward accuracy on the 10-K sentiment task, evaluated after sequential [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: Prompt evolution on the 10-K sentiment task: (L) how an idealised analyst may evolve [PITH_FULL_IMAGE:figures/full_fig_p009_5.png]
Figure 6
Figure 6. Figure 6: End-to-end chain success rate across chain lengths L1–L10 on the Gmail agentic suite. [PITH_FULL_IMAGE:figures/full_fig_p010_6.png]
Figure 7
Figure 7. Figure 7: TempWiki-Easy: object F1 across the three drift slices (Qwen3-8B, no thinking, seed 42). [PITH_FULL_IMAGE:figures/full_fig_p019_7.png]
Figure 8
Figure 8. Figure 8: A PnL constructed from retaining only Items 2, 5, 7, 7A, 8, 9, and 9A from the 10-K filings. [PITH_FULL_IMAGE:figures/full_fig_p021_8.png]
Figure 9
Figure 9. Figure 9: The change in abstain-rate per section. On the aggregate, predicting more often (abstain-rate [PITH_FULL_IMAGE:figures/full_fig_p021_9.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Macaron-V1: Towards Open Continual Learning with Self-Improvement and Mixture-of-LoRA

    cs.LG 2026-08 conditional novelty 6.0 of 10

    The released Macaron-V1-Venti model uses a frozen 744B base plus four per-turn-routed LoRA specialists and reports high internal benchmark scores, but it does not demonstrate cross-generation continual-learning gains.

Reference graph

Works this paper leans on

61 extracted references · 61 canonical work pages · cited by 1 Pith paper

  1. [1]

    gpt-oss-120b & gpt-oss-20b Model Card

    Sandhini Agarwal, Lama Ahmad, Jason Ai, Sam Altman, Andy Applebaum, Edwin Arbus, Rahul K Arora, Yu Bai, Bowen Baker, Haiming Bao, et al. gpt-oss-120b & gpt-oss-20b model card.arXiv preprint arXiv:2508.10925, 2025

    work page Pith review arXiv 2025

  2. [2]

    GEPA: Reflective Prompt Evolution Can Outperform Reinforcement Learning

    Lakshya A Agrawal, Shangyin Tan, Dilara Soylu, Noah Ziems, Rishi Khare, Krista Opsahl-Ong, Arnav Singhvi, Herumb Shandilya, Michael J Ryan, Meng Jiang, et al. Gepa: Reflective prompt evolution can outperform reinforcement learning.arXiv preprint arXiv:2507.19457, 2025

    work page Pith review arXiv 2025

  3. [3]

    Language models are few-shot learners.Advances in neural information processing systems, 33:1877–1901, 2020

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners.Advances in neural information processing systems, 33:1877–1901, 2020

    work page 1901

  4. [4]

    Adaevolve: Adaptive llm driven zeroth-order optimization.arXiv preprint arXiv:2602.20133, 2026

    Mert Cemri, Shubham Agrawal, Akshat Gupta, Shu Liu, Audrey Cheng, Qiuyang Mang, Ashwin Naren, Lutfi Eren Erdogan, Koushik Sen, Matei Zaharia, et al. Adaevolve: Adaptive llm driven zeroth-order optimization.arXiv preprint arXiv:2602.20133, 2026

    work page arXiv 2026

  5. [5]

    Finqa: A dataset of numerical reasoning over financial data

    Zhiyu Chen, Wenhu Chen, Charese Smiley, Sameena Shah, Iana Borova, Dylan Langdon, Reema Moussa, Matt Beane, Ting-Hao Huang, Bryan R Routledge, et al. Finqa: A dataset of numerical reasoning over financial data. InProceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 3697–3711, 2021

    work page 2021

  6. [6]

    Su- perposition of many models into one.Advances in neural information processing systems, 32, 2019

    Brian Cheung, Alexander Terekhov, Yubei Chen, Pulkit Agrawal, and Bruno Olshausen. Su- perposition of many models into one.Advances in neural information processing systems, 32, 2019

    work page 2019

  7. [7]

    Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities

    Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, et al. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities.arXiv preprint arXiv:2507.06261, 2025

    work page Pith review arXiv 2025

  8. [8]

    Loss of plasticity in deep continual learning.Nature, 632 (8026):768–774, 2024

    Shibhansh Dohare, J Fernando Hernandez-Garcia, Qingfeng Lan, Parash Rahman, A Rupam Mahmood, and Richard S Sutton. Loss of plasticity in deep continual learning.Nature, 632 (8026):768–774, 2024

    work page 2024

Show all 61 references
  1. [9]

    Adversarial continual learning

    Sayna Ebrahimi, Franziska Meier, Roberto Calandra, Trevor Darrell, and Marcus Rohrbach. Adversarial continual learning. InEuropean conference on computer vision, pages 386–402. Springer, 2020

    2020

  2. [10]

    Cartridges: Lightweight and general- purpose long context representations via self-study.arXiv preprint arXiv:2506.06266, 2025

    Sabri Eyuboglu, Ryan Ehrlich, Simran Arora, Neel Guha, Dylan Zinsley, Emily Liu, Will Tennien, Atri Rudra, James Zou, Azalia Mirhoseini, et al. Cartridges: Lightweight and general- purpose long context representations via self-study.arXiv preprint arXiv:2506.06266, 2025

    2025 arXiv

  3. [11]

    In-place test-time training.arXiv preprint arXiv:2604.06169, 2026

    Guhao Feng, Shengjie Luo, Kai Hua, Ge Zhang, Di He, Wenhao Huang, and Tianle Cai. In-place test-time training.arXiv preprint arXiv:2604.06169, 2026. 11

    2026 arXiv

  4. [12]

    Sciknoweval: Evaluating multi-level scientific knowledge of large language models

    Kehua Feng, Xinyi Shen, Weijie Wang, Xiang Zhuang, Yuqi Tang, Qiang Zhang, and Keyan Ding. Sciknoweval: Evaluating multi-level scientific knowledge of large language models. arXiv preprint arXiv:2406.09098, 2024

    2024

  5. [13]

    Catastrophic forgetting in connectionist networks.Trends in cognitive sciences, 3(4):128–135, 1999

    Robert M French. Catastrophic forgetting in connectionist networks.Trends in cognitive sciences, 3(4):128–135, 1999

    1999

  6. [14]

    Frontiermath: A benchmark for evaluating advanced mathematical reasoning in ai.arXiv preprint arXiv:2411.04872, 2024

    Elliot Glazer, Ege Erdil, Tamay Besiroglu, Diego Chicharro, Evan Chen, Alex Gunning, Caroline Falkman Olsson, Jean-Stanislas Denain, Anson Ho, Emily de Oliveira Santos, et al. Frontiermath: A benchmark for evaluating advanced mathematical reasoning in ai.arXiv preprint arXiv:2...

    2024 arXiv

  7. [15]

    Continual learning via neural pruning

    Siavash Golkar, Michael Kagan, and Kyunghyun Cho. Continual learning via neural pruning. arXiv preprint arXiv:1903.04476, 2019

    1903 arXiv

  8. [16]

    Kellybench: A benchmark for long-horizon sequential decision making.arXiv preprint arXiv:2604.27865, 2026

    Thomas Grady, Kip Parker, Iliyan Zarov, Henry Course, Chengxi Taylor, and Ross Taylor. Kellybench: A benchmark for long-horizon sequential decision making.arXiv preprint arXiv:2604.27865, 2026

    2026 arXiv

  9. [17]

    Measuring mathematical problem solving with the math dataset

    Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2103.03874, 2021

    2021 arXiv

  10. [18]

    Lora: Low-rank adaptation of large language models.Iclr, 1(2):3, 2022

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Liang Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models.Iclr, 1(2):3, 2022

    2022

  11. [19]

    Reinforcement learning via self-distillation.arXiv preprint arXiv:2601.20802, 2026

    Jonas Hübotter, Frederike Lübeck, Lejs Behric, Anton Baumann, Marco Bagatella, Daniel Marta, Ido Hakimi, Idan Shenfeld, Thomas Kleine Buening, Carlos Guestrin, et al. Reinforcement learning via self-distillation.arXiv preprint arXiv:2601.20802, 2026

    2026 arXiv

  12. [20]

    Editing models with task arithmetic.arXiv preprint arXiv:2212.04089, 2022

    Gabriel Ilharco, Marco Tulio Ribeiro, Mitchell Wortsman, Suchin Gururangan, Ludwig Schmidt, Hannaneh Hajishirzi, and Ali Farhadi. Editing models with task arithmetic.arXiv preprint arXiv:2212.04089, 2022

    2022 arXiv

  13. [21]

    Temporalwiki: A lifelong benchmark for training and evaluating ever-evolving language models

    Joel Jang, Seonghyeon Ye, Changho Lee, Sohee Yang, Joongbo Shin, Janghoon Han, Gyeonghun Kim, and Minjoon Seo. Temporalwiki: A lifelong benchmark for training and evaluating ever-evolving language models. InProceedings of the 2022 Conference on Empirical Methods in Natural Lan...

    2022

  14. [22]

    Meta-learning representations for continual learning.Ad- vances in neural information processing systems, 32, 2019

    Khurram Javed and Martha White. Meta-learning representations for continual learning.Ad- vances in neural information processing systems, 32, 2019

    2019

  15. [23]

    Swe-bench: Can language models resolve real-world github issues? In The twelfth international conference on learning representations, 2023

    Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R Narasimhan. Swe-bench: Can language models resolve real-world github issues? In The twelfth international conference on learning representations, 2023

    2023

  16. [24]

    Financial reports sec

    Aman Khan. Financial reports sec. URL https://huggingface.co/datasets/ JanosAudran/financial-reports-sec

  17. [25]

    Dspy: compiling declarative language model calls into state-of-the-art pipelines

    Omar Khattab, Arnav Singhvi, Paridhi Maheshwari, Zhiyuan Zhang, Keshav Santhanam, Saiful Haq, Ashutosh Sharma, Thomas T Joshi, Hanna Moazam, Heather Miller, et al. Dspy: compiling declarative language model calls into state-of-the-art pipelines. InThe Twelfth International Con...

    2023

  18. [26]

    Meta-harness: End-to-end optimization of model harnesses.arXiv preprint arXiv:2603.28052, 2026

    Yoonho Lee, Roshen Nair, Qizheng Zhang, Kangwook Lee, Omar Khattab, and Chelsea Finn. Meta-harness: End-to-end optimization of model harnesses.arXiv preprint arXiv:2603.28052, 2026

    2026 arXiv

  19. [27]

    Retrieval-augmented generation for knowledge-intensive nlp tasks.Advances in neural information processing systems, 33:9459–9474, 2020

    Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. Retrieval-augmented generation for knowledge-intensive nlp tasks.Advances in neural information processing systems,...

    2020

  20. [28]

    Combee: Scaling prompt learning for self-improving language model agents.arXiv preprint arXiv:2604.04247, 2026

    Hanchen Li, Runyuan He, Qizheng Zhang, Changxiu Ji, Qiuyang Mang, Xiaokun Chen, Lakshya A Agrawal, Wei-Liang Liao, Eric Yang, Alvin Cheung, et al. Combee: Scaling prompt learning for self-improving language model agents.arXiv preprint arXiv:2604.04247, 2026

    2026 arXiv

  21. [29]

    Compressing context to enhance inference efficiency of large language models

    Yucheng Li, Bo Dong, Frank Guerin, and Chenghua Lin. Compressing context to enhance inference efficiency of large language models. InProceedings of the 2023 conference on empirical methods in natural language processing, pages 6342–6353, 2023

    2023

  22. [30]

    Continual learning via sparse memory finetuning.arXiv preprint arXiv:2510.15103, 2025

    Jessy Lin, Luke Zettlemoyer, Gargi Ghosh, Wen-Tau Yih, Aram Markosyan, Vincent-Pierre Berges, and Barlas O ˘guz. Continual learning via sparse memory finetuning.arXiv preprint arXiv:2510.15103, 2025

    2025

  23. [31]

    Deepseek-v3

    Aixin Liu, Aoxue Mei, Bangcai Lin, Bing Xue, Bingxuan Wang, Bingzheng Xu, Bochao Wu, Bowei Zhang, Chaofan Lin, Chen Dong, et al. Deepseek-v3. 2: Pushing the frontier of open large language models.arXiv preprint arXiv:2512.02556, 2025

    2025 arXiv

  24. [32]

    Low-redundancy distillation for continual learning.Pattern Recognition, 167:111712, 2025

    Ruiqi Liu, Boyu Diao, Libo Huang, Zijia An, Hangda Liu, Zhulin An, and Yongjun Xu. Low-redundancy distillation for continual learning.Pattern Recognition, 167:111712, 2025

    2025

  25. [33]

    Gradient episodic memory for continual learning

    David Lopez-Paz and Marc’Aurelio Ranzato. Gradient episodic memory for continual learning. Advances in neural information processing systems, 30, 2017

    2017

  26. [34]

    Edgar- corpus, September 2021

    Lefteris Loukas, Manos Fergadiotis, Ion Androutsopoulos, and Prodromos Malakasiotis. Edgar- corpus, September 2021. URLhttps://doi.org/10.5281/zenodo.5528490

    2021 doi

  27. [35]

    On-policy distillation.Thinking Machines Lab: Con- nectionism, 2025

    Kevin Lu and Thinking Machines Lab. On-policy distillation.Thinking Machines Lab: Con- nectionism, 2025. doi: 10.64434/tml.20251026. https://thinkingmachines.ai/blog/on-policy- distillation

    2025 doi

  28. [36]

    Understanding and preventing capacity loss in reinforcement learning.arXiv preprint arXiv:2204.09560, 2022

    Clare Lyle, Mark Rowland, and Will Dabney. Understanding and preventing capacity loss in reinforcement learning.arXiv preprint arXiv:2204.09560, 2022

    2022 arXiv

  29. [37]

    Decoding risk sentiment in 10-k filings: Predictability for us stock indices.Finance Research Letters, 81:107472, 2025

    Nicolás Magner, Pablo A Henríquez, and Aliro Sanhueza. Decoding risk sentiment in 10-k filings: Predictability for us stock indices.Finance Research Letters, 81:107472, 2025

    2025

  30. [38]

    Frontiercs: Evolving challenges for evolving intelligence.arXiv preprint arXiv:2512.15699, 2025

    Qiuyang Mang, Wenhao Chai, Zhifei Li, Huanzhi Mao, Shang Zhou, Alexander Du, Hanchen Li, Shu Liu, Edwin Chen, Yichuan Wang, et al. Frontiercs: Evolving challenges for evolving intelligence.arXiv preprint arXiv:2512.15699, 2025

    2025

  31. [39]

    Some philosophical problems from the standpoint of artificial intelligence

    John McCarthy and Patrick J Hayes. Some philosophical problems from the standpoint of artificial intelligence. InReadings in artificial intelligence, pages 431–450. Elsevier, 1981

    1981

  32. [40]

    Catastrophic interference in connectionist networks: The sequential learning problem

    Michael McCloskey and Neal J Cohen. Catastrophic interference in connectionist networks: The sequential learning problem. InPsychology of learning and motivation, volume 24, pages 109–165. Elsevier, 1989

    1989

  33. [41]

    h1: Bootstrapping llms to reason over longer horizons via reinforcement learning.arXiv preprint arXiv:2510.07312, 2025

    Sumeet Ramesh Motwani, Alesia Ivanova, Ziyang Cai, Philip Torr, Riashat Islam, Shital Shah, Christian Schroeder de Witt, and Charles London. h1: Bootstrapping llms to reason over longer horizons via reinforcement learning.arXiv preprint arXiv:2510.07312, 2025

    2025

  34. [42]

    The primacy bias in deep reinforcement learning

    Evgenii Nikishin, Max Schwarzer, Pierluca D’Oro, Pierre-Luc Bacon, and Aaron Courville. The primacy bias in deep reinforcement learning. InInternational conference on machine learning, pages 16828–16847. PMLR, 2022

    2022

  35. [43]

    Alphaevolve: A coding agent for scientific and algorithmic discovery.arXiv preprint arXiv:2506.13131, 2025

    Alexander Novikov, Ngân V˜u, Marvin Eisenberger, Emilien Dupont, Po-Sen Huang, Adam Zsolt Wagner, Sergey Shirobokov, Borislav Kozlovskii, Francisco JR Ruiz, Abbas Mehrabian, et al. Alphaevolve: A coding agent for scientific and algorithmic discovery.arXiv preprint arXiv:2506.1...

    2025 arXiv

  36. [44]

    Optimizing instructions and demonstrations for multi-stage language model programs

    Krista Opsahl-Ong, Michael J Ryan, Josh Purtell, David Broman, Christopher Potts, Matei Zaharia, and Omar Khattab. Optimizing instructions and demonstrations for multi-stage language model programs. InProceedings of the 2024 Conference on Empirical Methods in Natural Language ...

    2024

  37. [45]

    Kernelbench: Can llms write efficient gpu kernels?arXiv preprint arXiv:2502.10517, 2025

    Anne Ouyang, Simon Guo, Simran Arora, Alex L Zhang, William Hu, Christopher Ré, and Azalia Mirhoseini. Kernelbench: Can llms write efficient gpu kernels?arXiv preprint arXiv:2502.10517, 2025

    2025 arXiv

  38. [46]

    Deepseekmath: Pushing the limits of mathematical reasoning in open language models.arXiv preprint arXiv:2402.03300, 2024

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models.arXiv preprint arXiv:2402.03300, 2024

    2024 arXiv

  39. [47]

    Openevolve: an open-source evolutionary coding agent, 2025

    Asankhaya Sharma. Openevolve: an open-source evolutionary coding agent, 2025. URL https://github.com/algorithmicsuperintelligence/openevolve

    2025

  40. [48]

    Self-distillation enables continual learning.arXiv preprint arXiv:2601.19897, 2026

    Idan Shenfeld, Mehul Damani, Jonas Hübotter, and Pulkit Agrawal. Self-distillation enables continual learning.arXiv preprint arXiv:2601.19897, 2026

    2026 arXiv

  41. [49]

    End-to-end test-time training for long context.arXiv preprint arXiv:2512.23675, 2025

    Arnuv Tandon, Karan Dalal, Xinhao Li, Daniel Koceja, Marcel Rød, Sam Buchanan, Xiaolong Wang, Jure Leskovec, Sanmi Koyejo, Tatsunori Hashimoto, et al. End-to-end test-time training for long context.arXiv preprint arXiv:2512.23675, 2025

    2025

  42. [50]

    Toolalpaca: Generalized tool learning for language models with 3000 simulated cases.arXiv preprint arXiv:2306.05301, 2023

    Qiaoyu Tang, Ziliang Deng, Hongyu Lin, Xianpei Han, Qiao Liang, Boxi Cao, and Le Sun. Toolalpaca: Generalized tool learning for language models with 3000 simulated cases.arXiv preprint arXiv:2306.05301, 2023

    2023 arXiv

  43. [51]

    Mmlu-pro: A more robust and challenging multi-task language understanding benchmark.Advances in Neural Information Processing Systems, 37:95266–95290, 2024

    Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, et al. Mmlu-pro: A more robust and challenging multi-task language understanding benchmark.Advances in Neural Information Processing Systems, 37:95...

    2024

  44. [52]

    Opti- mas: Optimizing compound ai systems with globally aligned local rewards.arXiv preprint arXiv:2507.03041, 2025

    Shirley Wu, Parth Sarthi, Shiyu Zhao, Aaron Lee, Herumb Shandilya, Adrian Mladenic Gro- belnik, Nurendra Choudhary, Eddie Huang, Karthik Subbian, Linjun Zhang, et al. Opti- mas: Optimizing compound ai systems with globally aligned local rewards.arXiv preprint arXiv:2507.03041, 2025

    2025

  45. [53]

    Qwen3 technical report.arXiv preprint arXiv:2505.09388, 2025

    An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report.arXiv preprint arXiv:2505.09388, 2025

    2025 arXiv

  46. [54]

    Meta context engineer- ing via agentic skill evolution.arXiv preprint arXiv:2601.21557, 2026

    Haoran Ye, Xuning He, Vincent Arak, Haonan Dong, and Guojie Song. Meta context engineer- ing via agentic skill evolution.arXiv preprint arXiv:2601.21557, 2026

    2026

  47. [55]

    differentiation

    Mert Yuksekgonul, Federico Bianchi, Joseph Boen, Sheng Liu, Zhi Huang, Carlos Guestrin, and James Zou. Textgrad: Automatic" differentiation" via text.arXiv preprint arXiv:2406.07496, 2024

    2024 arXiv

  48. [56]

    Recursive language models.arXiv preprint arXiv:2512.24601, 2025

    Alex L Zhang, Tim Kraska, and Omar Khattab. Recursive language models.arXiv preprint arXiv:2512.24601, 2025

    2025 arXiv

  49. [57]

    Side- tuning: a baseline for network adaptation via additive side networks

    Jeffrey O Zhang, Alexander Sax, Amir Zamir, Leonidas Guibas, and Jitendra Malik. Side- tuning: a baseline for network adaptation via additive side networks. InEuropean conference on computer vision, pages 698–714. Springer, 2020

    2020

  50. [58]

    Darwin godel machine: Open-ended evolution of self-improving agents.arXiv preprint arXiv:2505.22954, 2025

    Jenny Zhang, Shengran Hu, Cong Lu, Robert Lange, and Jeff Clune. Darwin godel machine: Open-ended evolution of self-improving agents.arXiv preprint arXiv:2505.22954, 2025

    2025 arXiv

  51. [59]

    Agentic context engineering: Evolving contexts for self-improving language models.arXiv preprint arXiv:2510.04618, 2025

    Qizheng Zhang, Changran Hu, Shubhangi Upasani, Boyuan Ma, Fenglu Hong, Vamsidhar Ka- manuru, Jay Rainton, Chen Wu, Mengmeng Ji, Hanchen Li, et al. Agentic context engineering: Evolving contexts for self-improving language models.arXiv preprint arXiv:2510.04618, 2025

    2025 arXiv

  52. [60]

    Self-distilled reasoner: On-policy self-distillation for large language models.arXiv preprint arXiv:2601.18734, 2026

    Siyan Zhao, Zhihui Xie, Mengchen Liu, Jing Huang, Guan Pang, Feiyu Chen, and Aditya Grover. Self-distilled reasoner: On-policy self-distillation for large language models.arXiv preprint arXiv:2601.18734, 2026

    2026 arXiv

  53. [61]

    Action":

    Shuyan Zhou. Webarena-infinity: Generating browser environments with verifiable tasks at scale.shuyanzhou.com, March 2026. URL https://webarena.dev/webarena-infinity/. 14 A Additional Experimental Details A.1 Compute and Hyperparameters Within each benchmark, every weight-upda...

    2026

Pith tools

Reviewed July 10, 2026 · model on record in the stance chip above.