{"id":"3cec9abc-4108-4f1c-8b00-a0eb3350470c","arxiv_id":"2607.07847","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Different patterns of environmental change (space vs time) require different LLM update behaviors; no single family of methods—prompts, distillation, RL, or compression—handles all regimes.","lead":"Continual learning for large language models is not one skill: domain shifts, fact updates, noisy temporal drift, and agent-driven state each demand different update behaviors. Prompt, distillation, RL, and compression methods succeed or fail in different regimes, so adaptation sometimes must live in weights and sometimes can stay external.","discovery_kind":"unification","skeptic_critique":{"model":"grok-4.5","headline":"The strongest claim over-reaches from method-family trade-offs under a single backbone and hand-adapted operators to a general rule about when weights are required.","rationale":"The reader correctly identifies the single-backbone design, fixed C, and sequential adaptations as the weakest assumption and assigns CONDITIONAL with moderate confidence. That is the right load-bearing concern: the paper’s comparative map is valuable and the qualitative trade-offs are clear and actionable, but the strongest claim generalizes from those specific operators and one model to a regime-dependent necessity of weight updates. The concrete multi-backbone re-run on the two time-axis tasks would settle whether the boundary is about the change pattern or about the particular method lifts and scale. No stronger internal inconsistency appears; the framing and protocol remain worth adopting. Verdict stays CONDITIONAL; agreement with the reader is full.","tokens_in":22680,"tokens_out":631,"duration_ms":7315,"concrete_test":"Re-run the TempWiki (fact-update) and 10-K (noisy drift) chains with at least one additional backbone (e.g., a 32B or reasoning-mode model) under the same protocol and budget C, keeping the sequential adaptations fixed. If the qualitative ranking of families (prompt vs distillation vs online RL vs compression) on drift-vs-stable F1 and on forward-vs-backward accuracy reverses or collapses, the claim that the change pattern itself dictates when weights are required is unsupported; if the same trade-offs reappear, the claim is strengthened.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim (Abstract, §5.1) is that continual learning is not a single capability and that different environmental-change patterns determine when adaptation must be learned inside model weights versus external scaffolding. The evidence is a comparative map of eight methods on four sequential regimes under one protocol (§3–4). That map shows clear qualitative trade-offs (prompt methods fit fast but forget; distillation accumulates but resists fact rewrite; GRPO updates facts but is reward-sensitive; compression is mostly flat). The load-bearing step is the leap from “these particular operators behave differently on these regimes with Qwen3-8B non-thinking and our sequential adaptations” to “therefore different patterns require fundamentally different update behaviors, determining when weights are required.” The protocol’s fairness hinges on (i) a single 8B non-thinking backbone, (ii) a fixed per-stage budget C with method-specific LRs chosen to avoid collapse (Table 2), and (iii) author-defined sequential lifts (θ_{k-1} as teacher for SDFT/SDPO; carried prompt/playbook for GEPA/ACE; new adapter per stage for Cartridges). Limitations (§5.2) already note relative behavior may change for larger or reasoning models. If those adaptations or the backbone choice drive the observed ranking more than the space/time axes themselves, the “when weights are required” boundary is not yet established as a property of the change pattern.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.5","summary":"The paper reframes continual learning for LLMs as increasing competence under environmental change, disentangled along space (domain shift) and time (discrete fact updates, continuous temporal drift, and agentic state accumulation). It introduces a mechanism-agnostic staged protocol (θ_k = U_k(θ_{k-1}, D_tr_k) with fixed per-stage budget C) and evaluates eight methods from four families—prompt optimization (GEPA, ACE), offline supervised updates (SFT, SDFT), online RL (GRPO, SDPO), and context compression (Cartridges, In-place TTT)—on four sequential recastings of standard LLM benchmarks (domain chain, TempWiki, 10-K sentiment, WebArena-style agentic chains). Empirically, prompt methods fit stages quickly but degrade on future tasks; distillation accumulates more stably but resists fact rewrite; compression improves efficiency without clear task acquisition; and online RL adapts best to knowledge updates but is sensitive to noisy rewards. The central claim is that continual learning is not a single capability and that different patterns of change require different update behaviors, determining when adaptation must occur in weights versus external scaffolding.","tokens_in":23112,"tokens_out":1290,"duration_ms":30088,"significance":"If the comparative map is reliable, this is a useful organizing contribution for LLM continual learning. The space/time framing separates regimes the literature often conflates; the protocol lets prompt, weight, and architectural updates be compared on equal ground; and the four sequential settings (especially TempWiki with a stable-fact probe, 10-K forward/backward transfer under weak signal, and agentic chains whose order is generated by the agent) are carefully constructed and more realistic than static CL task sequences. Strengths include an explicit sequential adaptation for each method (Table 1), standard transfer metrics (BWT/FWT), released code, and an honest limitations section. The work is diagnostic rather than algorithmic: its value is in guiding evaluation practice and method design. The main risk is over-generalizing family-level trade-offs under one backbone and author-defined sequential lifts into a general rule about when weights are required.","major_comments":[{"comment":"Abstract and §5.1 claim that different environmental-change patterns determine when adaptation must be learned inside model weights versus external scaffolding. The evidence is a qualitative map of eight operators on four regimes with a single backbone (Qwen3-8B non-thinking). §5.2 already notes relative behavior may change for larger or reasoning models. As written, the leap from “these operators trade off under our protocol” to a general “when weights are required” boundary is stronger than the multi-model support. Either soften the claim to a protocol-level empirical map, or add at least one additional backbone / reasoning-mode check on a subset of regimes so the boundary is not backbone-contingent.","section":null},{"comment":"§3.1–3.2 and Table 1: fairness of the mechanism-agnostic protocol hinges on author-defined sequential lifts (θ_{k-1} as teacher for SDFT; θ_{k-1} preference generator for SDPO; carried prompt/playbook for GEPA/ACE; new adapter per stage for Cartridges; resume from θ_{k-1} for SFT/GRPO). These choices are explicit and reasonable, but they are not unique, and they can drive rankings as much as the space/time axes. The manuscript should state more clearly which conclusions are robust to alternative sequential adaptations (e.g., fixed teacher, replay, or shared reference for DPO-style methods) and which are specific to the chosen lifts; a short sensitivity check on one regime would substantially strengthen the “fair comparison” claim.","section":null},{"comment":"Figures 2–4 (and the corresponding TempWiki/10-K curves): all primary comparisons are single-run accuracy/F1 trajectories without error bars, seeds, or confidence intervals. Appendix A.1 reports one epoch over 500 examples and method-specific outer learning rates chosen to avoid collapse (Table 2). Under a fixed nominal budget C but different LRs and loss geometries, small differences (e.g., GRPO’s +1.6 drift points, Cartridges near-baseline) are hard to interpret. Report multi-seed means and variability for at least the headline transfer quantities (final-stage scores, BWT/FWT, drift vs. stable F1), or justify single-run reporting with a stability argument.","section":null}],"minor_comments":[{"comment":"§3.1 defines BWT and FWT but the main figures emphasize per-stage accuracy curves; a compact table of BWT/FWT per method per benchmark would make the transfer claims easier to audit.","section":null},{"comment":"Figure 1 caption lists four axes (domain shift, fact update, temporal drift, agentic state) while the text organizes change as two axes (space and time) with time sub-regimes; align the figure language with §1/§4 to avoid a three- vs. four-axis reading.","section":null},{"comment":"Appendix A.2.3 (TempWiki-Easy) is important for the catastrophic-memorizing claim; a one-sentence pointer in §4.2 main text would help readers who skip the appendix.","section":null},{"comment":"Agentic results (§4.4) use Qwen-32B for ACE and Qwen-8B for SFT, unlike the 8B backbone elsewhere; state this more prominently when comparing to other sections so cross-regime conclusions are not over-read.","section":null},{"comment":"Typos/clarity: “abroadrange” (p.1), “and and agentic” (§4), “less sensitive to than prompt methods” (§4.1); fix for camera-ready.","section":null}],"recommendation":"major_revision","confidential_remarks":"Solid empirical systems paper with a useful framing; not a theory or new-algorithm paper. Fit is good for a methods/evaluation venue in ML. The over-claim risk is real but fixable with softer language plus multi-seed and/or multi-backbone checks—I would not reject on novelty grounds. Watch that sequential adaptations are not sold as uniquely canonical without sensitivity evidence."},"author_rebuttal":null,"desk_editor":{"model":"grok-4.5","letter":"The thing worth knowing is that this is a comparative systems paper with a clean reframe, not a theorem about when weights are mandatory. They split environmental change into space (new domains) and time (fact rewrite, noisy drift, agent-induced state), put eight methods under one staged protocol, and show the qualitative trade-offs side by side. That map is the contribution.\n\nWhat is actually new: the space/time cut for modern LLMs, the “catastrophic memorizing” probe (drift facts plus a held-out stable set), the sequential 10-K year chain with past/future accuracy, and agentic chains whose order comes from the agent’s own state. The protocol (θ_k = U_k(θ_{k-1}, D_tr_k) with fixed C, BWT/FWT, full forgetting matrices) is simple and fair enough that prompt, weight, and compression methods can be read on the same axes. Appendix detail on data, scorers, and LRs is solid; code is linked. The empirical story is consistent: prompts fit fast and overwrite; distillation accumulates and resists rewrite; compression is mostly flat on acquisition; GRPO moves facts when the reward is clean and falls over when it is not. That is actionable for people building post-training and agents.\n\nSoft spots, in proportion. Single backbone (Qwen3-8B non-thinking), method-specific outer LRs chosen to avoid collapse, no error bars or multi-seed, and author-defined sequential lifts (previous model as teacher, carried playbook, new adapter per stage). The stress-test is right that the leap from “these operators rank this way under our adaptations” to a general “when weights are required” rule is not fully earned. The limitations section already says relative behavior may change for larger or reasoning models. That weakens absolute rankings, not the usefulness of the regime map. Circularity is low; metrics are external.\n\nWho it is for: anyone designing continual adaptation for LLMs who is tired of anti-forgetting or RAG-only framings. Worth a serious referee. I would engage with the protocol and the four suites; I would not treat the weight-vs-scaffold boundary as settled without multi-model confirmation. Send it to review.","headline":"Useful regime map for LLM continual learning; the framing and protocol are the real product, not a settled rule about when weights are required.","tokens_in":23729,"tokens_out":558,"would_cite":true,"duration_ms":13606,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.5","headline":"Continual learning is not one skill: different kinds of world change demand different kinds of model updates.","keywords":["continual learning","large language models","domain shift","temporal drift","catastrophic forgetting","prompt optimization","self-distillation","reinforcement learning"],"falsifier":"Repeat the same four sequential suites on a substantially larger reasoning model (or with systematically permuted stage orders) and check whether the reported method-by-regime ranking reverses.","tokens_in":23547,"feed_emoji":"🔄","tokens_out":530,"duration_ms":5923,"temperature":0.7,"pith_summary":"The field has treated continual learning mainly as context management and avoiding forgetting. This paper reframes it as the harder problem of raising competence while the world itself changes. Change is split into space (new domains) and time (drift under a fixed task), with time further covering discrete fact revisions, slow noisy trends, and state that accumulates from an agent’s own actions. Under one shared protocol that scores prompt, weight, and compression methods the same way, eight techniques show consistent trade-offs: prompts lock onto the current stage and then forget; distillation accumulates steadily but resists rewriting outdated beliefs; compression saves tokens without teaching new tasks; online reinforcement rewrites facts best but collapses when rewards are noisy. The practical message is that designers must match the update mechanism to the pattern of environmental change rather than search for a single universal continual learner.","feed_headline":"Continual learning is not one skill","feed_subtitle":"Different kinds of world change demand different model updates—weights, prompts, or neither","key_machinery":"A mechanism-agnostic sequential protocol: stages arrive in fixed order, each method applies an unrestricted update operator under a shared compute budget, and performance is scored with a forgetting matrix that yields backward and forward transfer. This single yardstick lets prompt, weight, and architectural methods be compared on equal ground.","core_discovery":"Continual learning is not a single capability. Different patterns of environmental change—domain shift, discrete fact updates, noisy temporal drift, and agent-driven state accumulation—require fundamentally different update behaviors, which in turn decide when adaptation must live inside model weights and when external scaffolding is enough.","pith_inferences":[],"forward_implications":[],"fun_headline_variants":["Continual learning splits by domain shift and fact drift","When adaptation must enter weights vs stay external","Space vs time change demands different update behaviors","Not every continual task requires learning in the weights","Prompts fit fast; weights hold knowledge; RL updates facts"],"cache_read_input_tokens":19456,"weakest_assumption_plain":"That results on one mid-size non-reasoning backbone under a fixed per-stage budget and the authors’ sequential adaptations of each method will hold for larger or reasoning models and for other task orderings.","fun_headline_variants_meta":{"raw":{"variants":["Continual learning splits by domain shift and fact drift","When adaptation must enter weights vs stay external","Space vs time change demands different update behaviors","Not every continual task requires learning in the weights","Prompts fit fast; weights hold knowledge; RL updates facts"]},"model":"grok-4.5","effort":"low","cost_usd":0.004634,"raw_usage":{"total_tokens":1377,"prompt_tokens":811,"num_sources_used":0,"completion_tokens":75,"cost_in_usd_ticks":46340000,"prompt_tokens_details":{"text_tokens":811,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":491,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":811,"tokens_out":75,"duration_ms":5912,"temperature":1.0,"reasoning_tokens":491,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-10T16:47:46.853260+00:00","model_set":{"reader":"grok-4.5"},"falsifier":"Repeat the same four sequential suites on a substantially larger reasoning model (or with systematically permuted stage orders) and check whether the reported method-by-regime ranking reverses.","supporting_citations":[],"review_version":1}