Pith.

Machine Learning · arXiv:2611.19700 v1 · submitted 15 Nov 2026 · 45 references

Where Should LLM Agents Adapt at Test Time? In Context, in Weights, or Both

Parsa Mirtaheri, Mohammadreza Armandpour, Keivan Alizadeh, Iman Mirzadeh, Parshin Shojaee, Aryo Lotfi, David Harrison, Mehrdad Farajtabar

The paper says

“LLM agents operating repeatedly in the same environment, such as a code repository, a database, or an application suite, encounter tasks that share latent structure.”

more of the abstract

Experience can reveal this structure, allowing agents to reuse environment knowledge and learn from environmental feedback. Agents can adapt at test-time using this experience through two channels: retaining it in context and updating their weights. These channels are usually studied separately, leaving unclear what each contributes and when they complement each other. We compare in-context learning (ICL), reinforcement learning with verifiable rewards using GRPO, and self-distillation using SDPO, alone and in combination, using a shared online protocol and modular framework. Agents encounter matched task streams, are evaluated on each task before adapting to it, and learn only from their own attempts and environment feedback. We evaluate 14 adaptation mechanisms and a stateless baseline with Qwen3.5-9B on 109 tasks from four environments drawn from SWE-bench Verified, AppWorld, ScienceWorld, and BIRD-Critic-SQLite, using five task orders per environment. ICL provides the largest standalone gain: its best policy raises mean success across environments from 28.8% to 49.4%. Adding GRPO to this policy achieves the highest mean success, 51.6%. While adding GRPO improves the overall mean under every ICL policy, SDPO has mixed effects. Trace analyses show fewer lookups and shorter attempts under ICL, consistent with reusing knowledge of the shared structure. However, ICL also frequently skips inspection and verification for the current task. Adding GRPO to ICL helps restore these behaviors while preserving its reduced environment lookups. These observations suggest that reinforcement learning with verifiable rewards can refine how agents use retained experience, motivating the joint study of context management and weight updates.

from the abstract

Pith says

Machine review · deepseek-v4-flash · 17 Nov 2026 · read the full text (137,630 characters) · not checked by a person

It can be wrong. Challenge this review or reply to it.

Holds up. A careful empirical comparison of in-context learning, GRPO, and SDPO for test-time adaptation of LLM agents, finding that ICL gives the largest standalone gain and adding GRPO yields the best overall performance. This line is wrong

10 minor

How often objections from reviews like this one are right: not yet measured

Are you an author? What you can do.

The paper, first character to last

The reviewer was given the whole text, 137,630 characters. Not checked: figures, code, what the 45 references say.

10 minor objections

The paper says, at Abstract

Pith says: minor objection 1 of 10

The abstract states 'We evaluate 14 adaptation mechanisms and a stateless baseline with Qwen3.5-9B on 109 tasks from four environments.' The number 109 is the sum of tasks across worlds (22+21+22+44=109), but the paper also mentions additional experiments on scikit-learn (32 tasks) and Toxicology (64 tasks) in Appendix D. The abstract should clarify that the main evaluation uses 109 tasks.

The paper says, at Section 3.1 (Stream of related tasks)

Pith says: minor objection 2 of 10

The world names (Xarray, File System, Measurement, Financial) are descriptive but the mapping to benchmarks is not immediately clear from the text. Table 2 in Appendix A.2 clarifies this, but a brief reminder in the main text would help readability.

show 8 more

The paper says, at Section 3.2 (Adaptation mechanisms)

Pith says: minor objection 3 of 10

The description of GRPO says 'without a KL penalty or standard-deviation normalization.' This is a notable departure from standard PPO/GRPO. The paper should briefly justify this choice or cite prior work that uses this variant.

The paper says, at Section 4.2 (Results), Table 1

Pith says: minor objection 4 of 10

The table uses superscripts for standard errors (e.g., '13.4^±1.0'). The formatting is unusual and may be difficult to parse. Consider using parentheses or separate columns.

The paper says, at Section 5.1 (Outcomes and Interaction Length), Figure 3

Pith says: minor objection 5 of 10

The figure caption says 'Results weight the five stream seeds equally, with 16 attempts per task and stream.' This is fine, but the paper should clarify whether the error bars (if any) in the figure represent standard errors across seeds or across tasks within a seed. The figure as described does not show error bars, but the text refers to 'mean proportions' without variability.

The paper says, at Section 7 (Discussion and Limitations)

Pith says: minor objection 6 of 10

The paper states 'We study one model on subsets of four benchmarks, with five task orders, raw-trajectory context policies, and specific update recipes.' This is an honest limitation, but the paper should also note that the model (Qwen3.5-9B) is not the most recent or largest, and results may not generalize to other model families or sizes.

The paper says, at Appendix A.2 (Benchmark environments)

Pith says: minor objection 7 of 10

Table 2 reports turn limits: SWE-bench 40, AppWorld 40, ScienceWorld 30, BIRD 5. The BIRD limit of 5 turns is very short compared to the others. This may affect the behavioral metrics (e.g., querying) and should be discussed in the main text.

The paper says, at Appendix A.3 (Adaptation settings)

Pith says: minor objection 8 of 10

The paper states 'Our SDPO implementation supports only one optimizer step per rollout batch, so we allocate an additional round to provide more fresh on-policy experience and adaptation opportunities.' This is an implementation limitation, not a principled choice. The paper should note whether this could be addressed in future work.

The paper says, at Appendix B.1 (Success rates by seed)

Pith says: minor objection 9 of 10

The longtable in Appendix B.1 is very detailed but the formatting makes it hard to read. Consider using a more compact layout or highlighting the best results per seed more clearly.

The paper says, at Appendix D (Additional Environments)

Pith says: minor objection 10 of 10

The additional experiments on scikit-learn and Toxicology use only seed 4 and one task order. The paper should acknowledge that these results are preliminary and do not have the same statistical strength as the main results.

What would settle it

A replication on a different model (e.g., Llama 3 70B) or with different ICL policies (e.g., summarization-based retrieval instead of raw replay) that shows ICL no longer provides the largest standalone gain, or that GRPO does not improve over ICL alone, would falsify the paper's central claim.

from Pith's summary of the paper

How the review went

5 passes by deepseek-v4-flash · 17 Nov 2026

  1. Reader holds up confidence high
  2. Skeptic concern withdrawn: Pith's reviewer misread the paper 7 s · 45,893 tokens read, 994 written
  3. Referee 10 minor 12 s · 46,204 tokens read, 2,537 written
  4. Check against the paper kept 0 of 5 objections 15 s · 47,105 tokens read, 3,017 written
  5. Editor wrote the verdict above 4 s · 46,533 tokens read, 458 written

Not run on this paper: the simulated authors' defence.

Pith's passes raised 5 objections. A check against the paper's text kept 0 and withdrew 5 (5 Pith's reviewer misread the paper).

The review in full

The review in brief, machine-written

This paper asks a practical question: when an LLM agent works repeatedly in the same environment (a code repo, a database, an app suite), should it adapt by keeping experience in its context, by updating its weights, or both? The authors run a controlled comparison of 14 mechanisms on 109 tasks from four benchmarks, using one model (Qwen3.5-9B) and a shared online protocol where each task is scored before the agent adapts to it.

The headline result is clear: in-context learning (ICL) provides the largest standalone gain, raising mean success from 28.8% to 49.4%. Adding GRPO (a policy-gradient method) to ICL yields the highest overall success at 51.6%. The behavioral analyses are the most interesting part: ICL reduces environment lookups and shortens attempts, consistent with reusing knowledge of shared structure, but it also skips task-specific verification. Adding GRPO helps restore verification while preserving the efficiency gains. SDPO (self-distillation) has mixed effects.

The paper is thorough and honest about its limitations. It studies one model, specific hyperparameters, and raw-trajectory ICL policies. The SDPO family gets three rounds per task versus two for GRPO and ICL, a confound the authors acknowledge. The behavioral observations are correlational, not causal. These are real but bounded weaknesses; the paper does not overclaim.

Who should read this? Anyone working on continual learning for LLM agents, or on combining in-context and weight-based adaptation. The framework is extensible and the results provide a useful baseline for future work. The paper deserves a serious referee; the main questions for review are whether the conclusions hold under different ICL policies (e.g., with summarization or retrieval) and whether the GRPO advantage persists with more rounds or different models.

The stress test's concern

Withdrawn: Pith's reviewer misread the paper.

The paper says

“Raw-trajectory replay lets us study context adaptation without introducing additional summarization, retrieval, or judgment procedures, while keeping the source of experience consistent across mechanisms.”

The referee's report

Not published. Pith's referee wrote this assessment before its objections were checked against the paper, and 4 of its 4 major objections did not survive that check.

Pith's reading of the paper (a machine's summary, not the authors')

The paper investigates how LLM agents should adapt at test time when operating repeatedly in the same environment. It compares two adaptation channels: retaining experience in context (in-context learning, ICL) and updating model weights via reinforcement learning (GRPO) or self-distillation (SDPO). Using a shared online protocol across four benchmarks, the authors find that ICL alone raises mean success from 28.8% to 49.4%, while adding GRPO to ICL achieves the highest mean success of 51.6%. Behavioral analyses show that ICL reduces redundant exploration but can skip task-specific verification; adding GRPO helps restore verification while preserving efficiency gains.

What carries the argument. The central comparison is between two adaptation channels: in-context learning (ICL), which retains prior attempts as conversation history in the context window, and weight updates via GRPO (reinforcement learning with verifiable rewards) or SDPO (self-distillation). The framework uses a shared online protocol where agents encounter matched task streams, are evaluated on each task before adapting to it, and learn only from their own attempts and environment feedback. Four ICL policies (crossing FIFO/Prefix retention with All/Successful admission) and two weight-update rules are tested alone and in combination.

Core claim. The paper's central claim is that in-context learning (ICL) provides the largest standalone gain for test-time adaptation of LLM agents, and adding reinforcement learning with verifiable rewards (GRPO) to ICL yields the highest overall success rate. Across 109 tasks from four benchmarks, ICL raises mean success from 28.8% to 49.4%, and ICL+GRPO achieves 51.6%. The authors argue that the two channels are complementary: ICL reuses environment knowledge to reduce exploration, while GRPO refines behavior by restoring task-specific inspection and verification that ICL tends to skip.

The premise it rests on. The results depend on the specific implementations of ICL policies (raw-trajectory replay with FIFO/Prefix retention and All/Successful admission) and weight updates (GRPO and SDPO with LoRA adapters and fixed hyperparameters), and may not generalize to other ICL policies, other models, longer streams, or different weight-update recipes.

If the paper is right (Pith's inferences, not the authors')

If ICL+GRPO consistently outperforms either channel alone, then practical deployments of LLM agents should combine context management with reinforcement learning rather than relying on one channel.

The finding that ICL reduces environment lookups but skips verification suggests that pure ICL may lead to over-reliance on past experience at the expense of checking current task requirements.

The mixed effects of SDPO indicate that self-distillation does not reliably complement ICL, possibly because its self-teacher inherits ICL's limitations.

The result that GRPO restores verification behaviors while preserving ICL's efficiency gains suggests that reinforcement learning can correct undesirable shortcuts learned from context.

The paper's protocol could be extended to study longer streams, where context window pressure might shift the balance between channels further toward weight updates.

The observation that ICL reduces exploration but also verification suggests a potential failure mode: agents may become overconfident in familiar patterns and miss novel task requirements.

The framework could be used to test whether other weight-update methods (e.g., online fine-tuning with different objectives) also complement ICL, or whether the complementarity is specific to GRPO's reward-based learning.

The transfer experiments hint that retained memory can help or hurt across environments, raising questions about when context should be reset or curated for new domains.

Check this review

What was read

The passes were given all 137,630 characters of the paper's LaTeX source.

Fingerprint of that text (SHA-256): 1992089f8afbfba68b4900271fb4fa2553068c4642f9f61d57ea60ff7c705c1b

Pith does not keep that text. Rebuilt from the paper's source or PDF by the same code, it has this fingerprint if it is the same text. From a copy of Pith's code:

PYTHONPATH=. python scripts/review_input_check.py 2611.19700

The passes, in the order they ran

  1. Reader deepseek-v4-flash · medium effort · $0.012 · instructions of the Reader, fingerprint f9ead7300382…
  2. Skeptic deepseek-v4-flash · medium effort · $0.0044 · instructions of the Skeptic, fingerprint 8f88c0f61488…
  3. Referee deepseek-v4-flash · medium effort · $0.0047 · instructions of the Referee, fingerprint a852d99aea0d…
  4. Check against the paper deepseek-v4-flash · medium effort · $0.0048 · instructions of the Check against the paper, fingerprint 6c7a6883d551…
  5. Editor deepseek-v4-flash · medium effort · $0.0043 · instructions of the Editor, fingerprint e7211f6a3d27…
  6. Summary for the page deepseek-v4-flash · medium effort · temperature 0.2 · $0.0046 · instructions of the Summary for the page, fingerprint e4f56bf568e7…

The passes cost $0.035 in model calls together.

Version stamped on this review: pith-pipeline@v1.4.1-daily-deepseek-or-medium+deepseek/deepseek-v4-flash+p.0ef1a750a3e3

Each link opens a pass's instructions as plain text, the exact bytes its fingerprint was taken over. Every stored set of instructions.

curl -s https://pith.science/methodology/prompts/f9ead730038216984cfb56d7cb34332bdf691828296a1de4245f3d6667d4bf56 | sha256sum

prints the fingerprint of the reader's instructions. The same works for every pass.

The receipt

The review's fingerprint was taken on 17 Nov 2026. The review you are reading still matches it.

Fingerprint of the review (SHA-256): bf0918f7da4b85785a9425cc02c0c43059cde6165608c8a776f660b5aaadfecc

curl -s 'https://pith.science/api/v1/verdicts/2611.19700?form=canonical' | sha256sum

prints the fingerprint of the review as it stands now. The rule, in four steps.

A matching fingerprint shows the review has not changed since that date. It does not show that the review is right.

How often reviews like this one are right

Reviews like this one: written by deepseek-v4-flash, from the paper's LaTeX source.

How often objections from reviews like this one are right: not yet measured

Every value above, in the review's public record (JSON)

Reviews by people

This review was written by a machine. No person has reviewed this paper on Pith yet. A scientist listed on Pith can review it for a fee, and their review is published here whatever it concludes.

Request a human review

How Pith checks papers · The full audit of this paper · Review history (2 versions)

The paper

Abstract

LLM agents operating repeatedly in the same environment, such as a code repository, a database, or an application suite, encounter tasks that share latent structure. Experience can reveal this structure, allowing agents to reuse environment knowledge and learn from environmental feedback. Agents can adapt at test-time using this experience through two channels: retaining it in context and updating their weights. These channels are usually studied separately, leaving unclear what each contributes and when they complement each other. We compare in-context learning (ICL), reinforcement learning with verifiable rewards using GRPO, and self-distillation using SDPO, alone and in combination, using a shared online protocol and modular framework. Agents encounter matched task streams, are evaluated on each task before adapting to it, and learn only from their own attempts and environment feedback. We evaluate 14 adaptation mechanisms and a stateless baseline with Qwen3.5-9B on 109 tasks from four environments drawn from SWE-bench Verified, AppWorld, ScienceWorld, and BIRD-Critic-SQLite, using five task orders per environment. ICL provides the largest standalone gain: its best policy raises mean success across environments from 28.8% to 49.4%. Adding GRPO to this policy achieves the highest mean success, 51.6%. While adding GRPO improves the overall mean under every ICL policy, SDPO has mixed effects. Trace analyses show fewer lookups and shorter attempts under ICL, consistent with reusing knowledge of the shared structure. However, ICL also frequently skips inspection and verification for the current task. Adding GRPO to ICL helps restore these behaviors while preserving its reduced environment lookups. These observations suggest that reinforcement learning with verifiable rewards can refine how agents use retained experience, motivating the joint study of context management and weight updates.

The paper's own sentences, arranged, kept beside the review and separate from it. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. No sentence in this record was written by a model.

abstract compiled · 2026-11-17

The authors' main claim, as the classifier picked it out 0.91

“These observations suggest that reinforcement learning with verifiable rewards can refine how agents use retained experience, motivating the joint study of context management and weight updates.”

How the main claim is stated: hedged possibility 0.99

What the abstract itself states, as probabilities: claims first 0.07 · states limitation 0.38 · human participants 0.03 · controlled comparison 0.97 · randomized assignment 0.04 · states quantitative result 0.99 · compares with prior results 0.92 · mentions code or data availability 0.04

The abstract, sentence by sentence

This table labels the abstract only (arXiv listing abstract v1). The number is the probability the classifier gave the label.

SentenceRoleThe authors' own workStates the main conclusionStrength
LLM agents operating repeatedly in the same environment, such as a code repository, a database, or an application suite, encounter tasks that share latent structure. background 1.00 0.08 0.02 asserted 1.00
Experience can reveal this structure, allowing agents to reuse environment knowledge and learn from environmental feedback. background 1.00 0.07 0.02 hedged possibility 0.68
Agents can adapt at test-time using this experience through two channels: retaining it in context and updating their weights. background 0.95 0.06 0.04 hedged possibility 0.50
These channels are usually studied separately, leaving unclear what each contributes and when they complement each other. gap 1.00 0.06 0.02 asserted 0.98
We compare in-context learning (ICL), reinforcement learning with verifiable rewards using GRPO, and self-distillation using SDPO, alone and in combination, using a shared online protocol and modular framework. method 0.68 0.93 0.04 asserted 0.96
Agents encounter matched task streams, are evaluated on each task before adapting to it, and learn only from their own attempts and environment feedback. method 0.99 0.83 0.02 asserted 0.99
We evaluate 14 adaptation mechanisms and a stateless baseline with Qwen3.5-9B on 109 tasks from four environments drawn from SWE-bench Verified, AppWorld, ScienceWorld, and BIRD-Critic-SQLite, using five task orders per environment. method 0.99 0.96 0.02 asserted 0.97
ICL provides the largest standalone gain: its best policy raises mean success across environments from 28.8% to 49.4%. result 1.00 0.95 0.18 asserted 1.00
Adding GRPO to this policy achieves the highest mean success, 51.6%. result 1.00 0.95 0.22 asserted 1.00
While adding GRPO improves the overall mean under every ICL policy, SDPO has mixed effects. result 1.00 0.93 0.12 asserted 1.00
Trace analyses show fewer lookups and shorter attempts under ICL, consistent with reusing knowledge of the shared structure. result 0.81 0.93 0.05 hedged possibility 0.60
However, ICL also frequently skips inspection and verification for the current task. result 0.94 0.89 0.03 asserted 0.99
Adding GRPO to ICL helps restore these behaviors while preserving its reduced environment lookups. result 0.79 0.91 0.20 asserted 0.98
These observations suggest that reinforcement learning with verifiable rewards can refine how agents use retained experience, motivating the joint study of context management and weight updates. interpretation 1.00 0.71 0.69 hedged possibility 0.99

What kind of paper the classifier reports this is: empirical study 1.00 · evidence from observation or experiment 0.98

What was examined, and what was not

Sources read

  • arXiv listing abstract v1

Not examined

  • full text
  • figures and tables
  • references and citing sentences
  • relations to other papers

Levels run

  • P abstract

Not run

  • S sections
  • T body sentences
  • C citations
  • R cross-paper relations

How the labels were made

Compiled by classifier-1.13.0 via TypeSafe API, question bank abstract-p1-20260923, 2026-11-17: 1 request, 10,143 input tokens.

Every quoted sentence on this record is the authors'. Every label is from Pith's closed vocabulary, with the probability shown beside it. This record as JSON

Figures (23)
Figure 1 of the paper
Figure 1. On average across benchmarks, in-context learning (ICL) gives the largest standalone gain, while ICL with GRPO gives the highest overall gain. Points show five-seed mean success rates (%), using the best mechanism within each adaptation family for each world. Curves connect results without and with ICL for the same weight-update policy. Standard errors are reported in
Figure 2 of the paper
Figure 2. One round on a task. (1) The adaptation mechanism receives a task. (2) Prepare returns a context c, with which the agent makes k attempts yi in fresh sessions of the environment. (3) Observe receives the attempts and their outcomes oi and updates the mechanism’s non-parametric state. (4) TeacherContexts adds hindsight h for SDPO’s self-teacher. (5) The mechanism updates its parametric state using the weighted SDPO and GRPO losses. We derive four worlds from well-established benchmarks. In Xarray, the agent resolves SWE-bench Verified issues of pydata/xarray with shell commands,…
Figure 3 of the paper
Figure 3. Mean gains and wins from adding GRPO or SDPO. Boxes show mean success rates. Yellow arrows indicate adding ICL with the policies listed below; blue and red arrows add GRPO and SDPO, respectively. Horizontal arrow labels show the mean success-rate gain in percentage points (top) and wins out of 20 world–seed stream pairs (bottom), weighting all streams equally. 5 Individual Contributions of Context and Weights What does each adaptation channel contribute to test-time adaptation? Beyond the success rates in Section 4, we examine how context and weight updates separately change the…
Figure 4 of the paper
Figure 4. How single-channel adaptation changes outcomes and attempt length. Top: mean proportions of successful, wrong, and unfinished attempts. An unfinished attempt ends without success or submission. ScienceWorld ends on reaching the goal and has no separate submission decision. Bottom: mean turns per attempt at each task position. Results weight the five stream seeds equally, with 16 attempts per task and stream. SWE-bench: ICL reduces code lookup. We count each turn that searches or displays repository files, excluding test runs, diffs, and reads used only for editing, and report the…
Figure 5 of the paper
Figure 5. How single-channel adaptation changes behavior along the stream. SWE-bench: mean code-lookup turns per attempt. AppWorld: mean turns containing API-documentation lookups. ScienceWorld: mean explicit look around actions per attempt (environment lookups). BIRD: percentage of all attempts with at least one executed exploratory SQL query. All panels weight the five seeds equally at each position. Weight updates offer a useful contrast. Both methods learn from the agent’s own attempts. GRPO’s loss uses the probabilities of sampled tokens, whereas SDPO also matches self-teacher…
Figure 6 of the paper
Figure 6. Final-task examples across all four benchmarks in seed 4. Each column summarizes all 16 attempts: graded successes and mean code-lookup turns (SWE-bench), mean API lookups (AppWorld), mean environment lookups (ScienceWorld), or the percentage of all attempts with at least one executed exploratory SQL query (BIRD). Below the divider, excerpts summarize attempt 1; t denotes the turn. 11
Figure 7 of the paper
Figure 7. Hybrid outcomes and exploration: GRPO preserves reduced lookups while restoring querying. Top: mean proportions of successful, wrong, and unfinished attempts. An unfinished attempt ends without success or submission. Bottom: mean code-lookup turns per attempt on SWE-bench, mean turns containing API-documentation lookups on AppWorld, mean explicit look around actions on ScienceWorld, and percentage of all attempts with at least one executed exploratory SQL query on BIRD. All panels weight the five seeds equally. Adding GRPO preserves ICL’s reduced exploration. Revisiting the…
Figure 8 of the paper
Figure 8. GRPO maintains higher late-stream querying across ICL policies on BIRD. Each panel compares ICL with its GRPO and SDPO hybrids under one ICL policy; Frozen is repeated for reference. Querying counts all attempts with at least one executed exploratory SQL query. Rates average 16 attempts per task in a stream. SWE-bench: fewer code lookups, more rechecking after edits. Knowing where to edit does not establish that a new patch works. We measure the fraction of attempts that run the same Python probe or pytest selection before and after an attempted source edit. Both weight updates…
Figure 9 of the paper
Figure 9. How weight updates change ICL’s task-specific inspection and verification. (a) Rechecking after edits on SWE-bench; (b) final-state inspection on AppWorld; (c) inspection during melting-point experiments on ScienceWorld; (d) querying on BIRD. Bars show rates (%) averaged equally over five seeds. 13
Figure 10 of the paper
Figure 10. How weight updates change ICL behavior: final-task examples across all four benchmarks in seed 0. Each column summarizes all 16 attempts: graded successes and same-check rates (SWE-bench), final-state inspection (AppWorld), state inspection (ScienceWorld), or querying (BIRD), using the metrics of
Figure 11 of the paper
Figure 11. Mean gains and wins from adding ICL and GRPO. Boxes show mean success rates. Each arrow compares its destination with its source: yellow adds ICL, blue adds GRPO. Arrow labels show the mean success-rate gain in percentage points (top) and wins out of 20 world–seed stream pairs (bottom), weighting all streams equally. 26
Figure 12 of the paper
Figure 12. Mean gains and wins from adding ICL and SDPO. Boxes show mean success rates. Each arrow compares its destination with its source: yellow adds ICL, red adds SDPO. Arrow labels show the mean success-rate gain in percentage points (top) and wins out of 20 world–seed stream pairs (bottom), weighting all streams equally. C Interaction Length and Action Composition These analyses complement Sections 5 and 6. We separate interaction length from the number of operations written within an action. C.1 Interaction length by outcome
Figure 13 of the paper
Figure 13. separates mean attempt length by outcome for the single-channel mechanisms. ICL uses fewer turns than the frozen baseline even among successful attempts in each benchmark, so its lower overall mean is not solely due to a smaller share of unfinished attempts. Measurement. We average turns over eligible attempts within each stream seed, then weight the five seeds equally. ICL uses FIFO–All on SWE-bench and Prefix–Successful elsewhere, as in
Figure 14 of the paper
Figure 14. complements the turn counts by measuring how many operations are written within one action in SWE-bench and AppWorld. These measures describe written action structure, not executed operation counts, token cost, or wall-clock cost. Measurement. In SWE-bench, we count separate shell invocations, including pipeline components, but ex￾clude setup-only commands and final submissions; embedded Python counts as one invocation. In AppWorld, we count explicit apis.<app>.<endpoint>(...) call sites, including documentation and supervisor calls. Calls inside loops count once as written, not…
Figure 15 of the paper
Figure 15. compares all four frozen-weight ICL policies over the stream. It complements the representative policy used for each benchmark in Section 5, showing both cumulative gains over the no-ICL baseline and the tasks represented in retained context. 0 250 500 750 Cumulative gain (pp) (a) SWE-bench Verified 0 250 500 750 (b) AppWorld 0 250 500 750 (c) ScienceWorld 0 250 500 750 (d) BIRD-Critic-SQLite 1 10 22 Task position 0 2 4 6 Tasks in context 1 10 21 Task position 0 5 10 15 1 10 22 Task position 0 6 12 18 1 20 44 Task position 0 8 16 24 FIFO · All FIFO · Successful Prefix · All Prefix…
Figure 16 of the paper
Figure 16. Adding GRPO to ICL: cumulative gains and retained context. Top: cumulative success-rate gain over the matching frozen-weight ICL policy, in percentage points. Bottom: distinct tasks in the hybrid’s context before each task begins. Curves show means over five seeds, using the pairing and context counts defined above. 29
Figure 17 of the paper
Figure 17. Adding SDPO to ICL: cumulative gains and retained context. Top: cumulative success-rate gain over the matching frozen-weight ICL policy, in percentage points. Bottom: distinct tasks in the hybrid’s context before each task begins. Curves show means over five seeds, using the pairing and context counts defined above. D.2 Prefix versus FIFO retention
Figure 18 of the paper
Figure 18. Reward variation and context admission with GRPO. Each bar partitions training batches by reward variation and admission of a new context episode for one ICL policy. Percentages pool batches across five seeds; category definitions are given above. 0 50 100% Prefix · All Prefix · Successful FIFO · All FIFO · Successful (a) SWE-bench Verified 57 8 31 6 56 38 58 42 24 38 35 0 50 100% (b) AppWorld 9 46 14 31 12 26 11 51 31 69 17 11 16 56 0 50 100% (c) ScienceWorld 22 24 32 22 21 38 7 34 29 71 20 32 9 39 0 50 100% (d) BIRD-Critic-SQLite 14 42 13 31 18 34 47 45 55 23 24 51 Training…
Figure 19 of the paper
Figure 19. Reward variation and context admission with SDPO. Each bar partitions training batches by reward variation and admission of a new context episode for one ICL policy. Percentages pool batches across five seeds; category definitions are given above. 31
Figure 20 of the paper
Figure 20. Success rates in two additional environments. Results use seed 4 and 16 attempts per task, with Prefix–Successful for all ICL mechanisms. Bold labels mark the highest success rate in each environment. SWE-bench Verified scikit-learn · 32 tasks Frozen 31.1% GRPO 53.9% SDPO 34.0% ICL 56.4% ICL + GRPO 62.9% ICL + SDPO 40.6% +22.9 +2.9 +6.4 -15.8 +9.0 +25.4 +6.6 BIRD-CRITIC Toxicology · 64 tasks Frozen 42.0% GRPO 47.2% SDPO 45.9% ICL 48.8% ICL + GRPO 53.1% ICL + SDPO 50.3% +5.2 +3.9 +4.3 +1.5 +6.0 +6.8 +4.4 Add ICL Add GRPO Add SDPO
Figure 21 of the paper
Figure 21. Gains from adding context and weight adaptation. Boxes show success rates; arrows show the change in percentage points from source to destination for the experiments in
Figure 22 of the paper
Figure 22. evaluates each agent on its original world’s tasks. These tasks were encountered during adaptation, so the results can reflect task-specific memorization and do not establish held-out generalization. They instead show how removing either component changes the final agent’s performance on those tasks. 0 25 50 75 100 Success (%) 15.1 23.6 32.1 43.8 (a) SWE-bench 57.7 82.1 55.1 84.8 (b) AppWorld 15.1 75.3 34.7 77.6 (c) ScienceWorld 30.8 44.0 37.2 49.0 (d) BIRD 0 25 50 75 100 Success (%) 15.3 51.1 41.2 59.7 59.5 96.1 51.5 96.7 13.9 66.8 36.9 71.3 26.8 53.7 40.2 60.1 Prefix-All + GRPO…
Figure 23 of the paper
Figure 23. evaluates the same agents and their components on the other worlds. The diagonal is masked because it revisits tasks used during adaptation. Off-diagonal entries report gains over the fresh frozen baseline for the target world in the corresponding round. SWE￾bench AppWorld Science World BIRD SWE-bench AppWorld ScienceWorld BIRD −30.7 +5.4 −4.7 +15.6 −0.6 −5.3 +22.4 −18.5 +0.4 +26.7 −12.2 +2.3 Both SWE￾bench AppWorld Science World BIRD +2.1 −1.1 +0.4 +1.7 +1.1 −1.4 +2.0 +2.4 −0.6 +0.9 +0.6 −4.3 Adapter only SWE￾bench AppWorld Science World BIRD −29.8 +5.1 −10.8 +14.2 +0.9 −3.4 +15.9…

Figures are the authors', reproduced from the paper.

References (45)
  1. [1] Additional environments and evaluations of final agents are described separately in Appendices E and F. A.1 Framework interfaces and adaptation mechanisms A benchmark registers four contracts: aworlddefines the tasks and shared interaction rules; astreamorders those tasks; ata... on Pith
  2. [2] URLhttp://arxiv.org/abs/2411.07279. arXiv:2411.07279 [cs]. Marc-Antoine Allard, Arnaud Teinturier, Victor Xing, and Gautier Viaud. Experiential Reflective Learning for Self- Improving LLM Agents, March on PithPith's reviewarXiv
  3. [3] Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. SWE- bench: CanLanguageModelsResolveReal-WorldGitHubIssues?, November2024. URLhttp://arxiv.org/abs/2310.06770. arXiv:2310.06770 [cs]. Haechan Kim, Yoonho Lee, Gisang Lee,... on PithPith's reviewarXiv
  4. [4] URLhttp://arxiv.org/abs/2606.05661. arXiv:2606.05661 [cs.AI]. A. P. Dawid. Present position and potential developments: Some personal views: Statistical theory: The prequential approach.Journal of the Royal Statistical Society. Series A (General), 147(2):278–290, on PithPith's reviewarXiv
  5. [6] URLhttp://arxiv.org/abs/2603.01761. arXiv:2603.01761 [cs.LG]. Kion Fallah, Silen Naihin, Barak Widawsky, and Qingqing Mao. CLaaS: Continual learning as a service for sample efficient online learning, June on PithPith's reviewarXiv
  6. [7] URLhttp://arxiv.org/abs/2606.05559. arXiv:2606.05559 [cs.LG]. João Gama, Raquel Sebastião, and Pedro Pereira Rodrigues. On evaluating stream learning algorithms.Machine Learning, 90(3):317–346, on PithPith's reviewarXiv
  7. [9] URL http://arxiv.org/abs/2607.07847. arXiv:2607.07847 [cs.LG]. Zexue He, Yu Wang, Churan Zhi, Yuanzhe Hu, Tzu-Ping Chen, Lang Yin, Ze Chen, Tong Arthur Wu, Siru Ouyang, Zihan Wang, Jiaxin Pei, Julian McAuley, Yejin Choi, and Alex Pentland. MemoryArena: Benchmarking Agent Memor... on PithPith's reviewarXiv
  8. [10] URLhttp://arxiv.org/abs/2602.16313. arXiv:2602.16313 [cs]. Prannay Hebbar, Yogendra Manawat, Samuel Verboomen, Alesia Ivanova, Selvam Palanimalai, Kunal Bhatia, and Vignesh Baskaran. SIA: Self Improving AI with Harness & Weight Updates, May on PithPith's reviewarXiv
  9. [11] URLhttp://arxiv.org/abs/2605. 27276. arXiv:2605.27276 [cs.AI]. Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. InInternational Conference on Learning Representations, on PithPith's reviewarXiv
  10. [12] URLhttp://arxiv.org/abs/2505.20633. arXiv:2505.20633 [cs]. Qisheng Hu, Quanyu Long, and Wenya Wang. When Continual Learning Moves to Memory: A Study of Experience Reuse in LLM Agents, April on PithPith's reviewarXiv
  11. [13] URLhttp://arxiv.org/abs/2604.27003. arXiv:2604.27003 [cs.LG]. Jonas Hübotter, Leander Diaz-Bone, Ido Hakimi, Andreas Krause, and Moritz Hardt. Learning on the Job: Test-Time Curricula for Targeted Reinforcement Learning, October on PithPith's reviewarXiv
  12. [14] URLhttp://arxiv.org/abs/2510.04786. arXiv:2510.04786 [cs]. 16 Jonas Hübotter, Frederike Lübeck, Lejs Behric, Anton Baumann, Marco Bagatella, Daniel Marta, Ido Hakimi, Idan Shenfeld, Thomas Kleine Buening, Carlos Guestrin, and Andreas Krause. Reinforcement Learning via Self- Di... on PitharXiv
  13. [15] URLhttp://arxiv.org/abs/2512.16301. arXiv:2512.16301 [cs] version: on PitharXiv
  14. [17] URLhttp://arxiv.org/abs/2609.00196. arXiv:2609.00196 [cs]. Ben Krause, Emmanuel Kahembwe, Iain Murray, and Steve Renals. Dynamic evaluation of neural sequence models. InProceedings of the 35th International Conference on Machine Learning, volume 80 ofProceedings of Machine Lea... on PithPith's reviewarXiv
  15. [18] URLhttp://arxiv.org/abs/2505.00661. arXiv:2505.00661 [cs.CL]. Jinyang Li, Xiaolong Li, Ge Qu, Per Jacobsson, Bowen Qin, Binyuan Hui, Shuzheng Si, Nan Huo, Xiaohan Xu, Yue Zhang, Ziwei Tang, Yuanshuai Li, Florensia Widjaja, Xintong Zhu, Feige Zhou, Yongfeng Huang, Yannis Papako... on PitharXiv
  16. [19] URLhttps://arxiv.org/abs/2506.18951. arXiv:2506.18951 [cs.DB]; NeurIPS on PitharXiv
  17. [20] URLhttps://arxiv.org/abs/2602.23008v2. Haodong Lu and Dong Gong. ASCENT: Online Test-Time Training of Long-Horizon Agents via Self-Distillation of Verified Experience, October on PitharXiv
  18. [21] URLhttp://arxiv.org/abs/2610.05303. arXiv:2610.05303 [cs]. XufangLuo, YugeZhang, ZhiyuanHe, ZilongWang, SiyunZhao, DongshengLi, LunaK.Qiu, andYuqingYang. Agent Lightning: Train ANY AI agents with reinforcement learning, August 2025a. URLhttps://arxiv.org/abs/2508.03680. arXiv:... on PithPith's reviewarXiv
  19. [23] URL http://arxiv.org/abs/2606.04815. arXiv:2606.04815 [cs]. Siru Ouyang, Jun Yan, I.-Hung Hsu, Yanfei Chen, Ke Jiang, Zifeng Wang, Rujun Han, Long T. Le, Samira Daruki, Xiangru Tang, Vishy Tirumalashetty, George Lee, Mahsan Rofouei, Hangfei Lin, Jiawei Han, Chen-Yu Lee, and To... on PithPith's reviewarXiv
  20. [24] URLhttp: //arxiv.org/abs/2509.25140. arXiv:2509.25140 [cs]. 17 Joon Sung Park, Joseph O’Brien, Carrie Jun Cai, Meredith Ringel Morris, Percy Liang, and Michael S. Bernstein. Generative Agents: Interactive Simulacra of Human Behavior. InProceedings of the 36th Annual ACM Sympos... on PithPith's reviewarXiv
show 25 more
  1. [27] Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. HybridFlow: A flexible and efficient RLHF framework.arXiv preprint arXiv:2409.19256, on PithPith's reviewarXiv
  2. [28] doi: 10.48550/arXiv.2409.19256. URLhttps://arxiv.org/abs/2409.19256. Haizhou Shi, Zihao Xu, Hengyi Wang, Weiyi Qin, Wenyuan Wang, Yibin Wang, Zifeng Wang, Sayna Ebrahimi, and Hao Wang. Continual Learning of Large Language Models: A Comprehensive Survey, June on PithPith's reviewarXivdoi:10.48550/arxiv.2409.19256
  3. [29] URL http://arxiv.org/abs/2404.16789. arXiv:2404.16789 [cs] version: on PithPith's reviewarXiv
  4. [30] URLhttps://arxiv.org/abs/2303.11366. David Silver and Richard S. Sutton. Welcome to the era of experience, on PithPith's reviewarXiv
  5. [32] URLhttp://arxiv.org/abs/2607.11012. arXiv:2607.11012 [cs.CL]. Yu Sun, Xiaolong Wang, Zhuang Liu, John Miller, Alexei A. Efros, and Moritz Hardt. Test-time training with self- supervision for generalization under distribution shifts. InProceedings of the 37th International Conf... on PithPith's reviewarXiv
  6. [33] URLhttps://arxiv.org/abs/2512.23675. Rishabh Tiwari, Kusha Sareen, Lakshya A. Agrawal, Joseph E. Gonzalez, Matei Zaharia, Kurt Keutzer, Inderjit S. Dhillon, Rishabh Agarwal, and Devvrit Khatri. Learning, Fast and Slow: Towards LLMs That Adapt Continually, May on PitharXiv
  7. [34] URLhttp://arxiv.org/abs/2605.12484. arXiv:2605.12484 [cs.LG]. Harsh Trivedi, Tushar Khot, Mareike Hartmann, Ruskin Manku, Vinty Dong, Edward Li, Shashank Gupta, Ashish Sabharwal, andNiranjanBalasubramanian. AppWorld: AControllableWorldofAppsandPeopleforBenchmarking Interactive... on PithPith's reviewarXiv
  8. [35] URLhttp://arxiv.org/abs/2407.18901. arXiv:2407.18901 [cs]. Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandku- mar. Voyager: An open-ended embodied agent with large language models, on PithPith's reviewarXiv
  9. [36] URLhttps://arxiv.org/abs/2305.16291. Ruoyao Wang, Peter Jansen, Marc-Alexandre Côté, and Prithviraj Ammanabrolu. ScienceWorld: Is your agent smarter than a 5th grader? InProceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, on PithPith's reviewarXiv
  10. [37] URL http://arxiv.org/abs/2409.07429. arXiv:2409.07429 [cs]. Tianxin Wei, Noveen Sachdeva, Benjamin Coleman, Zhankui He, Yuanchen Bei, Xuying Ning, Mengting Ai, Yunzhe Li, Jingrui He, Ed H. Chi, Chi Wang, Shuo Chen, Fernando Pereira, Wang-Cheng Kang, and Derek Zhiyuan 18 Cheng.... on PithPith's reviewarXiv
  11. [38] URL http://arxiv.org/abs/2511.20857. arXiv:2511.20857 [cs.CL]. Cheng-Kuang Wu, Zhi Rui Tam, Chieh-Yen Lin, Yun-Nung Chen, and Hung-yi Lee. StreamBench: Towards Benchmarking Continuous Improvement of Language Agents, October on PithPith's reviewarXiv
  12. [39] URLhttp://arxiv.org/abs/2406.08747. arXiv:2406.08747 [cs]. Rong Wu, Xiaoman Wang, Jianbiao Mei, Pinlong Cai, Daocheng Fu, Cheng Yang, Licheng Wen, Xuemeng Yang, Yufan Shen, Yuxin Wang, and Botian Shi. EvolveR: Self-Evolving LLM Agents through an Experience-Driven Lifecycle. In... on PithPith's reviewarXiv
  13. [40] Tianzhu Ye, Li Dong, Qingxiu Dong, Xun Wu, Shaohan Huang, and Furu Wei. Online Experiential Learning for Language Models, June 2026a. URLhttp://arxiv.org/abs/2603.16856. arXiv:2603.16856 [cs.CL]. Tianzhu Ye, Li Dong, Xun Wu, Shaohan Huang, and Furu Wei. On-Policy Context Disti... on PithPith's reviewarXiv
  14. [41] URLhttps: //arxiv.org/abs/2601.16175v2. Qizheng Zhang, Changran Hu, Shubhangi Upasani, Boyuan Ma, Fenglu Hong, Vamsidhar Kamanuru, Jay Rainton, Chen Wu, Mengmeng Ji, Hanchen Li, Urmish Thakker, James Zou, and Kunle Olukotun. Agentic Context Engineer- ing: Evolving Contexts for... on PithPith's reviewarXiv
  15. [42] URLhttps://arxiv.org/abs/2510.04618v3. Andrew Zhao, Daniel Huang, Quentin Xu, Matthieu Lin, Yong-Jin Liu, and Gao Huang. ExpeL: LLM agents are experiential learners, on PithPith's reviewarXiv
  16. [43] URLhttps://arxiv.org/abs/2308.10144. Huichi Zhou, Yihang Chen, Siyuan Guo, Xue Yan, Kin Hei Lee, Zihan Wang, Ka Yiu Lee, Guchun Zhang, Kun Shao, Linyi Yang, and Jun Wang. Memento: Fine-tuning LLM Agents without Fine-tuning LLMs, August on PithPith's reviewarXiv
  17. [44] URL http://arxiv.org/abs/2508.16153. arXiv:2508.16153 [cs]. Yuxin Zuo, Kaiyan Zhang, Li Sheng, Shang Qu, Ganqu Cui, Xuekai Zhu, Haozhan Li, Yuchen Zhang, Xinwei Long, Ermo Hua, Biqing Qi, Youbang Sun, Zhiyuan Ma, Lifan Yuan, Ning Ding, and Bowen Zhou. TTRL: Test-time reinforce... on PithPith's reviewarXiv
  18. [1984] doi: 10.2307/2981683. Vaggelis Dorovatas, Malte Schwerin, Andrew D. Bagdanov, Lucas Caccia, Antonio Carta, Laurent Charlin, Barbara Hammer, Tyler L. Hayes, Timm Hess, Christopher Kanan, Dhireesha Kudithipudi, Xialei Liu, Vincenzo Lomonaco, Jorge Mendez-Mendez, Darshan Patil, A... on Pithdoi:10.2307/2981683
  19. [2013] doi: 10.1007/s10994-012-5320-9. Moritz Hardt and Yu Sun. Test-time training on nearest neighbors for large language models. InInternational Conference on Learning Representations, on Pithdoi:10.1007/s10994-012-5320-9
  20. [2017] Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. DeepSeekMath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300, on PithPith's reviewarXiv
  21. [2022] URLhttps://arxiv.org/abs/ 2209.15189. arXiv:2209.15189 [cs.CL]. Dilara Soylu, Christopher Potts, and Omar Khattab. Fine-Tuning and Prompt Optimization: Two Great Steps that Work Better Together. InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, on PithPith's reviewarXiv
  22. [2023] Association for Computing Machinery. ISBN 979-8-4007-0132-0. doi: 10.1145/3586183.3606763. URLhttps://dl.acm.org/doi/10.1145/ 3586183.3606763. John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms.arXiv preprint ... on PithPith's reviewarXiv
  23. [2024] URLhttps://proceedings.neurips.cc/paper_files/paper/2024/hash/ 877b40688e330a0e2a3fc24084208dfa-Abstract-Datasets_and_Benchmarks_Track.html. Bo Mao, Jie Zhou, Yutao Yang, Xin Li, Xian Wei, Qin Chen, Xingjiao Wu, and Liang He. Learning While Acting: A Skill-Enhanced Test-Time C... on Pith
  24. [2025] URLhttp://arxiv.org/abs/2510.07841. arXiv:2510.07841 [cs]. Lakshya A. Agrawal, Shangyin Tan, Dilara Soylu, Noah Ziems, Rishi Khare, Krista Opsahl-Ong, Arnav Singhvi, Herumb Shandilya, Michael J. Ryan, Meng Jiang, Christopher Potts, Koushik Sen, Alexandros G. Dimakis, Ion Stoic... on PitharXiv
  25. [2026] URLhttp://arxiv.org/abs/2603.24639. arXiv:2603.24639 [cs] version: on PitharXiv

Challenges and discussion

Disagree with the review, or know something it missed? Leave a comment, or challenge this review formally: a challenge becomes a permanent entry in this paper's record, and its decision is published here.

No comments yet.

Sign in to comment