Pith. sign in

REVIEW 3 major objections 5 minor 26 references

Contextual Experience Replay for Self-Improvement of Language Agents

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Contextual Experience Replay claims that replaying distilled memories in the context window lets language agents improve on web tasks with no training, lifting WebArena success by 51.0% relative.

desk verdict Training-free experience replay lifts web-agent success rates, but the headline 51% gain leans on a small human-curated offline set that the paper itself shows is fragile. read the letter →

arxiv 2506.06698 v1 pith:PPJ7WYVJ submitted 2025-06-07 cs.AI cs.CLcs.CVcs.LG

Yitao Liu , Chenglei Si , Karthik Narasimhan , Shunyu Yao This is my paper
classification cs.AIcs.CLcs.CVcs.LG
keywords contextualexperiencereplaylanguageagentswebnavigationArenaVisualin-contextlearningself-improvement
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper proposes Contextual Experience Replay (CER), a training-free way for language-model agents to get better at web tasks while they run. Instead of fine-tuning or waiting for ground-truth rewards, the agent distills completed trajectories—both successes and failures—into short natural-language entries about page dynamics and reusable skills, stores them in a buffer, retrieves the most relevant ones for the current goal, and injects them into its context window. On WebArena, the authors report that hybrid CER (offline human-annotated trajectories plus online self-generated ones) raises the GPT-4o/BrowserGym baseline from 24.3% to 36.7% average success, a 51.0% relative gain, at roughly 17% extra input tokens. On VisualWebArena, online CER reaches 31.9%, above the tree-search method's 26.4% at a much lower token cost. The point of the work is that continual, context-only memory can substitute for environment-specific training in complex interactive settings.

What carries the argument

The load-bearing object is the dynamic experience buffer containing two structured entry types: dynamics (a page name, description, possible usages, and URL) and skills (a named sub-goal followed by parameterized steps with concrete action examples). Distillation, retrieval, and replay are all implemented by prompting a vision-language model, so the whole improvement loop is a context-engineering process rather than a training run. The retriever caps replay at $k_d=5$ dynamics and $k_s=5$ skills per task, and the experiences are mapped into natural-language text added to the agent's system prompt. This is what carries the argument: the buffer converts raw trajectories into reusable, generalizable knowledge, and retrieval keeps the context overhead small enough that the reported gains come at roughly 5.8–17.3% additional input tokens.

What would settle it

Run the hybrid setting on WebArena with the human-authored training tasks replaced by an equal number of goal-oriented tasks written for websites not in the test set; if the 51.0% relative gain over the GPT-4o baseline largely vanishes, then the transferability of distilled experiences—the premise behind the headline result—is false.

Watch

Extended reading notes

Core claim

The central claim is that an LLM agent can self-improve in a complex environment without any weight updates or reward labels by replaying distilled experience inside its context window. CER treats each past trajectory as a source of two kinds of knowledge: dynamics—summaries of important pages, their URLs, and what they are for—and skills—abstracted step-by-step workflows with variable placeholders such as 'sort posts by {criterion}'. A distillation module written as VLM prompts converts trajectories into these entries, a retrieval module selects the top-k dynamics and top-k skills for the current goal, and the agent solves the task with those entries prepended to its context. The authors show the framework works offline from a small human-annotated trajectory set, online from self-generated trajectories accumulated task by task, and best in a hybrid mode that combines both, with the hybrid configuration reaching 36.7% average success on WebArena and online CER reaching 31.9% on VisualWebArena.

Load-bearing premise

The load-bearing premise is that a small set (5–10 per website) of human-authored, goal-oriented training tasks, written on the same websites as the test tasks, yields distilled experiences whose distribution matches unseen test tasks.

Editorial extensions

If this is right

  • Hybrid CER lifts the GPT-4o/BrowserGym baseline on WebArena from 24.3% to 36.7% average success, a 51.0% relative improvement, with about 17% extra input tokens.
  • On VisualWebArena, online CER reaches 31.9% average success, outperforming the tree-search baseline (26.4%) at a much lower token cost.
  • CER improves weaker open-source models too: on the GitLab split, Llama-3.1-70B rises from 17.3% to 22.0%, a 26.5% relative gain.
  • The framework retains old competence while adding new competence: on the Forum split, stability is 93% and plasticity is 141% in cross-template success-rate terms.
  • CER combines with trajectory sampling and reranking: CER with sampling reaches 52.6% on the Forum split versus 37.7% for CER alone, showing the memory mechanism is orthogonal to other agent methods.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • An implication the paper leaves implicit is that CER's gains should transfer across websites that share UI conventions, since distilled skills use placeholders like '{sort criterion}' rather than site-specific ids; a cheap test is to train on one site and evaluate on another.
  • The sensitivity to trajectory quality in Section A.5 suggests a testable extension: automatically generating goal-oriented training tasks from website documentation could replace human annotations and preserve most of the offline gain, while pure random exploration likely cannot.
  • Because retrieval is the only selection mechanism, buffer growth may eventually create a bottleneck; a natural next experiment is to measure CER's success as the number of accumulated tasks grows past the current benchmark sizes.
  • The online-setting premise that early task order matters implies that curriculum ordering—solving easier or more informative tasks first—could be engineered rather than left to the benchmark's fixed order.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes Contextual Experience Replay (CER), a training-free framework that distills past trajectories into two types of experience, environment dynamics and decision-making skills, stores them in a dynamic memory buffer, and retrieves the most relevant experiences to augment the prompt of an LLM-based web agent. CER is evaluated on WebArena in offline, online, and hybrid settings, and on VisualWebArena in the online setting. The headline results are a 36.7% average success rate on WebArena in the hybrid setting, a 51.0% relative improvement over the authors' reproduced GPT-4o/BrowserGym baseline (24.3%), and a 31.9% success rate on VisualWebArena, outperforming the tree-search baseline (26.4%). The paper also reports ablations on cross-template generalization, stability/plasticity, the contribution of dynamics versus skills, robustness to failed trajectories, and synergy with a sampling-based method.

Significance. If the headline results hold, CER is a meaningful contribution to web-agent self-improvement: it is simple, training-free, and adds only modest token overhead while improving a strong GPT-4o baseline by roughly half on WebArena. The paper has several strengths: evaluation on external public benchmarks, a reproduced baseline, multiple ablations that support the proposed mechanism, a cross-template analysis that speaks to the memorization concern, and cost measurements. The main caveat is the reliance on a small set of human-authored offline training tasks on the same websites as the test tasks, whose sensitivity is quantified only on the Forum split. The paper honestly acknowledges this dependence in Section 7, but the full-benchmark impact is not measured.

major comments (3)
  1. [Section 4.1.1, Section A.5, Table 9] The headline 36.7% hybrid WebArena result depends on a small set of human-authored training tasks written on the same websites that appear in the test set. The paper's own A.5 experiment shows that on the Forum split, replacing human annotations with self-guided exploration drops the hybrid success rate from 41.2% to 35.1%, which is below the online-only value of 37.7%. Since the abstract's 51.0% relative improvement is defined against the 24.3% baseline, a similar drop on the full benchmark would materially reduce the claimed gain. The paper should either report the full-benchmark hybrid result with a non-human offline data source, or explicitly scope the headline claim to the human-annotated setup and provide evidence that the Forum split is representative of the full benchmark. This is load-bearing because the central claim is the 51.0% relative improvement.
  2. [Section 5.1, Table 3] There is an inconsistency in the reported improvement. The abstract and Section 4.2 state a 51.0% relative improvement for CER hybrid over the baseline, and the numbers in Table 1 indeed yield (36.7 - 24.3) / 24.3 = 51.0%. However, Table 3 reports '52.5%' in the ΔSR (%) column for CER hybrid, and the text in Section 5.1 describes this as a 'significant improvement by 52.5%'. This is neither the relative increase (51.0%) nor the absolute percentage-point increase (12.4 points). Please clarify which quantity is intended and use consistent definitions throughout the paper.
  3. [Section 5.4 and Table 5] The cross-reference error in Section 5.4 ('As shown in Fig. 5') points to a nonexistent figure; the result is reported in Table 5. More substantively, the Llama-3.1-70B experiment is conducted only on the Gitlab split, and the paper should state explicitly that this result is split-specific rather than full-benchmark evidence of generality across models.
minor comments (5)
  1. [Section 2 and Table 1 footnote] The paper cites BrowserGym as (Drouin et al., 2024), which is the WorkArena paper; please either cite the BrowserGym repository/toolkit directly or clarify the relationship between the two citations.
  2. [Section 4.2, VisualWebArena results] The claim that tree search uses 'at least 3 times fewer tokens' than CER is not supported by any token-cost measurements in the paper; please either report the relevant token counts or soften the claim to a qualitative statement.
  3. [Table 3] The column labels in Table 3 should be clarified so that ΔTokens (%) and ΔSR (%) are defined explicitly as relative changes with respect to the ReAct baseline; otherwise the table is ambiguous.
  4. [Throughout] The benchmark names are typeset inconsistently as WEBARENA and VISUALWEBARENA in some places and WebArena and VisualWebArena in others; please standardize the notation.
  5. [Section 5.3, Table 4] The definitions of stability and plasticity are based on cross-template success rates, which is reasonable, but the paper should state explicitly that these are proxy measures and not direct measurements of knowledge retention or acquisition.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: CER's success rates are benchmark measurements; the offline task set is a separate human-authored input, not a fitted parameter, and self-citations are baselines or related work, not load-bearing premises.

full rationale

This is an empirical systems paper, not a derivation, so the circularity patterns (self-definition, fitted input called prediction, imported uniqueness theorem, ansatz smuggled via citation, renaming a known result) do not apply. The central claims are external benchmark success rates on WebArena and VisualWebArena. The offline component uses a small human-authored task set (Section 4.1.1, A.4) that is filtered to avoid overlap with the test set, and the cross-template analysis (Section 5.2, Table 4) is explicitly designed to rule out trajectory memorization. The sensitivity of the hybrid result to the offline data source is openly quantified in A.5/Table 9, where self-guided exploration drops Forum hybrid from 41.2 to 35.1; this is a robustness limitation that the authors acknowledge in Section 7, but it is not a circular reduction of the prediction to its inputs. Self-citations to ReAct, Reflexion, WebShop, and Tree of Thoughts are used as adopted output formats, baselines, or previously published benchmarks/related methods, not as an unverified premise that forces the result. No parameter is fitted to the test set and then renamed a prediction; no equation-level equivalence between input and output exists. The paper's contribution is a memory-and-retrieval system evaluated against public benchmarks with baselines reproduced by the authors, so the honest finding is no significant circularity.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The central claim rests on GPT-4o-generated experience text being beneficial when replayed, on the offline training tasks resembling the test distribution, and on single-run evaluations being representative. There are no invented physical or theoretical entities; the experience buffer is a software artifact. The hand-chosen parameters (kd, ks, training-task counts, max steps) are few and disclosed, but sensitivity to them is not reported.

free parameters (5)
  • retrieval top-k for dynamics (kd) = 5
    Set to kd=5 in Section 4.1.1; hand-chosen with no sensitivity analysis reported. The final success rates depend on how many dynamics fit in the context window.
  • retrieval top-k for skills (ks) = 5
    Set to ks=5 in Section 4.1.1; hand-chosen with no sensitivity analysis reported.
  • number of human-authored offline training tasks per website = 5 to 10 (about 5% of the test set)
    Section 4.1.1 and Appendix A.4; the headline hybrid gain (36.7%) depends on this curated offline data. Section A.5 shows self-guided exploration lowers hybrid success from 41.2% to 35.1% on the Forum split.
  • max steps per task = 30
    Section 4.2, inherited from prior benchmark settings; caps token cost and bounds the trajectories available for distillation.
  • sampling temperature = 0.1
    Section 4.1.1; standard low-temperature choice, but all conclusions rest on a single run at this value.
assumptions (4)
  • domain assumption GPT-4o-based distillation and retrieval modules produce faithful, non-misleading experiences when replayed into the same model family.
    Invoked throughout Section 3 and Section 5.6; the whole method presupposes that auto-generated text experiences improve rather than corrupt the agent's decisions. The paper tests robustness to noisy trajectories in Section 5.6 but only partially.
  • domain assumption The distribution of human-authored offline training tasks overlaps the test-task distribution on WebArena; otherwise offline and hybrid gains vanish.
    Section 4.1.1 introduces per-website human-designed tasks; Section A.5 demonstrates sensitivity of offline and hybrid results to the data source. This is the structural premise behind the main 36.7% result.
  • domain assumption A single run at temperature 0.1 is representative of agent performance, and online task ordering does not materially change the outcome.
    All tables report one run; the online setting accumulates experiences task-by-task so results depend on task order, which is not analyzed or reported.
  • domain assumption Cross-template success rate is a valid proxy separating genuine skill acquisition from task memorization.
    Section 5.2 relies on WebArena's template structure (at most five tasks per template) to argue improvements are not memorization. The argument depends on the template partition being a meaningful decomposition of generalization.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Contextual Experience Replay for Self-Improvement of Language Agents." pith.science (2026). https://pith.science/paper/PPJ7WYVJ

@misc{pith2026250606698,
  author       = {Pith},
  title        = {Pith review of: Contextual Experience Replay for Self-Improvement of Language Agents},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/PPJ7WYVJ}},
  note         = {Machine review of arXiv:2506.06698}
}
read the original abstract

Large language model (LLM) agents have been applied to sequential decision-making tasks such as web navigation, but without any environment-specific experiences, they often fail in these complex tasks. Moreover, current LLM agents are not designed to continually learn from past experiences during inference time, which could be crucial for them to gain these environment-specific experiences. To address this, we propose Contextual Experience Replay (CER), a training-free framework to enable efficient self-improvement for language agents in their context window. Specifically, CER accumulates and synthesizes past experiences into a dynamic memory buffer. These experiences encompass environment dynamics and common decision-making patterns, allowing the agents to retrieve and augment themselves with relevant knowledge in new tasks, enhancing their adaptability in complex environments. We evaluate CER on the challenging WebArena and VisualWebArena benchmarks. On VisualWebArena, CER achieves a competitive performance of 31.9%. On WebArena, CER also gets a competitive average success rate of 36.7%, relatively improving the success rate of the GPT-4o agent baseline by 51.0%. We also conduct a comprehensive analysis on it to prove its efficiency, validity and understand it better.

Figures

Figures reproduced from arXiv: 2506.06698 by the authors.

Figure 1
Figure 1. Overview of Contextual Experience Replay including offline and online settings. (1) In the online setting, [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Compare ReAct baseline with ReAct + CER. The experiences, including dynamics and skills, are obtained through multiple modules as in Fig.1. They are "replayed" in the context window of the model, helping the agent to make correct decisions. For simplicity, the thinking process is neglected in the figure. or skills in short. The dynamics provide useful state information to help the agent make state-aware de￾cisions o… view at source ↗
Figure 3
Figure 3. System message for dynamics distillation module in CER [PITH_FULL_IMAGE:figures/full_fig_p016_3.png] view at source ↗
Figures from the paper (6 more)
Figure 4
Figure 4. Figure 4: System message for skills distillation module in CER [PITH_FULL_IMAGE:figures/full_fig_p017_4.png]
Figure 5
Figure 5. Figure 5: System message for dynamics retrieval module in CER [PITH_FULL_IMAGE:figures/full_fig_p018_5.png]
Figure 6
Figure 6. Figure 6: System message for skills retrieval module in CER [PITH_FULL_IMAGE:figures/full_fig_p019_6.png]
Figure 7
Figure 7. Figure 7: Experience snippet example on WEBARENA # Environment dynamics (common pages to navigate to from previous experience): ## Dynamics 1: Video Games category page ### Video Games category page contents: This page lists products that are within the "Video Games" category, i…
Figure 8
Figure 8. Figure 8: Experience snippet example on VISUALWEBARENA # Instructions: Your objective is to discover diverse and interesting tasks (that a human might give to an agent) by interacting with the webpage through these actions. You’ve executed the following actions, and observed the…
Figure 9
Figure 9. Figure 9: The instruction part of prompt for random explore agent [PITH_FULL_IMAGE:figures/full_fig_p020_9.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

26 extracted references · 23 canonical work pages

  1. [1]

    DO NOT include any other words except url, think and page summary as the format stated above

    work page

  2. [2]

    Follow the example to think and summarize the page

    work page

  3. [3]

    You should only summarize once for each unique URL

    work page

  4. [4]

    good taste

    Reflexion: language agents with verbal re- inforcement learning. InNeural Information Pro- cessing Systems. Mohit Shridhar, Xingdi Yuan, Marc-Alexandre Cote, Yonatan Bisk, Adam Trischler, and Matthew Hausknecht. 2021. {ALFW}orld: Aligning text and embodied environments for interactive learning. In International Conference on Learning Representa- tions. Pa...

    arXiv 2021

  5. [5]

    Focus on the main content of the page and may ignore the modifications made by the user when generating the summary. Figure 3: System message for dynamics distillation module in CER 16 You will be given the state-action trajectory of a user interacting with a webpage and the overall goal of the trajectory. You need to summarize skills from the trajectory....

    work page

  6. [8]

    Summarized before

    Check existing pages before generating, do not summarize pages that have already been summarized, instead, use "Summarized before" in the steps

    work page

  7. [10]

    “‘click(sort by id)“‘

    To sort the products by sort criterion, I need to click on the "Sort by" dropdown menu. “‘click(sort by id)“‘

    work page

  8. [11]

    “‘click(sort criterion id)“‘ Human user trajectory: [neglected here for length] ##Output: [neglected here for length] IMPORTANT NOTES you should absolutely follow:

    To sort the products by sort criterion, I need to select the sort criterion option from the "Sort by" dropdown menu. “‘click(sort criterion id)“‘ Human user trajectory: [neglected here for length] ##Output: [neglected here for length] IMPORTANT NOTES you should absolutely follow:

    work page

Show all 26 references
  1. [12]

    DO NOT include any other words except skills and steps as the format stated above

  2. [13]

    Summarized before

    Check existing skills before generating; do not summarize skills that have already been summarized; instead, use "Summarized before" in the steps

  3. [14]

    You should break the overall goal into sub-goals and summarize each sub-goal as a skill. Figure 4: System message for skills distillation module in CER 17 You will be given a goal of a task to be executed on a website and a list of urls and the corresponding page summary to ch...

  4. [15]

    “‘click(forums id)“‘

    Click on the "Forums" menu item. “‘click(forums id)“‘

  5. [16]

    “‘click(forum name id)“‘ Skill 2: Submit a new post

    Click on the specific forum name. “‘click(forum name id)“‘ Skill 2: Submit a new post

  6. [17]

    ‘type(title text box id,

    Type the post title in the title text box. “‘type(title text box id, "Post Title")“‘

  7. [18]

    ‘type(content text box id,

    Type the post content in the content text box. “‘type(content text box id, "Post Content")“‘

  8. [19]

    Click on the "Submit" button.‘ “click(submit button id)“‘ Skill 3: Sort posts by sort criterion

  9. [20]

    “‘click(sort by dropdown id)“‘

    Click on the "Sort by" dropdown menu. “‘click(sort by dropdown id)“‘

  10. [21]

    “‘click(sort criterion id)“‘ Output: <think> The goal is to upvote the hottest post in r/books

    Select the sort criterion option from the "Sort by" dropdown menu. “‘click(sort criterion id)“‘ Output: <think> The goal is to upvote the hottest post in r/books. The user needs to navigate to the r/books page first or go to forums to find the r/books page. Then the user needs...

  11. [22]

    write a post to express happiness is useful in a task to write a post to express sadness

    Some skills might not be consistent with the current task but it is still useful to refer to, e.g. write a post to express happiness is useful in a task to write a post to express sadness. Figure 6: System message for skills retrieval module in CER 19 # Environment dynamics (c...

  12. [23]

    “‘goto(’http://localhost:9999/user/username/edit biography’)“‘

    Navigate to the Edit Biography page. “‘goto(’http://localhost:9999/user/username/edit biography’)“‘

  13. [24]

    “‘fill(biography text area id, ’new biography content’)“‘

    Fill in the biography text area with the new biography content. “‘fill(biography text area id, ’new biography content’)“‘

  14. [25]

    Video Games

    Click on the save button to update the biography. “‘click(save button id)“‘ ... Figure 7: Experience snippet example on WEBARENA # Environment dynamics (common pages to navigate to from previous experience): ## Dynamics 1: Video Games category page ### Video Games category pag...

  15. [26]

    {sort criterion}

    Select the "{sort criterion}" option from the sort dropdown menu. “‘select_option({sort dropdown button id}, {sort criterion})“‘ ... Figure 8: Experience snippet example on VISUALWEBARENA # Instructions: Your objective is to discover diverse and interesting tasks (that a human...

  16. [2021]

    InInternational Conference on Learning Representations

    Measuring massive multitask language under- standing. InInternational Conference on Learning Representations. Gary Klein. 1998.Sources of Power: How People Make Decisions. MIT Press. Jing Yu Koh, Robert Lo, Lawrence Jang, Vikram Duvvur, Ming Chong Lim, Po-Yu Huang, Graham Neub...

    1998 arXiv

  17. [2023]

    Alexandre Drouin, Maxime Gasse, Massimo Caccia, Is- sam H

    Mind2web: Towards a generalist agent for the web.Advances in neural information processing systems, Datasets and Benchmarks Track, 36. Alexandre Drouin, Maxime Gasse, Massimo Caccia, Is- sam H. Laradji, Manuel Del Verme, Tom Marty, Léo Boisvert, Megh Thakkar, Quentin Cappart, ...

  18. [2024]

    Stan Franklin and Art Graesser

    Workarena: How capable are web agents at solving common knowledge work tasks?Interna- tional Conference on Machine Learning. Stan Franklin and Art Graesser. 1996. Is it an agent, or just a program? a taxonomy for autonomous agents. InInternational Workshop on Agent Theories, A...

    1996 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.