Pith.

Artificial Intelligence · arXiv:2610.24453 v2 · submitted 17 Oct 2026

Hippo: Training-Free Agent Self-Evolution via Hippocampal Replay

Yutong Song, Jiang Wu, Yuxuan Ren, Tony Qin, Baoyuan Wang

The paper says

“Motivated by recursive self-improvement (RSI), we study how LLM agents can learn through external memory without updating model parameters.”

more of the abstract

Extracting useful experience from a single attempt is difficult: flawed reasoning can produce success, while poor execution can obscure a sound strategy. Preserving its value over time is also difficult: our accumulation baseline without consolidation initially improves but eventually performs below a no-memory baseline. Our key idea is to compare attempts at the same task to identify reusable decisions and refine when they apply. We introduce Hippo, a memory framework inspired by selective hippocampal encoding and replay that learns from these comparisons when shortcomings are diagnosed. It compares the resulting experience with existing memory to merge overlaps and resolve conflicts through explicit applicability conditions. This allows prior guidance to be revised without storing every lesson as a separate entry. Experiments on SWE-bench Verified, WebArena, and Mind2Web show performance gains in all three settings and substantially fewer retained experiences than the accumulation baseline. The gains extend across the evaluated base models of varying capabilities, indicating that memory-based adaptation remains useful even for highly capable models.

from the abstract

Pith says

Machine review · deepseek-v4-flash · 20 Oct 2026 · of version 1 of the paper; version 2 is newer · what was read was not recorded · not checked by a person

It can be wrong. Challenge this review or reply to it.

Holds up. Well-executed empirical paper on memory-based self-evolution for LLM agents; the core mechanism is novel and the evaluation is thorough, but the stress-test concern about unvalidated memory quality is real and acknowledged by the authors. This line is wrong

3 major objections · 6 minor

How often objections from reviews like this one are right: not yet measured

Are you an author? What you can do.

Not checked: figures, code, what the references say.

The paper says, at Section 4.2, Figure 4

The review did not quote this passage.

Pith says: Objection 1 of 3 · major

The long-horizon evaluation plots success rates over 400 tasks without any indication of variability. The main benchmark results report mean ± standard deviation over three complete runs, but Figure 4 appears to present a single run. Since learning dynamics can be sensitive to task order and stochasticity, the authors should either report error bars (e.g., over multiple seeds or shuffled task orders) or explicitly state if the figure reflects one run and discuss the potential variance.

more of objection 1

Without this, the robustness of the sustained gains claim is difficult to assess.

The paper says, at Section D.4, Table 15

The review did not quote this passage.

Pith says: Objection 2 of 3 · major

The human audit validates only the binary surprise/not-surprise classification of the judge. The quality, relevance, and correctness of the L1 reflections and L2 consolidated experiences—the content that actually drives agent behavior—are not systematically evaluated. While Section H acknowledges shared blind spots as a limitation, the central learning mechanism depends on the quality of these generated entries.

more of objection 2

The case studies (Section F) are illustrative but not representative. A more systematic assessment (e.g., human ratings of a stratified sample of L1/L2 entries for relevance, correctness, and actionability) would strengthen the claim that performance gains are due to the quality of memory content rather than to increased context or inference calls. The authors should either provide such an analysis or explicitly discuss its feasibility and why the current evidence suffices.

The paper says, at Table 5, memory update strategies

The review did not quote this passage.

Pith says: Objection 3 of 3 · major

The repeated-rewrite variant shows substantially lower performance (54.57%) than the append-only variant (70.10%) and the proposed add-and-delete (72.90%). This suggests that the rewrite policy's instructions may be suboptimal or that the LLM revises entries incorrectly when given broad freedom. While the paper compares policies, the exact rewrite prompt is only partially described in Appendix G.3.

more of objection 3

Providing the full rewrite prompt and analyzing a few examples of what went wrong (e.g., excessive modification, loss of useful details) would clarify why this policy fails and further justify the proposed restricted deletion policy.

6 minor objections

The paper says, at Abstract (line 1)

Pith says: minor objection 1 of 6

Typo: 'exmaple' should be 'example'.

The paper says, at Section 3.3, paragraph 2

Pith says: minor objection 2 of 6

The paper mentions 'evaluated basedline performance' — 'basedline' is a typo for 'baseline'.

show 4 more

The paper says, at Figure 5 caption

Pith says: minor objection 3 of 6

The figure caption says 'exmaple' — should be 'example'.

The paper says, at Table 2 caption

Pith says: minor objection 4 of 6

Table 2 is referenced in text before Table 1 is fully discussed; reordering may improve flow.

The paper says, at General

Pith says: minor objection 5 of 6

The paper uses both 'L1' and 'L2' in figures and text without consistently defining the abbreviations at first use (they are defined in Section 3.1 and 3.3, but could be more prominent).

The paper says, at Section 4.2, Figure 4

Pith says: minor objection 6 of 6

The y-axis label 'Success rate (%)' would benefit from stating 'on SWE-bench tasks' for clarity.

What would settle it

Human-annotate a random sample of 200 L2 memory entries across all benchmarks for correctness, relevance, and absence of misleading guidance. Then replace Hippo's L2 bank with a version where only entries judged as 'correct and applicable' are retained, and rerun the main experiments (Tables 1-2, Mind2Web). If performance drops significantly (more than one standard deviation from the original), it indicates that the LLM-generated memory quality is critical and that the current audit is insufficient.

from the skeptic's reading

How the review went

4 readings by deepseek-v4-flash · 20 Oct 2026

  1. Reader holds up confidence high
  2. Skeptic partly agrees 24 s · 27,590 tokens read, 2,094 written
  3. Editor wrote the verdict above 13 s · 28,384 tokens read, 1,140 written
  4. Referee minor revision: 3 major objections, 6 minor 40 s · 27,887 tokens read, 3,573 written

Not run on this paper: the simulated authors' defence.

Pith did not check this review's objections against the paper.

The review in full

The review in brief, machine-written

This paper does something genuinely useful: it addresses the performance degradation problem in unconsolidated agent memory by introducing surprise-driven selective reflection, multi-rollout contrastive induction, and consolidation with applicability-based merging. The empirical work is strong—three benchmarks, four base models, error bars, and a long-horizon study showing sustained gains while baselines degrade below no-memory. The insight that more attempts alone improve success (9.3 pp absolute, 28.8% normalized) is a practical baseline contribution. The ablation study cleanly separates the contributions of surprise detection, cross-rollout comparison, and memory update strategy.

The weakest point, as the stress-test flags, is that the only human audit (Section D.4, 194 traces) validates the judge's surprise classification but not the quality of the L1 reflections or L2 consolidated experiences. The authors acknowledge shared blind spots (Section H) but do not systematically check whether the generated guidance is correct, generalizable, or non-misleading. The case studies (Appendix F) are illustrative but not representative. Given that the same model serves as agent, judge, and memory writer, this is a real concern—though it doesn't undermine the main result, since the overall performance gains are clear. The paper would be stronger with a broader audit of memory entries.

Minor points: the free parameters (N=8, K=5) are reasonable but not optimized; the cost increase (12.4%) is modest but may not capture all overhead. The paper is honest about these.

Overall: this is a solid, novel contribution that deserves a serious referee. The central claim holds up; the soft spots are acknowledged and do not invalidate the results. I would assign it to an experienced reviewer who can push for a broader audit of memory quality but will recognize the empirical rigor otherwise.

The strongest objection

The paper's weakest point is that the only human audit (Section D.4) checks the judge's binary surprise/not-surprise classification on 194 WebArena traces, but does not evaluate the correctness, relevance, or generalizability of the L1 reflections or the L2 consolidated experiences that actually drive the agent's behavior. The memory formation pipeline (surprise-driven reflection, contrastive experience induction, memory consolidation) all use the same base model that serves as the agent, creating shared blind spots (Section H, b683). Without a systematic quality check on the generated memory entries, it is possible that the observed performance gains are not primarily due to the quality of the extracted guidance, but rather to other factors such as increased context or the additional inference calls. The case studies (Section F) are illustrative but not representative; a broader audit is needed to ensure the central learning mechanism works as intended.

The reader correctly identifies the reliability of LLM components as the weakest assumption. However, the specific gap is that the human audit covers only the judge's binary classification, not the content of the memories that drive agent behavior. The paper acknowledges shared blind spots and provides qualitative case studies, but does not systematically validate memory quality. Despite this gap, the overall empirical evidence across multiple benchmarks, models, ablations, and long-horizon evaluation is strong and consistent. The concern does not overturn the verdict, but highlights a specific area where additional validation would further strengthen the central claim.

The referee's report

The paper addresses a central challenge in building self-improving LLM agents: how to accumulate and refine experience without catastrophic forgetting or degradation. Hippo's approach of selective encoding based on surprise, contrastive experience induction from multiple rollouts, and conflict-resolving consolidation through applicability conditions is well-motivated by cognitive science and clearly operationalized. The empirical evaluation is unusually thorough, covering three diverse benchmarks (coding, interactive web, offline next-action), four base models including very capable ones like GPT-5.6 Sol, three independent runs with standard deviations, and ablations that validate each component. The long-horizon experiment (400 tasks) directly shows that naive accumulation leads to performance collapse while Hippo's consolidation avoids it, which is a significant empirical result. The paper also contributes a practical insight (multiple attempts alone improve performance) that should become a standard baseline. If the results hold, Hippo provides a principled and cost-effective way to equip LLM agents with persistent, reusable memory.

Pith's reading of the paper (a machine's summary, not the authors')

This paper argues that LLM agents can improve their own task performance by comparing multiple attempts at the same task and selectively retaining only the useful experience in an external memory, without any parameter updates. The key idea is that a single attempt is insufficient: a correct outcome may conceal flawed reasoning, while a failed attempt may reflect poor execution of a valid plan. By running multiple rollouts and contrasting successes and failures, Hippo distills reusable decisions and explicit applicability conditions. It then merges or revises existing memory entries to avoid duplication and conflict. The result is a persistently improving agent that stores far fewer entries than prior approaches and avoids the performance degradation observed with simple accumulation.

What carries the argument. The central machinery is the memory evolution engine, which operates in two steps. First, contrastive experience induction compares N parallel rollouts (default 8) of the same task to extract a candidate experience: a decision together with its conditions of use. The comparison handles four cases depending on the ratio of verified successes. Second, memory consolidation compares this candidate against related existing memories via an LLM comparator, which discards duplicates, merges overlaps by making applicability conditions explicit, and deletes replaced entries. This two-stage process ensures that memory remains compact and non-contradictory, avoiding the performance collapse observed when experience is simply appended.

Core claim. Hippo introduces a two-stage memory framework inspired by hippocampal encoding and replay. In the first stage, a 'surprise' judge identifies rollouts that fail or succeed for the wrong reasons; only those produce episodic reflections. In the second stage, the memory evolution engine compares multiple rollouts of the same task to derive a candidate experience, then compares this candidate against existing memory to merge overlaps and resolve conflicts through explicit applicability conditions. This process allows prior guidance to be revised without storing every lesson as a separate entry. On SWE-bench Verified, WebArena, and Mind2Web, Hippo improves task performance across all evaluated base models, including strong ones like GPT-5.6 Sol, while retaining substantially fewer experiences than the accumulation baseline. The method sustains gains over 400 tasks without the inverted-U degradation seen in baselines.

The premise it rests on. The LLM-based judge correctly identifies 'surprise' (failures and successes for wrong reasons), and the same model used for reflection and consolidation reliably produces accurate and generalizable guidance from multi-rollout comparisons, even when all attempts fail or a trace is mislabeled.

If the paper is right (Pith's inferences, not the authors')

Hippo provides a recipe for gradient-free self-improvement that works across diverse task domains (coding, web navigation, offline prediction).

The selective encoding and consolidation mechanism suggests that memory quality, not quantity, drives agent improvement.

Even strong models like GPT-5.6 benefit from memory-based adaptation, indicating that experience reuse remains valuable as base models improve.

The long-horizon results show that Hippo avoids the catastrophic forgetting observed in prior memory methods, enabling sustained learning over hundreds of tasks.

The surprise judge's reliance on the same model as the agent creates a potential blind spot: if the model cannot recognize its own flawed reasoning, incorrect lessons may enter memory. A separate validation model could mitigate this.

The framework could be extended to hierarchical consolidation over longer episodes, addressing the paper's acknowledged limitation on long individual tasks.

The explicit applicability conditions extracted by the LLM comparator may serve as a form of symbolic abstraction that could be transferred to other agents.

Check this review

Pith stored no record of this review's passes and took no fingerprint of it. Reviews made by code from 13 Nov 2026 on carry both.

How often reviews like this one are right

Reviews like this one: written by deepseek-v4-flash, where what was read was not recorded.

How often objections from reviews like this one are right: not yet measured

Every value above, in the review's public record (JSON)

Reviews by people

This review was written by a machine. No person has reviewed this paper on Pith yet. A scientist listed on Pith can review it for a fee, and their review is published here whatever it concludes.

Request a human review

How Pith checks papers · The full audit of this paper

The paper

Abstract

Motivated by recursive self-improvement (RSI), we study how LLM agents can learn through external memory without updating model parameters. Extracting useful experience from a single attempt is difficult: flawed reasoning can produce success, while poor execution can obscure a sound strategy. Preserving its value over time is also difficult: our accumulation baseline without consolidation initially improves but eventually performs below a no-memory baseline. Our key idea is to compare attempts at the same task to identify reusable decisions and refine when they apply. We introduce Hippo, a memory framework inspired by selective hippocampal encoding and replay that learns from these comparisons when shortcomings are diagnosed. It compares the resulting experience with existing memory to merge overlaps and resolve conflicts through explicit applicability conditions. This allows prior guidance to be revised without storing every lesson as a separate entry. Experiments on SWE-bench Verified, WebArena, and Mind2Web show performance gains in all three settings and substantially fewer retained experiences than the accumulation baseline. The gains extend across the evaluated base models of varying capabilities, indicating that memory-based adaptation remains useful even for highly capable models.

The paper's own sentences, arranged, kept beside the review and separate from it. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. No sentence in this record was written by a model.

abstract compiled · 2026-10-20

The authors' main claim, as the classifier picked it out 0.87

“Experiments on SWE-bench Verified, WebArena, and Mind2Web show performance gains in all three settings and substantially fewer retained experiences than the accumulation baseline.”

How the main claim is stated: asserted 1.00

What the abstract itself states, as probabilities: claims first 0.05 · states limitation 0.32 · human participants 0.03 · controlled comparison 0.96 · randomized assignment 0.02 · states quantitative result 0.11 · compares with prior results 0.96 · mentions code or data availability 0.03

The abstract, sentence by sentence

This table labels the abstract only (arXiv listing abstract v1). The number is the probability the classifier gave the label.

SentenceRoleThe authors' own workStates the main conclusionStrength
Motivated by recursive self-improvement (RSI), we study how LLM agents can learn through external memory without updating model parameters. background 0.54 0.38 0.04 asserted 0.57
Extracting useful experience from a single attempt is difficult: flawed reasoning can produce success, while poor execution can obscure a sound strategy. background 0.75 0.20 0.02 asserted 0.98
Preserving its value over time is also difficult: our accumulation baseline without consolidation initially improves but eventually performs below a no-memory baseline. result 0.69 0.90 0.04 asserted 1.00
Our key idea is to compare attempts at the same task to identify reusable decisions and refine when they apply. contribution 0.53 0.30 0.10 asserted 0.95
We introduce Hippo, a memory framework inspired by selective hippocampal encoding and replay that learns from these comparisons when shortcomings are diagnosed. contribution 0.71 0.79 0.10 asserted 0.99
It compares the resulting experience with existing memory to merge overlaps and resolve conflicts through explicit applicability conditions. method 0.99 0.81 0.05 asserted 1.00
This allows prior guidance to be revised without storing every lesson as a separate entry. method 0.90 0.68 0.07 asserted 0.89
Experiments on SWE-bench Verified, WebArena, and Mind2Web show performance gains in all three settings and substantially fewer retained experiences than the accumulation baseline. result 1.00 0.96 0.50 asserted 1.00
The gains extend across the evaluated base models of varying capabilities, indicating that memory-based adaptation remains useful even for highly capable models. interpretation 0.76 0.93 0.19 hedged possibility 0.81

What kind of paper the classifier reports this is: method or tool 1.00 · evidence from observation or experiment 0.99

What was examined, and what was not

Sources read

  • arXiv listing abstract v1

Not examined

  • full text
  • figures and tables
  • references and citing sentences
  • relations to other papers

Levels run

  • P abstract

Not run

  • S sections
  • T body sentences
  • C citations
  • R cross-paper relations

How the labels were made

Compiled by classifier-1.13.0 via TypeSafe API, question bank abstract-p1-20260923, 2026-10-20: 1 request, 6,784 input tokens.

Every quoted sentence on this record is the authors'. Every label is from Pith's closed vocabulary, with the probability shown beside it. This record as JSON

Figures (6)
Figure 1 of the paper
Figure 1. Correct outcomes can mask flawed reasoning, leading to erroneous or overly general
Figure 2 of the paper
Figure 2. Overview of Hippo. Hippo combines surprise-driven episodic reflection with memory
Figure 3 of the paper
Figure 3. Improvements in step success rate over the No Memory baseline for different methods across the three Mind2Web evalu￾ation splits. Long-term self-evolution requires experience that re￾mains useful across many tasks and repeated mem￾ory updates, growing memory bank alone does not demonstrate continued learning.. Prior work shows that later adaptation can weaken earlier capabilities despite fixed model parameters (Yu et al., 2026). We investigate this behavior using GPT-5.6 Sol on a collection of 450 tasks from SWE-bench Verified and SWE-bench Pro (Deng et al., 2025), with all methods…
Figure 4 of the paper
Figure 4. Long-horizon performance and memory size with GPT-5.6 Sol. (a) Success s
Figure 5 of the paper
Figure 5. Memory formation and reuse exmaple on WebArena. Excerpts use [. . . ] for omissions; dashed boxes mark L2 additions and refinements. Learning from more tries. This insight motivates Hippo to collect N concurrent trajectories per task us￾ing the same policy and retrieved memory. The dif￾ferent paths allow comparisons between successes and failures when both occur. Hippo records surprise￾triggered reflections in L1. A fix for one failure should not automatically become a general rule. We should check whether the problem recurs, specify when the rule applies, and test it on other…
Figure 6 of the paper
Figure 6. Candidate coverage on the 363 queries with a same-template experience available. Cover

Figures are the authors', reproduced from the paper.

Challenges and discussion

Disagree with the review, or know something it missed? Leave a comment, or challenge this review formally: a challenge becomes a permanent entry in this paper's record, and its decision is published here.

No comments yet.

Sign in to comment