Machine Learning · arXiv:2611.11342 v2 · submitted 8 Nov 2026
Agent Self-Improvement with Frozen Weights: The Added Value and Limits of Failure Traces
The paper says
“We evaluate whether an AI agent's harness, the notes around its model, can learn from the agent's failure traces beyond a matched search that never sees them. It can, but less than it looks.”
more of the abstract
We tested an open 9-billion-parameter model with fixed weights; a larger teacher model wrote its notes. On generated customer-service decisions, where both note writers were told the target rules, notes based on the agent's mistakes improved accuracy by 4.0 percentage points over notes from the same process with the mistakes withheld. That process alone, trying and keeping better-scoring notes, reached about 60% of the total improvement. In full customer-service conversations the same notes showed no demonstrated improvement. Separately, a strict screen of single lessons admitted two customer-service lessons, still awaiting tests on new data, and one tool-use lesson, written without seeing the agent's mistakes, which showed no demonstrated gain on new tool-use tasks. A neutral note with no procedure scored higher than the learned lessons (exploratory). A blind audit judged 146 of 240 tool-use test items defective.
from the abstract
Pith says
Machine review · deepseek-v4-flash · 10 Nov 2026 · of version 1 of the paper; version 2 is newer · what was read was not recorded · not checked by a person
It can be wrong. Challenge this review or reply to it.
Holds up. Honest, well-controlled empirical study that quantifies how much of agent self-improvement comes from search vs. failure traces—modest claims, solid methods, worth taking seriously. This line is wrong
5 minor
How often objections from reviews like this one are right: not yet measured
Not checked: figures, code, what the references say.
5 minor objections
The paper says, at Abstract
Pith says: minor objection 1 of 5
The phrase 'neutral note with no procedure' is initially unclear; consider specifying earlier that this is a fixed system note with no learning or search procedure, to avoid confusion with the no-evidence search.
The paper says, at Figure 1
Pith says: minor objection 2 of 5
In Figure 1(c), the pass^k curves for the learned arm reflect variation across different lesson libraries per trial, while the standard harness uses one prompt. The caption notes this but the figure itself could use a symbol or annotation to remind readers that the k>1 points are not directly comparable as a measure of agent consistency.
show 3 more
The paper says, at §2, Gate description
Pith says: minor objection 3 of 5
The description of the hierarchical Bayesian model and the three named priors is concise but could benefit from a brief justification of the priors' parameter choices (e.g., the sceptical prior's variance). This is minor because the registered document likely contains full details.
The paper says, at §3, BFCL results
Pith says: minor objection 4 of 5
When reporting the audited estimate for the gate winner (+3.9 [-0.4, +8.3]), the interval includes zero. The text says 'its interval includes zero' implicitly, but stating this explicitly would help the reader interpret the fragility of the result.
The paper says, at §4, Limitations
Pith says: minor objection 5 of 5
The limitation that the teacher replacement was registered before any result is mentioned but could be moved earlier (e.g., to §2) for readers who focus on the experimental design section. This is a minor organizational suggestion.
What would settle it
Re-run the τ2 decision-level comparison with an additional neutral-note control (a fixed non-procedural note) to test whether the trace lessons' advantage over the search is specific to learned content. If the neutral note also outperforms the search, the 4.0-point gain may partly reflect non-specific note effects.
from the skeptic's reading
How the review went
4 readings by deepseek-v4-flash · 10 Nov 2026
- Reader holds up
- Skeptic agrees
- Editor wrote the verdict above
- Referee accept: 5 minor
Not run on this paper: the simulated authors' defence.
Pith did not check this review's objections against the paper.
The review in full
The review in brief, machine-written
The paper does one thing that makes it worth your time: it runs a matched no-evidence search control that the agent self-improvement literature has been lacking. The headline result—failure traces add about +4 points over the control, but the control alone gets ~60% of the total gain—is clearly stated and honestly bounded. The tau2 result doesn't transfer to full conversations; the BFCL result is swallowed by benchmark defects; the neutral note beats the learned lessons descriptively. None of those are spun.
What's good: pre-registration with hash-chained commits, the gate procedure with per-context non-inferiority, the blind scorer audit reported alongside every BFCL estimate, and the explicit list of what they do not claim. The paper does not oversell. On tau2 decisions they note the writers were told the target rules, so it measures phrasing skill, not rule discovery. On BFCL they report the audit-adjusted intervals and let the primary stand. The discussion is unusually candid about selection noise and score defects.
Soft spots in proportion. The main one the authors own: the no-evidence control only partly isolates the effect of evidence from search because the search trajectories can differ. That's real, but they are transparent about it and the central claim (small added value from failure traces) does not require perfect isolation. The tau2 gate winners were not tested on held-out data and one restates a rule the control also wrote—this is acknowledged but means that part of the pipeline is unvalidated. The BFCL neutral note comparison was exploratory, so the observation that a fixed neutral note outperforms learned sets is suggestive but not confirmatory. The defect rate on BFCL (61%) is high enough that any claim on that benchmark should be read with caution; the paper handles this by reporting both raw and audited estimates and not changing the primary.
Who benefits: anyone working on harness evolution or agent self-improvement, especially people designing evaluations. The matched search control is a methodological contribution that should become standard practice. The results are a useful corrective to work that reports gains from trace-based learning without isolating the search component.
Recommendation: This paper deserves a serious referee. It is well-designed, honestly reported, and the central empirical contribution—the controlled quantification of search vs. evidence—is new and reproducible. I would accept it with minor comments (e.g., consider a neutral note control on tau2 decisions; clarify whether the 'matched' search could be made more tightly matched by tracking the search itself). No fatal weaknesses.
The strongest objection
The paper's central claim—that access to failure traces adds +4.0 points over a matched no-evidence search—is supported by the data, and the authors honestly discuss the limitations of the control. The main limitation (the control does not fully isolate the effect of evidence from search) is acknowledged and does not invalidate the conclusion. The study is well-designed with pre-registration, multiple controls, and blind audits. No internal inconsistency or critical flaw is apparent.
The paper honestly reports its limitations and the central claim is appropriately hedged. The reader's weakest assumption is correctly identified; I concur that it does not undermine acceptance. No new load-bearing concern emerges.
The referee's report
If the central claim holds—that access to failure traces provides a modest, domain-limited benefit over a matched search that never sees failures—the paper makes several methodological contributions: (i) a matched no-evidence control that isolates the effect of evidence (partially), (ii) a registered gate procedure with harm constraints, (iii) a blind scorer audit. These contributions strengthen the evidence base for what failure traces can and cannot do for small open models with frozen weights, and they provide reusable templates for controlled evaluation of agent self-improvement. The results also align with a growing sceptical literature on generalization from failure traces, and the high defect rate on BFCL raises important questions about benchmark reliability for tool-use evaluation.
Pith's reading of the paper (a machine's summary, not the authors')
The paper tests whether a teacher model can improve a frozen agent's notes by reading the agent's mistakes. On generated customer-service decisions, notes written from failures beat notes from a matched search that never saw failures by +4.0 percentage points. However, that search alone achieved about 60% of the total improvement over the standard harness, meaning the search process contributes most of the gain. In full conversations and on held-out tool-use tasks, the failure-derived notes showed no demonstrated improvement. The paper concludes that access to failure traces provides a small additional benefit, but the majority of the improvement comes from the process of searching for better notes, not from the evidence of failures.
What carries the argument. The matched no-evidence search: the same writer, prompt, loop, selection, budget, and development feedback as the trace-based search, but with the agent's episode contents replaced by a fixed sentence. This design isolates the effect of reading failure traces from the effect of the search process itself. The gate uses a hierarchical Bayesian model to screen single lessons for credible gain, per-context non-inferiority within a 1-point margin, and bounded harm. The scorer audit is a blind defect list frozen before outcomes were opened.
Core claim. The paper's central claim is that giving a teacher model access to the agent's failure traces yields a statistically significant but small improvement over a matched search that never sees failures. The improvement of +4.0 percentage points falls below the pre-registered worthwhile threshold of 5 points, and the matched search alone captures about 60% of the total gain over the standard harness. On full conversations and held-out BFCL tasks, no improvement from failure-derived notes is demonstrated, and a neutral note with no procedure scored higher than the learned lessons in a descriptive analysis. The paper also contributes a rigorous experimental protocol including a blind scorer audit and a gate for screening individual lessons, and finds that most BFCL test instances had clear scoring defects.
The premise it rests on. The control search only partly separates evidence from search because the actual search trajectories could differ, even though the budget and development feedback are matched.
If the paper is right (Pith's inferences, not the authors')
Most of the gain from learning from traces in this setup comes from the search process, not the content of failures.
Failure traces provide only a small incremental benefit on generated decisions and none on full conversations or held-out BFCL tasks.
Note effects may be non-specific: a neutral note outperformed both trace-derived and search-derived lessons in descriptive analyses.
Benchmark scorers need blind audits: over 60% of BFCL held-out instances had clear defects.
Screens on a single split do not certify lessons: the admitted lesson's gate gain did not replicate on held-out data.
The matched no-evidence search should become a standard baseline in harness evolution studies, as it separates search from evidence effects.
The neutral note's descriptive advantage suggests that any well-written, task-appropriate note may improve performance, not just those derived from agent traces.
The high defect rate in BFCL (60.8%) raises questions about the reliability of conclusions drawn from that leaderboard.
Future work could test whether the small gain from failure traces persists when the teacher is not given the target rules, isolating rule discovery from rule phrasing.
Check this review
Pith stored no record of this review's passes and took no fingerprint of it. Reviews made by code from 13 Nov 2026 on carry both.
How often reviews like this one are right
Reviews like this one: written by deepseek-v4-flash, where what was read was not recorded.
How often objections from reviews like this one are right: not yet measured
Reviews by people
This review was written by a machine. No person has reviewed this paper on Pith yet. A scientist listed on Pith can review it for a fee, and their review is published here whatever it concludes.
The paper
Abstract
We evaluate whether an AI agent's harness, the notes around its model, can learn from the agent's failure traces beyond a matched search that never sees them. It can, but less than it looks. We tested an open 9-billion-parameter model with fixed weights; a larger teacher model wrote its notes. On generated customer-service decisions, where both note writers were told the target rules, notes based on the agent's mistakes improved accuracy by 4.0 percentage points over notes from the same process with the mistakes withheld. That process alone, trying and keeping better-scoring notes, reached about 60% of the total improvement. In full customer-service conversations the same notes showed no demonstrated improvement. Separately, a strict screen of single lessons admitted two customer-service lessons, still awaiting tests on new data, and one tool-use lesson, written without seeing the agent's mistakes, which showed no demonstrated gain on new tool-use tasks. A neutral note with no procedure scored higher than the learned lessons (exploratory). A blind audit judged 146 of 240 tool-use test items defective.
The paper's own sentences, arranged, kept beside the review and separate from it. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. No sentence in this record was written by a model.
abstract compiled · 2026-11-10
The authors' main claim, as the classifier picked it out 1.00
“It can, but less than it looks.”
How the main claim is stated: asserted 1.00
What the abstract itself states, as probabilities: claims first 0.04 · states limitation 0.97 · human participants 0.06 · controlled comparison 0.96 · randomized assignment 0.03 · states quantitative result 0.99 · compares with prior results 0.91 · mentions code or data availability 0.04
The abstract, sentence by sentence
This table labels the abstract only (arXiv listing abstract v1). The number is the probability the classifier gave the label.
| Sentence | Role | The authors' own work | States the main conclusion | Strength |
|---|---|---|---|---|
| We evaluate whether an AI agent's harness, the notes around its model, can learn from the agent's failure traces beyond a matched search that never sees them. | contribution 0.63 | 0.42 | 0.07 | unclear 0.47 |
| It can, but less than it looks. | result 0.78 | 0.79 | 0.74 | asserted 1.00 |
| We tested an open 9-billion-parameter model with fixed weights; a larger teacher model wrote its notes. | method 1.00 | 0.89 | 0.04 | asserted 0.99 |
| On generated customer-service decisions, where both note writers were told the target rules, notes based on the agent's mistakes improved accuracy by 4.0 percentage points over notes from the same process with the mistakes withheld. | result 1.00 | 0.95 | 0.20 | asserted 1.00 |
| That process alone, trying and keeping better-scoring notes, reached about 60% of the total improvement. | result 0.94 | 0.94 | 0.10 | asserted 0.99 |
| In full customer-service conversations the same notes showed no demonstrated improvement. | result 0.90 | 0.93 | 0.14 | asserted 0.99 |
| Separately, a strict screen of single lessons admitted two customer-service lessons, still awaiting tests on new data, and one tool-use lesson, written without seeing the agent's mistakes, which showed no demonstrated gain on new tool-use tasks. | result 0.84 | 0.92 | 0.06 | asserted 0.97 |
| A neutral note with no procedure scored higher than the learned lessons (exploratory). | result 0.98 | 0.86 | 0.07 | asserted 0.86 |
| A blind audit judged 146 of 240 tool-use test items defective. | result 0.90 | 0.82 | 0.04 | asserted 1.00 |
What kind of paper the classifier reports this is: empirical study 1.00 · evidence from observation or experiment 0.99
What was examined, and what was not
Sources read
- arXiv listing abstract v1
Not examined
- full text
- figures and tables
- references and citing sentences
- relations to other papers
Levels run
- P abstract
Not run
- S sections
- T body sentences
- C citations
- R cross-paper relations
How the labels were made
Compiled by classifier-1.13.0 via TypeSafe API, question bank abstract-p1-20260923, 2026-11-10: 1 request, 6,784 input tokens.
Every quoted sentence on this record is the authors'. Every label is from Pith's closed vocabulary, with the probability shown beside it. This record as JSON
Figures (1)
Figures are the authors', reproduced from the paper.
Challenges and discussion
Disagree with the review, or know something it missed? Leave a comment, or challenge this review formally: a challenge becomes a permanent entry in this paper's record, and its decision is published here.
Come back by email
A short list of the week's newly reviewed papers. Nothing is sent until you confirm from your inbox, and every email has a link that stops it. No account.
No comments yet.
Sign in to comment