Pith. sign in

REVIEW 4 major objections 4 minor 47 references

A self-evolving loop improves a frozen model's harness by +9 to +15.5 percentage points on held-out tests, with every credited gain gated by a paired 2-sigma significance test on a sealed test set.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 04:03 UTC pith:ECFAJECQ

load-bearing objection A carefully disciplined self-evolution loop with a load-bearing audit gap: the sealed-test guarantee is procedural, not mechanical, but the protocol is honest enough to deserve a serious referee. the 4 major comments →

arxiv 2607.13683 v1 pith:ECFAJECQ submitted 2026-07-15 cs.CL

Self-Evolving Agent Harnesses via Gated Semantic Quality-Diversity

Xiaotian Luo , Fengxingyu Wang , Chuanrui Hu , Dizhan Xue , Yafeng Deng This is my paper
classification cs.CL
keywords self-evolving agentsharness optimizationgated semantic MAP-Elitesquality-diversity archivepaired significance testingsealed test generalizationfrozen language modelspathology-to-patch matching
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper tries to establish that the harness around a frozen language model — the prompts, injected knowledge, control loop, and configuration — is a reliable and auditable optimization lever. It argues the hard part is not generating better edits but deciding which one truly helped, since self-generated feedback is noisy and single-run gains are often artifacts. The authors build a loop that separates proposing from crediting: a language model diagnoses failures and writes patches, while deterministic code owns sampling, measurement, and significance testing, crediting only paired 2-sigma gains on a sealed test scored once after evolution. Across six benchmark domains with a frozen open-weight model, the train-selected harness achieves credited held-out gains of +9 to +15.5 percentage points, retaining 86–147% of the training gain, and the winning patch tracks the model's dominant failure pathology rather than its size or family. If correct, this makes harness self-evolution a dependable way to raise deployed performance without touching weights, and locates the transferable contribution in the diagnose-and-credit loop itself, not in any specific harness.

Core claim

The paper's central claim is that a frozen model's harness can be improved automatically by a self-evolution loop whose every credited improvement is real, in the specific sense of clearing a paired 2σ significance bar on a sealed test withheld from all decisions. The loop's GSME archive organizes candidate edits by the (where × why) pathology they address — the harness surface touched and the diagnosed failure mode — rather than by the tasks they fix, which is the anti-overfitting bias. The claimed empirical result is credited held-out gains of +9 to +15.5 percentage points on six domains, with 86–147% retention of training gain, and a pathology-to-patch matching law: the winning patch chan

What carries the argument

The central object is the Gated Semantic MAP-Elites (GSME) archive combined with a strict three-gate credit pipeline. Each candidate harness edit is placed in a categorical archive cell indexed by (where × why): the harness surface it touches (prompt, knowledge, runtime, config) and the diagnosed failure pathology it addresses (e.g., thinking-runaway, premature finalization). Credit is decided only by deterministic code: a validity gate re-runs infrastructure failures, an activation gate requires the mechanism to have fired, and a paired 2σ significance test on per-task differences decides acceptance. The sealed test, withheld from the evolver and scored once after evolution, is the arbiter

Load-bearing premise

The sealed test is never read by the evolving agent is enforced only by procedure, not mechanism — the evolver runs in the same environment that holds the test tasks, so any observed test signal would invalidate every credited gain and retention ratio; the paper flags mechanical enforcement as future work.

What would settle it

Re-run the evolution loop with the sealed test placed in a separate container with no read access and every filesystem read by the evolver logged; reproduce the six credited sealed gains and the 86–147% retention under mechanical withholding. Alternatively, audit the evolution tree's git history and the evolver's command history for any read of the held-out task files — a single such read would falsify the 'sealed' property and make the generalization claims uninterpretable.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • If the central claim holds, harness optimization is a dependable lever for frozen, mid-capability models: credited held-out gains of +9 to +15.5 percentage points, with no overfitting collapse, across tasks the evolving agent never saw.
  • The winning patch is portable exactly as far as the pathology it targets is shared: the same careless-to-submit-verify match recurs across two model families, while a patch mismatched to a model's pathology is near-zero in effect.
  • What reliably transfers is the diagnose-and-credit loop, not any specific harness; a new model still needs its own diagnosis even if a pathology recurs.
  • A deterministic 'mean improves' rule would have credited phantom gains, including a mechanism whose activation beacon never fired; the paired significance and activation gates are necessary, not cosmetic.
  • The evolved harness expands the set of solvable tasks, since best-of-3 pass also rises on every credited domain, not just per-attempt hit rate.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the pathology-to-patch law extends beyond the two families tested, harness evolution could become a two-step procedure: first diagnose the model's dominant failure mode cheaply, then fetch the matched patch from a library — turning an expensive per-model evolution run into a lookup.
  • The anti-overfitting design could transfer to other self-improvement loops with verifiable per-task rewards (skill acquisition, memory editing): key the archive by the failure type rather than the task identity, and gate credit with paired significance.
  • The paper's own caveat that mechanical sealing is future work implies a direct audit: instrument the evolver's filesystem access and re-run with the test set physically inaccessible; the retention numbers are only convincing if they survive that check.
  • The loop's reliance on per-task verifiers means the method inherits any verifier gameability; in weak-verifier domains, an extension would need a meta-verifier or reward-model calibration before the same guarantees apply.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper proposes a self-evolving harness framework for frozen LLM agents. A stronger evolver diagnoses failures and proposes patches, while deterministic code owns sampling, measurement, and paired per-task significance testing. Candidate patches populate a categorical MAP-Elites archive keyed on (where × why) pathology cells. The authors report that the train-selected harness improves over vanilla on a sealed test across six credited benchmarks (+9 to +15.5pp, retention 86–147%), with SWE-bench reported as preliminary. They also claim a cross-model pathology-to-patch matching law on AppWorld across Qwen and Gemini models.

Significance. If the sealed-test guarantees are trustworthy, the paper makes a strong methodological contribution: it demonstrates a disciplined loop in which the proposing model is separated from the crediting machinery, every credited gain passes a paired 2σ gate on a withheld set, and the archive keyed on pathology rather than task identity is a plausible anti-overfitting inductive bias. The honest reporting of SWE-bench, the documented rejection of a wrong hypothesis, and the release of per-task differences are commendable and increase confidence in the experimental core. The cross-model dissociation, if confirmed, is a valuable result for the harness-optimization literature. However, the central claim is currently undermined by unresolved questions about sealed-test integrity and by an internal contradiction in the number of times the test set was used.

major comments (4)
  1. [§3.4 and §3.2] The sealed-test guarantee is procedural rather than mechanical. The evolver (Claude Opus 4.8 via Claude Code) operates in the same git-tracked environment that contains the test tasks, and no access log for the test set is provided. The manuscript itself states 'Enforcing the withholding mechanically is future work.' Because the headline conclusion is that the gains generalize, the possibility that the evolving agent observed the test set during evolution makes the credited sealed-test numbers uninterpretable as currently presented. This needs to be fixed, either by sandboxing the test set with access logging or by rerunning the main experiments under mechanical separation.
  2. [§4.6 vs §3.4/§4.3] The protocol says the sealed test is 'scored once' and that 'each credited number is a single sealed comparison' (Algorithm 1, §4.3). Yet §4.6 reports that on GDPval 'a recovery+VF variant reached the highest sealed score (+11.5pp)' despite being ranked below on train. This implies that more than one candidate harness was scored on the sealed test. If true, the 'scored once' claim is false and the argument that no multiple-comparison correction is needed collapses; if false, the statement needs clarification. Please disclose exactly how many harnesses were evaluated on the sealed test and whether the reported +9.2pp is the single pre-registered comparison or a post-hoc selection.
  3. [Abstract and Table 1] The abstract claims 'Across seven domains ... its credited gains there are +9 to +15.5pp and retain 86–147% of the training gain.' Table 1 shows SWE-bench with +5.1pp, z=0.78, retention 37%, explicitly not credited. Thus the credited-gain claim covers six domains, not seven. The abstract and §1 should be revised to avoid overclaiming: six credited benchmarks plus one preliminary result.
  4. [§4.1] The six credited benchmarks are said to use 'disjoint train/sealed-test splits' and SWE-bench's split is 'disjoint by construction,' but the construction is not documented. For each benchmark, specify how the split was made (e.g., provided by EvoAgentBench, random with seed, temporal split) and report any overlap check. Without this, the held-out status of the test set cannot be independently audited.
minor comments (4)
  1. [§3.4] The definition of pass@1 with K=3 attempts is unusual (per-task success over K attempts, not the standard first-sample pass@1). Please flag this at first use to avoid confusion with standard pass@1 notation.
  2. [§4.4 / Table 2] The 'thinking-off' ablation is described as 'blanket thinking-off toggle' but the table footnote says 'thinking disabled only after a detected runaway.' Please make the condition unambiguous in the table caption.
  3. [App. B, Lemma 1] The lemma refers to a 'posterior mean' without stating the prior. If a Beta prior is assumed, give the prior parameters; otherwise clarify that the variance reduction is computed for the empirical Bernoulli variance.
  4. [§4.3] The sentence 'researcher degrees of freedom are spent on train' is true only if the sealed test is never used for any model selection. The GDPval discrepancy in §4.6 makes this sentence potentially misleading; after fully resolving the test-usage question, revisit this wording.

Circularity Check

0 steps flagged

No significant circularity: credit decisions are structurally separated from proposals by deterministic gates and a single sealed evaluation.

full rationale

The credit loop is not circular because the paper explicitly separates proposing from crediting: 'why is an LLM-assigned hypothesis, not ground truth... credit comes solely from the paired-2σ gate' (§4.4), and it shows a wrong hypothesis being rejected ('the loop diagnosed a knowledge gap... the gate rejected it'). The GSME why-label only steers exploration; Proposition 2 states deployment selects 'arg max f, whose definition contains no label' (App. B), so a mislabeled pathology cannot propagate into the final choice. The train-selected harness is scored once on a sealed test (§3.4, Algorithm 1), and the main results use full K=3 re-evaluation; the score-borrowing estimator is validated 'against ground truth rather than relied upon to produce it' (App. B). The one flagged weakness is that the seal is procedural, not mechanical: 'Enforcing the withholding mechanically is future work' (§3.4). If the evolving agent read the test set, the credited gains would be uninterpretable, but the paper provides no such evidence and this is a data-integrity limitation rather than a reduction of the conclusion to its inputs. A possible author-overlap citation (Zhang et al. 2025c, Agentic Context Engineering) appears only in a related-work list ('which our knowledge lever subsumes as one editable surface among four') and is not load-bearing. No equation equates a prediction to a fitted parameter or a renamed fit.

Axiom & Free-Parameter Ledger

6 free parameters · 7 axioms · 2 invented entities

The central claim is empirical, so the ledger is dominated by protocol and domain assumptions rather than fitted parameters. The most important entries are the two load-bearing domain assumptions the paper itself flags: the procedural (not mechanical) seal of the test set (§3.4) and verifier validity (§6). The free parameters are hand-chosen protocol constants (K=3, z≥1.96, 1.5σ screen, P/R), not fitted values. No new physical entities are introduced; the GSME archive and the where×why descriptor space are new system constructs whose causal payoff is asserted but not ablated.

free parameters (6)
  • K (attempts per task) = 3
    Chosen protocol constant; all pass@1 numbers and the paired test are defined at K=3 (§4.1). Changing K changes variance and possibly the credited set.
  • Paired-2σ credit threshold = z ≥ 1.96
    Standard two-sided 5% bar applied per benchmark on the sealed test (§3.4); the paper claims no multiple-comparison correction is needed because each credited number is one sealed comparison (§4.3).
  • Anchor screen margin = 1.5σ (K=1)
    Data-driven cull margin from Lemma 1; screens candidates out before full evaluation, so it affects which harnesses are eligible for credit.
  • Termination patience/cap = P=10, R=20
    Stopping rule; never consults the test set (§3.2).
  • Tree-aware borrowing constants = unspecified (λ, η, Δ, α0 cap)
    Theorem 1 (Appendix B) uses discount λ, ancestor-child gap bound η, candidate margin Δ and a bias cap, but the manuscript leaves them undefined; affects only the cost-reduction recipe, not the main full-re-evaluation results.
  • Retention ratio definition = sealedΔ ÷ trainΔ
    Headline metric summarizing generalization; reported without uncertainty, and on TB2 the trainΔ (+6.3, z=1.60) fails the paper's own credit bar (§4.3).
axioms (7)
  • standard math Paired per-task pass@1 differences are approximately normal, so a z ≥ 1.96 threshold on the sealed set is a valid significance test for the credited gain (§3.4).
    The paper treats per-task paired differences as Gaussian with z = mean/(σ/√n); no multiple-comparison correction across the six domains (§4.3).
  • standard math Beta-posterior variance identity (Lemma 1) and weighted Hoeffding bound (Theorem 1) hold as stated; proofs are sketches.
    Appendix B gives 'Proof sketch' for both, leaving symbols (λ, η, Δ) undefined in the provided text; these back the anchor-screen and cost-borrowing claims, not the main credited numbers.
  • domain assumption Per-task verifiers (harbor, lcb_runner, LLM judge, graded rubrics) faithfully measure task success.
    The paper's own Limitation: paired-2σ crediting 'certifies that a gain is real on the measured metric, not that the metric is valid'; a harness that exploits a weak checker would still be credited (§6).
  • domain assumption The sealed test is genuinely never read by the evolving agent during evolution.
    §3.4: 'Enforcing the withholding mechanically is future work.' The evolver (Claude Code) operates in the same repo environment that holds the test tasks; no access log or mechanical barrier is provided.
  • domain assumption Train and sealed-test tasks are drawn from the same distribution, making retention (sealedΔ/trainΔ) interpretable as generalization.
    Splits are described only as 'disjoint train/sealed-test splits' (§4.1); no split construction, contamination check, or distribution-shift analysis is given.
  • domain assumption The LLM diagnoser's why-labels are useful steering signals for search.
    The paper cites evidence that root-cause diagnosis accuracy is below 30% even for strong models (§2). Proposition 2 bounds the effect of random label noise on archive fill rate but does not handle systematic misdiagnosis.
  • ad hoc to paper The K=1 anchor screen with 1.5σ margin culls only egregious losers and never a future champion.
    Asserted via Lemma 1's proof sketch (§3.2, Appendix B); a candidate dropped by the screen can never be credited, so a wrong margin would bias the credited set.
invented entities (2)
  • GSME archive (Gated Semantic MAP-Elites) no independent evidence
    purpose: Categorical quality-diversity archive keyed on (where×why) pathology, one gated elite per cell, with cross-cell recombination (§3.3).
    The claimed anti-overfitting payoff is supported by descriptive run facts (credited harness is a cross-cell recombination on 4/6 domains; accepted edits span four levers) and the paper explicitly declines the task-keyed baseline comparison ('which overfits by construction', §4.6), so the archive's causal contribution is not isolated.
  • (where × why) descriptor space no independent evidence
    purpose: Two-axis coordinate binding each patch to a lever (prompt/knowledge/runtime/config) and an LLM-diagnosed failure class; the 'why' axis is open and evolvable (§3.3).
    The 'why' labels are the evolver's own hypotheses; the paper concedes 'why is an LLM-assigned hypothesis, not ground truth' (§4.4) and cites low diagnosis accuracy in the literature (§2). Label noise is bounded, not eliminated (Prop. 2).

pith-pipeline@v1.3.0-alltime-deepseek · 18081 in / 26622 out tokens · 221218 ms · 2026-08-02T04:03:31.005335+00:00 · methodology

0 comments
read the original abstract

An LLM agent's real-task performance is shaped as much by the harness around its model as by the frozen model itself: its prompts, injected knowledge, runtime control, and configuration. In deployment the harness is often the only lever available, so improving it automatically is the natural way to raise performance without touching the weights. The hard part is not generating changes but knowing which one truly helped. Self-generated feedback is noisy, and an apparent gain can be a measurement artifact or an edit that merely overfits the tasks it was tuned on. We present a self-evolving agent-harness framework that separates proposing changes from crediting them: a language model diagnoses failures and proposes patches, while all sampling, measurement, and significance testing are owned by deterministic code, so every credited improvement is trustworthy by construction. Patches populate a gated, categorical quality-diversity archive (GSME) keyed on the (WHERE x WHY) pathology an edit addresses rather than the tasks it fixes, an anti-overfitting inductive bias; generalization is measured on a sealed test scored only after evolution. Across seven domains with a frozen open-weight model, the harness is train-selected and scored once on a sealed test; its credited gains there are +9 to +15.5pp and retain 86-147% of the training gain, evidence they generalize rather than overfit. The winning patch tracks the model's dominant pathology, not its size or family: changing the model can change the pathology and the patch, while the same pathology-to-patch match recurs across two model families. What transfers is the diagnose-and-credit loop, not any specific harness.

Figures

Figures reproduced from arXiv: 2607.13683 by Chuanrui Hu, Dizhan Xue, Fengxingyu Wang, Xiaotian Luo, Yafeng Deng.

Figure 1
Figure 1. Figure 1: The self-evolution loop. A Task Agent runs the train tasks; Scoring returns per-task results and failure signals; the Evolver diagnoses (where×why) pathologies and designs patches, a preflight prune dropping inert candidates before any rollout. Persistent state (GSME archive + git tree) is written only by gated commits; the best harness is scored once on the sealed test. elite per descriptor cell. Recent w… view at source ↗
Figure 2
Figure 2. Figure 2: Evolution tree (running example, BrowseC￾omp+). Nodes are git commits (RBR = reasoning-budget recovery, VF = verify-finalize, QD = query-decompose). Of the three round-1 recombinations only VF+RBR is syner￾gistic (⋆; sealed +13.9 pp, credited); the others match their stronger component, and four follow-ups fail to beat it. Descriptor. Each edit is placed in a cell of a categorical archive indexed by (where… view at source ↗
Figure 3
Figure 3. Figure 3: Sealed-test success across seven domains: vanilla [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: GSME archive, aggregated across the six credited domains. Rows are the four where levers; columns are open, LLM-diagnosed why classes (the last marks model-capability limits no harness edit can fix); each cell prints a mechanism’s status and one-line reason (verdicts from training splits; sealed scores in [PITH_FULL_IMAGE:figures/full_fig_p012_4.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

47 extracted references · 22 linked inside Pith

  1. [1]

    Transactions on Machine Learning Research , year=

    A Survey of Self-Evolving Agents: What, When, How, and Where to Evolve on the Path to Artificial Super Intelligence , author=. Transactions on Machine Learning Research , year=

  2. [2]

    2026 , eprint=

    Self-Harness: Harnesses That Improve Themselves , author=. 2026 , eprint=

    2026

  3. [3]

    2026 , eprint=

    From Failed Trajectories to Reliable LLM Agents: Diagnosing and Repairing Harness Flaws , author=. 2026 , eprint=

    2026

  4. [4]

    2026 , eprint=

    Agentic Harness Engineering: Observability-Driven Automatic Evolution of Coding-Agent Harnesses , author=. 2026 , eprint=

    2026

  5. [5]

    International Conference on Machine Learning (ICML) , year=

    VeRO: A Harness for Agents to Optimize Agents , author=. International Conference on Machine Learning (ICML) , year=

  6. [6]

    2026 , eprint=

    Meta-Harness: End-to-End Optimization of Model Harnesses , author=. 2026 , eprint=

    2026

  7. [7]

    2026 , eprint=

    SkillOpt: Executive Strategy for Self-Evolving Agent Skills , author=. 2026 , eprint=

    2026

  8. [8]

    2026 , eprint=

    HarnessX: A Composable, Adaptive, and Evolvable Agent Harness Foundry , author=. 2026 , eprint=

    2026

  9. [9]

    2026 , eprint=

    Adapting the Interface, Not the Model: Runtime Harness Adaptation for Deterministic LLM Agents , author=. 2026 , eprint=

    2026

  10. [10]

    2025 , eprint=

    Agentic Context Engineering: Evolving Contexts for Self-Improving Language Models , author=. 2025 , eprint=

    2025

  11. [11]

    2026 , eprint=

    Next-Generation Agentic Reinforcement Learning Systems Enable Self-Evolving Agents , author=. 2026 , eprint=

    2026

  12. [12]

    International Conference on Learning Representations (ICLR) , year=

    Darwin G\"odel Machine: Open-Ended Evolution of Self-Improving Agents , author=. International Conference on Learning Representations (ICLR) , year=

  13. [13]

    2025 , eprint=

    A Self-Improving Coding Agent , author=. 2025 , eprint=

    2025

  14. [14]

    International Conference on Learning Representations (ICLR) , year=

    Automated Design of Agentic Systems , author=. International Conference on Learning Representations (ICLR) , year=. 2408.08435 , archivePrefix=

    Pith/arXiv arXiv

  15. [15]

    International Conference on Learning Representations (ICLR) , year=

    GEPA: Reflective Prompt Evolution Can Outperform Reinforcement Learning , author=. International Conference on Learning Representations (ICLR) , year=. 2507.19457 , archivePrefix=

    Pith/arXiv arXiv

  16. [16]

    International Conference on Learning Representations (ICLR) , year=

    AFlow: Automating Agentic Workflow Generation , author=. International Conference on Learning Representations (ICLR) , year=. 2410.10762 , archivePrefix=

    Pith/arXiv arXiv

  17. [17]

    2026 , eprint=

    Empirical-MCTS: Continuous Agent Evolution via Dual-Experience Monte Carlo Tree Search , author=. 2026 , eprint=

    2026

  18. [18]

    International Conference on Learning Representations (ICLR) , year=

    Large Language Models Are Human-Level Prompt Engineers , author=. International Conference on Learning Representations (ICLR) , year=. 2211.01910 , archivePrefix=

    Pith/arXiv arXiv

  19. [19]

    International Conference on Learning Representations (ICLR) , year=

    Large Language Models as Optimizers , author=. International Conference on Learning Representations (ICLR) , year=. 2309.03409 , archivePrefix=

    Pith/arXiv arXiv

  20. [20]

    arXiv preprint arXiv:2309.16797 , year=

    Promptbreeder: Self-Referential Self-Improvement Via Prompt Evolution , author=. arXiv preprint arXiv:2309.16797 , year=

    Pith/arXiv arXiv

  21. [21]

    International Conference on Learning Representations (ICLR) , year=

    DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelines , author=. International Conference on Learning Representations (ICLR) , year=. 2310.03714 , archivePrefix=

    Pith/arXiv arXiv

  22. [22]

    Nature , volume=

    Mathematical discoveries from program search with large language models , author=. Nature , volume=

  23. [23]

    arXiv preprint arXiv:1504.04909 , year=

    Illuminating search spaces by mapping elites , author=. arXiv preprint arXiv:1504.04909 , year=

    Pith/arXiv arXiv

  24. [24]

    IEEE Transactions on Evolutionary Computation , volume=

    Quality and Diversity Optimization: A Unifying Modular Framework , author=. IEEE Transactions on Evolutionary Computation , volume=

  25. [25]

    IEEE Congress on Evolutionary Computation (CEC) , year=

    Diverse Prompts: Illuminating the Prompt Space of Large Language Models with MAP-Elites , author=. IEEE Congress on Evolutionary Computation (CEC) , year=. 2504.14367 , archivePrefix=

    Pith/arXiv arXiv

  26. [26]

    arXiv preprint arXiv:2605.09781 , year=

    Parameter-Efficient Neuroevolution for Diverse LLM Generation: Quality-Diversity Optimization via Prompt Embedding Evolution , author=. arXiv preprint arXiv:2605.09781 , year=

    Pith/arXiv arXiv

  27. [27]

    arXiv preprint arXiv:2605.30621 , year=

    Harness Updating Is Not Harness Benefit: Disentangling Evolution Capabilities in Self-Evolving LLM Agents , author=. arXiv preprint arXiv:2605.30621 , year=

    Pith/arXiv arXiv

  28. [28]

    arXiv preprint arXiv:2605.29225 , year=

    BenchTrace: A Benchmark for Testing Reflection Ability and Controlled Evolution in LLM Agents , author=. arXiv preprint arXiv:2605.29225 , year=

    Pith/arXiv arXiv

  29. [29]

    arXiv preprint arXiv:2601.00828 , year=

    Decomposing LLM Self-Correction: The Accuracy-Correction Paradox and Error Depth Hypothesis , author=. arXiv preprint arXiv:2601.00828 , year=

    arXiv

  30. [30]

    arXiv preprint arXiv:2509.25370 , year=

    Where LLM Agents Fail and How They can Learn From Failures , author=. arXiv preprint arXiv:2509.25370 , year=

    arXiv

  31. [31]

    International Conference on Machine Learning (ICML) , year=

    Which Agent Causes Task Failures and When? On Automated Failure Attribution of LLM Multi-Agent Systems , author=. International Conference on Machine Learning (ICML) , year=. 2505.00212 , archivePrefix=

    Pith/arXiv arXiv

  32. [32]

    Annual Meeting of the Association for Computational Linguistics (ACL) , year=

    Seeing the Whole Elephant: A Benchmark for Failure Attribution in LLM-based Multi-Agent Systems , author=. Annual Meeting of the Association for Computational Linguistics (ACL) , year=. 2604.22708 , archivePrefix=

    Pith/arXiv arXiv

  33. [33]

    arXiv preprint arXiv:2603.25001 , year=

    Rethinking Failure Attribution in Multi-Agent Systems: A Multi-Perspective Benchmark and Evaluation , author=. arXiv preprint arXiv:2603.25001 , year=

    arXiv

  34. [34]

    arXiv preprint arXiv:2504.11343 , year=

    A Minimalist Approach to LLM Reasoning: from Rejection Sampling to Reinforce , author=. arXiv preprint arXiv:2504.11343 , year=

    Pith/arXiv arXiv

  35. [35]

    arXiv preprint arXiv:2505.14970 , year=

    Self-Evolving Curriculum for LLM Reasoning , author=. arXiv preprint arXiv:2505.14970 , year=

    arXiv

  36. [36]

    arXiv preprint arXiv:2603.26499 , year=

    AIRA\_2: Overcoming Bottlenecks in AI Research Agents , author=. arXiv preprint arXiv:2603.26499 , year=

    Pith/arXiv arXiv

  37. [37]

    arXiv preprint arXiv:2605.22505 , year=

    Towards Direct Evaluation of Harness Optimizers via Priority Ranking , author=. arXiv preprint arXiv:2605.22505 , year=

    Pith/arXiv arXiv

  38. [38]

    arXiv preprint arXiv:2606.03108 , year=

    EvoTrainer: Co-Evolving LLM Policies and Training Harnesses for Autonomous Agentic Reinforcement Learning , author=. arXiv preprint arXiv:2606.03108 , year=

    Pith/arXiv arXiv

  39. [39]

    arXiv preprint arXiv:2605.22568 , year=

    Measuring Security Without Fooling Ourselves: Why Benchmarking Agents Is Hard , author=. arXiv preprint arXiv:2605.22568 , year=

    Pith/arXiv arXiv

  40. [40]

    arXiv preprint arXiv:2505.18102 , year=

    CapBencher: Give Your LLM Benchmark a Built-in Alarm for Test-Set Overfitting , author=. arXiv preprint arXiv:2505.18102 , year=

    Pith/arXiv arXiv

  41. [41]

    arXiv preprint arXiv:2605.13941 , year=

    EvolveMem: Self-Evolving Memory Architecture via AutoResearch for LLM Agents , author=. arXiv preprint arXiv:2605.13941 , year=

    Pith/arXiv arXiv

  42. [42]

    International Conference on Learning Representations (ICLR) , year=

    ReasoningBank: Scaling Agent Self-Evolving with Reasoning Memory , author=. International Conference on Learning Representations (ICLR) , year=

  43. [43]

    arXiv preprint arXiv:2605.21951 , year=

    Dynamic Mixture of Latent Memories for Self-Evolving Agents , author=. arXiv preprint arXiv:2605.21951 , year=

    Pith/arXiv arXiv

  44. [44]

    arXiv preprint arXiv:2509.24771 , year=

    LatentEvolve: Self-Evolving Test-Time Scaling in Latent Space , author=. arXiv preprint arXiv:2509.24771 , year=

    arXiv

  45. [45]

    Maia Polo, Felipe and Weber, Lucas and Choshen, Leshem and Sun, Yuekai and Guo, Gui and Yurochkin, Mikhail , booktitle=. tiny. 2024 , note=

    2024

  46. [46]

    North American Chapter of the Association for Computational Linguistics (NAACL) , year=

    Efficient Benchmarking (of Language Models) , author=. North American Chapter of the Association for Computational Linguistics (NAACL) , year=

  47. [47]

    Biometrika , volume=

    Inference and missing data , author=. Biometrika , volume=