Pith.

Artificial Intelligence · arXiv:2610.00906 v1 · submitted 1 Oct 2026 · 44 references

ActiveSaddler: Automated Curriculum Learning for Agent Harness Optimization

Sungho Park, Wonjoong Kim, Jue Zhang, Wook-Shin Han, Pengfei Gao, Chanyoung Park, Yongqiang Yao, Rao Fu

show 3 more authors

Elsie Nallipogu, Qingwei Lin, Victor Rühle

The paper says

“Automated harness optimization can substantially improve LLM agents by iteratively updating their prompts, tool interfaces, and control logic from execution feedback.”

more of the abstract

However, existing methods primarily optimize how the harness is updated while largely fixing which training scenarios generate the feedback that drives those updates. As the harness evolves, the scenarios most useful for further optimization can change, suggesting that the training curriculum itself should adapt alongside the harness. We formulate this missing dimension of harness optimization as an automated curriculum learning problem and introduce ActiveSaddler. ActiveSaddler models the evolving curriculum as a non-stationary bandit with dynamically instantiated optimization targets. It abstracts recurring failures into reusable failure-pattern arms, estimates the potential learning progress from further targeting each pattern, and adaptively balances revisiting known weaknesses with exploring unseen scenarios for new ones. Optimization outcomes continually update both the set of discovered failure patterns and their priorities, allowing the curriculum to co-evolve with the harness. Experiments on GAIA2 and Terminal-Bench 2.0 show that ActiveSaddler consistently discovers stronger harnesses, improving test Pass@1 by 4.4 and 7.5 percentage points over the same harness optimizer using a scenario order fixed before optimization, respectively. Ablations further show that these gains depend on dynamically constructing optimization targets, estimating their evolving utility, and balancing continued optimization with new failure discovery. Together, these results establish automated curriculum learning as a new crucial optimization dimension for harness optimization.

from the abstract

Pith says

Machine review · deepseek-v4-flash · 2 Oct 2026 · what was read was not recorded · not checked by a person

It can be wrong. Challenge this review or reply to it.

Holds up, with conditions. Genuine method contribution with solid GAIA2 evidence; the Terminal-Bench 7.5pp headline is a 3-task swing from a single run and needs a second run or a qualified claim. This line is wrong

2 major objections · 6 minor

How often objections from reviews like this one are right: not yet measured

Are you an author? What you can do.

Not checked: figures, code, what the 44 references say.

The paper says, at Table 1 / Section 5.2 / Appendix C

The review did not quote this passage.

Pith says: Objection 1 of 2 · major

The Terminal-Bench 2.0 headline of +7.5 pp (72.5 to 80.0) is a 3-task change on a 40-task test set, and the manuscript reports a second independent optimization run only for GAIA2 (Appendix C). Every TB2 ablation row the mechanism attribution rests on — w/o Failure-Pattern arms (69.2/73.3), w/o Arm Prioritizer (72.5), w/o Exploration Controller (71.7) — likewise comes from a single optimization trajectory.

more of objection 1

The GAIA2 two-run data (AutoSaddler 55.4/56.6, ActiveSaddler 59.8/61.6) place run-to-run spread at roughly 1–2 pp over 300 tasks; by binomial scaling to n=40, the per-run standard error of a Pass@1 estimate is near 7–8 pp, so the TB2 delta and each single-run ablation row are inside the optimization-stochasticity noise the paper itself documents. The ± values in Table 1 are standard deviations over three test-time executions of one optimized harness, not over optimization runs, so they do not quantify this uncertainty. Before the 'consistent' cross-benchmark claim (abstract, [b7], [b9]) can stand, I ask for replicated TB2 runs for the headline and the three key ablation rows, with the number of runs stated explicitly per cell; alternatively, the TB2 claims should be explicitly downgraded and marked as preliminary single-run results.

The paper says, at Eq. (8) / Appendix K / Section 5.3 RQ2

The review did not quote this passage.

Pith says: Objection 2 of 2 · major

The mechanism attribution 'gains depend on ... estimating their evolving utility' presumes that the LLM score phi_t(a) in Eq. (8) is a faithful estimate of expected population-performance change from selecting arm a, but no calibration of phi_t against realized per-arm development-set improvement is reported.

more of objection 2

The score is a hand-chosen equal-weight mean — the Arm Scoring Prompt fixes (severity + fixability + breadth + (1 - side_effect)) / 4 — and the Exploration Controller's PULL/DRAW call is explicitly made 'with no fixed formula or threshold' (Appendix K). Appendix D's EMA (56.7) and UCB-AIR (55.6) comparisons are a genuine strength: they show the LLM components beat plausible mechanical alternatives, which rules out the weak version of the concern. What remains open is whether the specific four-axis weighting and the unthresholded exploration judgment are load-bearing design choices rather than implementation details. I ask the authors to either report a sensitivity check on the axis weights with at least one run, or state plainly that the mechanism attribution is contingent on un-calibrated LLM judgments that were not validated against measured per-arm improvement.

6 minor objections

The paper says, at Appendix K

Pith says: minor objection 1 of 6

The extraction prompt states that 'a single patch should be able to fix one pattern', i.e., it assumes harness weaknesses decouple into independently fixable atomic arms; composition via multi-tagging is described in the same prompt, so the assumption is partially operationalized, but non-atomic weaknesses that no single patch fixes are not analyzed — a sentence in the limitations section would help.

The paper says, at Table 1

Pith says: minor objection 2 of 6

AutoSaddler on Terminal-Bench 2.0 reports a standard deviation of 0.0 across three test-time executions; please confirm this is accurate and not a reporting artifact, since it contrasts with the several-point spreads of adjacent rows.

show 4 more

The paper says, at Reproducibility Statement / Abstract

Pith says: minor objection 3 of 6

The paper says code 'will be available' at a project URL rather than shipping an artifact; given the detailed reproducibility statement, a hosted repository link or artifact DOI in the current version would strengthen the verification story.

The paper says, at Figure 4

Pith says: minor objection 4 of 6

The in-panel transcripts of the exploration rationales are dense and hard to read at print size; moving the PULL/DRAW decision rationales into a table would make the adaptive-exploration argument easier to follow.

The paper says, at Section 4 / Appendix K

Pith says: minor objection 5 of 6

Terminology is inconsistent: phi_t(a) is introduced as the 'learning-progress score' in Eq. (8) but is called the 'cardinal learning-progress score' in the Arm Scoring Prompt; unify the naming.

The paper says, at Section 5.1

Pith says: minor objection 6 of 6

The fixed difficulty curricula are constructed from 'five evaluations of the base harness on the training set'; Appendix F correctly counts this cost toward the budget, but stating this explicitly in Section 5.1 would avoid the impression that these evaluations are free.

What would settle it

Replicate the complete ActiveSaddler-vs-AutoSaddler comparison on Terminal-Bench 2.0 under the identical protocol and 490-rollout budget for at least 3 (ideally 5) independent optimization runs, running the three attribution ablations (w/o Failure-Pattern Arms, w/o Arm Prioritizer, w/o Exploration Controller) across the same runs. Report mean ± std of test Pass@1 over runs, the per-task delta, and an effect size with confidence interval (or a paired test over run outcomes) for the 7.5 pp gap versus the fixed shuffled AutoSaddler baseline. If the TB2 delta does not replicate at the reported magnitude, or the ablation rows collapse under run variance, the headline claim and the Table 1 mechanism attribution need revision.

from the skeptic's reading

How the review went

4 readings by deepseek-v4-flash · 2 Oct 2026

  1. Reader holds up, with conditions confidence moderate
  2. Skeptic partly agrees 1 min 31 s · 37,596 tokens read, 14,281 written
  3. Editor wrote the verdict above 20 s · 38,775 tokens read, 2,592 written
  4. Referee major revision: 2 major objections, 6 minor 48 s · 38,114 tokens read, 7,776 written

Not run on this paper: the simulated authors' defence.

Pith did not check this review's objections against the paper.

The review in full

The review in brief, machine-written

Quick read: the adaptive-curriculum idea is real, the GAIA2 evidence carries it, and the paper has one genuine soft spot — the Terminal-Bench 2.0 headline is a 3-task improvement from a single optimization run.

What's actually new: prior adapters (Task-CoEvolve, HarnessLens) select tasks on the evaluation side after candidates are proposed. This paper adapts which training scenarios generate the feedback for each harness update, using dynamically instantiated failure-pattern arms, adaptive arm prioritization, and an adaptive explore/exploit decision. That is a real distinction, and the design is clean. The paper is also honest about scope: the optimizer is held fixed, and Appendix L explicitly disclaims safety, security, and reward-hacking readiness. I don't hold that against them.

The empirical package is well above average. Two independent GAIA2 runs (59.8/61.6 vs 55.4/56.6 on a 300-task test set), a port to a second optimizer (GEPA, +3.0pp), non-LLM baselines (EMA scoring, UCB-AIR) that underperform the LLM modules, a cost analysis, and an ablation ladder whose ordering is internally consistent. Circularity burden is low: the comparison is against fixed-order baselines on external benchmarks, not fitted parameters. The case studies showing an arm retargeted after a failed first patch make the mechanism story plausible.

The soft spot is TB2. The 7.5pp delta is 3 tasks on a 40-task test set, from a single run, and every TB2 ablation row is also single-run. The paper's own GAIA2 data put run-to-run spread at ~1–2pp on 300 tasks; scaled to 40 tasks that becomes several pp of noise, so the TB2 number is closer to 2 sigma than to a confirmed effect. The paper discloses the 40-task size, which I credit, but quoting 7.5pp as a headline overstates what one run on 40 tasks supports. The mechanism attribution does not collapse — GAIA2 independently supports it — but the TB2 number needs a second run or a qualified write-up.

Secondary concern: the curriculum decisions are LLM judgments (equal-weight four-axis scores, an exploration call with explicitly no formula) with no calibration check against population performance. Partly mitigated by the EMA/UCB-AIR ablations, which show the LLM modules beat mechanical alternatives, but a calibration analysis would tighten it.

Who it's for: anyone working on offline harness or prompt optimization for LLM agents. This deserves a serious referee. My recommendation: send to review, and require a TB2 replication run and either calibration evidence for the scoring or a pruned headline claim.

The strongest objection

The central claim requires the 4.4/7.5 pp deltas to be real and attributable to the adaptive curriculum. The GAIA2 part is well supported: Appendix C reports two independent runs (ActiveSaddler 59.8/61.6 vs AutoSaddler 55.4/56.6) on a 300-task test set, and the ablation ladder (uniform scoring 55.3, EMA 56.7, UCB-AIR 55.6, LLM 59.8) plus the GEPA port (Appendix E, +3.0 pp) and cost analysis (Appendix F) are genuine strengths. The Terminal-Bench 2.0 part is not comparably supported. TB2 has 40 test tasks (Table 1), so the 7.5 pp headline delta (72.5→80.0, [b7], [b134]) is a 3-task change from a single optimization run, and every TB2 row the mechanism attribution rests on — w/o Failure-Pattern arms (69.2/73.3), w/o Arm Prioritizer (72.5), w/o Exploration Controller (71.7) — is also single-run. The paper's own GAIA2 two-run data place run-to-run spread at ~1.2 pp (AutoSaddler 55.4/56.6) and ~1.8 pp (ActiveSaddler 59.8/61.6) on 300 tasks; binomial scaling to 40 tasks grows variance ~2.7x, implying several-pp run noise, so the TB2 delta is only ~1.5-2 sigma. The reward signal does not shore this up: phi_t(a) in Eq. 8 is an equally weighted LLM judgment of severity/fixability/breadth/side-effect risk, and the Exploration Controller makes the PULL/DRAW call with explicitly 'no fixed formula or threshold' (Appendix K), with no reported calibration of either against actual per-arm development-set improvement. The authors disclose the 40-task test size and run GAIA2 twice only; the gap is that the larger claimed gain and the entire attribution table are unverified against run-to-run variance.

The paper's central claim has two parts: an empirical improvement (4.4 pp GAIA2, 7.5 pp TB2) and a mechanism attribution (failure-pattern arms, adaptive prioritization, adaptive exploration). The GAIA2 evidence is robust: two independent runs with a 300-task test set, and ablations placing the LLM curriculum above mechanical alternatives (EMA 56.7, UCB-AIR 55.6, uniform 55.3 vs LLM 59.8); the GEPA port (+3.0 pp) and the end-to-end cost analysis are supporting evidence, and Appendix L appropriately disclaims safety and security readiness, which I credit rather than penalize. The load-bearing weakness is confined to the Terminal-Bench 2.0 portion and to the single-run status of the attribution table: 40 test tasks, a 3-task headline delta, and all four TB2 ablation rows from one optimization run each. The paper's own two-run GAIA2 data (spread ~1.2-1.8 pp on 300 tasks) imply several-pp run noise when scaled to 40 tasks, making the 7.5 pp delta marginal. I distinguish my concern from the reader's weakest assumption (LLM calibration faithfulness). That concern is real but partially mitigated by the ablation ladder showing the LLM scores beat mechanical signals; the sharper gap is that the larger claimed gain and the entire attribution are unverified against run-to-run variance, which only replication can settle. This sharpens the condition on the existing CONDITIONAL verdict rather than changing its level, so I recommend UNCHANGED.

The referee's report

The GAIA2 evidence is strong: two independent optimization runs (Appendix C, 55.4/56.6 vs 59.8/61.6) on a 300-task test set of disjoint universes, a consistent ablation ladder (55.3, 55.3, 55.6, 56.7, 59.8), a cross-optimizer GEPA port, non-LLM alternative comparisons, and a careful end-to-end cost analysis showing that the curriculum's optimizer-side overhead is offset by better allocation of expensive task-agent rollouts. The appendices ship unusually complete detail: prompt templates for extraction, scoring, and exploration; a pattern CLI; per-arm discovery tables; and per-iteration pull-probability trajectories. These are genuine reproducibility assets. If the effect holds, the contribution is meaningful: it cleanly isolates the 'which feedback' dimension of harness optimization from 'how feedback is used' and shows it matters under a fixed rollout budget. The main weakness is that the Terminal-Bench 2.0 half of the headline rests on a single optimization run on a 40-task test set, leaving the cross-benchmark 'consistent' claim and the TB2 mechanism attributions within the run-noise range the paper itself documents on GAIA2.

Pith's reading of the paper (a machine's summary, not the authors')

This paper argues that in offline LLM-agent harness optimization, the training scenarios that generate the feedback for each harness update should adapt as the harness itself improves, rather than being fixed before optimization begins. Harness optimization iteratively patches prompts, tool interfaces, and runtime control logic from execution traces, but prior methods schedule those training scenarios in advance. The paper introduces ActiveSaddler, which models the evolving curriculum as a non-stationary bandit whose arms are failure-pattern weaknesses instantiated on the fly from failed executions, and at each step either revisits a known weakness or probes an unseen scenario to discover new ones. On GAIA2 and Terminal-Bench 2.0, the same AutoSaddler optimizer driven by ActiveSaddler's curriculum beats a fixed scenario order by 4.4 and 7.5 percentage points of test Pass@1, and ablations attribute the gains to failure-pattern arm granularity, adaptive arm prioritization, and adaptive exploration timing.

What carries the argument. The failure-pattern arm is the load-bearing abstraction: a harness weakness shared across failed trajectories, constructed by an extractor that first abstracts each failure into a candidate symptom without consulting existing arms, then normalizes candidates by linking them to an existing atomic pattern, composing several patterns, or instantiating a new arm. These arms form the arm set of a non-stationary bandit in which selection is a softmax over LLM-estimated learning-progress scores with temperature $\tau_{\text{sel}}$, and the Exploration Controller makes a binary draw-versus-pull call on whether the known arms still contain a worthwhile unresolved target. Because the arm pool, the scores, and the exploration decision are all recomputed from the latest optimization history, the curriculum co-evolves with the harness.

Core claim. The central claim is that scenario selection is a distinct, exploitable optimization dimension for harness optimization: under a fixed rollout budget, test Pass@1 rises from 55.4% to 59.8% on GAIA2 and from 72.5% to 80.0% on Terminal-Bench 2.0 compared with the same AutoSaddler optimizer using a scenario order fixed before optimization. The mechanism is a bandit over dynamically instantiated failure-pattern arms: recurring failures are abstracted through a two-stage extractor into reusable patterns, each arm is scored by an LLM estimate of learning progress $\phi_t(a)\in[0,1]$ based on severity, fixability, breadth, and inverted side-effect risk, and an exploration controller decides each iteration whether to pull a known arm or draw unseen scenarios. The paper further claims the gains decompose into three design principles, each validated by ablation: failure-pattern granularity rather than category- or scenario-level arms, adaptive priority rescoring as the harness evolves, and state-dependent exploration timing that repairs when unresolved weaknesses remain and explores when they do not.

The premise it rests on. The load-bearing premise is that the LLM's judgments are faithful: the four-axis learning-progress scores are treated as estimates of the true expected population-performance change from each arm, and the draw-versus-pull call, made with explicitly no fixed formula or threshold, picks the scenario batch that actually helps, with no calibration check against realized performance.

If the paper is right (Pith's inferences, not the authors')

Under a fixed rollout budget, adaptively selecting training scenarios yields stronger final harnesses than any fixed order: +4.4 percentage points on GAIA2 and +7.5 on Terminal-Bench 2.0 over AutoSaddler with a fixed order, and larger margins over difficulty-ordered curricula.

The curriculum benefit transfers to a different optimizer: applying the same scenario-selection policy to GEPA raises its average test Pass@1 from 54.2% to 57.2%.

Adaptive curricula improve end-to-end cost efficiency despite added optimizer overhead, reaching a given development accuracy at a fraction of the cost of fixed orders, for instance 58.5% at $298 on GAIA2 versus $1,360 for AutoSaddler.

Arm granularity matters: failure-pattern arms more often sample scenarios on which the harness still fails than category arms (too coarse) or scenario arms (too fragmented), by 16.5 and 35.3 points on GAIA2.

Curriculum decisions are state-dependent: the draw rate falls from 44% in the first half of GAIA2 optimization to 20% in the second half, and pulls occur with 2.0x to 6.9x as many unresolved arms as draws do.

The gains depend on all three components, since removing the failure-pattern arms, the arm prioritizer, or the exploration controller each costs several percentage points of test Pass@1.

A natural next experiment is an oracle substitution: replace the LLM arm scorer with the true realized arm-level performance gains and see how much of the gap over fixed order remains; if the oracle adds little, the curriculum structure, not scorer calibration, is doing the work, whereas a large gap would mean the method leans on subjective judgment that a worse scorer could erase.

Because the four-axis score is a hand-chosen equal-weight combination and the draw/pull call is explicitly made with no fixed formula or threshold, the method's resilience to weaker or cheaper LLM judges is untested; a smaller model or a lower reasoning setting could plausibly collapse the 4.4 and 7.5 point gains even though the bandit structure is sound.

The failure-pattern abstraction presumes weaknesses decouple into independently fixable atomic patterns; nothing in the method validates that premise, so the approach is a testable bet that harness defects do not interact in ways that make composed patterns degenerate into a single unresolved megatarget.

A portability prediction follows from the design: because the curriculum only chooses which scenarios feed a fixed optimizer, it should transfer to any offline harness optimizer that exposes per-batch failure traces; the GEPA result is initial support, and an optimizer with very different update semantics would settle whether the benefit is universal.

Check this review

Pith stored no record of this review's passes and took no fingerprint of it. Reviews made by code from 13 Nov 2026 on carry both.

How often reviews like this one are right

Reviews like this one: written by deepseek-v4-flash, where what was read was not recorded.

How often objections from reviews like this one are right: not yet measured

Every value above, in the review's public record (JSON)

Reviews by people

This review was written by a machine. No person has reviewed this paper on Pith yet. A scientist listed on Pith can review it for a fee, and their review is published here whatever it concludes.

Request a human review

How Pith checks papers · The full audit of this paper

The paper

Abstract

Automated harness optimization can substantially improve LLM agents by iteratively updating their prompts, tool interfaces, and control logic from execution feedback. However, existing methods primarily optimize how the harness is updated while largely fixing which training scenarios generate the feedback that drives those updates. As the harness evolves, the scenarios most useful for further optimization can change, suggesting that the training curriculum itself should adapt alongside the harness. We formulate this missing dimension of harness optimization as an automated curriculum learning problem and introduce ActiveSaddler. ActiveSaddler models the evolving curriculum as a non-stationary bandit with dynamically instantiated optimization targets. It abstracts recurring failures into reusable failure-pattern arms, estimates the potential learning progress from further targeting each pattern, and adaptively balances revisiting known weaknesses with exploring unseen scenarios for new ones. Optimization outcomes continually update both the set of discovered failure patterns and their priorities, allowing the curriculum to co-evolve with the harness. Experiments on GAIA2 and Terminal-Bench 2.0 show that ActiveSaddler consistently discovers stronger harnesses, improving test Pass@1 by 4.4 and 7.5 percentage points over the same harness optimizer using a scenario order fixed before optimization, respectively. Ablations further show that these gains depend on dynamically constructing optimization targets, estimating their evolving utility, and balancing continued optimization with new failure discovery. Together, these results establish automated curriculum learning as a new crucial optimization dimension for harness optimization.

The paper's own sentences, arranged, kept beside the review and separate from it. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. No sentence in this record was written by a model.

abstract compiled · 2026-10-02

The authors' main claim, as the classifier picked it out 1.00

“Together, these results establish automated curriculum learning as a new crucial optimization dimension for harness optimization.”

How the main claim is stated: asserted 1.00

What the abstract itself states, as probabilities: claims first 0.20 · states limitation 0.11 · human participants 0.03 · controlled comparison 0.95 · randomized assignment 0.02 · states quantitative result 0.99 · compares with prior results 0.95 · mentions code or data availability 0.03

The abstract, sentence by sentence

This table labels the abstract only (arXiv listing abstract v1). The number is the probability the classifier gave the label.

SentenceRoleThe authors' own workStates the main conclusionStrength
Automated harness optimization can substantially improve LLM agents by iteratively updating their prompts, tool interfaces, and control logic from execution feedback. background 1.00 0.09 0.04 asserted 0.57
However, existing methods primarily optimize how the harness is updated while largely fixing which training scenarios generate the feedback that drives those updates. gap 0.86 0.07 0.03 asserted 1.00
As the harness evolves, the scenarios most useful for further optimization can change, suggesting that the training curriculum itself should adapt alongside the harness. gap 0.70 0.10 0.06 hedged possibility 0.99
We formulate this missing dimension of harness optimization as an automated curriculum learning problem and introduce ActiveSaddler. contribution 1.00 0.90 0.06 asserted 0.98
ActiveSaddler models the evolving curriculum as a non-stationary bandit with dynamically instantiated optimization targets. method 0.96 0.88 0.04 asserted 1.00
It abstracts recurring failures into reusable failure-pattern arms, estimates the potential learning progress from further targeting each pattern, and adaptively balances revisiting known weaknesses with exploring unseen scenarios for new ones. method 0.99 0.87 0.05 asserted 1.00
Optimization outcomes continually update both the set of discovered failure patterns and their priorities, allowing the curriculum to co-evolve with the harness. method 0.97 0.79 0.10 asserted 1.00
Experiments on GAIA2 and Terminal-Bench 2.0 show that ActiveSaddler consistently discovers stronger harnesses, improving test Pass@1 by 4.4 and 7.5 percentage points over the same harness optimizer using a scenario order fixed before optimization, respectively. result 1.00 0.96 0.24 asserted 1.00
Ablations further show that these gains depend on dynamically constructing optimization targets, estimating their evolving utility, and balancing continued optimization with new failure discovery. result 0.74 0.95 0.08 asserted 1.00
Together, these results establish automated curriculum learning as a new crucial optimization dimension for harness optimization. interpretation 1.00 0.81 0.90 asserted 1.00

What kind of paper the classifier reports this is: method or tool 1.00 · evidence from observation or experiment 1.00

What was examined, and what was not

Sources read

  • arXiv listing abstract v1

Not examined

  • full text
  • figures and tables
  • references and citing sentences
  • relations to other papers

Levels run

  • P abstract

Not run

  • S sections
  • T body sentences
  • C citations
  • R cross-paper relations

How the labels were made

Compiled by classifier-1.13.0 via TypeSafe API, question bank abstract-p1-20260923, 2026-10-02: 1 request, 7,576 input tokens.

Every quoted sentence on this record is the authors'. Every label is from Pith's closed vocabulary, with the probability shown beside it. This record as JSON

Figures (16)
Figure 1 of the paper
Figure 1. Optimization efficiency across benchmarks. Best-so-far development accuracy versus cumulative task-agent rollouts for ActiveSaddler and baseline methods. 1 INTRODUCTION Large Language Models (LLMs) have become increasingly capable of powering autonomous agents, yet reliable performance on multi-step and long-horizon tasks still depends heavily on the external ∗Work done during an internship at Microsoft. †Corresponding authors. 1 arXiv:2610.00906v1 [cs.AI] 1 Oct 2026
Figure 2 of the paper
Figure 2. Overview of ActiveSaddler. At each iteration, the curriculum chooses between exploring unseen scenarios and revisiting an instantiated arm, optimizes the harness on the selected scenario batch, and updates its state based on the resulting execution and optimization outcomes. across multiple scenarios and traces. Because such weaknesses are not directly observable, we infer them through failure diagnosis and group failures attributed to the same deficiency into a common failure-pattern arm. This lets related failures contribute evidence toward a shared optimization target. This…
Figure 3 of the paper
Figure 3. Effect of adaptive arm scoring. Adaptive scoring dynamically reallocates optimization priority and yields more effective repair of observed failures. Together, these results show that adapting scenario selection during optimization improves harness quality over fixed curricula under the same optimizer. We further examine these gains through an independent optimization run, a different offline optimizer, and alternative curriculum strategies. In an independent GAIA2 run, ActiveSaddler again outperforms all fixed-order baselines, showing robustness to optimization stochasticity…
Figure 4 of the paper
Figure 4. Adaptive exploration over the optimization trajectory. The policy chooses DRAW when discovery is more valuable and PULL when promising repair opportunities remain. the same score to every arm while keeping the remaining curriculum procedure unchanged. As shown in
Figure 5 of the paper
Figure 5. End-to-end optimization efficiency. Best-so-far development accuracy against cumulative monetary cost, LLM calls, input tokens, and output tokens on GAIA2 (top) and Terminal-Bench 2.0 (bottom). Cumulative usage includes both optimizer-side and task-agent computation. is broadly comparable across methods on each benchmark. This suggests that differences in total task-agent cost are driven primarily by how rollouts are allocated over the optimization trajectory rather than by substantially cheaper individual executions. End-to-End Cost Efficiency. We next combine optimizer-side and…
Figure 6 of the paper
Figure 6. Case study on tnxtee (GAIA2), where a failure-pattern arm instantiated from the initial failure enables an unsuccessful repair to be retargeted and refined in a subsequent iteration. The second repair extended the trigger condition to multiline affirmative replies, allowing the hook to correctly append the required closing and real-name signature; the scenario then passed. In the category-arm variant, tnxtee was grouped with 10 other scenarios under the much broader category-time arm. After the first sampled time batch passed, the arm’s severity dropped sharply from 0.62 to 0.12,…
Figure 7 of the paper
Figure 7. Case study on 4bytkx (GAIA2), where retargeting the same failure-pattern arm corrects an executor hook that was initially inserted too late in the execution flow
Figure 8 of the paper
Figure 8. Failure hit rate by arm definition. Fraction of optimization iterations whose sampled mini-batch contains at least one scenario with an unresolved failure. Budgets are 1,400 rollouts for GAIA2 and 490 for TB2. Failure Hit Rate
Figure 9 of the paper
Figure 9. Case study of a resolved weakness (P3 of GAIA2). After an accepted patch at Iteration 5, the arm’s priority drops sharply. A later pull finds all supporting scenarios successful and produces no further patch, after which the arm remains at low priority. 50 55 60 65 Dev accuracy (%) 0 12 24 36 48 0.00 0.25 0.50 0.75 1.00 P rio rit y ϕ P8 arm Priority ϕ Arm pulled Scenario joins the pattern Patch accepted No patch: all passed Dev accuracy (shaded) Best dev so far
Figure 10 of the paper
Figure 10. Case study of a recurring weakness (P8 of GAIA2). Early pulls produce no patch because the associated scenarios succeed when re-executed, yet the arm remains highly prioritized. The same failure pattern later appears in additional scenarios, providing new evidence that the weakness remains active; a subsequent pull produces an accepted patch. Cross markers indicate scenarios newly associated with P8. I ADDITIONAL ANALYSIS FOR RQ2 Case Studies. To illustrate how LLM-based arm scoring responds to evolving optimization evidence,
Figure 11 of the paper
Figure 11. Evolution of pull-probability mass across failure-pattern arms. The ten arms with the highest average pull probability over the run are shown individually, while the remaining 25 arms are aggregated into Other arms. The upper panel shows the number of instantiated arms, which grows from 6 to 35 as new failure patterns are discovered. Pull probabilities are shown only at the 34 iterations where arm selection is performed. Evolution of Arm Priorities. To further examine the curriculum-level dynamics summarized in Figure 3a, we analyze the pull-probability trajectories of all…
Figure 12 of the paper
Figure 12. Pull-probability trajectories for all failure-pattern arms. Rows show all 35 discovered arms ordered by instantiation time, and columns correspond to the 34 iterations in which arm selection is performed. Cell intensity denotes pull probability, orange markers indicate the arm sampled at each iteration, and gray cells denote iterations before an arm entered the pool. their supporting scenarios subsequently succeed, while new observations of the same failure pattern across additional scenarios can increase their priority again. This evidence-dependent prioritization explains the…
Figure 13 of the paper
Figure 13. Dependence of exploration decisions on unresolved weaknesses. Ratio of the average number of unresolved arms at PULL to DRAW decisions. Values above one indicate more unresolved weaknesses at PULL, while values near one indicate little difference between the two decision types. J ADDITIONAL ANALYSIS FOR RQ3 Dependence of Exploration Decisions on Unresolved Weaknesses. To quantify whether explo￾ration decisions respond to unresolved weaknesses, we compare how many arms remain unresolved at PULL and DRAW decisions. An arm is considered unresolved when its associated weakness remains…
Figure 14 of the paper
Figure 14. Failure Pattern Extraction Prompt: the agent independently abstracts failed execution
Figure 15 of the paper
Figure 15. Arm Scoring Prompt: the agent estimates a learning-progress score
Figure 16 of the paper
Figure 16. Unseen Scenario Exploration Prompt: the agent decides whether to continue optimizing a

Figures are the authors', reproduced from the paper.

References (44)
  1. [1] Pierre. Intrinsic Motivation Systems for Autonomous Mental Development , journal =. 2007 , url =. doi:10.1109/TEVC.2006.890271 , timestamp = on Pithdoi:10.1109/tevc.2006.890271
  2. [2] Manuel Lopes and Pierre. The strategic student approach for life-long exploration and learning , booktitle =. 2012 , url =. doi:10.1109/DEVLRN.2012.6400807 , timestamp = on Pithdoi:10.1109/devlrn.2012.6400807
  3. [3] Alex Graves and Marc G. Bellemare and Jacob Menick and R. Automated Curriculum Learning for Neural Networks , booktitle =. 2017 , url = on Pith
  4. [4] Yifan Yang and Ziyang Gong and Weiquan Huang and Qihao Yang and Ziwei Zhou and Zisu Huang and Yan Li and Xuemei Gao and Qi Dai and Bei Liu and Kai Qiu and Yuqing Yang and Dongdong Chen and Xue Yang and Chong Luo , title =. CoRR , volume =. 2026 , url =. doi:10.48550/ARXIV.2605... on PithPith's reviewarXivdoi:10.48550/arxiv.2605.23904
  5. [5] Tambet Matiisen and Avital Oliver and Taco Cohen and John Schulman , title =. CoRR , volume =. 2017 , url =. 1707.00183 , timestamp = on PithPith's reviewarXiv
  6. [6] R. Teacher algorithms for curriculum learning of Deep. 3rd Annual Conference on Robot Learning, CoRL 2019, Osaka, Japan, October 30 - November 1, 2019, Proceedings , series =. 2019 , url = on Pith
  7. [7] Wenyi Wang and Piotr Piekos and Nanbo Li and Firas Laakom and Yimeng Chen and Mateusz Ostaszewski and Mingchen Zhuge and J. Huxley-G. CoRR , volume =. 2025 , url =. doi:10.48550/ARXIV.2510.21614 , eprinttype =. 2510.21614 , timestamp = on Pithdoi:10.48550/arxiv.2510.21614
  8. [8] Yizao Wang and Jean. Algorithms for Infinitely Many-Armed Bandits , booktitle =. 2008 , url = on Pith
  9. [9] Chenyu Zhou and Huacan Chai and Wenteng Chen and Zihan Guo and Rong Shan and Yuanyi Song and Tianyi Xu and Yingxuan Yang and Aofan Yu and Weiming Zhang and Congming Zheng and Jiachen Zhu and Zeyu Zheng and Zhuosheng Zhang and Xingyu Lou and Changwang Zhang and Zhihui Fu and Ju... on PithPith's reviewarXivdoi:10.48550/arxiv.2604.08224
  10. [10] Effective harnesses for long-running agents , author =. 2025 , howpublished = on Pith
  11. [11] Harness engineering: leveraging Codex in an agent-first world , author =. 2026 , howpublished = on Pith
  12. [12] Improving Deep Agents with harness engineering , author =. 2026 , howpublished = on Pith
  13. [13] Romain Froger and Pierre Andrews and Matteo Bettini and Amar Budhiraja and Ricardo Silveira Cabral and Virginie Do and Emilien Garreau and Jean. Gaia2: Benchmarking. CoRR , volume =. 2026 , url =. doi:10.48550/ARXIV.2602.11964 , eprinttype =. 2602.11964 , timestamp = on Pithdoi:10.48550/arxiv.2602.11964
  14. [14] Mike A. Merrill and Alexander Glenn Shaw and Nicholas Carlini and Boxuan Li and Harsh Raj and Ivan Bercovich and Lin Shi and Jeong Yeon Shin and Thomas Walshe and Estefany Kelly Buchanan and Junhong Shen and Guanghao Ye and Haowei Lin and Jason Poulos and Maoyu Wang and Marian... on PithPith's reviewarXivdoi:10.48550/arxiv.2601.11868
  15. [15] Yoonho Lee and Roshen Nair and Qizheng Zhang and Kangwook Lee and Omar Khattab and Chelsea Finn , title =. CoRR , volume =. 2026 , url =. doi:10.48550/ARXIV.2603.28052 , eprinttype =. 2603.28052 , timestamp = on PithPith's reviewarXivdoi:10.48550/arxiv.2603.28052
  16. [16] Sungho Park and Wonjoong Kim and Rongyuan Tan and Jue Zhang and Wook. AutoSaddler: Automatic Harness Optimization with Durable Updates from Agent Execution Traces , journal =. 2026 , url =. doi:10.48550/ARXIV.2608.23041 , eprinttype =. 2608.23041 , timestamp = on PithPith's reviewarXivdoi:10.48550/arxiv.2608.23041
  17. [17] Lakshya A. Agrawal and Shangyin Tan and Dilara Soylu and Noah Ziems and Rishi Khare and Krista Opsahl. The Fourteenth International Conference on Learning Representations,. 2026 , url = on Pith
  18. [18] Jeff Clune , title =. CoRR , volume =. 2019 , url =. 1905.10985 , timestamp = on PithPith's reviewarXiv
  19. [19] Rui Wang and Joel Lehman and Jeff Clune and Kenneth O. Stanley , title =. CoRR , volume =. 2019 , url =. 1901.01753 , timestamp = on PithPith's reviewarXiv
  20. [20] Felipe Petroski Such and Aditya Rawal and Joel Lehman and Kenneth O. Stanley and Jeffrey Clune , title =. Proceedings of the 37th International Conference on Machine Learning,. 2020 , url = on Pith
show 24 more
  1. [21] Richard S. Sutton and Michael H. Bowling and Patrick M. Pilarski , title =. CoRR , volume =. 2022 , url =. doi:10.48550/ARXIV.2208.11173 , eprinttype =. 2208.11173 , timestamp = on PithPith's reviewarXivdoi:10.48550/arxiv.2208.11173
  2. [22] Yoshua Bengio and J. Curriculum learning , booktitle =. 2009 , url =. doi:10.1145/1553374.1553380 , timestamp = on Pithdoi:10.1145/1553374.1553380
  3. [23] Wojciech Zaremba and Ilya Sutskever , title =. CoRR , volume =. 2014 , url =. 1410.4615 , timestamp = on PithPith's reviewarXiv
  4. [24] Yuxin Wu and Yuandong Tian , title =. 5th International Conference on Learning Representations,. 2017 , url = on Pith
  5. [25] Jiahang Lin and Shichun Liu and Chengjun Pan and Lizhi Lin and Shihan Dou and Xuanjing Huang and Hang Yan and Zhenhua Han and Tao Gui , title =. CoRR , volume =. 2026 , url =. doi:10.48550/ARXIV.2604.25850 , eprinttype =. 2604.25850 , timestamp = on PithPith's reviewarXivdoi:10.48550/arxiv.2604.25850
  6. [26] Mengzhuo Chen and Junjie Wang and Zhe Liu and Yawen Wang and Qing Wang , title =. CoRR , volume =. 2026 , url =. doi:10.48550/ARXIV.2606.06324 , eprinttype =. 2606.06324 , timestamp = on PithPith's reviewarXivdoi:10.48550/arxiv.2606.06324
  7. [27] Hangfan Zhang and Shao Zhang and Kangcong Li and Chen Zhang and Yang Chen and Yiqun Zhang and Lei Bai and Shuyue Hu , title =. CoRR , volume =. 2026 , url =. doi:10.48550/ARXIV.2606.09498 , eprinttype =. 2606.09498 , timestamp = on PithPith's reviewarXivdoi:10.48550/arxiv.2606.09498
  8. [28] Biswa Sengupta and Jinhua Wang , title =. CoRR , volume =. 2026 , url =. doi:10.48550/ARXIV.2604.20938 , eprinttype =. 2604.20938 , timestamp = on PithPith's reviewarXivdoi:10.48550/arxiv.2604.20938
  9. [29] Hanghui Guo and Weijie Shi and Zhangze Chen and Shengxiang Xu and Yishu Wang and Yimei Zhang and Wangze Ni and Jia Zhu and Shimin Di , title =. CoRR , volume =. 2026 , url =. doi:10.48550/ARXIV.2607.26722 , eprinttype =. 2607.26722 , timestamp = on PithPith's reviewarXivdoi:10.48550/arxiv.2607.26722
  10. [30] Wenbo Pan and Shujie Liu and Chin. Evolving Agents in the Dark: Retrospective Harness Optimization via Self-Preference , journal =. 2026 , url =. doi:10.48550/ARXIV.2606.05922 , eprinttype =. 2606.05922 , timestamp = on PithPith's reviewarXivdoi:10.48550/arxiv.2606.05922
  11. [31] Luan Zhang and Ruochen Zhou and Dandan Song and Zhengyu Chen and Yuhang Tian and Jun Yang and Huipeng Ma and Chenhao Li and Guangyuan Feng and Xudong Li and Yizhou Jin and Yan Xu , title =. CoRR , volume =. 2026 , url =. doi:10.48550/ARXIV.2608.01918 , eprinttype =. 2608.01918... on PithPith's reviewarXivdoi:10.48550/arxiv.2608.01918
  12. [32] Yue Huang and Wenjie Wang and Han Bao and Yuchen Ma and Xiaonan Luo and Yi Nian and Haomin Zhuang and Zheyuan Liu and Yue Zhao and Zhaoxiang Zhang , title =. CoRR , volume =. 2026 , url =. doi:10.48550/ARXIV.2607.14159 , eprinttype =. 2607.14159 , timestamp = on PithPith's reviewarXivdoi:10.48550/arxiv.2607.14159
  13. [33] Prannay Hebbar and Yogendra Manawat and Samuel Verboomen and Alesia Ivanova and Selvam Palanimalai and Kunal Bhatia and Vignesh Baskaran , title =. CoRR , volume =. 2026 , url =. doi:10.48550/ARXIV.2605.27276 , eprinttype =. 2605.27276 , timestamp = on PithPith's reviewarXivdoi:10.48550/arxiv.2605.27276
  14. [34] Zhengyu Chen and Teng Xiao and Huaisheng Zhu and Yige Yuan and Luan Zhang and Jingang Wang , title =. CoRR , volume =. 2026 , url =. doi:10.48550/ARXIV.2607.22688 , eprinttype =. 2607.22688 , timestamp = on PithPith's reviewarXivdoi:10.48550/arxiv.2607.22688
  15. [35] Seth Karten and Joel Zhang and Tersoo Upaa Jr and Ruirong Feng and Wenzhe Li and Chengshuai Shi and Chi Jin and Kiran Vodrahalli , title =. CoRR , volume =. 2026 , url =. doi:10.48550/ARXIV.2605.09998 , eprinttype =. 2605.09998 , timestamp = on PithPith's reviewarXivdoi:10.48550/arxiv.2605.09998
  16. [36] Tianxin Wei and Zhan Shi and Minhua Lin and Bing He and Zewen Liu and Yisi Sang and Yuanchen Bei and Xuying Ning and Jiaru Zou and Ting. Evo-Harness: Context-to-Harness Skill Compilation for Self-Evolving Agents , journal =. 2026 , url =. doi:10.48550/ARXIV.2608.15071 , eprint... on PithPith's reviewarXivdoi:10.48550/arxiv.2608.15071
  17. [37] Jun Nie and Yonggang Zhang and Jun Song and Qianshu Cai and Dahai Yu and Yike Guo and Xinmei Tian and Bo Han , title =. CoRR , volume =. 2026 , url =. doi:10.48550/ARXIV.2607.08124 , eprinttype =. 2607.08124 , timestamp = on PithPith's reviewarXivdoi:10.48550/arxiv.2607.08124
  18. [38] Minqi Jiang and Edward Grefenstette and Tim Rockt. Prioritized Level Replay , booktitle =. 2021 , url = on Pith
  19. [39] Shidong Yang and Ziyu Ma and Tongwen Huang and Yiming Hu and Yong Wang and Xiangxiang Chu , editor =. CoEvolve: Training. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers),. 2026 , url =. doi:10.18653/V1/2026.ACL-LO... on Pithdoi:10.18653/v1/2026.acl-long.1055
  20. [40] Atsuyuki Miyai and Kiyoharu Aizawa and Toshihiko Yamasaki , title =. CoRR , volume =. 2026 , url =. doi:10.48550/ARXIV.2608.20169 , eprinttype =. 2608.20169 , timestamp = on PithPith's reviewarXivdoi:10.48550/arxiv.2608.20169
  21. [41] Jinghan Xu and Yikai Zhang and Aili Chen and Weiyuan Li and Jiaqing Liang and Deqing Yang , title =. CoRR , volume =. 2026 , url =. doi:10.48550/ARXIV.2608.27311 , eprinttype =. 2608.27311 , timestamp = on PithPith's reviewarXivdoi:10.48550/arxiv.2608.27311
  22. [42] GitHub Copilot SDK , author=. 2026 , howpublished= on Pith
  23. [43] WHALE: A Simple Recipe for Joint Harness-Weight Optimization , author=. arXiv preprint arXiv:2609.00196 , year= on PithPith's reviewarXiv
  24. [44] Terminus-KIRA: Boosting Frontier Model Performance on Terminal-Bench with Minimal Harness , author=. 2026 , howpublished= on Pith
Cited by 2 Pith papers

Challenges and discussion

Disagree with the review, or know something it missed? Leave a comment, or challenge this review formally: a challenge becomes a permanent entry in this paper's record, and its decision is published here.

No comments yet.

Sign in to comment