Artificial Intelligence · arXiv:2609.38143 v2 · submitted 29 Sep 2026
Learning Meta-Skills for Agent Harness Design in Test-Time AI4AI
The paper says
“Agent performance depends on both reasoning ability and the environment in which it acts. We study test-time AI-for-AI, asking how a Builder can learn to construct better execution environments for a Target while both models' weights remain fixed.”
more of the abstract
To make the Builder's experience reusable, we introduce Meta-Skill: principles specifying when support is needed and what resources to provide. The Builder learns these principles from Target's execution feedback on the development set, then uses the frozen skill bank to construct harnesses for unseen tasks. Across Harness-Bench and NewtonBench, full-bank meta-skills improve macro-average performance by 8.95 percentage points over no-skill construction, and 12.02 points over direct delivery of the same bank to the Target. These results highlight the value of translating experience into executable support. Gains when the same model serves both roles further suggest a path to system level self-improvement through learning to build better environments.
from the abstract
Pith says
Machine review · deepseek-v4-flash · 30 Sep 2026 · of version 1 of the paper; version 2 is newer · what was read was not recorded · not checked by a person
It can be wrong. Challenge this review or reply to it.
Holds up, with conditions. The central claim survives scrutiny but the 8.95-point headline is NewtonBench-driven; the controls are unusually clean and the paper deserves a serious referee, conditional on matching code release and run-level variance. This line is wrong
3 major objections · 3 minor
How often objections from reviews like this one are right: not yet measured
Not checked: figures, code, what the references say.
The paper says, at Table 3 / §4.2
The review did not quote this passage.
Pith says: Objection 1 of 3 · major
The four headline comparisons (full-bank meta-skills vs no-skill Builder, and vs direct delivery) are reported without uncertainty intervals, although the paper already uses paired task-bootstrap CIs elsewhere (Fig 3(b), Table 4, Appendix B.3). Per-cell full-bank vs no-skill gains in Table 3 range from +1.13 (Qwen, Harness-Bench) and +5.20 (GPT-OSS, Harness-Bench) to +12.68–+13.36 on NewtonBench, so the 8.95-point aggregate (b4, b106) …
more of objection 1
… is largely driven by NewtonBench, and several Harness-Bench gains are likely consistent with zero at the interval widths seen in the paper's own transfer analyses. Please add paired task-bootstrap CIs for the main results, or state explicitly that the headline gain is benchmark-dependent, so the central quantitative claim becomes assessable.
The paper says, at §5.2 / Fig 3(a)
The review did not quote this passage.
A classifier read this sentence of the paper as acknowledging the concern (probability 0.61):
“Our evidence spans one main Builder, two benchmarks, and one adopted execution per task and condition.”
Pith says: Objection 2 of 3 · major
The same-model result of an 18.71-point average gain (abstract, b35; §5.2, b134) is computed over three settings whose individual gains range from +8.9 to +25.3, with no confidence intervals and no run-level variance reported. The paper's limitations section acknowledges one main Builder, two benchmarks, and one adopted execution per task (b195), but that qualifier appears only in §6; the result is asserted without uncertainty in the abstract and §5.2.
more of objection 2
Given that this claim motivates the 'path to system-level self-improvement' framing, please attach per-setting uncertainty estimates or explicitly qualify the claim as preliminary.
The paper says, at §3.1 Eq. (2) / Table 2
The review did not quote this passage.
Pith says: Objection 3 of 3 · major
The skill bank is learned in a single run from small development sets (11 Harness-Bench and 32 NewtonBench tasks, Table 2), with all keep/revise/add decisions made by one Builder model (GPT-5.6-Sol) from one batch of Target executions per task, and with no independent verification of those decisions.
more of objection 3
Section 5.1 shows this reflection is imperfect — Qwen drops 4.06 points on later passes as revisions become overly specific (b128). The paper defers partition- and run-level variance to future work (Appendix A.3), yet presents the 8.95-point claim as a point estimate without sensitivity analysis. At minimum, a check across alternative development partitions would establish that the claim is not an artifact of the specific 10% split.
3 minor objections
The paper says, at §1
Pith says: minor objection 1 of 3
Typo: 'each meta-skill shuold specify' should read 'should specify'.
The paper says, at Table 3
Pith says: minor objection 2 of 3
In the 'Mined Free-form Target skills, retrieved' row, the Harness-Bench and NewtonBench numbers run together without separators (e.g., '63.5171.1246.92'), making the row unreadable; the entries need to be properly separated.
show 1 more
The paper says, at Fig 3
Pith says: minor objection 3 of 3
The subfigure labels are confusing: panel (b) 'Cross-Builder transfer' internally contains panels labeled (a) and (b), and panel (a)'s bars are referenced in the text only as 'the three settings in Figure 3(a)'. Renaming the inner panels would improve readability.
What would settle it
Recompute the macro-average gain after removing Harness-Bench entirely, reporting only NewtonBench; if the NewtonBench-only gain is, say, >5 points while the Harness-Bench-only gain is near zero, the pooled 8.95 claim does not support cross-benchmark generality. Also run four independent seeds of the full pipeline (GPT-5.6-Sol Builder, Gemini-3.6-Flash Target, Harness-Bench) to obtain run-level bootstrap intervals on the no-skill versus full-bank difference and on the same-model 18.71-point figure.
from the skeptic's reading
How the review went
4 readings by deepseek-v4-flash · 30 Sep 2026
- Reader holds up, with conditions
- Skeptic partly agrees
- Editor wrote the verdict above
- Referee major revision: 3 major objections, 3 minor
Not run on this paper: the simulated authors' defence.
Pith did not check this review's objections against the paper.
The review in full
The review in brief, machine-written
The one thing you should know: this is a careful, unusually well-controlled paper for the harness-design line, and the central claim largely holds — but the 8.95-point headline is narrower than the abstract implies. It is driven mostly by NewtonBench. On Harness-Bench, two of three Targets gain only +1.13 (Qwen) and +5.20 (GPT-OSS) over no-skill construction, and the paper's own task-bootstrap intervals for those settings are wide. The paper is transparent — Table 3 shows every number — but the abstract presents the macro-average as the general result, and that framing overstates the breadth. The 18.71-point same-model claim is the same story: no run-level variance at all.
What is genuinely new: the when/provide/use meta-skill representation, and the choice to teach the Builder rather than the Target. The controls are the best part. No-skill Builder isolates construction capability, direct delivery isolates semantic knowledge, and all six settings improve with the full bank. That internal consistency is real evidence, not a single point estimate. The cross-Builder and same-model extensions are useful, though exploratory — the authors say so themselves, and they report the transfer confidence intervals honestly, including the ones that contain zero.
Soft spots, in proportion. First, a concrete contradiction: the title-page footnote says the code is released at a GitHub URL, while Appendix A.2 says full code and prompts will be released 'upon paper acceptance.' Those cannot both be true, and nobody can reproduce the headline numbers today either way. Second, the mechanism's weakest premise — that the Builder's self-directed keep/revise/add reflection is competent — is partially contradicted by the paper's own refinement analysis: Qwen drops 4.06 points when revisions become overly specific to recent evidence. The authors report this openly and build a selective-retention lesson around it, but it is the spot a referee should push hardest on: the whole pipeline's value depends on an unverified introspection step the paper shows is imperfect. Third, the headline numbers rest on a single main Builder and one adopted execution per task-condition; the limitations sentence acknowledges this, so it is an abstract framing problem, not concealment.
The citation pattern looks sound — Meta-Harness, Evo-Harness, and Meta Context Engineering are all cited, and the distinction from Meta-Harness (searching harness implementations versus learning reusable principles) is real.
Bottom line: send it to peer review. It deserves a serious referee. The fixes are deliverable — align the code-release claim with reality, add run-level variance to the headline numbers, and restate the abstract so the per-benchmark imbalance is visible. I would cite the NewtonBench result and the when/provide/use representation.
The strongest objection
The paper's central claim (b4, b106) is that full-bank meta-skills improve macro-average performance by 8.95 points over no-skill construction across Harness-Bench and NewtonBench. Table 3 shows this is not uniform: on Harness-Bench the per-Target gains are +14.52 (Gemini), +1.13 (Qwen), and +5.20 (GPT-OSS); on NewtonBench they are +12.68, +13.36, +6.84. The pooled 8.95 macro-average is therefore driven by NewtonBench, where the paper itself reports the largest gains, while two of three Harness-Bench gains are small or within the unadjusted 95% task-bootstrap intervals (b146, b167). The same-model setting (b35, b134) claims an 18.71-point average gain, but no run-level variance or confidence interval is reported for this figure; the paper's own limitations (b195) concede one main Builder, two benchmarks, and one adopted execution per task. The refinement analysis (b128) shows Qwen drops 4.06 points on later passes, so gains are not monotonic. If the headline gain is presented as general, the per-benchmark imbalance and single-run same-model results leave the breadth of the claim under-supported.
The reader's CONDITIONAL verdict is appropriate given the paper's acknowledged single-run evidence and its disclosed intervals that include zero. The reader's weakest_assumption focused on the Builder's self-directed keep/revise/add reflection being competent and generalizable, which is a real mechanism-level concern but is partially acknowledged by the paper (b128). My stress-test identifies a more load-bearing, underappreciated issue: the headline 8.95-point gain is pooled over six settings, yet Harness-Bench gains for Qwen and GPT-OSS are small or within the reported CIs, so the aggregate depends heavily on NewtonBench. The same-model 18.71-point claim has no reported task- or run-level variance, and the paper concedes one adopted execution per task-condition (b195). A benchmark-exclusion (remove Harness-Bench) plus independent-seeds test would settle whether the claim generalizes or should be explicitly scoped to NewtonBench and to the single evaluated execution. The paper's controls are clean and the science is careful, but the headline breadth should be tightened until run-level intervals are reported.
The referee's report
If the claims hold, the paper establishes learned harness design as a transferable capability: a pinned-weight Builder converts its own construction outcomes into reusable support principles that improve a Target's end-to-end performance, including for itself. The experimental design is a genuine strength: the no-skill Builder isolates construction capability, direct delivery of the same bank isolates semantic content, and full-bank meta-skills beat both in all six settings; the learning/deployment protocol is external (bank frozen before testing, held-out evaluation), keeping circularity low; benchmarks, budgets, and scoring rules are fixed; and the paper ships paired task-bootstrap CIs for its transfer and ablation analyses. The main weakness is that the headline quantitative claims are not accompanied by the same uncertainty quantification, the aggregate 8.95-point figure is driven heavily by NewtonBench with two near-zero Harness-Bench cells, and the mechanism rests on a single Builder's self-directed reflection over small development sets.
Pith's reading of the paper (a machine's summary, not the authors')
This paper tries to establish that an AI agent can become better at a task without any change to its own weights, by learning how to build the execution environment around the model that actually solves the task. It introduces meta-skills: short principles stating when a solver needs support, what resource to supply, and how to use it, which a Builder model learns from the solver's execution feedback on development tasks and then freezes to guide harness construction for unseen tasks. The central claim is that enacting a lesson as executable support — tools, memory, control logic, verification gates — is worth more than delivering the same advice as text: across two benchmarks, full-bank meta-skills improve macro-average performance by 8.95 percentage points over skill-free harness construction and 12.02 points over giving the identical bank directly to the solver. When the same model plays both Builder and solver, the measured gain is larger still, which the authors read as evidence that learning to build better support is a distinct route to system-level self-improvement.
What carries the argument. The central object is the meta-skill, a $(when, provide, use)$ triple that states the observable condition calling for support, the capability or resource to supply, and how the solver should apply that support while retaining responsibility for judgment. The argument runs on a construction–execution–reflection loop, which the paper formalizes as $H_x^j = B(H_0, x, S_j)$, $e_x^j = T(x; H_x^j)$, and $S_{j+1} = \mathrm{Revise}_B\big(S_j, \{(H_x^j, F(e_x^j))\}\big)$: the Builder builds a harness from the neutral environment, shared component interfaces, and its current bank; the Target executes and leaves a public trace and score; and the same Builder reviews that feedback to keep, revise, or add at most one skill per task over two passes before the bank is frozen. At test time, the Builder compiles the chosen skills into seven editable component families — instructions, memory, context organization, composed tools, execution control, verification and recovery, and workspace preparation — and this compilation step is the mechanism that turns declarative advice into executable support.
Core claim. The core discovery is that translating learned guidance into executable environment components carries most of the value. With the semantic knowledge held fixed — the same frozen meta-skill bank is given either to the Builder, which implements it as harness components, or directly to the solver — the Builder-implemented version wins in all six model-benchmark settings, by 12.02 points on average and up to 25.43 points on a single pair. The authors propose that meta-skills function as a compact language for provisioning task-specific support: a lesson such as 'revalidate an artifact after it is edited' becomes a version-tracking memory store and an audit gate that controls submission, offloading mechanical bookkeeping from the solver so it can spend its limited interaction budget on judgment. Starting from an empty bank, two development passes of build-and-reflect produce the frozen skill set, and the second pass contributes most of the gain on the scientific-discovery benchmark, while on the workflow benchmark over-refinement can regress one solver by 4.06 points. The mechanism is therefore not better reasoning but better provisioning, and this holds even when no stronger teacher is available.
The premise it rests on. The mechanism rests on the Builder's self-directed reflection being competent and generalizable: one model, from a single batch of executions per pass, decides with no independent verification which skills to keep, revise, or add, and the paper's own analysis shows revisions can become overfit to recent evidence, with one solver losing 4.06 points.
If the paper is right (Pith's inferences, not the authors')
Harness design becomes its own learning target: a fixed-weight agent can improve a second agent, or itself, by learning what support to build rather than how to reason.
Because the same advice is far more effective when implemented as tools and control than delivered as text, the bottleneck in raising agent performance may often be provisioning, not knowledge.
Same-model experiments, with average gains of 18.71 points over skill-free construction, identify environment-building as a route to self-improvement without weight updates or a stronger teacher.
Skill banks can transfer across Builders and Targets, although the benefit depends on how the receiving Builder implements shared guidance; recipient-side learning aligns skills better, with a model's own bank beating an imported one by 3.69 to 9.93 points.
The 192-token skill format and two-pass minimal-supervision loop make reflection cheap enough to run per domain; a natural test the paper explicitly leaves open is whether banks generalize across datasets, not just across held-out tasks within a benchmark.
The consistent superiority of full-bank access over lexical retrieval suggests the bank behaves like a small parameter-free library whose components interact; a testable extension is retrieval that scores skill complementarity rather than lexical match.
The result reframes agent-skill research as a design-space problem: teaching a scaffold-builder may complement teaching a task-solver, and the two could be co-evolved, a direction the paper names but does not pursue.
Check this review
Pith stored no record of this review's passes and took no fingerprint of it. Reviews made by code from 13 Nov 2026 on carry both.
How often reviews like this one are right
Reviews like this one: written by deepseek-v4-flash, where what was read was not recorded.
How often objections from reviews like this one are right: not yet measured
Reviews by people
This review was written by a machine. No person has reviewed this paper on Pith yet. A scientist listed on Pith can review it for a fee, and their review is published here whatever it concludes.
The paper
Abstract
Agent performance depends on both reasoning ability and the environment in which it acts. We study test-time AI-for-AI, asking how a Builder can learn to construct better execution environments for a Target while both models' weights remain fixed. To make the Builder's experience reusable, we introduce Meta-Skill: principles specifying when support is needed and what resources to provide. The Builder learns these principles from Target's execution feedback on the development set, then uses the frozen skill bank to construct harnesses for unseen tasks. Across Harness-Bench and NewtonBench, full-bank meta-skills improve macro-average performance by 8.95 percentage points over no-skill construction, and 12.02 points over direct delivery of the same bank to the Target. These results highlight the value of translating experience into executable support. Gains when the same model serves both roles further suggest a path to system level self-improvement through learning to build better environments.
The paper's own sentences, arranged, kept beside the review and separate from it. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. No sentence in this record was written by a model.
abstract compiled · 2026-09-30
The authors' main claim, as the classifier picked it out 0.81
“These results highlight the value of translating experience into executable support.”
How the main claim is stated: asserted 0.93
What the abstract itself states, as probabilities: claims first 0.04 · states limitation 0.17 · human participants 0.03 · controlled comparison 0.96 · randomized assignment 0.02 · states quantitative result 0.99 · compares with prior results 0.94 · mentions code or data availability 0.03
The abstract, sentence by sentence
This table labels the abstract only (arXiv listing abstract v1). The number is the probability the classifier gave the label.
| Sentence | Role | The authors' own work | States the main conclusion | Strength |
|---|---|---|---|---|
| Agent performance depends on both reasoning ability and the environment in which it acts. | background 1.00 | 0.08 | 0.03 | asserted 1.00 |
| We study test-time AI-for-AI, asking how a Builder can learn to construct better execution environments for a Target while both models' weights remain fixed. | contribution 0.89 | 0.48 | 0.04 | asserted 0.53 |
| To make the Builder's experience reusable, we introduce Meta-Skill: principles specifying when support is needed and what resources to provide. | contribution 0.65 | 0.90 | 0.07 | asserted 1.00 |
| The Builder learns these principles from Target's execution feedback on the development set, then uses the frozen skill bank to construct harnesses for unseen tasks. | method 0.99 | 0.91 | 0.09 | asserted 1.00 |
| Across Harness-Bench and NewtonBench, full-bank meta-skills improve macro-average performance by 8.95 percentage points over no-skill construction, and 12.02 points over direct delivery of the same bank to the Target. | result 1.00 | 0.95 | 0.39 | asserted 1.00 |
| These results highlight the value of translating experience into executable support. | interpretation 1.00 | 0.59 | 0.49 | asserted 0.93 |
| Gains when the same model serves both roles further suggest a path to system level self-improvement through learning to build better environments. | interpretation 0.99 | 0.81 | 0.27 | hedged possibility 0.99 |
What kind of paper the classifier reports this is: method or tool 0.99 · evidence from observation or experiment 0.95
What was examined, and what was not
Sources read
- arXiv listing abstract v1
Not examined
- full text
- figures and tables
- references and citing sentences
- relations to other papers
Levels run
- P abstract
Not run
- S sections
- T body sentences
- C citations
- R cross-paper relations
How the labels were made
Compiled by classifier-1.13.0 via TypeSafe API, question bank abstract-p1-20260923, 2026-09-30: 1 request, 5,531 input tokens.
Every quoted sentence on this record is the authors'. Every label is from Pith's closed vocabulary, with the probability shown beside it. This record as JSON
Cited by 1 Pith paper
Reviewed papers on Pith that cite this one. The full citation record
- Flex-RSI: Benchmarking Recursive Self-Improvement of Frontier Agents cs.AI · Nov 2026
Challenges and discussion
Disagree with the review, or know something it missed? Leave a comment, or challenge this review formally: a challenge becomes a permanent entry in this paper's record, and its decision is published here.
Come back by email
A short list of the week's newly reviewed papers. Nothing is sent until you confirm from your inbox, and every email has a link that stops it. No account.
No comments yet.
Sign in to comment