Pith.

Artificial Intelligence · arXiv:2609.09646 v2 · submitted 9 Sep 2026

RobustSGPO: Search-Space Control for Agent Harness Evolution

Zibo Zhao, Jijun Shi, Mo Zhou, Zhongyuan Wang, Shifu Bie, Yunfei Zhang, Xuanting Zhou, Xiangyu Wu

show 4 more authors

Bin Liu, Ruiming Tang, Wenwu Ou, Kun Gai

The paper says

“Semantic-gradient-based prompt optimization (SGPO) improves agent harnesses using execution feedback, but its local update rule leaves the choice of edit scope and operation unresolved.”

more of the abstract

We introduce RobustSGPO, which specifies the requested edit, constructs and checks the patch, and continues search from either the incumbent or retained snapshots. We evaluate permission scheduling, cumulative controls, and task-family transfer in the AgentX brainstorming workflow using 120 tasks, 95 runs, and 7,350 candidate attempts. Periodic 1 → 2 → 3 scheduling exceeds fixed maximum permission by 0.28 test-score points. RobustSGPO increases completion on 30 held-out tasks from 60.0% to 80.0% and improves test quality from 3.77 to 4.14 under a 20-million-token budget. Category retention reduces source-task degradation after a shift, whereas random retention reaches a higher destination endpoint. Search-space control benefits quality through executable edits and alternative starting points, with measurable retention overhead.

from the abstract

Pith says

Machine review · deepseek-v4-flash · 10 Sep 2026 · of version 1 of the paper; version 2 is newer · what was read was not recorded · not checked by a person

It can be wrong. Challenge this review or reply to it.

Holds up, with conditions. RobustSGPO proposes explicit search-space control for harness evolution, a sensible idea, but the headline gains lack statistical support and the baseline is the authors' own system. This line is wrong

4 major objections · 8 minor

How often objections from reviews like this one are right: not yet measured

Are you an author? What you can do.

Not checked: figures, code, what the references say.

The paper says, at Section 6.2–6.5, Table 3

The review did not quote this passage.

Pith says: Objection 1 of 4 · major

No confidence intervals, error bars, or statistical significance tests are reported for any of the headline point estimates. The completion gain from 18/30 to 24/30 (60% to 80%) under a two-sided binomial test yields p≈0.10, not significant at conventional thresholds.

more of objection 1

The quality difference of 0.37 points (3.77 to 4.14) and the 0.04-point gap between RobustSGPO and Structured Search at equal token cost (Section 6.5) could easily lie within per-seed or per-task variance. Without uncertainty quantification, the reader cannot assess whether these differences are systematic or noise. The authors should provide confidence intervals or standard deviations over seeds and, where applicable, effect-size estimates.

The paper says, at Section 6.5, Figure 7 and Table 3

The review did not quote this passage.

Pith says: Objection 2 of 4 · major

The paper claims 'the full method outperforms free search at equal token cost,' but the margin between RobustSGPO and the Structured Search baseline at 20 million tokens is only 0.04 points (4.14 vs 4.10). Given the absence of error bars, this difference is effectively indistinguishable from noise and does not support the claim of superiority over Structured Search. The conclusion should be tempered or the analysis should show that the difference is robust.

The paper says, at Section 6.3, Figure 5(b) and surrounding text

The review did not quote this passage.

Pith says: Objection 3 of 4 · major

Effective-category coverage is defined as the fraction of the fixed scope–operation vocabulary reached by at least one valid edit. However, the total size of this vocabulary is never stated, so the reported percentages (48%, 68%, etc.) are not interpretable. Without the denominator, the reader cannot evaluate whether coverage differences are meaningful. The authors should specify the vocabulary size and, ideally, the number of categories that are even feasible.

The paper says, at Section 6.4, E3 results (Figure 6 and text)

The review did not quote this passage.

Pith says: Objection 4 of 4 · major

The transfer experiment compares four conditions over only 10 rounds per phase. The claims about retention versus adaptation tradeoffs rest on small differences: source-task drops of 0.15 vs 0.31–0.33 points, and destination endpoints of 4.18 vs 3.98. With five seeds and no confidence intervals, it is unclear whether these differences are robust.

more of objection 4

Moreover, the random archive achieves the highest destination endpoint (4.18) while category retention shows the smallest source drop (0.15). Without error bars, the tradeoff narrative is speculative. The authors should provide per-seed trajectories or at least standard deviations.

8 minor objections

The paper says, at Section 6.1

Pith says: minor objection 1 of 8

The phrase 'five paired seeds' is ambiguous. Clarify whether seeds are paired across conditions (e.g., same seed for all configurations in a replicate block) or only within each condition. If paired, explain how the pairing is used in analysis.

The paper says, at Table 3

Pith says: minor objection 2 of 8

The column header 'Success 20M tokens' suggests a success rate, but the values are test quality scores (e.g., 3.77, 4.14). Rename the column to 'Test score at 20M tokens' or similar, and make the metric explicit.

show 6 more

The paper says, at Section 5.1, Eq. (3) and surrounding text

Pith says: minor objection 3 of 8

The admission threshold epsilon = 0.05 is introduced without justification. Authors should discuss how this value was chosen and whether results are sensitive to it (e.g., ablation with different epsilon values).

The paper says, at Section 5.3

Pith says: minor objection 4 of 8

The archive capacity is set to ten category slots. This choice is not motivated. Provide a sensitivity analysis or at least comment on how the capacity affects behavior.

The paper says, at General (reproducibility)

Pith says: minor objection 5 of 8

The paper does not include a high-level algorithm or pseudocode for RobustSGPO's edit selection, patch construction, or archive update rules. Adding an algorithmic description (even in the appendix) would significantly improve reproducibility.

The paper says, at Figure 1 and Figure 2 captions

Pith says: minor objection 6 of 8

These figures are reproduced from the AgentX technical report [4]. Ensure that the captions explicitly state 'Reproduced from [4]' rather than 'Prior-work foundation' to clarify attribution.

The paper says, at Section 5.2, line 184

Pith says: minor objection 7 of 8

The statement 'The current program does not automatically update caller instructions' is an important limitation that should be noted in the conclusion or future work section.

The paper says, at Table 1

Pith says: minor objection 8 of 8

The total attempts sum to 7,350, which is consistent with the abstract. This internal consistency is appreciated.

What would settle it

From the same 5 × 30 test-score matrix (configurations × seeds), compute bootstrapped 95% confidence intervals for completion rate and mean test quality for each configuration. Report the overlap. Additionally, perform a one-sided paired permutation test comparing RobustSGPO to SGPO on the 30 held-out tasks (average across seeds). If the p-value exceeds 0.05 or the intervals overlap substantially, the central claim is weakened.

from the skeptic's reading

How the review went

4 readings by deepseek-v4-flash · 10 Sep 2026

  1. Reader holds up, with conditions confidence moderate
  2. Skeptic agrees 25 s · 8,363 tokens read, 2,646 written
  3. Editor wrote the verdict above 16 s · 9,251 tokens read, 1,526 written
  4. Referee major revision: 4 major objections, 8 minor 26 s · 8,623 tokens read, 2,797 written

Not run on this paper: the simulated authors' defence.

Pith did not check this review's objections against the paper.

The review in full

The review in brief, machine-written

The paper introduces RobustSGPO, which makes edit scope, operation, and targets explicit in semantic-gradient-based harness optimization. The core design—edit planning, exact-target checking, and a MAP-Elites-style archive of harness snapshots—is cleanly motivated and clearly described. The authors run a systematic sweep (E1–E3) isolating each component's contribution, and the internal arithmetic checks out. That is real work, and the method section is detailed enough to reconstruct the algorithm.

The soft spots are proportional but real. The headline claim—24/30 vs 18/30 tasks completed—rests on five seeds with no error bars or significance test. With 30 held-out tasks, a shift of 6 successes is in binomial noise territory (p ≈ 0.10). The quality difference of 0.37 points on a 0–5 scale could live inside per-task variance. The baseline is the authors' own AgentX, and the evaluation uses one internal brainstorming workflow. No code or data are shipped. The paper itself admits these limits: the margin over Structured Search at equal token cost shrinks to 0.04, and the workflow is not externally ranked.

So the method is plausible and well-reasoned, but the evidence is weaker than the abstract suggests. This is a useful architecture note for researchers working on automated harness optimization. I would send it to peer review with a strong request for confidence intervals and, ideally, a reproducible artifact.

The strongest objection

The central claim that RobustSGPO increases held-out completion from 60.0% to 80.0% (18/30 to 24/30) and test quality from 3.77 to 4.14 rests on point estimates without reported uncertainty. The paper reports "five paired seeds" (Section 6.1) but Table 3 provides a single value per condition. With 30 held-out tasks, a shift of 6 tasks is not automatically significant: a two-sided binomial test of proportions (18/30 vs 24/30) yields p ≈ 0.10. The quality difference of 0.37 points on a 0–5 scale could be within per-task variance. Without error bars, the reader cannot distinguish systematic improvement from seed- or task-specific noise. This concern is amplified by the small margin between RobustSGPO and Structured Search at equal token cost (0.04 points) and the subjective nature of the test rubric. If the variance is large, the headline improvements may not replicate.

The reader's weakest assumption was that five paired seeds make point differences systematic rather than noise. I identify the same load-bearing concern: the absence of error bars or significance tests for the headline completion and quality gains. The paper is transparent about its limitations (single workflow, no external benchmarks) and the method is coherent, but the statistical grounding of the central claim is insufficient. A CONDITIONAL verdict is appropriate; the authors should supply confidence intervals or significance tests. If they do, the current verdict stands; if not, the claims would need to be downgraded. Thus verdict_should_be = UNCHANGED.

The referee's report

If the reported improvements are systematic, the paper demonstrates that explicit control over edit selection, patch construction, and retention can meaningfully improve prompt-optimization outcomes in multi-agent harness evolution. The contribution is primarily empirical and practical: it identifies which components of search-space control matter (periodic scheduling, predefined add/remove operations, category retention) and quantifies their overhead. The work is situated within a specific in-house workflow (AgentX brainstorming), with all tasks, rubrics, and baselines originating from the same group, which limits generality but still provides useful engineering insight for similar industrial settings.

Pith's reading of the paper (a machine's summary, not the authors')

The paper tries to establish that how an agent-harness optimizer is allowed to edit its own specification matters as much as the optimizer's learning rule. In the AgentX workflow, semantic-gradient prompt optimization (SGPO) improves agent prompts by replaying candidate edits against the current version and keeping only those that score better, but it never decides in advance which agents to touch, what operation to perform, or where to continue search from. RobustSGPO makes those decisions explicit: a controller picks the scope, operation, and targets before content is generated; a checker rejects edits that touch the wrong objects while predefined code constructs agent additions and removals; and an archive of the best snapshot per edit category offers alternative starting points. The paper reports that these controls raise completion on 30 held-out tasks from 60.0% to 80.0% and test quality from 3.77 to 4.14 within a fixed 20-million-token budget, and that periodically widening the permission level beats always running with the widest permission. If true, the result locates the gain in search-space discipline rather than in larger permissions or a stronger optimizer.

What carries the argument. The machinery is a factored edit-selection rule plus three enforcement and memory components. Before generation, the controller draws a feasible category $c=(s,o)$—the set of agents to change and the operation to perform—and eligible targets, so candidates are sampled from a distribution summed over scope–operation–target combinations rather than free-form refinement. An exact-target check rejects any realized edit that changes objects outside the request; a predefined patch constructor, the Typed Compiler, writes and deletes the files for agent additions and removals so structural edits pass validity checks without relying on the model to emit executable code; and a MAP-Elites-style archive stores the best valid snapshot per scope–operation category, seeded by a reservoir of at most ten slots, with search alternating between incumbent and archived parents and every descendant required to beat the incumbent for admission. These components jointly implement the paper's three controls: choose the edit, construct and check the patch, and continue from a retained version.

Core claim. On the paper's own terms, the central claim is that free-form semantic-gradient updates underuse the permitted edit space, and explicit control of edit selection, patch execution, and search starting points repairs that. RobustSGPO factors the update distribution into a scope–operation–target choice $$q(\$\Delta$ h \mid h, g)= extstyle{\sum_{s,o,T}} p(s,o mid h)\, p(T mid s,o,h)\, q_{\mathrm{LLM}}(\$\Delta$ h mid h,g,s,o,T)$$ made before generation, enforces the requested edit with an exact-target check, constructs agent additions and removals with predefined code, and archives the highest-validation snapshot per scope–operation category so search alternates between incumbent and retained parents. Given five paired seeds on 120 tasks in the AgentX brainstorming workflow, the paper reports that the full method completes 24 of 30 held-out tasks versus 18 of 30 for the SGPO baseline, improves test quality from 3.77 to 4.14 at equal 20-million-token cost, and that periodic $1\to2\to3$ permission scheduling finishes 0.28 test-score points above fixed maximum permission. After a task shift, category retention loses only 0.15 points on source tasks versus 0.31–0.33 for other configurations, while the random-retention archive reaches a higher destination endpoint (4.18).

The premise it rests on. The comparison rests on five paired seeds per condition with no error bars or significance tests, and on the premise that admitting candidates by a mean validation gain above $\epsilon = 0.05$ predicts quality on held-out tasks; if either premise gives way, the reported gains could be seed-specific or task-specific artifacts rather than properties of RobustSGPO.

If the paper is right (Pith's inferences, not the authors')

Structured edits become reliable search steps: stating a requested edit and building structural patches with predefined code raises the validity rate of structural proposals from 48.9% (Constrained Search) to 77.8% (Structured Search), and the full method reaches 68% final coverage of the edit-category vocabulary versus 48% for the baseline.

Budget-constrained users gain more from controlled search than from free-form rewriting: within 20 million charged tokens RobustSGPO scores 4.14 against 3.77 for SGPO, and still beats Structured Search (4.10) despite archive overhead.

Permission schedules matter on their own: a periodic narrow-to-wide schedule ($1\to2\to3$) finishes 0.28 points above always-wide permission, which suggests that when an edit is allowed shapes final quality alongside what is allowed.

Retention policy is a real tradeoff control: category-wise retention minimizes source-task loss (0.15 points) after a task shift, while random retention maximizes the destination endpoint (4.18) at the cost of more source degradation.

The gain decomposes cumulatively: round-30 scores rise monotonically from 3.82 (SGPO) to 3.94, 4.04, 4.20, and 4.30 as each control is added, so each of the three controls contributes something at the margin.

A cheap transfer to test: applying the same 'change exactly those named objects' enforcement to instruction-only optimizers that rewrite single prompts, without structural patches and without an archive, may capture part of the gain; the paper's cumulative configurations do not isolate that component alone.

Because coverage counts only whether a category was reached by a valid edit, filling rarely visited categories with weak snapshots could raise coverage without raising held-out quality; a per-category quality analysis would separate exploration from optimization, and the paper does not report one.

The 0.28-point scheduling effect, if it generalizes, implies permission schedules are a tunable hyperparameter comparable to step size, and one could optimize the schedule itself rather than fixing 1-to-2-to-3 by hand.

A testable extension: since the archive stores snapshots per scope–operation category but the paper reports only aggregate transfer curves, tracking which categories' snapshots are actually reused after the task shift would show whether transfer gains come from structural categories (add and remove) or from instruction categories.

Check this review

Pith stored no record of this review's passes and took no fingerprint of it. Reviews made by code from 13 Nov 2026 on carry both.

How often reviews like this one are right

Reviews like this one: written by deepseek-v4-flash, where what was read was not recorded.

How often objections from reviews like this one are right: not yet measured

Every value above, in the review's public record (JSON)

Reviews by people

This review was written by a machine. No person has reviewed this paper on Pith yet. A scientist listed on Pith can review it for a fee, and their review is published here whatever it concludes.

Request a human review

How Pith checks papers · The full audit of this paper

The paper

Abstract

Semantic-gradient-based prompt optimization (SGPO) improves agent harnesses using execution feedback, but its local update rule leaves the choice of edit scope and operation unresolved. We introduce RobustSGPO, which specifies the requested edit, constructs and checks the patch, and continues search from either the incumbent or retained snapshots. We evaluate permission scheduling, cumulative controls, and task-family transfer in the AgentX brainstorming workflow using 120 tasks, 95 runs, and 7,350 candidate attempts. Periodic 1 → 2 → 3 scheduling exceeds fixed maximum permission by 0.28 test-score points. RobustSGPO increases completion on 30 held-out tasks from 60.0% to 80.0% and improves test quality from 3.77 to 4.14 under a 20-million-token budget. Category retention reduces source-task degradation after a shift, whereas random retention reaches a higher destination endpoint. Search-space control benefits quality through executable edits and alternative starting points, with measurable retention overhead.

The paper's own sentences, arranged, kept beside the review and separate from it. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. No sentence in this record was written by a model.

abstract compiled · 2026-09-23

The authors' main claim, as the classifier picked it out 0.93

“Search-space control benefits quality through executable edits and alternative starting points, with measurable retention overhead.”

How the main claim is stated: asserted 1.00

What the abstract itself states, as probabilities: claims first 0.05 · states limitation 0.89 · human participants 0.08 · controlled comparison 0.82 · randomized assignment 0.06 · states quantitative result 0.99 · compares with prior results 0.82 · mentions code or data availability 0.03

The abstract, sentence by sentence

This table labels the abstract only (arXiv listing abstract v1). The number is the probability the classifier gave the label.

SentenceRoleThe authors' own workStates the main conclusionStrength
Semantic-gradient-based prompt optimization (SGPO) improves agent harnesses using execution feedback, but its local update rule leaves the choice of edit scope and operation unresolved. gap 0.65 0.09 0.04 asserted 1.00
We introduce RobustSGPO, which specifies the requested edit, constructs and checks the patch, and continues search from either the incumbent or retained snapshots. method 0.57 0.90 0.12 asserted 0.99
We evaluate permission scheduling, cumulative controls, and task-family transfer in the AgentX brainstorming workflow using 120 tasks, 95 runs, and 7,350 candidate attempts. method 0.91 0.96 0.04 asserted 0.97
Periodic $1\to2\to3$ scheduling exceeds fixed maximum permission by 0.28 test-score points. result 1.00 0.92 0.07 asserted 1.00
RobustSGPO increases completion on 30 held-out tasks from 60.0% to 80.0% and improves test quality from 3.77 to 4.14 under a 20-million-token budget. result 1.00 0.96 0.42 asserted 1.00
Category retention reduces source-task degradation after a shift, whereas random retention reaches a higher destination endpoint. result 0.98 0.91 0.09 asserted 1.00
Search-space control benefits quality through executable edits and alternative starting points, with measurable retention overhead. interpretation 0.98 0.85 0.75 asserted 1.00

What kind of paper the classifier reports this is: method or tool 1.00 · evidence from observation or experiment 0.96

What was examined, and what was not

Sources read

  • arXiv listing abstract v1

Not examined

  • full text
  • figures and tables
  • references and citing sentences
  • relations to other papers

Levels run

  • P abstract

Not run

  • S sections
  • T body sentences
  • C citations
  • R cross-paper relations

How the labels were made

Compiled by classifier-1.13.0 via TypeSafe API, question bank abstract-p1-20260923, 2026-09-23: 1 request, 5,614 input tokens.

Every quoted sentence on this record is the authors'. Every label is from Pith's closed vocabulary, with the probability shown beside it. This record as JSON

Figures (7)
Figure 1 of the paper
Figure 1. AgentX changes industrial recommendation iteration from a manually handed-off pipeline into an agent-driven closed loop. Brain￾storming, development, and evaluation agents consume online feedback and trajectory data, while harness evolution improves each agent over time. Prior-work foundation, reproduced from the AgentX technical report [4]. 3.2 Trace-to-Update Loop The report’s SGPO-I updates one target agent 𝑖, keeping the model, tools, orchestration, and other agents fixed [4]. It sam￾ples traces T, derives rubrics R and standalone replay tasks, and computes a natural-language…
Figure 2 of the paper
Figure 2. Prior-work SGPO-I loop: trace sampling and task construction, semantic-gradient refinement, and paired replay with accept/rollback. Reproduced from the AgentX technical report [4]. 4 The Search-Space Problem The nominal space Snom contains permitted edits; the realized space Sreal contains actually generated edits. Validity checks select the effective space: Seff = {Δ𝐻 ∈ Sreal : 𝐶path ∧ 𝐶semantic ∧ 𝐶budget ∧ 𝐶safe}. (4) Effective means valid, not admitted by replay. Broader per￾missions need not yield more valid edits or higher quality. We use three levels: 𝛼1 edits one designated…
Figure 3 of the paper
Figure 3. RobustSGPO extends
Figure 4 of the paper
Figure 4. E1 test-score trajectories on a 0–5 scale. Periodic 1 → 2 → 3 has the highest final score; early and final rankings differ. 6.3 E2: Cumulative Search-Space Controls
Figure 5 of the paper
Figure 5. E2 search behavior: (a) held-out quality, (b) cumulative effective-category coverage, (c) cumulative structural validity with valid/attempted counts, and (d) occupied archive categories and cumulative admitted archive-origin updates. 6.4 E3: Adaptation and Retention After a Task Shift We evaluate local-to-cross-agent failures and the reverse. Each run spends ten rounds on source tasks and ten on des￾tination tasks, using the respective optimization and valida￾tion splits. Retained snapshots are rescored on the destination validation set before reuse. We compare SGPO, Structured…
Figure 6 of the paper
Figure 6. E3 adaptation and retention after the round-ten task shift. Dashed curves use random retention; category retention reduces source￾task loss, while random retention reaches a higher destination endpoint. 5
Figure 7 of the paper
Figure 7. Test quality versus cumulative charged tokens, including all search-side model calls. Dots mark the last affordable checkpoints within 20 million tokens. Per-round costs in

Figures are the authors', reproduced from the paper.

Cited by 1 Pith paper

Challenges and discussion

Disagree with the review, or know something it missed? Leave a comment, or challenge this review formally: a challenge becomes a permanent entry in this paper's record, and its decision is published here.

No comments yet.

Sign in to comment