Artificial Intelligence · arXiv:2609.09646 v2 · submitted 9 Sep 2026
RobustSGPO: Search-Space Control for Agent Harness Evolution
The paper says
“Semantic-gradient-based prompt optimization (SGPO) improves agent harnesses using execution feedback, but its local update rule leaves the choice of edit scope and operation unresolved.”
more of the abstract
We introduce RobustSGPO, which specifies the requested edit, constructs and checks the patch, and continues search from either the incumbent or retained snapshots. We evaluate permission scheduling, cumulative controls, and task-family transfer in the AgentX brainstorming workflow using 120 tasks, 95 runs, and 7,350 candidate attempts. Periodic 1 → 2 → 3 scheduling exceeds fixed maximum permission by 0.28 test-score points. RobustSGPO increases completion on 30 held-out tasks from 60.0% to 80.0% and improves test quality from 3.77 to 4.14 under a 20-million-token budget. Category retention reduces source-task degradation after a shift, whereas random retention reaches a higher destination endpoint. Search-space control benefits quality through executable edits and alternative starting points, with measurable retention overhead.
from the abstract
Pith says
Machine review · deepseek-v4-flash · 10 Sep 2026 · of version 1 of the paper; version 2 is newer · what was read was not recorded · not checked by a person
It can be wrong. Challenge this review or reply to it.
Holds up, with conditions. RobustSGPO proposes explicit search-space control for harness evolution, a sensible idea, but the headline gains lack statistical support and the baseline is the authors' own system. This line is wrong
4 major objections · 8 minor
How often objections from reviews like this one are right: not yet measured
Not checked: figures, code, what the references say.
The paper says, at Section 6.2–6.5, Table 3
The review did not quote this passage.
Pith says: Objection 1 of 4 · major
No confidence intervals, error bars, or statistical significance tests are reported for any of the headline point estimates. The completion gain from 18/30 to 24/30 (60% to 80%) under a two-sided binomial test yields p≈0.10, not significant at conventional thresholds.
more of objection 1
The quality difference of 0.37 points (3.77 to 4.14) and the 0.04-point gap between RobustSGPO and Structured Search at equal token cost (Section 6.5) could easily lie within per-seed or per-task variance. Without uncertainty quantification, the reader cannot assess whether these differences are systematic or noise. The authors should provide confidence intervals or standard deviations over seeds and, where applicable, effect-size estimates.
The paper says, at Section 6.5, Figure 7 and Table 3
The review did not quote this passage.
Pith says: Objection 2 of 4 · major
The paper claims 'the full method outperforms free search at equal token cost,' but the margin between RobustSGPO and the Structured Search baseline at 20 million tokens is only 0.04 points (4.14 vs 4.10). Given the absence of error bars, this difference is effectively indistinguishable from noise and does not support the claim of superiority over Structured Search. The conclusion should be tempered or the analysis should show that the difference is robust.
The paper says, at Section 6.3, Figure 5(b) and surrounding text
The review did not quote this passage.
Pith says: Objection 3 of 4 · major
Effective-category coverage is defined as the fraction of the fixed scope–operation vocabulary reached by at least one valid edit. However, the total size of this vocabulary is never stated, so the reported percentages (48%, 68%, etc.) are not interpretable. Without the denominator, the reader cannot evaluate whether coverage differences are meaningful. The authors should specify the vocabulary size and, ideally, the number of categories that are even feasible.
The paper says, at Section 6.4, E3 results (Figure 6 and text)
The review did not quote this passage.
Pith says: Objection 4 of 4 · major
The transfer experiment compares four conditions over only 10 rounds per phase. The claims about retention versus adaptation tradeoffs rest on small differences: source-task drops of 0.15 vs 0.31–0.33 points, and destination endpoints of 4.18 vs 3.98. With five seeds and no confidence intervals, it is unclear whether these differences are robust.
more of objection 4
Moreover, the random archive achieves the highest destination endpoint (4.18) while category retention shows the smallest source drop (0.15). Without error bars, the tradeoff narrative is speculative. The authors should provide per-seed trajectories or at least standard deviations.
8 minor objections
The paper says, at Section 6.1
Pith says: minor objection 1 of 8
The phrase 'five paired seeds' is ambiguous. Clarify whether seeds are paired across conditions (e.g., same seed for all configurations in a replicate block) or only within each condition. If paired, explain how the pairing is used in analysis.
The paper says, at Table 3
Pith says: minor objection 2 of 8
The column header 'Success 20M tokens' suggests a success rate, but the values are test quality scores (e.g., 3.77, 4.14). Rename the column to 'Test score at 20M tokens' or similar, and make the metric explicit.
show 6 more
The paper says, at Section 5.1, Eq. (3) and surrounding text
Pith says: minor objection 3 of 8
The admission threshold epsilon = 0.05 is introduced without justification. Authors should discuss how this value was chosen and whether results are sensitive to it (e.g., ablation with different epsilon values).
The paper says, at Section 5.3
Pith says: minor objection 4 of 8
The archive capacity is set to ten category slots. This choice is not motivated. Provide a sensitivity analysis or at least comment on how the capacity affects behavior.
The paper says, at General (reproducibility)
Pith says: minor objection 5 of 8
The paper does not include a high-level algorithm or pseudocode for RobustSGPO's edit selection, patch construction, or archive update rules. Adding an algorithmic description (even in the appendix) would significantly improve reproducibility.
The paper says, at Figure 1 and Figure 2 captions
Pith says: minor objection 6 of 8
These figures are reproduced from the AgentX technical report [4]. Ensure that the captions explicitly state 'Reproduced from [4]' rather than 'Prior-work foundation' to clarify attribution.
The paper says, at Section 5.2, line 184
Pith says: minor objection 7 of 8
The statement 'The current program does not automatically update caller instructions' is an important limitation that should be noted in the conclusion or future work section.
The paper says, at Table 1
Pith says: minor objection 8 of 8
The total attempts sum to 7,350, which is consistent with the abstract. This internal consistency is appreciated.
What would settle it
From the same 5 × 30 test-score matrix (configurations × seeds), compute bootstrapped 95% confidence intervals for completion rate and mean test quality for each configuration. Report the overlap. Additionally, perform a one-sided paired permutation test comparing RobustSGPO to SGPO on the 30 held-out tasks (average across seeds). If the p-value exceeds 0.05 or the intervals overlap substantially, the central claim is weakened.
from the skeptic's reading
How the review went
4 readings by deepseek-v4-flash · 10 Sep 2026
- Reader holds up, with conditions
- Skeptic agrees
- Editor wrote the verdict above
- Referee major revision: 4 major objections, 8 minor
Not run on this paper: the simulated authors' defence.
Pith did not check this review's objections against the paper.
The review in full
The review in brief, machine-written
The paper introduces RobustSGPO, which makes edit scope, operation, and targets explicit in semantic-gradient-based harness optimization. The core design—edit planning, exact-target checking, and a MAP-Elites-style archive of harness snapshots—is cleanly motivated and clearly described. The authors run a systematic sweep (E1–E3) isolating each component's contribution, and the internal arithmetic checks out. That is real work, and the method section is detailed enough to reconstruct the algorithm.
The soft spots are proportional but real. The headline claim—24/30 vs 18/30 tasks completed—rests on five seeds with no error bars or significance test. With 30 held-out tasks, a shift of 6 successes is in binomial noise territory (p ≈ 0.10). The quality difference of 0.37 points on a 0–5 scale could live inside per-task variance. The baseline is the authors' own AgentX, and the evaluation uses one internal brainstorming workflow. No code or data are shipped. The paper itself admits these limits: the margin over Structured Search at equal token cost shrinks to 0.04, and the workflow is not externally ranked.
So the method is plausible and well-reasoned, but the evidence is weaker than the abstract suggests. This is a useful architecture note for researchers working on automated harness optimization. I would send it to peer review with a strong request for confidence intervals and, ideally, a reproducible artifact.
The strongest objection
The central claim that RobustSGPO increases held-out completion from 60.0% to 80.0% (18/30 to 24/30) and test quality from 3.77 to 4.14 rests on point estimates without reported uncertainty. The paper reports "five paired seeds" (Section 6.1) but Table 3 provides a single value per condition. With 30 held-out tasks, a shift of 6 tasks is not automatically significant: a two-sided binomial test of proportions (18/30 vs 24/30) yields p ≈ 0.10. The quality difference of 0.37 points on a 0–5 scale could be within per-task variance. Without error bars, the reader cannot distinguish systematic improvement from seed- or task-specific noise. This concern is amplified by the small margin between RobustSGPO and Structured Search at equal token cost (0.04 points) and the subjective nature of the test rubric. If the variance is large, the headline improvements may not replicate.
The reader's weakest assumption was that five paired seeds make point differences systematic rather than noise. I identify the same load-bearing concern: the absence of error bars or significance tests for the headline completion and quality gains. The paper is transparent about its limitations (single workflow, no external benchmarks) and the method is coherent, but the statistical grounding of the central claim is insufficient. A CONDITIONAL verdict is appropriate; the authors should supply confidence intervals or significance tests. If they do, the current verdict stands; if not, the claims would need to be downgraded. Thus verdict_should_be = UNCHANGED.
The referee's report
If the reported improvements are systematic, the paper demonstrates that explicit control over edit selection, patch construction, and retention can meaningfully improve prompt-optimization outcomes in multi-agent harness evolution. The contribution is primarily empirical and practical: it identifies which components of search-space control matter (periodic scheduling, predefined add/remove operations, category retention) and quantifies their overhead. The work is situated within a specific in-house workflow (AgentX brainstorming), with all tasks, rubrics, and baselines originating from the same group, which limits generality but still provides useful engineering insight for similar industrial settings.
Pith's reading of the paper (a machine's summary, not the authors')
The paper tries to establish that how an agent-harness optimizer is allowed to edit its own specification matters as much as the optimizer's learning rule. In the AgentX workflow, semantic-gradient prompt optimization (SGPO) improves agent prompts by replaying candidate edits against the current version and keeping only those that score better, but it never decides in advance which agents to touch, what operation to perform, or where to continue search from. RobustSGPO makes those decisions explicit: a controller picks the scope, operation, and targets before content is generated; a checker rejects edits that touch the wrong objects while predefined code constructs agent additions and removals; and an archive of the best snapshot per edit category offers alternative starting points. The paper reports that these controls raise completion on 30 held-out tasks from 60.0% to 80.0% and test quality from 3.77 to 4.14 within a fixed 20-million-token budget, and that periodically widening the permission level beats always running with the widest permission. If true, the result locates the gain in search-space discipline rather than in larger permissions or a stronger optimizer.
What carries the argument. The machinery is a factored edit-selection rule plus three enforcement and memory components. Before generation, the controller draws a feasible category $c=(s,o)$—the set of agents to change and the operation to perform—and eligible targets, so candidates are sampled from a distribution summed over scope–operation–target combinations rather than free-form refinement. An exact-target check rejects any realized edit that changes objects outside the request; a predefined patch constructor, the Typed Compiler, writes and deletes the files for agent additions and removals so structural edits pass validity checks without relying on the model to emit executable code; and a MAP-Elites-style archive stores the best valid snapshot per scope–operation category, seeded by a reservoir of at most ten slots, with search alternating between incumbent and archived parents and every descendant required to beat the incumbent for admission. These components jointly implement the paper's three controls: choose the edit, construct and check the patch, and continue from a retained version.
Core claim. On the paper's own terms, the central claim is that free-form semantic-gradient updates underuse the permitted edit space, and explicit control of edit selection, patch execution, and search starting points repairs that. RobustSGPO factors the update distribution into a scope–operation–target choice $$q(\$\Delta$ h \mid h, g)= extstyle{\sum_{s,o,T}} p(s,o mid h)\, p(T mid s,o,h)\, q_{\mathrm{LLM}}(\$\Delta$ h mid h,g,s,o,T)$$ made before generation, enforces the requested edit with an exact-target check, constructs agent additions and removals with predefined code, and archives the highest-validation snapshot per scope–operation category so search alternates between incumbent and retained parents. Given five paired seeds on 120 tasks in the AgentX brainstorming workflow, the paper reports that the full method completes 24 of 30 held-out tasks versus 18 of 30 for the SGPO baseline, improves test quality from 3.77 to 4.14 at equal 20-million-token cost, and that periodic $1\to2\to3$ permission scheduling finishes 0.28 test-score points above fixed maximum permission. After a task shift, category retention loses only 0.15 points on source tasks versus 0.31–0.33 for other configurations, while the random-retention archive reaches a higher destination endpoint (4.18).
The premise it rests on. The comparison rests on five paired seeds per condition with no error bars or significance tests, and on the premise that admitting candidates by a mean validation gain above $\epsilon = 0.05$ predicts quality on held-out tasks; if either premise gives way, the reported gains could be seed-specific or task-specific artifacts rather than properties of RobustSGPO.
If the paper is right (Pith's inferences, not the authors')
Structured edits become reliable search steps: stating a requested edit and building structural patches with predefined code raises the validity rate of structural proposals from 48.9% (Constrained Search) to 77.8% (Structured Search), and the full method reaches 68% final coverage of the edit-category vocabulary versus 48% for the baseline.
Budget-constrained users gain more from controlled search than from free-form rewriting: within 20 million charged tokens RobustSGPO scores 4.14 against 3.77 for SGPO, and still beats Structured Search (4.10) despite archive overhead.
Permission schedules matter on their own: a periodic narrow-to-wide schedule ($1\to2\to3$) finishes 0.28 points above always-wide permission, which suggests that when an edit is allowed shapes final quality alongside what is allowed.
Retention policy is a real tradeoff control: category-wise retention minimizes source-task loss (0.15 points) after a task shift, while random retention maximizes the destination endpoint (4.18) at the cost of more source degradation.
The gain decomposes cumulatively: round-30 scores rise monotonically from 3.82 (SGPO) to 3.94, 4.04, 4.20, and 4.30 as each control is added, so each of the three controls contributes something at the margin.
A cheap transfer to test: applying the same 'change exactly those named objects' enforcement to instruction-only optimizers that rewrite single prompts, without structural patches and without an archive, may capture part of the gain; the paper's cumulative configurations do not isolate that component alone.
Because coverage counts only whether a category was reached by a valid edit, filling rarely visited categories with weak snapshots could raise coverage without raising held-out quality; a per-category quality analysis would separate exploration from optimization, and the paper does not report one.
The 0.28-point scheduling effect, if it generalizes, implies permission schedules are a tunable hyperparameter comparable to step size, and one could optimize the schedule itself rather than fixing 1-to-2-to-3 by hand.
A testable extension: since the archive stores snapshots per scope–operation category but the paper reports only aggregate transfer curves, tracking which categories' snapshots are actually reused after the task shift would show whether transfer gains come from structural categories (add and remove) or from instruction categories.
Check this review
Pith stored no record of this review's passes and took no fingerprint of it. Reviews made by code from 13 Nov 2026 on carry both.
How often reviews like this one are right
Reviews like this one: written by deepseek-v4-flash, where what was read was not recorded.
How often objections from reviews like this one are right: not yet measured
Reviews by people
This review was written by a machine. No person has reviewed this paper on Pith yet. A scientist listed on Pith can review it for a fee, and their review is published here whatever it concludes.
The paper
Abstract
Semantic-gradient-based prompt optimization (SGPO) improves agent harnesses using execution feedback, but its local update rule leaves the choice of edit scope and operation unresolved. We introduce RobustSGPO, which specifies the requested edit, constructs and checks the patch, and continues search from either the incumbent or retained snapshots. We evaluate permission scheduling, cumulative controls, and task-family transfer in the AgentX brainstorming workflow using 120 tasks, 95 runs, and 7,350 candidate attempts. Periodic 1 → 2 → 3 scheduling exceeds fixed maximum permission by 0.28 test-score points. RobustSGPO increases completion on 30 held-out tasks from 60.0% to 80.0% and improves test quality from 3.77 to 4.14 under a 20-million-token budget. Category retention reduces source-task degradation after a shift, whereas random retention reaches a higher destination endpoint. Search-space control benefits quality through executable edits and alternative starting points, with measurable retention overhead.
The paper's own sentences, arranged, kept beside the review and separate from it. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. No sentence in this record was written by a model.
abstract compiled · 2026-09-23
The authors' main claim, as the classifier picked it out 0.93
“Search-space control benefits quality through executable edits and alternative starting points, with measurable retention overhead.”
How the main claim is stated: asserted 1.00
What the abstract itself states, as probabilities: claims first 0.05 · states limitation 0.89 · human participants 0.08 · controlled comparison 0.82 · randomized assignment 0.06 · states quantitative result 0.99 · compares with prior results 0.82 · mentions code or data availability 0.03
The abstract, sentence by sentence
This table labels the abstract only (arXiv listing abstract v1). The number is the probability the classifier gave the label.
| Sentence | Role | The authors' own work | States the main conclusion | Strength |
|---|---|---|---|---|
| Semantic-gradient-based prompt optimization (SGPO) improves agent harnesses using execution feedback, but its local update rule leaves the choice of edit scope and operation unresolved. | gap 0.65 | 0.09 | 0.04 | asserted 1.00 |
| We introduce RobustSGPO, which specifies the requested edit, constructs and checks the patch, and continues search from either the incumbent or retained snapshots. | method 0.57 | 0.90 | 0.12 | asserted 0.99 |
| We evaluate permission scheduling, cumulative controls, and task-family transfer in the AgentX brainstorming workflow using 120 tasks, 95 runs, and 7,350 candidate attempts. | method 0.91 | 0.96 | 0.04 | asserted 0.97 |
| Periodic $1\to2\to3$ scheduling exceeds fixed maximum permission by 0.28 test-score points. | result 1.00 | 0.92 | 0.07 | asserted 1.00 |
| RobustSGPO increases completion on 30 held-out tasks from 60.0% to 80.0% and improves test quality from 3.77 to 4.14 under a 20-million-token budget. | result 1.00 | 0.96 | 0.42 | asserted 1.00 |
| Category retention reduces source-task degradation after a shift, whereas random retention reaches a higher destination endpoint. | result 0.98 | 0.91 | 0.09 | asserted 1.00 |
| Search-space control benefits quality through executable edits and alternative starting points, with measurable retention overhead. | interpretation 0.98 | 0.85 | 0.75 | asserted 1.00 |
What kind of paper the classifier reports this is: method or tool 1.00 · evidence from observation or experiment 0.96
What was examined, and what was not
Sources read
- arXiv listing abstract v1
Not examined
- full text
- figures and tables
- references and citing sentences
- relations to other papers
Levels run
- P abstract
Not run
- S sections
- T body sentences
- C citations
- R cross-paper relations
How the labels were made
Compiled by classifier-1.13.0 via TypeSafe API, question bank abstract-p1-20260923, 2026-09-23: 1 request, 5,614 input tokens.
Every quoted sentence on this record is the authors'. Every label is from Pith's closed vocabulary, with the probability shown beside it. This record as JSON
Figures (7)
Figures are the authors', reproduced from the paper.
Cited by 1 Pith paper
Reviewed papers on Pith that cite this one. The full citation record
Challenges and discussion
Disagree with the review, or know something it missed? Leave a comment, or challenge this review formally: a challenge becomes a permanent entry in this paper's record, and its decision is published here.
Come back by email
A short list of the week's newly reviewed papers. Nothing is sent until you confirm from your inbox, and every email has a link that stops it. No account.
No comments yet.
Sign in to comment