{"id":"0ac6913e-4924-4d8e-8b91-0664fa5019c1","arxiv_id":"2608.06301","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A new benchmark, HarnessOpt-Bench, measures how well frontier LLMs can improve the prompts, tools, and control flow of target agents under a fixed evaluation budget with a held-out test set.","lead":"AI researchers built a benchmark that tests whether one AI model can improve another AI agent by editing its prompts, tools, and control flow under a strict evaluation budget. It matters because AI agents are becoming common, and we currently lack a standard way to measure how well a model can make another agent better.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The headline 1.8x model-vs-harness contrast has no sampling uncertainty: the paper runs each optimizer configuration only twice, and its resolution band measures evaluation noise for a fixed candidate, not optimizer-run variability.","rationale":"The reader's weakest assumption (verifier gaming) is a legitimate validity concern, but the most load-bearing issue for the stated headline is that the 1.8x contrast has no quantification of optimizer-run sampling error; with n=2 per cell the ratio may not survive. The paper's own caveats—'two rounds do not estimate dispersion' and resolution bands described as descriptive—show the authors know this, which makes the numeric claim in Section 5.1 over-precise. The qualitative conclusions (models separate, native harnesses show no consistent advantage) are better supported by direction consistency and the LSS-lambda ordering, so I do not recommend rejecting the paper. I would keep the CONDITIONAL verdict and make the condition (a) artifact release and (b) a bootstrap or variance-component analysis of the headline ratio. This is an addressable gap, not a fatal flaw.","tokens_in":18488,"tokens_out":11372,"duration_ms":118424,"concrete_test":"Release the promised run store (Appendix E) and bootstrap the replicate runs: for each model-harness-task cell, resample its two replicate means with replacement, recompute the Section 5.1 model contrast, harness contrast, and their ratio over at least 10,000 resamples, and report a percentile interval. Separately fit a mixed model with crossed random effects for optimizer model and harness on the same balanced shared-vs-native subset and report the variance-component ratio. If the 95% interval for the ratio includes 1.0, or the variance-component ratio is not stably above 1, the central claim should be reported as a qualitative ordering, not as the 1.8x numeric contrast.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing empirical claim in Section 5.1 is the ratio 0.142/0.079 for model versus harness changes, reported as about 1.8x. That ratio is not supported by the reported statistics. Section 4 states that each optimizer configuration is run only twice, and Table 2 says a parenthetical range is 'not a confidence interval — two rounds do not estimate dispersion.' The resolution band is estimated by scoring the same candidate twice on the same cases; it therefore covers rollout stochasticity for a fixed candidate, not the sampling variability of the optimization process, which produces a different candidate on each run. Many adjacent cells in Table 2 have overlapping ranges, so the 1.8x ratio could be a small-sample artifact. In addition, the two contrasts are computed from different quantities: the model contrast averages pairwise differences among five models under the shared opencode harness, which includes extreme pairs such as claude-opus-5 versus gpt-5.6-terra; the harness contrast averages absolute opencode-versus-native differences within each model. The 1.8x figure is therefore not a variance-component or matched-contrast estimate, and it could change substantially under a commensurable definition. The acknowledged verifier-gaming risk in the Limitations is real, but it is a separate validity threat; even setting it aside, the numeric headline lacks the uncertainty analysis needed to be load-bearing.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper introduces HarnessOpt-Bench, a benchmark and evaluation protocol for measuring how well LLM-based optimizers can improve the harness (prompts, tools, control flow, memory, orchestration code) of a target agent under a fixed evaluation budget. The protocol pins seed harnesses, fixes non-overlapping dev/val/test partitions, places held-out test scoring behind a trusted execution environment, and reports normalized gain over the seed. The authors evaluate five frontier LLMs under a shared coding harness and their native harnesses on four downstream tasks, over 111 scored runs. Their main reported findings are that optimizer model choice has a larger average effect than coding-harness choice (about 1.8x), that native harnesses provide no consistent advantage, and that achievable gains vary substantially across tasks and seed regimes. The paper also contributes release infrastructure: immutable candidate versions, pinned seeds and split manifests, and a reproducible run store.","tokens_in":18713,"tokens_out":6538,"duration_ms":65095,"significance":"If the empirical claims hold, HarnessOpt-Bench is a valuable controlled protocol for a capability that is currently measured in method-specific and incomparable ways. The protocol's strengths are real: a held-out test partition that is inaccessible during search, environment- rather than instruction-level enforcement of the evaluation boundary, a transparent normalized-gain statistic, and explicit descriptive resolution bands. The paper is also commendably careful in several places: GAIA is kept separate because its baseline is zero, Table 2 parentheticals are explicitly labeled as not confidence intervals, and the Limitations section acknowledges that the benchmark is 'hack-resistant, not hackproof.' The release design (pinned splits, immutable Git commits, reproducible run artifacts) is a significant asset for future work. However, the headline quantitative claim that model choice is about 1.8x more important than harness choice is not supported by the statistics as presented, and the manuscript's own statements about two-round runs and resolution-band methodology make this gap clear.","major_comments":[{"comment":"The headline claim that 'changing the optimizer model moves gain by 0.142 on average' and 'changing the harness moves it by 0.079,' making the model contrast 'about 1.8x larger,' has no uncertainty quantification for the optimization process. Section 4 states that each optimizer configuration is run only twice, and Table 2's caption explicitly says the parenthetical range is 'not a confidence interval — two rounds do not estimate dispersion.' The resolution band is estimated by scoring the same candidate twice on the same cases, so it measures evaluation noise for a fixed candidate, not the run-to-run variability of the optimizer, which produces a different candidate each time. Adjacent entries in Table 2 frequently have overlapping ranges; for example, Terminal-Bench claude-sonnet-5 is 0.10 (0.02–0.18) with claude-code and 0.15 (0.09–0.22) with opencode. The 1.8x ratio could change substantially under a modest amount of optimizer-run variability. The authors should either provide an uncertainty analysis for the 0.142 and 0.079 estimates (for instance, bootstrap across runs/cells or a variance-component model) or explicitly downgrade the claim to a descriptive, non-inferential observation.","section":"Section 5.1, Table 2"},{"comment":"The two contrasts being compared are not commensurable. The model contrast appears to average pairwise differences among five models under the shared opencode harness, whereas the harness contrast averages absolute opencode-versus-native differences within each model. These are different estimands: the model contrast includes extreme pairs (e.g., claude-opus-5 vs gpt-5.6-terra) and uses signed differences, while the harness contrast uses absolute differences and therefore discards direction. As a result, the ratio 0.142/0.079 is not a variance-component estimate or a matched-contrast estimate, and it is not a direct output of the additive LSS-lambda model in Eq. (5), which contains a model term but no harness term. The paper should define the exact quantities being averaged, report their uncertainty, or derive the model-versus-harness comparison from a single model that includes both factors.","section":"Section 5.1 and Eq. (5)"},{"comment":"The validity of the benchmark's scores depends on held-out normalized gain reflecting genuine task improvement rather than exploitation of the fixed evaluation harness. The manuscript acknowledges in the Limitations that the benchmark is 'designed to be hack-resistant, not hackproof,' and the Ethics statement cites evidence that comparable agent benchmarks can be gamed via evaluator hijacking or judge prompt-injection [24]. Because the optimizer can edit the target agent's output formatting, prompting, and verifier-facing behavior, there is a concrete risk that some of the measured gain comes from adapting to the verifier rather than improving task ability. The paper does not report an audit of nominated candidates for such exploits (for example, changes to answer formatting, verifier-specific keywords, or attempts to manipulate the scoring function). A concrete safeguard or analysis, such as checking whether final candidates altered output formatting or verifier-facing behavior and reporting any such cases, would substantially strengthen the validity of the conclusions.","section":"Limitations and Ethics Statement, Section 3.2"}],"minor_comments":[{"comment":"The manuscript uses '111 scored runs' and later '7 of the 111 cells'; please define whether a 'run' and a 'cell' are the same unit, since Table 2 reports means over two rounds per configuration and the count of cells is smaller than 111.","section":"Abstract and Section 5.3"},{"comment":"The sentence 'Both exceed the task resolution bands, although the harness contrast does so narrowly' compares aggregate cross-task averages to per-task resolution bands; please clarify which task's band is meant, since the bands differ by task and the averages are not task-specific.","section":"Section 5.1"},{"comment":"The figure caption lists colors for the Claude and GPT series, but the x-axis labels and legend are crowded; adding a clear legend and marking the release points by family would improve readability.","section":"Figure 3"},{"comment":"Several rows have very narrow observed ranges (e.g., 0.63 (0.60–0.67) and 0.51 (0.46–0.56)), and the text does not explain why two runs can give such tight ranges without a confidence interval; a sentence noting that these are raw observed ranges rather than precision estimates would prevent over-reading.","section":"Table 2, OfficeQA rows"}],"recommendation":"major_revision","confidential_remarks":"The paper is a reasonable fit for the journal's AI/ML scope. The benchmark contribution is real, and the authors are honest about many limitations, but the central 1.8x claim is the load-bearing point for RQ3 and is not statistically supported as presented. A major revision that either quantifies the uncertainty of the model-versus-harness contrast or removes the quantitative claim from the abstract/headline, plus an audit of verifier-gaming risk, would make the paper acceptable. The reliance on the authors' own VERO infrastructure is disclosed and appears appropriate."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"This is a genuinely useful benchmark paper with an unusually honest set of caveats, but the headline number does not carry the weight the abstract puts on it. The real contribution is the protocol: fixed target agent, held-out test partition, trusted execution boundary, budget enforcement, and a paired model-by-harness grid across five frontier models and four tasks. That is new and badly needed. It makes harness optimization a measurable target rather than a collection of one-off demonstrations. The paper also does something rare: it reports ranges, keeps GAIA separate because its baseline is zero, and explicitly calls resolution bands descriptive thresholds rather than significance tests.\n\nNow the soft spots, in proportion. The central claim in Section 5.1—model contrast 0.142 vs harness contrast 0.079, roughly 1.8x—is not supported by the reported statistics. Each optimizer configuration is run only twice. The resolution band is estimated by scoring the same candidate twice on the same cases, so it covers rollout stochasticity for a fixed candidate, not the much larger variability of the optimization process itself. Two runs produce different candidates; you can see the spread in Table 2. On top of that, the two contrasts are not defined the same way. The model contrast averages pairwise model differences under the shared opencode harness. The harness contrast averages opencode-vs-native differences within each model. Those are different objects. The 1.8x is a ratio of two point estimates with no uncertainty attached, and it could shift substantially under a matched definition. The paper should present it as an observation to be confirmed, not a load-bearing finding, unless it adds more replicate runs and a real variance analysis.\n\nThe artifacts are promised but no URL or commit hash appears. That is a release-blocker, not a research blocker. The acknowledged verifier-gaming risk in the Limitations is real but secondary; the held-out partition and trusted execution do contain the main failure mode. Citation pattern is fine; the related work is the right set.\n\nWho is this for? Anyone working on agent evaluation, self-improvement, or harness engineering. It deserves a serious referee: the protocol is important, the data are useful, and the weaknesses are addressable. My recommendation is conditional acceptance—require more replicates on the headline contrast, report uncertainty properly, and release the artifacts.","headline":"Solid, honest benchmark protocol; the 1.8x model-vs-harness headline is a point estimate without uncertainty and should be treated as such.","tokens_in":19287,"tokens_out":2654,"would_cite":true,"duration_ms":27265,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"HarnessOpt-Bench measures how well LLMs improve other agents' harnesses, and finds the optimizer model matters ~1.8× more than the coding harness.","keywords":["harness optimization","agent harnesses","LLM benchmarks","agent self-improvement","trusted execution","held-out evaluation","normalized gain","coding agents"],"falsifier":"Re-score the submitted candidates after perturbing the verifier—for example, rounding thresholds, case weights, or per-case reward noise—while holding everything else fixed. If the ranking of optimizer models collapses or the model contrast falls below the task's resolution band, the measured capability is partly an artifact of the stable evaluator rather than robust harness improvement.","tokens_in":18275,"feed_emoji":"🧰","tokens_out":7656,"duration_ms":68152,"temperature":0.7,"pith_summary":"HarnessOpt-Bench is a controlled benchmark for measuring how well an LLM-based optimizer can improve the \"harness\" of another agent—the prompts, tools, control flow, memory, and orchestration code that surround it. The paper's central claim is that harness optimization is a measurable and discriminative capability: with task and target fixed, swapping the optimizer model changes a candidate's normalized gain by 0.142 on average, while swapping the coding harness the optimizer acts through changes it by 0.079, making the model contrast about 1.8 times larger. To make this measurement trustworthy, the benchmark holds out a test partition that is never visible during search, imposes fixed evaluation budgets, and runs every candidate in a trusted execution environment that preserves versions for audit. The authors evaluate five frontier optimizer models under a shared and a native harness across four downstream tasks, and report that native harnesses give no consistent advantage and that gains vary widely across tasks and seed regimes. If the benchmark delivers what it promises, it gives the community a common protocol for comparing optimizer models, harnesses, and search algorithms on a task that is increasingly central to building capable AI systems.","feed_headline":"Optimizer model matters 1.8x more than coding harness","feed_subtitle":"HarnessOpt-Bench holds the harness fixed and shows which frontier LLMs can actually improve other agents.","key_machinery":"The load-bearing object is the evaluation protocol itself. An optimizer receives a pinned seed harness $H_0$, can request evaluations on development cases (which reveal per-case outcomes and traces) and on validation cases (which reveal only an aggregate score), and must nominate a final candidate $H^+$ that is scored on a held-out test partition that remains inaccessible during search. The metric is normalized gain $g=(E_\\theta(H^+)-E_\\theta(H_0))/(1-E_\\theta(H_0))$, the fraction of headroom above the pinned seed baseline that a candidate captures. A trusted execution environment enforces the boundary: the optimizer can edit only the harness, not the target model, environment, verifier, or data; every model call passes through an allow-listed gateway with per-scope budgets; and each candidate is preserved as an immutable version for audit. Task-specific resolution bands, estimated by re-scoring the same candidate, mark differences that are too small to be treated as real. For cross-task comparison, the paper defines LSS-λ, a task-adjusted model effect in normalized-gain units, computed from a balanced additive decomposition over tasks and models.","core_discovery":"The paper's central discovery is that, under a fixed evaluation protocol with held-out scoring, frontier LLMs separate more by their own optimization capability than by the coding harness they act through. Averaged over the balanced shared-harness grid on the three tasks with competent seeds, the task-adjusted model effect (LSS-λ) places the top model at +0.228 normalized-gain units and the bottom at −0.174, with intermediate models clustered within a resolution band that the authors treat as unresolved. Across the 20 model–task pairs run under both a shared harness and each model's native harness, the shared harness wins 11, the native wins 9, and none tie, so native tooling is not a reliable estimator of optimization ability. The paper also finds that broader exploration of harness levers is positively associated with gain on every task (Spearman ρ between +0.34 and +0.88), that detailed trace reading is not positively associated with gain, and that the case-pass allowance, not the evaluation-call cap, is the binding budget constraint.","pith_inferences":["If the model-versus-harness contrast generalizes beyond the five models and four tasks tested, then routine comparison of agent optimizers under each developer's own harness can overstate the optimizers' relative standing; a shared-harness control may become standard practice.","The negative association between trace-reading and gain suggests that richer per-case summary feedback might substitute for full traces; a testable extension would vary the development-disclosure policy and measure whether gains improve.","Because the benchmark is explicitly \"hack-resistant, not hackproof,\" the metric's long-run validity depends on evaluator jitter; introducing per-run noise in cases, tool behavior, and verifier implementation would test whether the measured capability survives such perturbation."],"forward_implications":["Comparisons of harness-optimization systems should fix the coding harness; otherwise model and harness effects are conflated, and the model effect is roughly 1.8 times larger.","A model's native coding harness should not be assumed superior; the shared harness won as often as the native one across the evaluated pairs.","Held-out test scoring is necessary: visible validation scores were optimistic relative to the submitted candidate's test score on most cells.","Search strategy matters: broader exploration of harness levers is associated with higher gain, while reading full traces is not; and optimizers should watch case-pass budgets rather than evaluation-call caps."],"supporting_citations":[{"why":"Supplies the trusted-execution infrastructure and evaluation protocol that HarnessOpt-Bench builds on.","marker":"[23]"},{"why":"An end-to-end harness optimizer that this benchmark is designed to evaluate and compare.","marker":"[12]"},{"why":"Documents how comparable agent benchmarks can be gamed via evaluator hijacking or judge prompt-injection, motivating the trusted boundary.","marker":"[24]"},{"why":"Defines the general-assistant task whose data, verifier, and environment are reused as one downstream task.","marker":"[16]"},{"why":"Defines the document-QA task whose data and verifier set one of the four optimization targets.","marker":"[18]"},{"why":"Defines the deep-research task whose data and verifier provide another optimization target.","marker":"[6]"},{"why":"Defines the terminal-use task whose data and verifier provide the fourth optimization target.","marker":"[15]"},{"why":"Shows why comparing agents without fixing the harness conflates effects, the problem HarnessOpt-Bench addresses.","marker":"[38]"}],"fun_headline_variants":["For agent harness tuning, the optimizer LLM beats the coding harness","HarnessOpt-Bench: optimizer model matters more than coding harness","Agent optimization: model choice outranks harness, benchmark says","In agent harness optimization, the model is 1.8x more important"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"Measured held-out gain reflects genuine improvement in the target agent's task ability, not exploitation of the fixed verifier or evaluation artifacts; the paper itself concedes the design is \"hack-resistant, not hackproof.\"","fun_headline_variants_meta":{"raw":{"variants":["For agent harness tuning, the optimizer LLM beats the coding harness","HarnessOpt-Bench: optimizer model matters more than coding harness","Agent optimization: model choice outranks harness, benchmark says","In agent harness optimization, the model is 1.8x more important"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000257,"raw_usage":{"total_tokens":1619,"prompt_tokens":1026,"completion_tokens":593,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":642,"completion_tokens_details":{"reasoning_tokens":517}},"tokens_in":642,"tokens_out":593,"duration_ms":6049,"temperature":1.0,"reasoning_tokens":517,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T05:56:06.350589+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-score the submitted candidates after perturbing the verifier—for example, rounding thresholds, case weights, or per-case reward noise—while holding everything else fixed. If the ranking of optimizer models collapses or the model contrast falls below the task's resolution band, the measured capability is partly an artifact of the stable evaluator rather than robust harness improvement.","supporting_citations":[{"cited_title":"Ursekar, A","cited_arxiv_id":null,"evidence_quote":"Supplies the trusted-execution infrastructure and evaluation protocol that HarnessOpt-Bench builds on."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Documents how comparable agent benchmarks can be gamed via evaluator hijacking or judge prompt-injection, motivating the trusted boundary."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the deep-research task whose data and verifier provide another optimization target."}],"review_version":1}