{"id":"833e7f4f-f7c3-4b0a-9902-c2a3e7490cdb","arxiv_id":"2607.14004","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"In a two-phase Terminal-Bench evaluation, only regression-aware RELAI-VCL compounded optimization gains, reaching the highest pass rate at every stage.","lead":"This paper compares three agent-harness optimizers in a two-phase continual-learning setup on Terminal-Bench 2.0, asking whether optimization gains survive new tasks and a second optimization round. It reports that only the authors' own regression-aware method (RELAI-VCL) both transfers to unseen tasks and keeps improving, while GEPA overfits and Meta Harness stalls.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Causal attribution to regression control is confounded by search space: RELAI-VCL is the only method with both the broadest search space and the in-loop constraint.","rationale":"The reader’s weakest assumption is sampling noise—two trials per task, no confidence intervals. That is a real concern, but it is secondary to a more fundamental identification problem: even if every reported difference were statistically robust, the design cannot attribute RELAI-VCL’s success to regression control because that method is also the only one with the broadest search space. The causal claim in the abstract is therefore unsupported independently of noise. I disagree with the reader’s choice of weakest assumption, but the appropriate verdict remains conditional: the protocol is a useful contribution, and the descriptive results are worth reporting, but the causal claim requires either the proposed ablation or a substantial hedge to ‘associated with’ rather than ‘only when.’ The current paper already contains some hedging (§6.6, §8), but the abstract states the strong version. Thus no verdict change is needed beyond the reader’s conditional acceptance, but the condition should include addressing the confound, not merely adding more trials.","tokens_in":14364,"tokens_out":5401,"duration_ms":54319,"concrete_test":"Run a two-arm ablation on the same T1/T2 split: (1) RELAI-VCL with its in-loop no-regression constraint removed, accepting any candidate that improves aggregate pass rate on the current task set; (2) GEPA with prompt+code search enabled (repairing the GEPA-code failure) and with the same no-regression constraint added. Keep the proposer model, 200-rollout budget per phase, and evaluation protocol identical. If arm 1 still shows positive transfer and Phase-2 improvement at comparable levels, the constraint is not necessary for compounding; if arm 2 still fails to compound, search breadth remains insufficient. Either outcome would settle whether regression control is the causal ingredient.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—that gains compound only when regression control is inside the optimization loop—is not identifiable from this comparison because RELAI-VCL differs from GEPA and Meta Harness in more than the constraint. Per Table 2 and §4.2–4.4, RELAI-VCL searches over prompts, tools, workflow, memory, skills, and code, while GEPA searches only prompts and Meta Harness only harness code. GEPA-code, the one arm that would have provided a broad-search/no-constraint comparison, failed to produce a valid candidate (§4.2). Thus the observed difference could be due to search expressiveness rather than the no-regression mechanism. The paper’s own qualitative evidence supports this: GEPA’s prompt-only search accreted task-specific lessons and transferred negatively; Meta Harness’s code-only search made generic but conservative fixes and stalled; RELAI-VCL found generic infrastructure improvements. Without a direct manipulation of the constraint (or of search breadth), the abstract’s causal statement is a single confounded observation, not a tested mechanism.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a two-phase continual-learning evaluation protocol for agent-harness optimizers, instantiated on 22 hard Terminal-Bench 2.0 tasks split into an initial 12-task set and a 10-task set. Three optimizers—GEPA, Meta Harness, and RELAI-VCL—are compared against a shared baseline under equal per-phase rollout budgets (200 rollouts per phase). The authors report static Phase 1 pass rates, transfer to unseen tasks after Phase 1, final pass rates after Phase 2 re-optimization, and a lifelong average. They find that all three optimizers improve over the baseline in the static setting, but only RELAI-VCL both transfers positively and continues to improve under re-optimization. The abstract attributes this compounding behavior to regression control being enforced inside the search loop.","tokens_in":14647,"tokens_out":3833,"duration_ms":39852,"significance":"The protocol is a useful methodological contribution: it separates static optimization strength, transfer to unseen tasks, and the ability to continue improving under re-optimization—properties that a single benchmark score conflates. The paper also releases its artifacts and implementation details, which supports reproducibility. If the causal attribution were established, the result would be practically important for designing continual agent-optimization systems. However, as presented, the headline causal claim is not supported: RELAI-VCL differs from the comparison methods in both the breadth of its search space and the presence of in-loop regression control, and all pass-rate comparisons rest on point estimates from only two trials per task. The paper's own limitations section explicitly acknowledges that the protocol cannot fully distinguish the proposed mechanism from other explanations, yet the abstract states it as the key finding.","major_comments":[{"comment":"The central attribution—that compounding gains arise specifically from in-loop regression control—is confounded by search-space breadth. Per Table 2, RELAI-VCL searches over prompts, tools, workflow, memory, skills, and code, whereas GEPA (prompt) searches only prompts and Meta Harness only harness code. The one arm that would have provided a broad-search/no-constraint comparison, GEPA-code, failed to produce a valid candidate during Phase 1 and was excluded (§4.2). The observed differences could therefore reflect expressiveness of the search space rather than the no-regression mechanism. A direct manipulation of the constraint, holding the search space fixed, is needed to support the abstract's causal claim.","section":"§4.2–4.4, Table 2"},{"comment":"Every task is scored from only two trials (R=2), and the paper compares point estimates without confidence intervals or significance tests. With 12–22 tasks, the headline gaps are small on a per-task basis: on transfer, RELAI-VCL's 72.7% vs. Meta Harness's 68.2% on the 22-task union is a difference of one task out of 22. Phase 1 differences are similarly within roughly one to two tasks. Without variance estimates or more trials, the claim that RELAI-VCL 'leads at every stage' is not statistically supported. At minimum, bootstrap confidence intervals or per-task score breakdowns should be reported.","section":"§3.3, Tables 4–8"},{"comment":"The paper's own limitations state that the current protocol 'cannot fully distinguish' the generalization-filter interpretation from other explanations, and note that the tasks are only loosely related and that only two optimization rounds are studied. Despite these caveats, the abstract and conclusion assert, as the 'key observation,' that gains compounded only when regression control was built into the optimization loop. This overstates the evidentiary strength of a single confounded three-way comparison. The conclusion should be reframed as an observed association, with the causal claim explicitly deferred until a controlled manipulation is performed.","section":"§6.6, §7.1–7.3; abstract; §8"}],"minor_comments":[{"comment":"The 'Max. agent timeout' entry for T1∪T2 is listed as '900s / 1800s'; this is ambiguous. Since the union's maximum timeout is 1800s, reporting a single value would be clearer.","section":"Table 3"},{"comment":"GEPA-code's failure to produce a valid candidate is described only briefly. A short explanation of the failure mode (e.g., invalid syntax, evaluation crashes, budget exhaustion) would help readers interpret the exclusion.","section":"§4.2, Appendix B.2"},{"comment":"The pass rate on the unseen Phase 2 tasks alone (65.0%) is reported only in text. Including it in Table 5 or a separate column would make the transfer result more transparent.","section":"§6.2"},{"comment":"Several references are to 2026 preprints or software repositories. Please confirm that all cited items are publicly accessible and include version/accession information where applicable.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The paper is authored by RELAI.ai staff and RELAI-VCL is the authors' own method. The favorable comparison is consistent with that affiliation. This is not a reason to reject, but the editor may wish to consider whether the reported causal framing should receive extra scrutiny. The methodological protocol is valuable; the main risk is overclaiming in the abstract relative to the confounded and statistically underpowered evidence."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe thing to know: the two-phase continual-learning evaluation is a real contribution, but the abstract's causal claim is not supported by the comparison as run. The paper separates static optimization, transfer to unseen tasks, and re-optimization—exactly what one-shot benchmarks conflate—and shows sharp divergence among three optimizers. That's a valuable template for future work.\n\nWhat it does well: common baseline, identical 200-rollout budgets, released artifacts, and careful qualitative inspection. GEPA's prompt accreted per-task lessons with explicit task IDs and transferred below baseline; Meta Harness made generic but conservative code fixes and stalled in Phase 2; RELAI-VCL's changes look general and it kept improving. That story is plausible and worth having.\n\nThe soft spots are real, though. The headline mechanism—'gains compounded only when regression control was built into the loop'—is an overclaim relative to the experiment. RELAI-VCL differs from GEPA and Meta Harness in search space as well as in the regression constraint: it searches prompts, tools, workflow, memory, skills, and code, while GEPA is prompt-only and Meta Harness is code-only. The arm that would have separated the variables, GEPA with code mutation, failed to produce a valid candidate, so the design cannot attribute the difference to the constraint. The paper itself acknowledges this in Section 6.6 and 7, but the abstract doesn't. On top of that, two trials per task leaves pass-rate differences of ~4.5 points corresponding to about two tasks out of 22; no confidence intervals or tests are reported. The task split (all 900s tasks versus all 1800s tasks) is also an arbitrary line that could plausibly affect transfer.\n\nVerdict: worth a serious referee. The protocol is a contribution and the empirical pattern, however preliminary, is interesting. The authors should soften the causal language or run an ablation that varies the constraint within a fixed search space. I'd bring it to a reading group to discuss evaluation design for agent optimizers.","headline":"A genuinely useful evaluation protocol, but the causal headline is confounded by search-space breadth and thin statistics; read it as a preliminary methodology, not a mechanism test.","tokens_in":15055,"tokens_out":3573,"would_cite":true,"duration_ms":34236,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Agent-optimizer gains compound only when regression control lives inside the search loop.","keywords":["agent optimization","continual learning","catastrophic forgetting","regression-aware search","harness optimization","transfer generalization","benchmark overfitting","two-phase evaluation"],"falsifier":"Repeat the two-phase protocol with many trials per task (say 20) and report confidence intervals on Phase-1, transfer, and final pass rates; if the regression-aware method's transfer and continued-improvement advantages over the other two shrink to noise, the compounding claim fails. A sharper test: run the same regression-aware search with the no-regression rejection disabled but the same search space and budget; if transfer and second-phase gains disappear, the mechanism is confirmed.","tokens_in":14309,"feed_emoji":"🔁","tokens_out":5523,"duration_ms":54684,"temperature":0.7,"pith_summary":"Agent-optimization results are usually reported as one-shot benchmark scores, which do not tell whether an optimized agent can be optimized again as new tasks arrive without losing earlier gains. This paper studies that compounding question with a two-phase protocol on hard terminal tasks: optimize on an initial set, test transfer to a new set, then re-optimize on the union. All three optimizers beat the unoptimized baseline in the static first phase, but only the regression-aware method transfers positively to unseen tasks and then continues improving in the second phase; the other two either overfit the first task set or stall when re-optimized. The paper's central observation is that gains compounded only when a no-regression constraint was enforced inside the optimization loop, rejecting candidate harness edits that trade new-task gains for regressions on tasks already solved. A reader should care because deployed agents are optimized recursively, and this protocol measures whether optimizer improvements survive that reality.","feed_headline":"Only in-loop regression control makes agent-optimizer gains compound","feed_subtitle":"A two-phase benchmark shows one optimizer transfers and improves; the others overfit or stall.","key_machinery":"The load-bearing machinery is the two-phase protocol itself, which separates three conflated properties: static optimization strength (Phase 1 score), transfer to unseen tasks (score on the expanded task union after optimizing only on the initial set), and continued improvement (final score after re-optimizing on the union). The mechanism the paper credits for the only method that passes all three is an in-loop no-regression constraint: candidates that improve newly targeted tasks while regressing on previously solved tasks are rejected during search, biasing the optimizer toward small, generalizable harness edits. In this evaluation those edits were generic additions—a task-type classifier,","core_discovery":"In a two-phase evaluation on 22 hard terminal tasks, each optimizer receives 200 rollouts to improve the same baseline agent, then another 200 after new tasks arrive. All three improve on the baseline's 62.5% Phase-1 pass rate (70.8%, 66.6%, 79.2%). Without new-task optimization, the prompt-evolution method falls to 54.5% on the 22-task union (below the 56.8% baseline), the harness-code method rises to 68.2%, and the regression-aware method reaches 72.7%. After re-optimization, the harness-code method drops to 59.1% because every second-round candidate scored worse than the existing agent; the regression-aware method improves to 77.3% and has the highest lifelong average (76.4% vs 66.0%, 64.","pith_inferences":["The no-regression constraint may function as an implicit complexity prior: in this task population, an edit that fixes a new task without breaking old ones is more likely to be general; this could be tested by removing the constraint while holding search space and budget fixed.","Because the benchmark tasks are only loosely related, the observed differences are not a stress test; on a stream of correlated failures within one application domain, shortcut exploitation could make both overfitting and stall effects more extreme.","The two-trial-per-task scoring leaves the reported gaps between close methods (e.g., 66.0% vs 64.6% lifelong average) potentially within sampling noise; a re-run with more trials per task would sharpen which differences are real.","For production settings with imperfect verifiers and single-trajectory feedback, an in-loop no-regression constraint would need a probabilistic or approximate form; the paper's clean verifier-based setting may overstate how easily the mechanism transfers."],"forward_implications":["A high static benchmark score does not predict transfer: the top Phase-1 method transferred below the unoptimized baseline once new tasks appeared.","Transfer alone does not imply compounding: the method that transferred well stalled in Phase 2, with every generated candidate worse than the existing agent.","Repeated optimization alone does not imply compounding: the method that overfit Phase 1 regained performance in Phase 2 only by optimizing directly against the new tasks.","Regression control placed inside the search loop, not checked afterward, was associated with both positive transfer and continued improvement in this task population.","The lifelong average is informative only when decomposed into Phase 1, transfer, and final components; similar averages can hide opposite failure modes."],"fun_headline_variants":["Agent optimizer gains compound only with regression control","Only regression-aware loops compound agent-optimizer gains","In continual learning, agent gains compound only with regression control","Two phases prove: only regression control compounds agent gains","On Terminal-Bench 2.0, only regression-aware optimizer compounds"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The load-bearing premise is that the reported pass-rate differences reflect real optimizer behavior rather than sampling noise, since each task is scored from only two trials and comparisons are made without confidence intervals or significance tests.","fun_headline_variants_meta":{"raw":{"variants":["Agent optimizer gains compound only with regression control","Only regression-aware loops compound agent-optimizer gains","In continual learning, agent gains compound only with regression control","Two phases prove: only regression control compounds agent gains","On Terminal-Bench 2.0, only regression-aware optimizer compounds"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.0008,"raw_usage":{"total_tokens":3423,"prompt_tokens":882,"completion_tokens":2541,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":626,"completion_tokens_details":{"reasoning_tokens":2462}},"tokens_in":626,"tokens_out":2541,"duration_ms":15683,"temperature":1.0,"reasoning_tokens":2462,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-02T03:01:55.431469+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Repeat the two-phase protocol with many trials per task (say 20) and report confidence intervals on Phase-1, transfer, and final pass rates; if the regression-aware method's transfer and continued-improvement advantages over the other two shrink to noise, the compounding claim fails. A sharper test: run the same regression-aware search with the no-regression rejection disabled but the same search space and budget; if transfer and second-phase gains disappear, the mechanism is confirmed.","supporting_citations":[],"review_version":1}