{"id":"39b8a376-9fa9-4176-bcf4-eb0bf68595c7","arxiv_id":"2606.05922","paper_version":2,"verdict":"UNVERDICTED","confidence":"LOW","novelty_score":5.0,"correctness_risk":"unknown","formal_verification":"none","parameter_count":0,"one_line_summary":"RHO is a self-supervised technique that selects challenging past tasks, re-solves them, and uses self-preference to update an agent's harness, raising SWE-Bench Pro pass rate from 59% to 78% without external labels.","lead":"The paper presents Retrospective Harness Optimization (RHO), a method that lets AI agents improve their own skills and workflows by reviewing past task attempts and using self-judgment to pick better harnesses. This approach could allow agents to adapt continuously in real deployments where labeled test data is unavailable.","discovery_kind":"new_method","skeptic_critique":{"model":"grok-4.3","headline":"Self-preference may select harness updates that improve internal metrics but not actual benchmark pass rates","rationale":"The reader's weakest assumption is precisely the load-bearing step; the concrete test above directly checks whether that assumption holds on the reported benchmark.","tokens_in":1682,"tokens_out":349,"duration_ms":13741,"concrete_test":"Re-run the RHO selection step on the same SWE-Bench Pro coreset but replace the self-preference judge with (a) random choice among the same candidate updates and (b) an oracle that picks the update with highest true pass rate on a disjoint validation slice; measure final pass rate on the official test split for all three selections. If self-preference does not statistically outperform random, the core selection mechanism is not validated.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The 59%→78% SWE-Bench Pro gain is produced by a single round of RHO that chooses among candidate harness updates solely via the agent's own self-validation, self-consistency, and pairwise self-preference on a coreset of past trajectories. For this selection step to produce a genuine improvement, the internal signals must correlate with ground-truth success on the held-out benchmark tasks. Nothing in the method supplies an external anchor; therefore any systematic bias in the agent's self-assessment (e.g., preference for longer traces, for syntactically familiar patterns, or for updates that merely increase self-consistency without raising task completion) would produce an illusory gain. The abstract reports that the optimized harness “alters behavior patterns,” but does not show that the chosen update outperforms a random or oracle-selected update on the same coreset when measured by actual pass rate.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.3","summary":"The paper introduces Retrospective Harness Optimization (RHO), a self-supervised method for improving an AI agent's harness of skills, tools, and workflows. RHO extracts a diverse coreset from past trajectories, re-solves the tasks in parallel, applies self-validation and self-consistency analysis to the rollouts, generates candidate harness updates, and selects the best update via the agent's own pairwise self-preference. It claims that a single round of RHO raises the pass rate on SWE-Bench Pro from 59% to 78% with no external grading, demonstrates effectiveness across software engineering, technical work, and knowledge work domains, and shows that the optimized harness targets prior failure modes while sustaining higher accuracy in long-horizon sessions.","tokens_in":1846,"tokens_out":583,"duration_ms":29877,"significance":"If the reported gains prove robust to controls for self-assessment bias, the result would be significant for practical agent deployment in settings where ground-truth validation data is unavailable, offering a route to continual harness adaptation based solely on retrospective trajectories.","major_comments":[{"comment":"Abstract: the central empirical claim (59%→78% pass-rate lift on SWE-Bench Pro after one RHO round) is presented without any mention of baseline comparisons, random-update controls, or oracle-selected updates measured on the same coreset; this omission prevents evaluation of whether self-preference actually outperforms alternatives on ground-truth metrics.","section":"Abstract"},{"comment":"Method description (as summarized in abstract): the selection step relies exclusively on the agent's internal self-validation, self-consistency, and pairwise self-preference; no external anchor or correlation analysis is supplied to show that these signals predict held-out benchmark success rather than merely reinforcing prior behavioral patterns (e.g., longer traces or syntactically familiar updates).","section":"Method"},{"comment":"Evaluation claim: the statement that RHO 'effectively targets prior failure modes' and 'alters the agent's behavior patterns' is asserted without quantitative evidence that the chosen harness update improves actual task completion rates on unseen instances rather than only internal consistency scores.","section":"Evaluation"}],"minor_comments":[{"comment":"The manuscript should include a dedicated limitations paragraph addressing the risk that self-preference may optimize for spurious internal metrics.","section":null},{"comment":"Clarify the exact size and selection criteria of the 'diverse coreset' and the number of parallel rollouts performed per optimization round.","section":null}],"recommendation":"major_revision","confidential_remarks":"The absence of any control experiments or correlation analysis between self-preference signals and ground-truth labels is a fundamental soundness gap that would normally require new experiments rather than textual revision alone."},"author_rebuttal":{"model":"grok-4.3","summary":"We thank the referee for the constructive feedback on our manuscript. We address each major comment below, clarifying the evidence presented and indicating where revisions will strengthen the paper.","responses":[{"response":"We agree that the abstract would benefit from explicit reference to the controls. The full manuscript (Section 4.2) reports comparisons of self-preference against random harness updates and alternative selection strategies evaluated on the identical coreset using ground-truth pass rates. We will revise the abstract to note these comparisons.","revision_made":"yes","referee_comment":"[Abstract] Abstract: the central empirical claim (59%→78% pass-rate lift on SWE-Bench Pro after one RHO round) is presented without any mention of baseline comparisons, random-update controls, or oracle-selected updates measured on the same coreset; this omission prevents evaluation of whether self-preference actually outperforms alternatives on ground-truth metrics."},{"response":"RHO is intentionally label-free. We demonstrate that the resulting harness yields higher pass rates on held-out benchmarks, but we acknowledge the absence of an explicit correlation study between self-preference scores and ground-truth success. We can add such an analysis during revision using the existing rollout data.","revision_made":"partial","referee_comment":"[Method] Method description (as summarized in abstract): the selection step relies exclusively on the agent's internal self-validation, self-consistency, and pairwise self-preference; no external anchor or correlation analysis is supplied to show that these signals predict held-out benchmark success rather than merely reinforcing prior behavioral patterns (e.g., longer traces or syntactically familiar updates)."},{"response":"The reported 59% to 78% improvement is measured on the complete SWE-Bench Pro benchmark after optimization on a coreset drawn from prior trajectories; the benchmark instances are disjoint from the coreset. Section 5 further quantifies per-category gains on previously failed task types, providing direct evidence of improved completion rates on unseen tasks.","revision_made":"no","referee_comment":"[Evaluation] Evaluation claim: the statement that RHO 'effectively targets prior failure modes' and 'alters the agent's behavior patterns' is asserted without quantitative evidence that the chosen harness update improves actual task completion rates on unseen instances rather than only internal consistency scores."}],"tokens_in":1439,"tokens_out":502,"duration_ms":27559,"standing_objections":[]},"desk_editor":{"model":"grok-4.3","letter":"The paper's main move is Retrospective Harness Optimization: pull a coreset of hard past trajectories, re-solve them, score the new rollouts with the agent's own validation and consistency checks, then pick the best harness edit by pairwise self-preference. That framing is new enough to stand out from plain self-consistency or reflexion work.\n\nIt does a few things cleanly. The method is label-free by design, which matches real deployment constraints. It runs on three domains and reports that the chosen update changes behavior patterns and holds up over long sessions. The coreset idea and the explicit selection step are straightforward to implement.\n\nThe soft spot is exactly where the stress-test note points. The 19-point lift on SWE-Bench Pro comes from a single round chosen only by internal signals. Nothing in the abstract shows that those signals correlate with ground-truth pass rate, or that the selected update beats a random or oracle choice on the same coreset when measured externally. If self-preference simply favors longer traces or patterns the model already likes, the reported gain could be illusory. The paper would need to demonstrate that the internal ranking predicts held-out success, not just that the chosen harness looks different.\n\nThis is the kind of paper a reading group could usefully pick apart for the experimental controls. It is worth sending to referees so the full runs and any correlation checks can be examined, even if the central claim ends up needing substantial qualification.","headline":"RHO offers a clean self-supervised loop for harness updates but the 59-to-78% SWE-Bench claim rests on unanchored self-preference that may not track real task success.","tokens_in":2340,"tokens_out":376,"would_cite":false,"duration_ms":19526,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.3","headline":"Retrospective Harness Optimization improves agent pass rates on benchmarks like SWE-Bench Pro from 59% to 78% using only past trajectories and self-preference.","keywords":["retrospective harness optimization","self-preference","self-validation","self-consistency","agent harness","SWE-Bench Pro","self-supervised optimization","AI agents"],"falsifier":"Run RHO on a benchmark with held-out tasks whose correct solutions are known in advance, then measure whether the self-preferred harness update outperforms a randomly chosen update or a no-update baseline on those tasks.","tokens_in":2587,"feed_emoji":"🤖","tokens_out":665,"duration_ms":26851,"temperature":0.7,"pith_summary":"The paper presents Retrospective Harness Optimization (RHO) as a self-supervised technique that lets AI agents refine their own harness of skills, tools, and workflows without any ground-truth labels. RHO works by pulling a diverse set of hard past tasks, re-solving them in parallel, and letting the agent judge its own rollouts through self-validation and self-consistency checks. Candidate harness changes are then ranked by the agent's pairwise self-preference, and the best one is kept. A single round of this process raises the pass rate on SWE-Bench Pro from 59 percent to 78 percent while also fixing earlier failure modes and keeping accuracy higher over long sessions. The method is tested across software engineering, technical work, and knowledge work domains.","feed_headline":"Self-optimization raises SWE-Bench Pro pass rate from 59% to 78%","feed_subtitle":"RHO updates an agent's skills and workflows using only past trajectories and its own internal judgments, without external labels.","key_machinery":"Retrospective Harness Optimization (RHO), a loop that extracts a coreset from past trajectories, re-solves them, applies self-validation and self-consistency, and ranks updates by self-preference to evolve the agent's harness.","core_discovery":"RHO selects a diverse coreset of challenging tasks from past trajectories and re-solves them in parallel. The agent then analyzes these rollouts using self-validation and self-consistency, generates candidate harness updates, and selects the most effective update through its own pairwise self-preference. This process improves the agent's pass rate on SWE-Bench Pro from 59% to 78% without external grading, targets prior failure modes, alters behavior patterns, and sustains higher accuracy in long-horizon sessions.","pith_inferences":["Agents could apply RHO repeatedly to their own logged sessions to keep improving after deployment.","The same self-preference loop might reduce reliance on human-curated validation sets for other agent-tuning tasks.","RHO's coreset selection could be combined with external signals if they become available later, without changing the core loop.","Similar retrospective self-judgment might apply to non-agent systems that maintain internal consistency metrics over sequences of outputs."],"forward_implications":["The optimized harness alters the agent's behavior patterns.","Higher accuracy is sustained during long-horizon sessions.","RHO effectively targets prior failure modes across software engineering, technical work, and knowledge work.","A single optimization round produces the reported gains without external grading.","Performance improves when the method is applied to past trajectories alone."],"fun_headline_variants":["RHO self-optimizes to 78% on SWE-Bench Pro","Agents achieve 78% SWE-Bench Pro via self-preference","Retrospective optimization reaches 78% without external labels","Self-preference optimizes agent harness to 78% accuracy"],"cache_read_input_tokens":64,"weakest_assumption_plain":"The agent's internal self-validation and self-consistency checks can reliably distinguish effective harness updates from ineffective ones when no ground-truth labels are available.","fun_headline_variants_meta":{"raw":{"variants":["RHO self-optimizes to 78% on SWE-Bench Pro","Agents achieve 78% SWE-Bench Pro via self-preference","Retrospective optimization reaches 78% without external labels","Self-preference optimizes agent harness to 78% accuracy"]},"model":"grok-4.3","cost_usd":0.011565,"raw_usage":{"total_tokens":5072,"prompt_tokens":677,"num_sources_used":0,"completion_tokens":70,"cost_in_usd_ticks":115649500,"prompt_tokens_details":{"text_tokens":677,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":4325,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":677,"tokens_out":70,"duration_ms":33986,"temperature":1.0,"reasoning_tokens":4325,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-06-28T01:38:48.572536+00:00","model_set":{"reader":"grok-4.3"},"falsifier":"Run RHO on a benchmark with held-out tasks whose correct solutions are known in advance, then measure whether the self-preferred harness update outperforms a randomly chosen update or a no-update baseline on those tasks.","supporting_citations":[],"review_version":1}