{"id":"ce6f697e-f3a1-4355-a376-10def1d7419d","arxiv_id":"2610.08432","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A frozen embodied model gets better at navigation and manipulation when the harness around it, the planning and tool-use logic, is iteratively revised from its own experience traces.","lead":"This paper keeps the AI model itself frozen and instead lets the agent's operating instructions, called its harness, improve itself from recorded successes and failures. On a standard embodied-AI benchmark the approach reports success-rate gains of up to 23.3 points in navigation and 33.3 in manipulation.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek/deepseek-v4-flash","headline":"The headline held-out gains are best-over-10-iterations maxima on small samples (n≈30–48, no error bars), not the held-out performance of the harness EMHO's own selection rule actually deploys; the §4.1 generalization claim rests on this single unmeasured quantity.","rationale":"The paper is coherent and honestly reported: the harness is a well-defined editable object, the optimizer prompts and worked diffs (Appendix E, Figures 9–13) are concrete, evaluation is on the fixed EmbodiedBench environment with a frozen model, and the authors explicitly disclose in Appendix B that held-out numbers are retrospective maxima and in Section 6 that the best-search harness is often not the best-held-out harness (b163–b164). The mechanism is a genuine external evaluation, not a circular construction. My concern lands on exactly the reader's weakest assumption: the held-out generalization claim in Section 4.1 (b77) is asserted from max-over-iterations held-out scores, while the method's only selection signal is search success (Eq. 3). Under a null where harness revisions are driven by the trajectories that generated them but transfer nothing to held-out episodes, the maximum of ten correlated held-out evaluations still tends to beat the initial harness, so the headline held-out gains are partly expected under the null. The paper's own limitation statement concedes the emerging-overfitting regime where selection inflation matters most, which is why the missing measurement — the deployed parent's held-out success — is decisive. This is not a contradiction and does not impugn the search-set improvements, which are the method's most solid result; it is simply the gap between what EMHO's selection procedure returns and what is claimed about it. The proposed test is cheap and decisive: evaluating the already-existing iteration-10 parent with a bootstrap CI plus a permutation-based selection-inflation estimate settles whether the generalization claim survives. The reader's CONDITIONAL verdict already anticipates this condition, so my read does not move the verdict. Credit is due where the evidence is independent of this gap: the one-shot-revision baselines, the skill-only ablation in Table 2, and the qualitative behavioral measurements (depth usage ~70%→~100% after blocked moves, valid-next-action 44%→68%, wrong-object selection 50%→34%) are concrete, falsifiable, and support the mechanism even where headline generalization is underdetermined.","tokens_in":27330,"tokens_out":4534,"duration_ms":33256,"concrete_test":"Report the held-out success of the iteration-10 parent harness H_par^(10), which is what EMHO's search-success selection rule (Eq. 3) actually returns, on the same held-out splits as Table 1 for the 27B runs: navigation (5 subtasks × 30 = 150 episodes) and manipulation (5 × 48 = 240 episodes). Give per-subtask and pooled success with a bootstrap 95% CI over episodes. Then compute the expected maximum held-out score of the ten candidates under a permutation null that shuffles candidate held-out outcomes relative to search outcomes, to quantify how much of the reported held-out gain is selection inflation. If the deployed harness's held-out CI contains the initial-harness held-out rate, the generalization claim in §4.1 should be re-scoped to search-set optimization with a pre-registered held-out rule (e.g., last parent, or best candidate under a regression-based validation score).","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 4.1 / Table 1 reports 'EMHO (Held-out)' as the best score over ten optimization iterations, which Appendix B explicitly labels a 'retrospective summary' of candidate performance. The per-subtask denominators are small — navigation percentages move in steps of 3.3 (n≈30), manipulation in steps of 2.1 (n≈48) — and no error bars or repeated seeds are given. The load-bearing gap is that the method's actual selection rule in Eq. (3) keeps a harness only when its search-set success improves; nothing in EMHO selects on held-out performance. Therefore the harness the method would actually deploy, the iteration-10 parent H_par^(10), is not the harness whose held-out number is reported. The central claim (b77: 'the evolved harnesses also achieve higher success than the initial harness on held-out episodes') is asserted from a maximum over unselected candidates. Under a null model in which harness revisions improve search outcomes only through the trajectories that generated them and transfer nothing to held-out episodes, the maximum of ten correlated held-out evaluations would still tend to exceed the initial harness, so the headline held-out gains are expected under that null to some unquantified degree. The authors concede the mechanism this inflation rides on (Section 6, b163–b164: later iterations improve search while held-out plateaus or declines), which makes the missing quantity — held-out success of the returned harness — decisive rather than cosmetic.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek/deepseek-v4-flash","summary":"The paper introduces EMHO, a framework that keeps an embodied vision-language model frozen and iteratively revises the surrounding 'harness' (planning prompts, tool orchestration, progress monitoring, memory, validation logic) using execution traces and prior revision history, with a variant EMHO-Merge for jointly optimizing a single harness across subtasks. The method is evaluated on EmbodiedBench navigation and manipulation with Qwen 9B and 27B models, reporting search-set and held-out task-success improvements, ablations of the optimization context, and qualitative analyses of discovered edits. The search-set gains are externally validated by environment success, and the paper includes a detailed appendix of harness modifications.","tokens_in":27657,"tokens_out":3093,"duration_ms":20239,"significance":"If the held-out generalization claim were supported, this would be a notable contribution: it would show that harness-level revision alone can materially improve a frozen embodied agent, and that the improvement goes beyond skill or recovery-prompt curation to include tool selection and progress monitoring. The search-set improvements, e.g., Qwen3.8-27B manipulation search success rising from 23.9% to 44.2%, are credible and externally grounded. The primary weakness is the evidence for generalization to held-out episodes, which rests on a quantity the paper itself labels a retrospective summary rather than the performance of the harness the method actually returns.","major_comments":[{"comment":"The central held-out claim (claim b77: 'the evolved harnesses also achieve higher success than the initial harness on held-out episodes') is not supported by the reported numbers. The 'EMHO (Held-out)' entries in Table 1 are, as Appendix B states, the best scores over the ten optimization iterations, not the held-out performance of the harness that EMHO's own selection rule (Eq. 3) would deploy. Since the optimizer selects a parent only by search-set success, the iteration-10 parent is the actual returned harness, and its held-out performance is never reported. Under a null model in which harness revisions transfer nothing to held-out episodes, the maximum of ten correlated held-out evaluations would still tend to exceed the initial harness. The paper must report the held-out success of the final returned harness (and ideally its standard error across repeated optimization runs) before the generalization claim in the abstract, §4.1, and Conclusion can be made.","section":"§4.1, Table 1; Appendix B","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"No uncertainty quantification is provided. Per-subtask denominators are small (navigation percentages move in steps of 3.3, suggesting n≈30; manipulation in steps of 2.1, suggesting n≈48), and no error bars, confidence intervals, or repeated-seed results are given. Many reported held-out gaps are within one or two episodes of the initial-harness value (e.g., the Qwen3.5-9B navigation 'Complex' held-out gain is 65.0 vs. 58.3, a 2-episode difference; the Qwen3.8-27B manipulation 'Base' held-out gain is 41.7 vs. 25.0, a 4-episode difference). The strength of the claims, especially the held-out generalization claim, depends on separating genuine transfer from sampling noise, and the current evidence cannot do so.","section":"§4.1, Tables 1–3; Appendix B","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The abstract and introduction state 'gains of up to 23.3% in navigation and 33.3% in manipulation' without specifying that these are search-set gains on single subtasks (the 23.3% figure is the Qwen3.5-9B Common Sense search-set gain; the 33.3% is the Qwen3.8-27B Base search-set gain). As written, the claim appears to describe held-out or average performance. The qualifier 'search set' should be added, and the only claim for held-out performance should be tied to the actually returned harness.","section":"Abstract, §4.1","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The paper's own Section 6 acknowledges that later iterations improve or maintain search performance while held-out performance plateaus or declines, and Appendix B labels the held-out maxima as 'retrospective summaries'. These acknowledgments are present but are not integrated into the abstract or conclusion, where the unqualified held-out claim appears. The authors should either report the final-harness held-out number (if it indeed improves) or revise the claims to state that only the best-observed candidate, not the method's selection, improves held-out performance.","section":"§6, Appendix B","authors_acknowledge":"These results motivate the need to consider generalization when selecting among harness iterations. 9 7 CONCLUSION We introduced EMHO, a framework for improving embodied agents by evolving the harness around a frozen underlying model from experience traces and sparse environmental feedback.","authors_acknowledge_p":0.75}],"minor_comments":[{"comment":"The manipulation held-out results show EMHO-Merge (avg 33.8) is worse than EMHO (Separate) (avg 40.7), yet the conclusion says EMHO-Merge 'mitigates trade-offs'. The claim is only supported on the search set; the held-out comparison should be reported to avoid overstating the merge benefit.","section":"§4.4, Table 3","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"In the navigation held-out split, EMHO (Separate) and EMHO-Merge have the same average (75.6), with the merge benefit coming almost entirely from the Base subtask (93.3 vs. 86.7). The claim that EMHO-Merge better manages cross-subtask trade-offs would be strengthened by a discussion of the per-subtask variance.","section":"§4.4, Table 3","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The qualitative percentages (e.g., 'repeated grasps decrease from 30% to 26%') are reported without any indication of the number of episodes or statistical significance; as descriptive statistics they are fine, but they should be clearly labeled as exploratory and not as evidence of a quantitative improvement.","section":"§5.1","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The claim that the evolved 9B navigation harness 'reaches performance competitive with larger published models' should specify which models and which split (search vs. held-out) are being compared; the italicized numbers in Table 1 are maxima over iterations, so the comparison is not apples-to-apples.","section":"§4.1","authors_acknowledge":null,"authors_acknowledge_p":null}],"recommendation":"major_revision"},"author_rebuttal":null,"desk_editor":{"model":"deepseek/deepseek-v4-flash","letter":"Worth a careful read, with one significant caveat. The search-set gains are plausible and the method is coherent, but the headline held-out gains are best-over-ten-iterations maxima, not the held-out performance of the harness EMHO's own selection rule would return. Appendix B says this plainly—held-out results are “retrospective summaries of candidate performance”—and Section 6 concedes search gains come with held-out plateaus. The abstract and Section 4.1 still present those maxima as “EMHO improves task success… on held-out episodes,” which overstates what has been measured.\n\nWhat is genuinely new: the transfer of trace-guided harness optimization from coding agents to embodied agents, with the full harness—planning, context, memory, vision-tool orchestration, validation, recovery—as the editable object and the model frozen. The prompts and diffs in the appendices are concrete, the optimizer constraints (no heldout data, no hidden simulator state, regression-aware editing) are sensible, and the qualitative analysis distinguishes recovery edits from progress-monitoring, tool-selection, and grounding changes. The skill-only comparison in Table 2 supports the claim that optimizing the whole harness buys more than adding recovery skills. Related work is fairly positioned.\n\nSoft spots, in order of importance. First, the held-out generalization claim is load-bearing and unmeasured. Equation (3) accepts a candidate only if search success improves; nothing selects on held-out. The reported held-out numbers are maxima over candidates at possibly different iterations, so the harness a user would actually deploy is not the harness whose number is in the table. Under the authors' own acknowledged overfitting pattern, the maximum of ten correlated held-out evaluations is expected to look better than the initial harness even if no revision transfers. The fix is simple: report the held-out success of the final accepted harness, with repeated runs or confidence intervals.\n\nSecond, sample sizes are small—navigation percentages move in 3.3-point steps (n≈30 per subtask), manipulation in 2.1-point steps (n≈48)—and no repeated seeds appear anywhere. The strongest search-set deltas (e.g., 23.9 to 44.2) are large enough to survive that noise, but the 10–15 point held-out claims are shakier. Third, EMHO-Merge's shared-harness advantage is shown only on the search set; its held-out manipulation average is below EMHO Separate, so gain-loss repair's generalization benefit is not established. Fourth, the code link is a placeholder.\n\nCitation pattern is fine; they engage with the coding-harness and embodied self-improvement lines. The math is simple but the loop is honestly benchmarked against a fixed external environment. This deserves a serious referee, with the request to report the selected harness's held-out performance before the headline claims can be taken at face value.","headline":"Coherent harness-optimization method with credible search-set gains, but the headline held-out numbers are retrospective maxima over ten candidates, not the held-out performance of the harness the method would actually deploy.","tokens_in":28503,"tokens_out":2406,"duration_ms":29470},"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek/deepseek-v4-flash","headline":"This paper claims that the scaffolding around an embodied agent — the prompts and runtime logic controlling what it observes, plans, and does — can be improved automatically from execution traces alone, with the underlying…","keywords":["embodied agent","harness optimization","experience traces","frozen model self-improvement","EmbodiedBench","tool orchestration","failure recovery","EMHO-Merge"],"falsifier":"Run the same ten-iteration EMHO procedure several times from scratch, then take the single harness that the search criterion would actually select and evaluate it on the held-out episodes with confidence intervals; if across seeds the search-selected harness shows no significant held-out gain over the initial harness, the paper's central generalization claim fails even though its optimization story is intact.","tokens_in":28701,"feed_emoji":"🤖","tokens_out":4789,"duration_ms":28067,"temperature":0.7,"pith_summary":"The paper claims that the scaffolding around an embodied agent — the prompts and runtime logic that decide what it observes, plans, and does — can be improved automatically from nothing but past execution traces and a binary success signal, with the underlying vision-language model left completely frozen. On EmbodiedBench navigation and manipulation tasks, this harness evolution raises task success by up to 23.3 and 33.3 percentage points respectively for both a 9B and a 27B Qwen model. The claim matters because it suggests that a meaningful share of embodied-agent capability lives outside the model weights, in the system that wraps them, and that this share can be harvested without any training. A companion mechanism, EMHO-Merge, extends the same idea to a single harness shared across several subtasks by using episode-level gains and losses to decide when and how each revised behavior should apply.","feed_headline":"Traces alone lift frozen embodied agents up to 33 percent","feed_subtitle":"Rewriting a frozen model's planning wrapper lifts navigation and manipulation success on EmbodiedBench.","key_machinery":"The central mechanism is a two-loop optimization over harnesses. In the inner loop the agent executes the current harness on a search set and records trajectories of observations, plans, tool calls, actions, feedback, and final outcomes; in the outer loop a harness optimizer — the same frozen model — reads those traces together with the complete history of previous harness edits and their evaluation scores, identifies recurring behavioral weaknesses, and writes targeted edits to harness files such as planner prompts, the tool orchestrator, memory rendering, and repair logic. A candidate replaces the parent only when its search-set success rate is higher. EMHO-Merge adds gain-loss refinement: comparing a candidate with its parent exposes episodes that changed from failure to success (gains) and success to failure (losses), and the optimizer examines the paired trajectories to find an observable condition separating benefit from harm, then narrows the trigger of the revised behavior. The paper reports that this gain-loss information is what lets a single shared harness retain useful adaptations across subtasks while limiting regressions.","core_discovery":"On its own terms, the paper establishes that a frozen embodied model's task performance can be materially improved by iteratively editing the harness — the instructions and runtime code controlling planning, context and memory, vision-tool selection, and failure recovery — guided by experience traces alone. In ten optimization iterations, with the same model acting as both agent and harness optimizer, EMHO improves search-set success over the initial EmbodiedBench harness in both navigation and manipulation for the 9B and 27B Qwen models, and the paper reports that the evolved harnesses also outperform the initial harness on held-out episodes. The improvement is not only better failure recovery: the qualitative analysis shows the evolved harness reshapes progress monitoring, perceptual-tool orchestration, and visual grounding, converting recurring failure patterns into concrete operating conditions rather than generic advice. The paper further claims that EMHO-Merge, which contrasts episodes that turn from failure to success with those that turn from success to failure and repairs the revision's triggering condition, manages the cross-subtask trade-offs of a shared harness better than simply pooling experience.","pith_inferences":["The divergence the authors report between search and held-out maxima points to a selection problem: reporting the best held-out candidate inflates expected deployment performance, and a practical pipeline would need a validation-based rule for choosing which harness iteration to deploy.","The same trace-guided loop could extend beyond prompt text to parser validation, memory-eviction policies, or the agent's own tool inventory; the redundancy-guard edit in the appendix already shows the mechanism editing runtime logic, not just instructions.","If gain-loss refinement scales to more subtasks, it amounts to a weak form of credit assignment over behaviors rather than episodes: the harness learns which triggers are safe, a step toward compositional harness policies.","With per-subtask denominators near thirty episodes, the double-digit percentage gains represent only a handful of episodes each, so an immediate check is whether the qualitative behaviors, such as depth-call gating, reproduce at larger scale."],"forward_implications":["Harness evolution becomes a cheap alternative or complement to model training: any frozen model can be made more capable on embodied tasks using only logged trajectories and a success signal.","Self-improvement methods that rely on explicit failures can be ported to settings with sparse, delayed feedback, as long as the harness writes and executes concrete operating rules rather than broad advice.","Failure recovery is the easiest first target, while progress monitoring, tool orchestration, and grounding rules arrive later, suggesting a natural curriculum for automated harness design.","If the held-out gains hold, measurable agent capability can live outside model weights, and benchmarks should report harness quality alongside model quality."],"supporting_citations":[{"why":"Supplies the EmbodiedBench benchmark, the navigation and manipulation tasks, the initial harness baseline, and the evaluation protocol.","marker":"(Yang et al., 2025)"},{"why":"Meta-Harness provides the coding-agent template of searching over executable harness code using traces and scores that EMHO adapts to embodied settings.","marker":"(Lee et al., 2026)"},{"why":"Self-Harness contributes the pattern of identifying model-specific weaknesses and proposing targeted modifications retained through regression-based validation.","marker":"(Zhang et al., 2026)"},{"why":"EmbodiSkill is the skill-only trajectory-reflection baseline that full-harness optimization is compared against in Table 2.","marker":"(Ju et al., 2026)"},{"why":"ASPIRE is a comparison baseline that evolves task-specific code and accumulates successful repairs as reusable skills.","marker":"(Lu et al., 2026)"},{"why":"Zetta is a comparison baseline focused on failure detection and recovery by evolving runtime critics and recovery behaviors.","marker":"(Ding et al., 2026)"},{"why":"Depth Anything 3 is one of the vision tools the harness learns to orchestrate, particularly for obstacle and proximity reasoning.","marker":"(Lin et al., 2025)"},{"why":"SAM 3 is the segmentation tool the harness learns to gate, using it only when a specific object target requires visual grounding.","marker":"(Carion et al., 2026)"}],"fun_headline_variants":["Frozen agents lift 33% by rewriting their own harness","Self-editing harness boosts frozen embodied agents 33%","Traces alone reshape frozen agent harness for 33% gain","Harness self-evolution lifts frozen agents up to 33%","Frozen models gain 33% from experience-trace harness edits"],"cache_read_input_tokens":0,"weakest_assumption_plain":"The load-bearing premise is that harness revisions chosen on the search set generalize to never-seen episodes: the reported held-out gains are the best over ten optimization iterations on per-subtask sets of roughly thirty episodes with no error bars, and the authors themselves acknowledge that the best search-set harness is often not the best held-out harness.","fun_headline_variants_meta":{"raw":{"variants":["Frozen agents lift 33% by rewriting their own harness","Self-editing harness boosts frozen embodied agents 33%","Traces alone reshape frozen agent harness for 33% gain","Harness self-evolution lifts frozen agents up to 33%","Frozen models gain 33% from experience-trace harness edits"]},"model":"deepseek/deepseek-v4-flash","effort":"low","cost_usd":0.000143,"raw_usage":{"cost":0.000142688,"is_byok":false,"cost_details":{"upstream_inference_cost":0.000142688,"upstream_inference_prompt_cost":0.000105728,"upstream_inference_completions_cost":3.696e-05},"total_tokens":1092,"prompt_tokens":960,"completion_tokens":132,"prompt_tokens_details":{"audio_tokens":0,"video_tokens":0,"cached_tokens":256,"cache_write_tokens":0},"completion_tokens_details":{"audio_tokens":0,"image_tokens":0,"reasoning_tokens":58}},"tokens_in":960,"tokens_out":132,"cost_source":"openrouter_usage","duration_ms":2002,"image_pages":[],"temperature":1.0,"billed_attempts":1,"reasoning_tokens":58,"transport_provider":"openrouter","cache_read_input_tokens":0,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-10-07T10:27:32.465594+00:00","model_set":{"reader":"deepseek/deepseek-v4-flash"},"pipeline_version":"pith-pipeline@v1.4.1-daily-deepseek-or-medium","input_kind":null,"input_chars":null,"input_total_chars":null,"passes_missing":null,"published_standing":null,"published_confidence":null,"objection_audit":null,"run_record":null,"basis":{"kind":"not_recorded","source":null,"chars_given":null,"chars_total":null},"receipt":null,"falsifier":"Run the same ten-iteration EMHO procedure several times from scratch, then take the single harness that the search criterion would actually select and evaluate it on the held-out episodes with confidence intervals; if across seeds the search-selected harness shows no significant held-out gain over the initial harness, the paper's central generalization claim fails even though its optimization story is intact.","supporting_citations":[],"lane_accuracy":{"status":"not yet measured","lane":{"model":"deepseek/deepseek-v4-flash","input_kind":null},"url":"https://pith.science/accuracy"},"review_version":1}