{"id":"f7b57e20-f620-4224-91eb-b312c1c475c7","arxiv_id":"2610.20474","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Changing the harness first and fine-tuning only the sub-steps that remain unreliable raises RoboCasa365 unseen-composite success from 4.6% to 13.8% with 28% of the GPU time of fine-tuning every sub-step.","lead":"Robots that try to improve themselves usually either rewrite their own instructions or retrain their model. This paper tests a middle path: improve the instructions first, then retrain only the exact step that still fails, and it roughly tripled success on new multi-step kitchen tasks in simulation.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek/deepseek-v4-flash","headline":"Pilot tasks are improved without any training limit (§4.3) yet are included in the all-50 headline results (Table 1), mixing evaluation protocols; non-pilot breakouts exist only for Composite-Unseen (Table 8). The claimed gains and cost ratios need to be re-reported with pilot tasks excluded.","rationale":"The reader's weakest_assumption is the attribution step: \"executor-limited\" is defined relative to the agent's own search (§4.2), so a weak search could mislabel harness-fixable failures as executor-limited. I agree this is real, but the paper already acknowledges it in §4.2 and §6, and all compared systems share the same agent search, so it bounds absolute performance rather than threatening the relative comparison against always-train or lowest-first. The more actionable weakness is the pilot-task protocol: §4.3 explicitly allows unlimited training on pilot tasks, §5.1 places pilots in every suite, and Table 1 reports all 50 tasks without separating them. If pilot tasks are excluded, the headline numbers may shift, and the cost advantage is part of the central claim. Table 8 provides one non-pilot breakout for Composite-Unseen and it supports the result, which is why I would not reject or downgrade the paper; the concern should be resolved by reporting the same breakout for all suites and for the cost tables. This matches the reader's conditional request to list pilot tasks, though it is not the reader's named weakest assumption, hence \"partial\" agreement. The verdict stays conditional pending that reporting.","tokens_in":19449,"tokens_out":12236,"duration_ms":89704,"concrete_test":"Publish the pilot-task list per suite and recompute Table 1, Table 5 and Table 7 restricted to the non-pilot tasks, the same operation already performed for Table 8 on Composite-Unseen. If the suite-level gains and the cost ratios persist outside the pilot tasks, the headline stands; if the gains concentrate in the pilot tasks, the abstract's \"on all 50 target tasks\" claim must be revised to non-pilot tasks or re-evaluated under a single budget-limited protocol.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 4.3 says \"The pilot tasks are improved without any limit on training\" and that their logs set the experience document and training budget for later tasks; Section 5.1 confirms pilots exist in each suite and are among the 50 target tasks summarized in Table 1. The headline comparisons—48.8% vs 42.4% overall, 13.8% vs 4.6% on Composite-Unseen, and 28% GPU / 22% demos relative to always-training—therefore pool budget-limited non-pilot tasks with uncapped pilot tasks. The pilot set is not listed, and pilots were selected by the authors on development data to span high, moderate and low base-VLA success (§4.3), so they are not a random subset. A reader cannot tell how much of the gain comes from tasks that received unlimited training and whose logs shaped the experience document. The paper isolates non-pilot Composite-Unseen in Table 8 (Agent+harness 12.0% to AdaRSI 16.6%), which suggests the on-demand result survives there, but no equivalent non-pilot breakouts are given for Atomic-Seen, Composite-Seen, or the cost comparison in Table 7. This is an evaluation-protocol confound in the headline empirical claim, not merely the acknowledged attribution limitation of Sec. 6: even with perfect attribution, the \"all 50\" claim is unverifiable until pilot and non-pilot results are separated.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek/deepseek-v4-flash","summary":"The paper asks when a self-improving embodied system should change the weights of its frozen vision-language-action executor. It proposes AdaRSI, in which a coding agent first refines the harness (skills, tools, procedure) around a frozen VLA on development episodes, then attributes each persistent sub-step failure by counterfactually retrying the sub-step from saved states under alternative instructions and approaches, and fine-tunes only sub-steps that remain unreliable, using self-collected demonstrations and a validation/adoption rule for every new policy. Pilot tasks are improved with unlimited training to produce an experience document and a training budget; all later tasks are improved under that budget. On the 50 target tasks of RoboCasa365 with GR00T N1.5 and no human demonstrations, the paper reports improvement on all three suites (42.4% to 48.8% overall; Composite-Unseen 4.6% to 13.8%) and claims slightly higher success than always-train at 28% of the GPU time and 22% of the demonstrations. A lowest-first training control under the same budget gains nothing over the harness alone, which the paper presents as evidence for the attribution step.","tokens_in":19938,"tokens_out":15211,"duration_ms":109708,"significance":"If the reported results survive the pilot/non-pilot re-analysis, this is a solid empirical contribution: a clear, falsifiable design principle (attribute before training; harness first) instantiated on a standard benchmark with unusually careful evaluation hygiene. I credit the paper for testing on the official target split with disjoint development/validation/test scenes and held-out starts, for paired sub-step tests from identical states (Table 4) that directly measure what the weights change, for cost accounting that includes trained-but-unadopted policies, and for two genuinely informative external controls on the attribution claim (agent-without-harness and lowest-first under the same budget, Table 8). The case studies in Table 9 honestly document training failures, and Sec. 6 explicitly acknowledges the dependence of the method on the coding agent's capability. That said, the headline result is currently unverifiable as stated because pilot tasks trained without any budget limit are pooled into the main tables, and several load-bearing contrasts are reported without any uncertainty quantification. The paper's own limitation section does not mention the pilot-pooling issue.","major_comments":[{"comment":"The pilot-task protocol confound is real and load-bearing. Section 4.3 states that pilot tasks are 'improved without any limit on training,' §5.1 states that pilots exist in each suite, and Tables 1 and 7 pool all 50 tasks (respectively all 16 Composite-Unseen tasks) into the headline numbers; at least GarnishPancake and HeatKebabSandwich are identifiable as Composite-Unseen pilots from Table 9, and the pilots were selected on development data to span base-VLA success levels. The headline claims (48.8 vs 42.4 overall, 13.8 vs 4.6 on Composite-Unseen, 28% GPU time and 22% demonstrations vs Always train) therefore mix budget-limited non-pilot tasks with uncapped pilot tasks, and the pilot set is never enumerated. To the authors' credit, the two identifiable Composite-Unseen pilots contribute no successes in Table 2, so the confound's magnitude is unclear rather than certainly harmful, and the non-pilot breakout in Table 8 (harness 12.0% to AdaRSI 16.6%) suggests the on-demand result survives for that suite. Nevertheless, this must be fixed before the central claim can be verified: enumerate the pilot tasks per suite, and re-report the results of Tables 1, 2, and 7, including the 3.5×/4.5× cost ratios, on non-pilot tasks alone or side-by-side with the pooled numbers.","section":"§4.3, §5.1, Tables 1, 2, 7","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"Suite-level and task-level comparisons are reported as point estimates with no confidence intervals or paired tests. Several load-bearing contrasts are small relative to 50-scene-per-task sampling noise: the weight-round increments on Atomic-Seen (+1.4 pp) and Composite-Seen (+1.0 pp) in Table 1, the 'slightly higher success' of AdaRSI over Always train (13.8 vs 12.6 in Table 5), and the lowest-first versus harness comparison (11.8 vs 12.0 in Table 8). Please report per-task paired differences with intervals (bootstrap or exact binomial tests), so the reader can see which of the small suite-level increments are distinguishable from noise; per-task tables such as Table 2 would also benefit from flags on differences of a few points that are within noise.","section":"§5.2, §5.3, Tables 1, 5, 8","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The definition of an executor-limited sub-step as one for which 'no harness the agent can find' lets the executor perform it reliably folds the coding agent's search capability into the ground truth. The paper explicitly acknowledges this dependence in Sec. 6, and the lowest-first ablation in Table 8 is the right kind of external control, showing that selecting the same sub-steps without retrying from other states or instructions gains nothing. What remains unresolved is quantitative: the paper never reports how often the retry probe actually found a working instruction or approach (that is, how often attribution correctly redirected effort back to the harness), nor the number of probe attempts per trained sub-step. Please report these probe statistics and, if feasible, the sensitivity of the chosen training set to the probe budget; without this, the reader cannot tell how often the weight round trained a failure that was in fact harness-callable by a stronger search.","section":"§4.2, §6, Table 8","authors_acknowledge":null,"authors_acknowledge_p":null}],"minor_comments":[{"comment":"Report how many tasks in the 'lowest first' arm actually trained at least one policy; with four tasks exhausting the development session before any policy was trained, the comparison partly measures failed sessions rather than selection quality alone.","section":"§5.3, Table 8","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The statement that the counts of Always train include the policies of AdaRSI, which it reuses, should appear in the cost comparison itself; since AdaRSI's 966 GPU minutes are counted in both columns, the marginal cost of the extra training is about 2.5× (3,411 minus 966 minutes), so the reported 3.5× is conservative and should be labeled as such.","section":"§5.3, Table 7","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"Clarify that 'the official protocol' refers to the evaluation protocol (target split, task horizons, success check, 50 test scenes), because the improvement phase, with privileged snapshots, saved states, and uncapped pilot training, is not part of that protocol and should be described as an additional development procedure.","section":"§5.1","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"Specify the two rules that currently drive the whole pipeline: the reliability threshold for declaring a sub-step executor-limited (and for adopting a policy), and the fixed rule that maps pilot training logs to the per-task training budget; both are described only qualitatively, which limits reproducibility.","section":"§4.2, §4.3","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The 'training cost' axis counts GPU minutes and demonstrations but excludes the language-model tokens of the improvement phase, which are large (154M input tokens per Composite-Unseen task); state explicitly in the figure caption and the cost section that the comparison is in training compute, with LM-token costs reported separately.","section":"§5.4, Figure 4","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"Document the exact model versions and decoding settings for the coding agent (GPT-6.1-sol via Codex) and the two GR00T N1.5 checkpoints, to aid reproducibility of the agent-harness component.","section":"§5.1","authors_acknowledge":null,"authors_acknowledge_p":null}],"recommendation":"major_revision"},"author_rebuttal":null,"desk_editor":{"model":"deepseek/deepseek-v4-flash","letter":"The paper asks the right question—when should a self-improving embodied system change the executor's weights instead of the harness around it—and answers with a mechanism that is actually new: save state, retry the failing sub-step with different instructions and approaches, and train only if it stays unreliable. The \"lowest first\" ablation is the strongest evidence: same budget, same harness, same number of trained sub-steps, no attribution, and it gains nothing over the harness alone. That control keeps the on-demand claim from being circular.\n\nThe experiments are mostly careful. Official RoboCasa365 protocol, disjoint scenes, held-out starts, paired sub-step tests from identical states, and cost accounting that includes trained-but-unadopted policies. The paired tests—fine-tuning teaches missing motions like the microwave door, degrades motions pi0 already completes—are the cleanest evidence, and the paper honestly reports the CuttingToolSelection regression and discusses it in Section 6.\n\nThe stress-test concern checks out. Section 4.3 says pilots run without any training limit and their logs set the budget; Section 5.1 confirms pilots are part of the 50 tasks; Table 1 pools them. So the headline 48.8% vs 42.4%, and the 28%/22% cost ratios, mix uncapped pilots with budget-limited tasks, and the pilots were chosen to span easy-to-hard, not randomly. The only non-pilot breakout, Table 8 on Composite-Unseen, is reassuring (16.6% vs 12.0% for the harness alone, and the budget-matched lowest-first rule gains nothing), but there are no equivalents for the other suites or the cost comparison. This must be fixed; I don't think it is fatal.\n\nSmaller issues: no confidence intervals or paired tests anywhere, on per-task counts that are often 0–7 out of 50; no code, prompts, or logs; and the harness is written by a proprietary model (GPT-6.1-sol via Codex), so the pipeline is only partly reproducible. The attribution caveat—\"executor-limited\" is defined relative to what this agent can find—is real, but the authors state it in Section 6, and the lowest-first control gives it teeth.\n\nFor people working on VLA adaptation, agentic harnesses, and self-improving robots, this is worth a serious referee. The question is well posed, the non-pilot evidence supports the principle, and the fixes are clear: list the pilot tasks, re-report all main numbers with pilots excluded, add intervals, and release the harness and logs. Send it to review.","headline":"A sound empirical answer to when an embodied system should retrain its executor, with a genuinely new attribution mechanism and a strong control—but the headline numbers pool uncapped pilot tasks with budget-limited ones and need re-separation.","tokens_in":20443,"tokens_out":6421,"duration_ms":46044},"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek/deepseek-v4-flash","headline":"A self-improving robot should change its executor's weights only for sub-steps that stay unreliable after every other way of calling the executor has failed; the paper shows this rule triples unseen-task success at a fraction of the…","keywords":["recursive self-improvement","vision-language-action models","embodied agents","failure attribution","on-demand fine-tuning","harness refinement","RoboCasa365","long-horizon tasks"],"falsifier":"Take a sub-step that AdaRSI labels executor-limited, such as opening the microwave door in WaffleReheat, and search over instructions, robot poses, and approach trajectories far more exhaustively than the paper's agent did while keeping the base policy frozen; if any such search makes the sub-step reliably succeed without a weight change, the attribution step has misclassified a harness failure as a missing motion, and the benefit of training on demand depends on the coding agent's search capability rather than on what the executor can actually do.","tokens_in":20951,"feed_emoji":"🤖","tokens_out":12887,"duration_ms":101502,"temperature":0.7,"pith_summary":"When a long-horizon robot task fails midway, the failure itself does not say whether the robot was called the wrong way or simply cannot perform the required motion. AdaRSI's answer is to improve the calling layer first, an agent-written harness of tools, instructions, and procedures, and to change the executor's weights only for sub-steps that stay unreliable after every alternative way of calling has been tried. The paper's evidence is that this on-demand rule works: on all 50 target tasks of the RoboCasa365 benchmark, with a GR00T N1.5 vision-language-action model as executor and no human demonstrations, success improves on every suite, most sharply on unseen composite tasks, from 4.6% to 13.8%. Compared with fine-tuning every sub-step, on-demand training reaches slightly higher success while using 28% of the GPU time and 22% of the self-collected demonstrations. The load-bearing insight is that attribution, deciding whether a failure belongs to the harness or to the executor, is what makes training pay off, because it also determines the states from which a missing motion is learned.","feed_headline":"Train only the motions a robot lacks; unseen tasks triple","feed_subtitle":"On all 50 RoboCasa365 tasks, on-demand fine-tuning beats training everything at 28% of the GPU cost.","key_machinery":"The mechanism that carries the argument is failure attribution by counterfactual re-execution. From saved simulator snapshots, the agent replays a failing sub-step from the same state under different instructions, poses, and approaches, and a sub-step that no such retry makes reliable is labelled executor-limited, separating a poor way of calling the executor from a motion the executor lacks. The harness that precedes this step is not merely a way to fix calling failures; it sets the states from which the executor is called, so a policy trained afterward learns the missing motion from the states in which it will actually be used. Validation completes the loop: every fine-tuned policy is compared with the base policy on held-out starts, and the procedure records recommendations so a new policy is called only where it is known to help, with the base policy kept as the default elsewhere.","core_discovery":"The paper's central claim is a principle: a self-improving embodied system should change the executor's weights only for a sub-step that stays unreliable after the agent has tried other ways of calling the executor, such as other instructions, robot poses, or approaches. It instantiates this as AdaRSI, a loop in which a coding agent first refines its harness, then attributes persistent failures by retrying failing sub-steps from saved simulator states, and finally fine-tunes only the executor-limited steps, those no harness the agent can find lets the base policy perform reliably, using demonstrations it collects itself. Each new policy is measured against the base policy from held-out states and is called only where it helps, so a weight change enters the system only after both attribution and validation. Empirically, on all 50 target tasks of RoboCasa365 with GR00T N1.5 as the executor and no human demonstrations, AdaRSI improves success on all three suites, from 4.6% to 13.8% on unseen composite tasks and from 75.4% to 79.7% and 43.0% to 49.0% on the seen suites, while reaching slightly higher success than fine-tuning every sub-step at 28% of the GPU time and 22% of the self-collected demonstrations.","pith_inferences":["Beyond the paper: the executor-limited set is defined relative to the coding agent's search capability, so a stronger searcher should shrink it; a testable prediction is that shifting budget from fine-tuning toward the retry search would move the optimal harness/weight split toward the harness at the same total cost.","Beyond the paper: the same attribute-before-train rule applies to any system with a frozen policy and a separate caller, such as an LLM tool-user or a planner, where a failure's cause should be probed by changing the call before changing the model; the saved-state retry here is the embodied instance of that test.","Beyond the paper: the paired-state measurements suggest fine-tuning can degrade sub-steps that already work, so the method needs not only a gate that admits trained policies but a stopping rule for the retry search; monitoring how fast the retry success rate saturates with effort would provide one."],"forward_implications":["Weight changes should be the last resort, not the default: fine-tuning every sub-step reaches lower success than on-demand training (12.6% vs 13.8% on Composite-Unseen) while using 3.5x the GPU time and 4.5x the demonstrations, and it can degrade sub-steps the executor already completes, as WaffleReheat falls from 38% to 28% when every sub-step is retrained.","Attribution decides not only which sub-steps to train but from which states to learn them: a rule that trains the lowest-success sub-steps first, under the same budget, gains nothing over the harness alone even though it selects mostly the same sub-steps (11 of 13).","The harness is the larger lever: on unseen composite tasks it more than doubles success, from 4.6% to 9.9%, before any training, and it makes later fine-tuning effective by fixing the poses and calls from which the executor operates.","Fine-tuning carries the tasks the harness cannot solve, such as opening a microwave door the base VLA never opens on its own (WaffleReheat, 0% to 38%), and there the trained weights decide whether the task succeeds.","Improvement compounds across rounds and tasks: an experience document and a training budget derived from pilot tasks guide later tasks, so the system improves not only itself but also the way it improves."],"supporting_citations":[{"why":"RoboCasa365: the benchmark whose 50 target tasks, three suites, and official protocol generate every reported success number.","marker":"[20]"},{"why":"The pretrained vision-language-action model used as the base executor that every compared system starts from and that AdaRSI fine-tunes.","marker":"[21]"},{"why":"Supplies the recursive self-improvement principle that every self-modification must be validated empirically, which AdaRSI's policy-adoption rule implements.","marker":"[37]"},{"why":"Learning by cheating: the privileged-information scheme the agent uses to generate and label its own demonstrations from simulator state.","marker":"[2]"},{"why":"The formal statement of learning using privileged information that grounds the teacher-student labelling of self-collected demonstrations.","marker":"[28]"},{"why":"The authors' earlier skill-harness evolution work that AdaRSI's harness rounds build on and extend.","marker":"[31]"}],"fun_headline_variants":["Retrain only the sub-steps a robot cannot fix by retrying","Robot fine-tunes only its stuck motions; unseen composite success nearly triples","AdaRSI: fine-tune only sub-steps still unreliable after retries","On-demand weight tuning triples unseen robot task success at 28% GPU","Change robot weights only for failures retrying cannot fix"],"cache_read_input_tokens":0,"weakest_assumption_plain":"The load-bearing premise is that the agent's search over instructions, poses, and approaches is a sound test of what the executor can do: a sub-step is judged executor-limited only if no harness this agent can find makes it reliable, so a failure to discover a working call is counted as a missing motion, a dependence on agent capability the paper itself notes in its discussion.","fun_headline_variants_meta":{"raw":{"variants":["Retrain only the sub-steps a robot cannot fix by retrying","Robot fine-tunes only its stuck motions; unseen composite success nearly triples","AdaRSI: fine-tune only sub-steps still unreliable after retries","On-demand weight tuning triples unseen robot task success at 28% GPU","Change robot weights only for failures retrying cannot fix"]},"model":"deepseek/deepseek-v4-flash","effort":"low","cost_usd":0.000332,"raw_usage":{"cost":0.0003323502,"is_byok":false,"cost_details":{"upstream_inference_cost":0.0003323502,"upstream_inference_prompt_cost":3.75102e-05,"upstream_inference_completions_cost":0.00029484},"total_tokens":2765,"prompt_tokens":1145,"completion_tokens":1620,"prompt_tokens_details":{"audio_tokens":0,"video_tokens":0,"cached_tokens":916,"cache_write_tokens":0},"completion_tokens_details":{"audio_tokens":0,"image_tokens":0,"reasoning_tokens":1537}},"tokens_in":1145,"tokens_out":1620,"cost_source":"openrouter_usage","duration_ms":13511,"image_pages":[],"temperature":1.0,"billed_attempts":1,"reasoning_tokens":1537,"transport_provider":"openrouter","cache_read_input_tokens":0,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-10-15T05:55:57.786261+00:00","model_set":{"reader":"deepseek/deepseek-v4-flash"},"pipeline_version":"pith-pipeline@v1.4.1-daily-deepseek-or-medium","input_kind":null,"input_chars":null,"input_total_chars":null,"passes_missing":null,"published_standing":null,"published_confidence":null,"objection_audit":null,"run_record":null,"basis":{"kind":"not_recorded","source":null,"chars_given":null,"chars_total":null},"receipt":null,"falsifier":"Take a sub-step that AdaRSI labels executor-limited, such as opening the microwave door in WaffleReheat, and search over instructions, robot poses, and approach trajectories far more exhaustively than the paper's agent did while keeping the base policy frozen; if any such search makes the sub-step reliably succeed without a weight change, the attribution step has misclassified a harness failure as a missing motion, and the benefit of training on demand depends on the coding agent's search capability rather than on what the executor can actually do.","supporting_citations":[{"cited_title":"RoboCasa365: A large-scale simulation framework for training and benchmarking generalist robots","cited_arxiv_id":null,"evidence_quote":"RoboCasa365: the benchmark whose 50 target tasks, three suites, and official protocol generate every reported success number."},{"cited_title":"GR00T N1: An Open Foundation Model for Generalist Humanoid Robots","cited_arxiv_id":"2503.14734","evidence_quote":"The pretrained vision-language-action model used as the base executor that every compared system starts from and that AdaRSI fine-tunes."},{"cited_title":"Darwin Godel Machine: Open-Ended Evolution of Self-Improving Agents","cited_arxiv_id":"2505.22954","evidence_quote":"Supplies the recursive self-improvement principle that every self-modification must be validated empirically, which AdaRSI's policy-adoption rule implements."},{"cited_title":"Learning by cheating","cited_arxiv_id":null,"evidence_quote":"Learning by cheating: the privileged-information scheme the agent uses to generate and label its own demonstrations from simulator state."},{"cited_title":"A new learning paradigm: Learning using privileged information.Neural Networks, 22(5–6):544–557, 2009","cited_arxiv_id":null,"evidence_quote":"The formal statement of learning using privileged information that grounds the teacher-student labelling of self-collected demonstrations."},{"cited_title":"Self-Evolving Embodied Agents via Skill-Harness Evolution","cited_arxiv_id":"2608.11350","evidence_quote":"The authors' earlier skill-harness evolution work that AdaRSI's harness rounds build on and extend."}],"lane_accuracy":{"status":"not yet measured","lane":{"model":"deepseek/deepseek-v4-flash","input_kind":null},"url":"https://pith.science/accuracy"},"review_version":1}