{"id":"adfa34e5-d163-4284-b029-aa6bd01c642c","arxiv_id":"2610.18792","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"A controlled benchmark of self-improving robot agents shows agents do retain experience, but much apparent learning is resampling, and libraries help mainly when a model succeeds between 30 and 80 percent of the time.","lead":"SEAR is a benchmark and test harness that checks whether AI agents improve robots for real, rather than just retrying and getting lucky. It finds that saved skills help most for mid-tier models, that code transfers better than task-specific solutions, and that a model's preferred way of acting matters as much as raw capability.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek/deepseek-v4-flash","headline":"env.check_success is branched on by 47% of GPT-5.6-sol's graded RoboLab policies versus 5% of Opus's, and asset geometry is read in most RoboLab runs; the paper never quantifies these seal openings, which could distort model rankings and the 8-10-point library margin.","rationale":"This is the same weakest point the reader identified: the sealed-judge premise is only partly true. I sharpen the mechanism: the danger is not mainly uniform inflation of the T1-T4 absolute levels, because every condition shares the openings; it is differential exploitation across models (check_success 47% vs 5%; USD reads concentrated in Opus), which can distort the Section 5 model findings, plus an unquantified magnitude that the paper's own audit shows can be material (the hidden-instruction leak that forced a harness fix). I considered and rejected three alternatives. (1) The 8-10-point library claim: Table 7's split-repetition design is a genuinely matched control — a third Vanilla repetition against a two-run reference is exchangeable — and the reported above/below rates indicate a location shift, not merely extra Lineage variance, so I found no internal hole in it. (2) Single-run hardware claims: acknowledged in A.8 and supported by the detailed Section K audit of failure mechanisms, so they are replication risks, not hidden flaws. (3) The unpublished LIBERO-RIG companion (Cui et al., 2026): Section N specifies the design fully, and SEAR computes its own cells, so I treat it as secondary under the citation rules. Credit is due where the paper independently supports its claims: the rig-defect finding was corrected and rerun rather than hidden, the withdrawn Inkling-small model is fully documented in Section O, and the early-stop audit in Section K is unusually concrete. I also note that the abstract's 'easier tasks' framing (b12) is a simplification of the body's actual driver — model time to first success (Figure 5, Section 3.1) — but the body is careful, so this is a framing issue, not a load-bearing flaw. Given the disclosure and the relational structure of the main findings, the correct posture remains conditional acceptance, not rejection: the seal's materiality is the specific unresolved condition that would move the verdict if it landed.","tokens_in":53285,"tokens_out":23078,"duration_ms":164120,"concrete_test":"Re-grade the already-frozen policies in the sealed judge with env.check_success stubbed to always return False: for every RoboLab and LIBERO-PRO graded policy that calls it, re-run the ten TEST rollouts identically otherwise, and compare against the original grades. Then recompute Table 6's per-model scores, Figure 13a's rankings, and Table 7's Lineage-minus-Vanilla margins, first with all policies and then excluding policies that parse USD/XML at runtime. If the margins stay within a point or two and adjacent-model orderings (GPT-5.6-sol vs Kimi-k3 on RoboLab; Gemini vs DeepSeek) are unchanged, the concern is bounded and the conditional reading stands; if the margin drops toward rerun noise or a ranking flips, the affected model-comparison and library headlines need re-qualification.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing question is not whether the seal leaks — the paper says it does (Section A.9, Section C; limitations b491, b496) — but whether the openings change the headline numbers. Every condition can read assets, so uniform inflation would spare the relational claims; what threatens them is the documented asymmetric exploitation. A.9 reports that GPT-5.6-sol branches on env.check_success in 47% of its graded RoboLab policies against 5% for Opus, and asset-file reads vary sharply by model (Opus parsed the scene USD in 1,533 of 1,743 RoboLab runs). During judge grading, env.check_success is an oracle 'is the task done' signal, so policies that branch on it can stop or retry optimally; the paper gives no estimate of its score contribution. The one material leak that was quantified — the post-LIBERO Opus library quoting a hidden instruction verbatim, and the post-LIBERO-PRO library matching 27 of 160 hidden instructions — forced a harness fix, demonstrating that these openings can matter. Because the sealed judge is the benchmark's distinguishing control (Table 2) and the model-comparison findings (Sections 5.1-5.2) are headline claims, the unmeasured magnitude is a load-bearing gap. The authors' unusual candor (A.8, A.9) does not close it; only a materiality estimate would.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek/deepseek-v4-flash","summary":"SEAR is a framework and benchmark for measuring whether LM agents that improve robot policies from their own experience are genuinely learning, and whether what they learn transfers. The paper runs seven frontier LMs through a self-improvement loop over 330 simulated tasks in five families (robosuite, LIBERO, LIBERO-PRO, LIBERO-RIG, RoboLab) and five physical tasks, in three policy forms (code-as-policy, agent-as-policy, auto-research VLA fine-tuning). Its methodological core is a set of matched controls: best-of-3 restarts at a matched budget, a no-library Vanilla condition, a sealed judge grading frozen policies on held-out TEST states, three repetitions (with disclosed exceptions), and five generalization tiers T1-T5 from new initial states to a real robot. The three headline findings are: (1) a long session beats restarts only when a short attempt ends before the model's first success, and an inherited library helps most for models scoring between roughly 30 and 80% without it; (2) task-general control code and routines transfer across tasks and simulators while task-specific solutions do not, and hardware practice success overstates test success on operator-staged scenes; (3) models differ in how they prefer to act, and a text-only model can beat a comparable multimodal model. The paper also documents the seal's breaches (Section A.9) and provides detailed failure audits for the hardware campaign (Section K).","tokens_in":53739,"tokens_out":14254,"duration_ms":94509,"significance":"If the results hold, SEAR is a substantial contribution: it provides a reusable harness and benchmark in which the two confounds that plague self-improvement claims, resampling and privileged access, are addressed by explicit matched controls rather than by assertion. The Equation (1) decomposition of the Lineage-versus-best-of-3 margin into session and library terms is clean, and the split-repetition comparison of Table 7, which correctly avoids bucketing by the full Vanilla mean because of regression to the mean, is a notably careful treatment of noise. The main claims are falsifiable and mostly relational; the three-run repetitions on the core simulation conditions give honest error bars, and the disclosures of the seal openings, the single-trial AR and hardware results, and the withdrawal of Inkling-small are unusually candid. The hardware audit of Section K is a genuine methodological contribution in its own right.","major_comments":[{"comment":"The paper discloses the seal's openings and their usage frequencies but does not bound their score contribution. Section A.9 reports that 47% of GPT-5.6-sol's graded RoboLab policies branch on env.check_success against 5% for Opus, and that Opus parsed the scene USD in 1,533 of 1,743 RoboLab runs, and the authors state that this \"was allowed, and it helped\" [b496]. Because the sealed judge is the named control for \"privileged information\" in Table 2, and because the model comparisons of Sections 5.1-5.2 are headline claims, the missing materiality estimate is load-bearing: the documented asymmetry means the model ordering and the absolute T1-T4 scores could partly reflect differential use of an oracle stop signal and static asset geometry rather than agent capability. The one leak whose effect was quantified, the post-LIBERO Opus library quoting a hidden instruction verbatim with 27 of 160 LIBERO-PRO instructions matched, forced a harness redesign, showing that openings can be non-negligible in this benchmark. I am not asking that every opening be closed; I am asking for (a) a bounded materiality estimate, for example re-grading a sample of RoboLab and LIBERO policies with env.check_success masked or with asset files withheld, or (b) an explicit re-scoping of the exposed claims. The report should also state what the estimate does not threaten: the Lineage-versus-Vanilla and best-of-3 comparisons of Sections 3.1-3.2 and the library crossover are largely insulated, because every compared condition runs under the same judge with the same asset access.","section":"§A.9, §C, Table 2","authors_acknowledge":"The sealed judge protects runtime state, but it does not hide everything an agent can read (Section A.9).","authors_acknowledge_p":0.65},{"comment":"The code-versus-VLA comparison on LIBERO, LIBERO-PRO, and RoboLab rests on one auto-research trial per family (Table 8 note; Section A.8), yet the main text reports it without the qualifier at the point of use: \"On LIBERO and LIBERO-PRO the VLA scores within two points of code (94.0 vs. 94.4 and 90.6 vs. 91.8%)\", \"falls well short on robosuite (61.0 vs. 97.1) and RoboLab (47.2 vs. 68.0)\", and the camera-move numbers of Figure 9(b) (code 9.0%, VLA 68.9%) come from that single lineage. The caveat is honestly recorded in Section A.8, and the qualitative direction is corroborated by the cited LIBERO-RIG companion work, but the main text should either carry the \"one run\" qualifier inline at these sentences or present the quantities explicitly as single-run observations rather than settled gaps.","section":"§4.3, Table 8, Figure 9","authors_acknowledge":null,"authors_acknowledge_p":null}],"minor_comments":[{"comment":"The tool name is rendered as \"env.check success()\" with the underscore lost; elsewhere it is env.check_success. Please fix the formatting artifact so the oracle call is unambiguous.","section":"§A.9","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The introduction states that GPT-6-astra \"more than doubles its score when it acts live (from 41% to 88%)\" without the caveat that Section 5.3 supplies (\"one practice run per model and five staged scenes per task\"). Please repeat or point to the caveat at the first occurrence, since this is a headline claim.","section":"Section 1 (Introduction)","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The withdrawal of Inkling-small is described after the fact (\"we withdrew the model\"); for a benchmark meant to be reused, please state whether model inclusion was governed by a pre-specified rule or by an in-study decision, so that future users know the selection protocol.","section":"Section O","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The eight-model VLA degradation numbers are attributed to Cui et al. (2026), which is listed as \"to appear on arXiv\". Since that source is not yet published, please mark those numbers as corroboration from an unpublished companion and make clear that the paper's own VLA camera-move figures are single-lineage observations.","section":"§A.5","authors_acknowledge":null,"authors_acknowledge_p":null}],"recommendation":"major_revision"},"author_rebuttal":null,"desk_editor":{"model":"deepseek/deepseek-v4-flash","letter":"Read SEAR. The thing to know: the benchmark controls are the contribution. Best-of-3 against one long session, the no-library control, the split-repetition library test, and five generalization tiers form a measurement instrument that did not exist in this combination. Equation (1) is simple and correct, and Table 7's comparison of a Lineage repetition against a third Vanilla repetition is the cleanest thing in the paper. The library gain of 8-10 points over a rerun's noise, the U-shaped benefit, and the cross-model library crossover are real findings.\n\nCredit where due: the authors are unusually candid. Section A.9 lists the seal openings, asset reads and the callable success predicate, and says they helped. Three runs per condition, single-run hardware, one withdrawn model, model-tool pairing: all disclosed. The hardware audit in Section K, where frozen policies stop at self-imposed limits, is good evidence and well presented.\n\nSoft spots, in proportion. The biggest is exactly what the stress-test flags: the seal openings are enumerated but not quantified. The asymmetry matters more than the leakage itself. GPT-5.6-sol branches on env.check_success in 47% of graded RoboLab policies, Opus in 5%. During grading that predicate is an oracle for task completion, and the paper gives no estimate of how many points it buys. Because model rankings and the library margin are headline claims, that missing materiality estimate is a load-bearing gap. The authors' candor does not close it; only an estimate would.\n\nThe hardware table is one practice run per agent, with no Opus AaP row and five staged scenes per task. The paper says so, but that still makes those numbers reports from a pilot, not settled measurements. The AR budget-matched cells on LIBERO, LIBERO-PRO, and RoboLab are also single trials. Minor: LIBERO-RIG is listed as 'to appear', though Appendix N specifies enough that the dependence is not severe.\n\nVerdict: conditional, matching the reader. The central argument, that rising success mixes retention and resampling and the controls separate them, holds up. I would send this to a serious referee. The referee should ask for a materiality analysis of each seal opening and repeated trials on the single-run cells. This paper is for people building or benchmarking self-evolving robot agents; they should read A.2, A.9, and Section 4.4.","headline":"The controls are the contribution: SEAR separates resampling from retention more cleanly than prior work, but unquantified seal openings and single-run hardware scores keep the headline numbers conditional.","tokens_in":54250,"tokens_out":4636,"duration_ms":36239},"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek/deepseek-v4-flash","headline":"SEAR argues that a rising success rate in a self-evolving robot agent is not proof of learning, and it separates the true gain from the noise: with resampling controlled, an inherited library adds 8 to 10 points over a rerun's baseline.","keywords":["self-evolving agents","robot manipulation","language-model agents","skill libraries","resampling controls","code-as-policy","generalization tiers","sim-to-real transfer"],"falsifier":"Re-run the Lineage and Vanilla campaigns with a hermetic seal — asset files stripped from the work image, the success predicate uncallable, and validation feedback cleared of layout descriptions — and compare TEST scores; if the 8 to 10 point library margin or the generalization-tier numbers shrink materially, the privileged reads were doing the work. A cheaper check is to re-grade the frozen policies with the success-check call blocked and count how many graded successes depended on it.","tokens_in":54807,"feed_emoji":"🤖","tokens_out":12883,"duration_ms":78107,"temperature":0.7,"pith_summary":"SEAR asks whether a language-model agent that writes robot-control code, tests it, and saves what works is genuinely learning or merely trying more often, and whether what it keeps helps anywhere beyond the practiced task. The paper argues that both effects are real and separable: with time budgets matched, an inherited library of code skills and memories adds 8 to 10 points over the noise of a rerun, and that gain concentrates in models that already succeed sometimes (roughly 30 to 80 percent) without a library. It also argues that one long session beats three short restarts only when a short attempt would end before the model's first success, and that what transfers is general machinery (control code and grasp routines), not task-specific solutions; on a physical robot, a ten-success practice streak overstates performance on operator-staged scenes. The practical upshot is that a rising success curve in any self-improving robot agent should be read through matched controls before it is credited as learning.","feed_headline":"Robot agents' real learning gain: 8 to 10 points","feed_subtitle":"Matched controls separate kept skills from lucky retries; on easy tasks three short tries beat one long session.","key_machinery":"The load-bearing object is the margin identity Lineage $-$ BO3 $=$ (Vanilla $-$ BO3) $+$ (Lineage $-$ Vanilla), where best-of-3 means three independent no-library attempts at a third of the budget each, chosen by validation score; this control converts 'trying more often' into a measured term rather than an excuse. Around the identity sits a matched-control protocol: a sealed judge that grades frozen policies on TEST initial states the agent never saw, DEV/VAL/TEST seed bands, three repetitions per condition to expose luck, and five generalization tiers measuring how far a gain reaches (new initial states, changed conditions, new tasks, a new simulator, a real robot). The identity's two terms are then read against each model's median time to first success and its no-library success rate, which is what turns the control design into an explanation instead of just a benchmark.","core_discovery":"The central claim, stated the way the authors would state it, is that self-evolution in robotic agents splits into two measurable effects and that the split is visible in one identity: the margin of a full session with an inherited library over three fresh short attempts equals (one long session minus three short attempts) plus (inherited library minus no library). The first term is what sustained effort buys over resampling, and it is positive only when the model's median time to first success outlasts a short attempt; the second term is what the library carries forward, and across 1,510 matched model–task pairs it is a genuine 8 to 10 points above what a third rerun of the no-library baseline achieves, concentrated where the model is neither hopeless nor near-perfect. The paper's companion claims are that transfer follows the content of saved experience (control code and grasp routines cross tasks and physics engines, task-specific fixes do not), that a wrong belief saved in the library is inherited like a skill and can hurt, and that the choice of language model and its preferred mode of acting — writing code, acting live, or fine-tuning a vision-language-action model — shapes outcomes at least as much as whether the model can see.","pith_inferences":["Editorial extension: the session/library decomposition is a general template for claims of self-improvement anywhere — any agent that reports learning from its own experience should be re-read through a best-of-N resampling control and a no-inheritance baseline, and the identity itself transfers beyond robotics.","Editorial extension: the model portraits support a testable engineering thesis — spending on grounded perception tools (segmentation plus depth lifting) buys more manipulation ability than a multimodal model that can see, because the measured failures were physical rather than perceptual.","Editorial extension: the hardware finding implies a general evaluation rule for self-improving physical agents: start states for testing must be staged independently of the agent's own resets, or a success streak in practice will systematically overstate performance.","Editorial extension: the dexterous-hand demo was explicitly not a transfer result — the harness moved, the skills did not — which suggests a direct experiment: carry the same skill library to a new embodiment, retrain on hardware, and test whether the 8 to 10 point library gain reappears."],"forward_implications":["For a fixed budget, three short restarts match one long session whenever each attempt outlives the model's median time to first success; for slow-to-succeed models the long session is worth 5 to 24 points on the hardest family.","An inherited library is worth 8 to 10 points over a rerun's noise at matched budgets, with the gain concentrated in the 30 to 80 percent no-library band; below 30 percent the model rarely reuses what it saves, and above 80 percent there is little left to gain.","Saved experience transfers across models and can also harm: one model scored 42.4 percent with its own library versus 60.6 percent with another model's, and a false memory saved as fact was inherited by later sessions in the same way a skill is.","General control and grasp routines carry across tasks and simulators (one skill was imported by 66 later policies), while task-specific solutions do not; on hardware, practice overstates held-out performance because the agent's own resets rarely produce the hard starting states an operator stages.","Perception is not the bottleneck: a text-only model beat a comparable multimodal one on the hardest family, 78 percent of failed episodes had closed the gripper on air, and only 2 percent of no-library policies used the pretrained action model."],"supporting_citations":[{"why":"Supplies the resampling baseline (more independent attempts solve more problems) that motivates the best-of-3 control separating learning from trying more often.","marker":"Brown et al., 2024"},{"why":"Provides robosuite, the first task family and one of the two physics engines in the 330-task testbed.","marker":"Zhu et al., 2020"},{"why":"Provides the 40 LIBERO language-conditioned tasks that anchor the middle families and the lifelong-learning framing.","marker":"Liu et al., 2023"},{"why":"Provides LIBERO-PRO's reworded and re-laid-out tasks used as the changed-conditions and new-task material.","marker":"Zhou et al., 2025"},{"why":"Supplies LIBERO-RIG's hardware-fault axes (moved camera, calibration error, worn gripper pads) behind the fault results.","marker":"Cui et al., 2026"},{"why":"Provides RoboLab, the hardest task family on Isaac Sim that serves as the new-simulator tier and much of the library-gain evidence.","marker":"Yang et al., 2026a"},{"why":"Defines the vision-language-action model that the auto-research mode collects rollouts to fine-tune.","marker":"Black et al., 2025"},{"why":"Supplies the MuJoCo physics engine underlying four of the five simulated families.","marker":"Todorov et al., 2012"},{"why":"Source of the sealed-judge design: separate work and judge containers adapted for grading frozen policies.","marker":"Zhu et al., 2026"},{"why":"Provides the segmentation tool that code agents compose with depth images to localize objects.","marker":"Carion et al., 2026"}],"fun_headline_variants":["SEAR: real learning gains 8 to 10 points after saved skills","Matched controls split robot learning from lucky retries","Library beats third rerun: 8-10 point edge across 1,510 pairs","Robot agents learn when saved skills beat repeated tries","Self-evolving robot agents gain 8 to 10 points from kept skills"],"cache_read_input_tokens":0,"weakest_assumption_plain":"The load-bearing premise is that the sealed judge actually seals: agents were allowed to read static object geometry from asset files in most runs and to call the environment's own success predicate as a stop signal, both of which helped them, so the premise is that this privileged information did not materially inflate the measured learning and transfer gains.","fun_headline_variants_meta":{"raw":{"variants":["SEAR: real learning gains 8 to 10 points after saved skills","Matched controls split robot learning from lucky retries","Library beats third rerun: 8-10 point edge across 1,510 pairs","Robot agents learn when saved skills beat repeated tries","Self-evolving robot agents gain 8 to 10 points from kept skills"]},"model":"deepseek/deepseek-v4-flash","effort":"low","cost_usd":0.000153,"raw_usage":{"cost":0.0001533168,"is_byok":false,"cost_details":{"upstream_inference_cost":0.0001533168,"upstream_inference_prompt_cost":3.62908e-05,"upstream_inference_completions_cost":0.000117026},"total_tokens":1829,"prompt_tokens":1186,"completion_tokens":643,"prompt_tokens_details":{"audio_tokens":0,"video_tokens":0,"cached_tokens":984,"cache_write_tokens":0},"completion_tokens_details":{"audio_tokens":0,"image_tokens":0,"reasoning_tokens":564}},"tokens_in":1186,"tokens_out":643,"cost_source":"openrouter_usage","duration_ms":5413,"image_pages":[],"temperature":1.0,"billed_attempts":1,"reasoning_tokens":564,"transport_provider":"openrouter","cache_read_input_tokens":0,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-10-14T07:06:27.358881+00:00","model_set":{"reader":"deepseek/deepseek-v4-flash"},"pipeline_version":"pith-pipeline@v1.4.1-daily-deepseek-or-medium","input_kind":null,"input_chars":null,"input_total_chars":null,"passes_missing":null,"published_standing":null,"published_confidence":null,"objection_audit":null,"run_record":null,"basis":{"kind":"not_recorded","source":null,"chars_given":null,"chars_total":null},"receipt":null,"falsifier":"Re-run the Lineage and Vanilla campaigns with a hermetic seal — asset files stripped from the work image, the success predicate uncallable, and validation feedback cleared of layout descriptions — and compare TEST scores; if the 8 to 10 point library margin or the generalization-tier numbers shrink materially, the privileged reads were doing the work. A cheaper check is to re-grade the frozen policies with the success-check call blocked and count how many graded successes depended on it.","supporting_citations":[{"cited_title":"Large Language Monkeys: Scaling Inference Compute with Repeated Sampling","cited_arxiv_id":"2407.21787","evidence_quote":"Supplies the resampling baseline (more independent attempts solve more problems) that motivates the best-of-3 control separating learning from trying more often."},{"cited_title":"LIBERO : Benchmarking knowledge transfer for lifelong robot learning","cited_arxiv_id":null,"evidence_quote":"Provides the 40 LIBERO language-conditioned tasks that anchor the middle families and the lifelong-learning framing."},{"cited_title":"LIBERO-RIG : A hardware-setup robustness benchmark for vision-language-action models, 2026","cited_arxiv_id":null,"evidence_quote":"Supplies LIBERO-RIG's hardware-fault axes (moved camera, calibration error, worn gripper pads) behind the fault results."},{"cited_title":"IndoorUAV: Benchmarking Vision-Language UAV Navigation in Continuous Indoor Environments","cited_arxiv_id":"2504.16054","evidence_quote":"Defines the vision-language-action model that the auto-research mode collects rollouts to fine-tune."},{"cited_title":"MuJoCo: A physics engine for model-based control","cited_arxiv_id":null,"evidence_quote":"Supplies the MuJoCo physics engine underlying four of the five simulated families."},{"cited_title":"SAM 3: Segment anything with concepts","cited_arxiv_id":null,"evidence_quote":"Provides the segmentation tool that code agents compose with depth images to localize objects."}],"lane_accuracy":{"status":"not yet measured","lane":{"model":"deepseek/deepseek-v4-flash","input_kind":null},"url":"https://pith.science/accuracy"},"review_version":1}