Pith.

Robotics · arXiv:2610.18792 v1 · submitted 13 Oct 2026 · 50 references

SEAR: Self-Evolving Agentic Robotics

Xinyi Wang, Bangzheng Li, Hanyu Wang, Lin Zhang, Andrea Yaoyun Cui, Zhenggang Tang, Shyamal Buch, Dejia Xu

The paper says

“Frontier language-model agents can now improve robots on their own: they write control code, test it, and keep what works as skills and memories for the next task.”

more of the abstract

We introduce SEAR (Self-Evolving Agentic Robotics), a framework and benchmark built around two questions. Is the agent really learning, or just trying more often? And does what it learns still help on new tasks, in a new simulator, and on a real robot? The framework runs an agent in three modes: it writes code that becomes the policy (code-as-policy), acts live in the control loop (agent-as-policy), or works as a researcher that fine-tunes a vision-language-action model (auto-research). The benchmark adds controls that separate retained experience from repeated attempts, familiar tasks, privileged simulator state, larger budgets, and luck. Across seven frontier language models, 330 simulated and five physical tasks, and over four years of agent runtime, we highlight three findings. First, the learning curve of a robotic agent depends both on the overall time budget and the task difficulty relative to model capability. Harder tasks make longer sessions worthwhile; on easier tasks, re-rolling short attempts does just as well. Second, control code and shared routines transfer across tasks and simulators, while task-specific solutions do not. On a real robot, we observe a performance gap between agent-designed practice and novel human-designed scenes. Third, different models evolve more effectively in different ways. Some do best acting live, others writing code first. And a text-only model keeps pace with a comparable multimodal one: perception helps with error assessment, but agentic reasoning and tool use can matter more. We release the benchmark and codebase to support future research.

from the abstract

Pith says

Machine review · deepseek-v4-flash · 14 Oct 2026 · what was read was not recorded · not checked by a person

It can be wrong. Challenge this review or reply to it.

Holds up, with conditions. The controls are the contribution: SEAR separates resampling from retention more cleanly than prior work, but unquantified seal openings and single-run hardware scores keep the headline numbers conditional. This line is wrong

2 major objections · 4 minor

How often objections from reviews like this one are right: not yet measured

Are you an author? What you can do.

Not checked: figures, code, what the 50 references say.

The paper says, at §A.9, §C, Table 2

The review did not quote this passage.

A classifier read this sentence of the paper as acknowledging the concern (probability 0.65):

“The sealed judge protects runtime state, but it does not hide everything an agent can read (Section A.9).”

Pith says: Objection 1 of 2 · major

The paper discloses the seal's openings and their usage frequencies but does not bound their score contribution. Section A.9 reports that 47% of GPT-5.6-sol's graded RoboLab policies branch on env.check_success against 5% for Opus, and that Opus parsed the scene USD in 1,533 of 1,743 RoboLab runs, and the authors state that this "was allowed, and it helped" [b496].

more of objection 1

Because the sealed judge is the named control for "privileged information" in Table 2, and because the model comparisons of Sections 5.1-5.2 are headline claims, the missing materiality estimate is load-bearing: the documented asymmetry means the model ordering and the absolute T1-T4 scores could partly reflect differential use of an oracle stop signal and static asset geometry rather than agent capability. The one leak whose effect was quantified, the post-LIBERO Opus library quoting a hidden instruction verbatim with 27 of 160 LIBERO-PRO instructions matched, forced a harness redesign, showing that openings can be non-negligible in this benchmark. I am not asking that every opening be closed; I am asking for (a) a bounded materiality estimate, for example re-grading a sample of RoboLab and LIBERO policies with env.check_success masked or with asset files withheld, or (b) an explicit re-scoping of the exposed claims. The report should also state what the estimate does not threaten: the Lineage-versus-Vanilla and best-of-3 comparisons of Sections 3.1-3.2 and the library crossover are largely insulated, because every compared condition runs under the same judge with the same asset access.

The paper says, at §4.3, Table 8, Figure 9

The review did not quote this passage.

Pith says: Objection 2 of 2 · major

The code-versus-VLA comparison on LIBERO, LIBERO-PRO, and RoboLab rests on one auto-research trial per family (Table 8 note; Section A.8), yet the main text reports it without the qualifier at the point of use: "On LIBERO and LIBERO-PRO the VLA scores within two points of code (94.0 vs. 94.4 and 90.6 vs. 91.8%)", "falls well short on robosuite (61.0 vs. …

more of objection 2

… 97.1) and RoboLab (47.2 vs. 68.0)", and the camera-move numbers of Figure 9(b) (code 9.0%, VLA 68.9%) come from that single lineage. The caveat is honestly recorded in Section A.8, and the qualitative direction is corroborated by the cited LIBERO-RIG companion work, but the main text should either carry the "one run" qualifier inline at these sentences or present the quantities explicitly as single-run observations rather than settled gaps.

4 minor objections

The paper says, at §A.9

Pith says: minor objection 1 of 4

The tool name is rendered as "env.check success()" with the underscore lost; elsewhere it is env.check_success. Please fix the formatting artifact so the oracle call is unambiguous.

The paper says, at Section 1 (Introduction)

Pith says: minor objection 2 of 4

The introduction states that GPT-6-astra "more than doubles its score when it acts live (from 41% to 88%)" without the caveat that Section 5.3 supplies ("one practice run per model and five staged scenes per task"). Please repeat or point to the caveat at the first occurrence, since this is a headline claim.

show 2 more

The paper says, at Section O

Pith says: minor objection 3 of 4

The withdrawal of Inkling-small is described after the fact ("we withdrew the model"); for a benchmark meant to be reused, please state whether model inclusion was governed by a pre-specified rule or by an in-study decision, so that future users know the selection protocol.

The paper says, at §A.5

Pith says: minor objection 4 of 4

The eight-model VLA degradation numbers are attributed to Cui et al. (2026), which is listed as "to appear on arXiv". Since that source is not yet published, please mark those numbers as corroboration from an unpublished companion and make clear that the paper's own VLA camera-move figures are single-lineage observations.

What would settle it

Re-grade the already-frozen policies in the sealed judge with env.check_success stubbed to always return False: for every RoboLab and LIBERO-PRO graded policy that calls it, re-run the ten TEST rollouts identically otherwise, and compare against the original grades. Then recompute Table 6's per-model scores, Figure 13a's rankings, and Table 7's Lineage-minus-Vanilla margins, first with all policies and then excluding policies that parse USD/XML at runtime. If the margins stay within a point or two and adjacent-model orderings (GPT-5.6-sol vs Kimi-k3 on RoboLab; Gemini vs DeepSeek) are unchanged, the concern is bounded and the conditional reading stands; if the margin drops toward rerun noise or a ranking flips, the affected model-comparison and library headlines need re-qualification.

from the skeptic's reading

How the review went

4 readings by deepseek-v4-flash · 14 Oct 2026

  1. Reader holds up, with conditions confidence moderate
  2. Skeptic agrees 2 min 44 s · 53,285 tokens read, 23,078 written
  3. Editor wrote the verdict above 36 s · 54,250 tokens read, 4,636 written
  4. Referee major revision: 2 major objections, 4 minor 1 min 35 s · 53,739 tokens read, 14,254 written

Not run on this paper: the simulated authors' defence.

Pith did not check this review's objections against the paper.

The review in full

The review in brief, machine-written

Read SEAR. The thing to know: the benchmark controls are the contribution. Best-of-3 against one long session, the no-library control, the split-repetition library test, and five generalization tiers form a measurement instrument that did not exist in this combination. Equation (1) is simple and correct, and Table 7's comparison of a Lineage repetition against a third Vanilla repetition is the cleanest thing in the paper. The library gain of 8-10 points over a rerun's noise, the U-shaped benefit, and the cross-model library crossover are real findings.

Credit where due: the authors are unusually candid. Section A.9 lists the seal openings, asset reads and the callable success predicate, and says they helped. Three runs per condition, single-run hardware, one withdrawn model, model-tool pairing: all disclosed. The hardware audit in Section K, where frozen policies stop at self-imposed limits, is good evidence and well presented.

Soft spots, in proportion. The biggest is exactly what the stress-test flags: the seal openings are enumerated but not quantified. The asymmetry matters more than the leakage itself. GPT-5.6-sol branches on env.check_success in 47% of graded RoboLab policies, Opus in 5%. During grading that predicate is an oracle for task completion, and the paper gives no estimate of how many points it buys. Because model rankings and the library margin are headline claims, that missing materiality estimate is a load-bearing gap. The authors' candor does not close it; only an estimate would.

The hardware table is one practice run per agent, with no Opus AaP row and five staged scenes per task. The paper says so, but that still makes those numbers reports from a pilot, not settled measurements. The AR budget-matched cells on LIBERO, LIBERO-PRO, and RoboLab are also single trials. Minor: LIBERO-RIG is listed as 'to appear', though Appendix N specifies enough that the dependence is not severe.

Verdict: conditional, matching the reader. The central argument, that rising success mixes retention and resampling and the controls separate them, holds up. I would send this to a serious referee. The referee should ask for a materiality analysis of each seal opening and repeated trials on the single-run cells. This paper is for people building or benchmarking self-evolving robot agents; they should read A.2, A.9, and Section 4.4.

The strongest objection

The load-bearing question is not whether the seal leaks — the paper says it does (Section A.9, Section C; limitations b491, b496) — but whether the openings change the headline numbers. Every condition can read assets, so uniform inflation would spare the relational claims; what threatens them is the documented asymmetric exploitation. A.9 reports that GPT-5.6-sol branches on env.check_success in 47% of its graded RoboLab policies against 5% for Opus, and asset-file reads vary sharply by model (Opus parsed the scene USD in 1,533 of 1,743 RoboLab runs). During judge grading, env.check_success is an oracle 'is the task done' signal, so policies that branch on it can stop or retry optimally; the paper gives no estimate of its score contribution. The one material leak that was quantified — the post-LIBERO Opus library quoting a hidden instruction verbatim, and the post-LIBERO-PRO library matching 27 of 160 hidden instructions — forced a harness fix, demonstrating that these openings can matter. Because the sealed judge is the benchmark's distinguishing control (Table 2) and the model-comparison findings (Sections 5.1-5.2) are headline claims, the unmeasured magnitude is a load-bearing gap. The authors' unusual candor (A.8, A.9) does not close it; only a materiality estimate would.

This is the same weakest point the reader identified: the sealed-judge premise is only partly true. I sharpen the mechanism: the danger is not mainly uniform inflation of the T1-T4 absolute levels, because every condition shares the openings; it is differential exploitation across models (check_success 47% vs 5%; USD reads concentrated in Opus), which can distort the Section 5 model findings, plus an unquantified magnitude that the paper's own audit shows can be material (the hidden-instruction leak that forced a harness fix). I considered and rejected three alternatives. (1) The 8-10-point library claim: Table 7's split-repetition design is a genuinely matched control — a third Vanilla repetition against a two-run reference is exchangeable — and the reported above/below rates indicate a location shift, not merely extra Lineage variance, so I found no internal hole in it. (2) Single-run hardware claims: acknowledged in A.8 and supported by the detailed Section K audit of failure mechanisms, so they are replication risks, not hidden flaws. (3) The unpublished LIBERO-RIG companion (Cui et al., 2026): Section N specifies the design fully, and SEAR computes its own cells, so I treat it as secondary under the citation rules. Credit is due where the paper independently supports its claims: the rig-defect finding was corrected and rerun rather than hidden, the withdrawn Inkling-small model is fully documented in Section O, and the early-stop audit in Section K is unusually concrete. I also note that the abstract's 'easier tasks' framing (b12) is a simplification of the body's actual driver — model time to first success (Figure 5, Section 3.1) — but the body is careful, so this is a framing issue, not a load-bearing flaw. Given the disclosure and the relational structure of the main findings, the correct posture remains conditional acceptance, not rejection: the seal's materiality is the specific unresolved condition that would move the verdict if it landed.

The referee's report

If the results hold, SEAR is a substantial contribution: it provides a reusable harness and benchmark in which the two confounds that plague self-improvement claims, resampling and privileged access, are addressed by explicit matched controls rather than by assertion. The Equation (1) decomposition of the Lineage-versus-best-of-3 margin into session and library terms is clean, and the split-repetition comparison of Table 7, which correctly avoids bucketing by the full Vanilla mean because of regression to the mean, is a notably careful treatment of noise. The main claims are falsifiable and mostly relational; the three-run repetitions on the core simulation conditions give honest error bars, and the disclosures of the seal openings, the single-trial AR and hardware results, and the withdrawal of Inkling-small are unusually candid. The hardware audit of Section K is a genuine methodological contribution in its own right.

Pith's reading of the paper (a machine's summary, not the authors')

SEAR asks whether a language-model agent that writes robot-control code, tests it, and saves what works is genuinely learning or merely trying more often, and whether what it keeps helps anywhere beyond the practiced task. The paper argues that both effects are real and separable: with time budgets matched, an inherited library of code skills and memories adds 8 to 10 points over the noise of a rerun, and that gain concentrates in models that already succeed sometimes (roughly 30 to 80 percent) without a library. It also argues that one long session beats three short restarts only when a short attempt would end before the model's first success, and that what transfers is general machinery (control code and grasp routines), not task-specific solutions; on a physical robot, a ten-success practice streak overstates performance on operator-staged scenes. The practical upshot is that a rising success curve in any self-improving robot agent should be read through matched controls before it is credited as learning.

What carries the argument. The load-bearing object is the margin identity Lineage $-$ BO3 $=$ (Vanilla $-$ BO3) $+$ (Lineage $-$ Vanilla), where best-of-3 means three independent no-library attempts at a third of the budget each, chosen by validation score; this control converts 'trying more often' into a measured term rather than an excuse. Around the identity sits a matched-control protocol: a sealed judge that grades frozen policies on TEST initial states the agent never saw, DEV/VAL/TEST seed bands, three repetitions per condition to expose luck, and five generalization tiers measuring how far a gain reaches (new initial states, changed conditions, new tasks, a new simulator, a real robot). The identity's two terms are then read against each model's median time to first success and its no-library success rate, which is what turns the control design into an explanation instead of just a benchmark.

Core claim. The central claim, stated the way the authors would state it, is that self-evolution in robotic agents splits into two measurable effects and that the split is visible in one identity: the margin of a full session with an inherited library over three fresh short attempts equals (one long session minus three short attempts) plus (inherited library minus no library). The first term is what sustained effort buys over resampling, and it is positive only when the model's median time to first success outlasts a short attempt; the second term is what the library carries forward, and across 1,510 matched model–task pairs it is a genuine 8 to 10 points above what a third rerun of the no-library baseline achieves, concentrated where the model is neither hopeless nor near-perfect. The paper's companion claims are that transfer follows the content of saved experience (control code and grasp routines cross tasks and physics engines, task-specific fixes do not), that a wrong belief saved in the library is inherited like a skill and can hurt, and that the choice of language model and its preferred mode of acting — writing code, acting live, or fine-tuning a vision-language-action model — shapes outcomes at least as much as whether the model can see.

The premise it rests on. The load-bearing premise is that the sealed judge actually seals: agents were allowed to read static object geometry from asset files in most runs and to call the environment's own success predicate as a stop signal, both of which helped them, so the premise is that this privileged information did not materially inflate the measured learning and transfer gains.

If the paper is right (Pith's inferences, not the authors')

For a fixed budget, three short restarts match one long session whenever each attempt outlives the model's median time to first success; for slow-to-succeed models the long session is worth 5 to 24 points on the hardest family.

An inherited library is worth 8 to 10 points over a rerun's noise at matched budgets, with the gain concentrated in the 30 to 80 percent no-library band; below 30 percent the model rarely reuses what it saves, and above 80 percent there is little left to gain.

Saved experience transfers across models and can also harm: one model scored 42.4 percent with its own library versus 60.6 percent with another model's, and a false memory saved as fact was inherited by later sessions in the same way a skill is.

General control and grasp routines carry across tasks and simulators (one skill was imported by 66 later policies), while task-specific solutions do not; on hardware, practice overstates held-out performance because the agent's own resets rarely produce the hard starting states an operator stages.

Perception is not the bottleneck: a text-only model beat a comparable multimodal one on the hardest family, 78 percent of failed episodes had closed the gripper on air, and only 2 percent of no-library policies used the pretrained action model.

Editorial extension: the session/library decomposition is a general template for claims of self-improvement anywhere — any agent that reports learning from its own experience should be re-read through a best-of-N resampling control and a no-inheritance baseline, and the identity itself transfers beyond robotics.

Editorial extension: the model portraits support a testable engineering thesis — spending on grounded perception tools (segmentation plus depth lifting) buys more manipulation ability than a multimodal model that can see, because the measured failures were physical rather than perceptual.

Editorial extension: the hardware finding implies a general evaluation rule for self-improving physical agents: start states for testing must be staged independently of the agent's own resets, or a success streak in practice will systematically overstate performance.

Editorial extension: the dexterous-hand demo was explicitly not a transfer result — the harness moved, the skills did not — which suggests a direct experiment: carry the same skill library to a new embodiment, retrain on hardware, and test whether the 8 to 10 point library gain reappears.

Check this review

Pith stored no record of this review's passes and took no fingerprint of it. Reviews made by code from 13 Nov 2026 on carry both.

How often reviews like this one are right

Reviews like this one: written by deepseek-v4-flash, where what was read was not recorded.

How often objections from reviews like this one are right: not yet measured

Every value above, in the review's public record (JSON)

Reviews by people

This review was written by a machine. No person has reviewed this paper on Pith yet. A scientist listed on Pith can review it for a fee, and their review is published here whatever it concludes.

Request a human review

How Pith checks papers · The full audit of this paper

The paper

Abstract

Frontier language-model agents can now improve robots on their own: they write control code, test it, and keep what works as skills and memories for the next task. We introduce SEAR (Self-Evolving Agentic Robotics), a framework and benchmark built around two questions. Is the agent really learning, or just trying more often? And does what it learns still help on new tasks, in a new simulator, and on a real robot? The framework runs an agent in three modes: it writes code that becomes the policy (code-as-policy), acts live in the control loop (agent-as-policy), or works as a researcher that fine-tunes a vision-language-action model (auto-research). The benchmark adds controls that separate retained experience from repeated attempts, familiar tasks, privileged simulator state, larger budgets, and luck. Across seven frontier language models, 330 simulated and five physical tasks, and over four years of agent runtime, we highlight three findings. First, the learning curve of a robotic agent depends both on the overall time budget and the task difficulty relative to model capability. Harder tasks make longer sessions worthwhile; on easier tasks, re-rolling short attempts does just as well. Second, control code and shared routines transfer across tasks and simulators, while task-specific solutions do not. On a real robot, we observe a performance gap between agent-designed practice and novel human-designed scenes. Third, different models evolve more effectively in different ways. Some do best acting live, others writing code first. And a text-only model keeps pace with a comparable multimodal one: perception helps with error assessment, but agentic reasoning and tool use can matter more. We release the benchmark and codebase to support future research.

The paper's own sentences, arranged, kept beside the review and separate from it. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. No sentence in this record was written by a model.

abstract compiled · 2026-10-14

The authors' main claim, as the classifier picked it out 0.28

“Frontier language-model agents can now improve robots on their own: they write control code, test it, and keep what works as skills and memories for the next task.”

How the main claim is stated: asserted 0.91

What the abstract itself states, as probabilities: claims first 0.07 · states limitation 0.84 · human participants 0.07 · controlled comparison 0.79 · randomized assignment 0.02 · states quantitative result 0.89 · compares with prior results 0.50 · mentions code or data availability 0.98

The abstract, sentence by sentence

This table labels the abstract only (arXiv listing abstract v1). The number is the probability the classifier gave the label.

SentenceRoleThe authors' own workStates the main conclusionStrength
Frontier language-model agents can now improve robots on their own: they write control code, test it, and keep what works as skills and memories for the next task. background 1.00 0.11 0.05 asserted 0.91
We introduce SEAR (Self-Evolving Agentic Robotics), a framework and benchmark built around two questions. contribution 1.00 0.84 0.03 asserted 0.89
Is the agent really learning, or just trying more often? gap 0.64 0.08 0.02 unclear 0.95
And does what it learns still help on new tasks, in a new simulator, and on a real robot? gap 0.52 0.15 0.03 unclear 0.95
The framework runs an agent in three modes: it writes code that becomes the policy (code-as-policy), acts live in the control loop (agent-as-policy), or works as a researcher that fine-tunes a vision-language-action model (auto-research). method 0.96 0.76 0.03 asserted 1.00
The benchmark adds controls that separate retained experience from repeated attempts, familiar tasks, privileged simulator state, larger budgets, and luck. method 0.79 0.84 0.04 asserted 0.99
Across seven frontier language models, 330 simulated and five physical tasks, and over four years of agent runtime, we highlight three findings. method 0.48 0.96 0.05 asserted 1.00
First, the learning curve of a robotic agent depends both on the overall time budget and the task difficulty relative to model capability. result 1.00 0.93 0.23 asserted 1.00
Harder tasks make longer sessions worthwhile; on easier tasks, re-rolling short attempts does just as well. result 0.80 0.93 0.13 asserted 1.00
Second, control code and shared routines transfer across tasks and simulators, while task-specific solutions do not. result 1.00 0.92 0.21 asserted 1.00
On a real robot, we observe a performance gap between agent-designed practice and novel human-designed scenes. result 0.96 0.95 0.07 asserted 1.00
Third, different models evolve more effectively in different ways. result 1.00 0.92 0.13 asserted 1.00
Some do best acting live, others writing code first. result 0.99 0.90 0.08 asserted 0.99
And a text-only model keeps pace with a comparable multimodal one: perception helps with error assessment, but agentic reasoning and tool use can matter more. result 0.80 0.92 0.10 asserted 0.91
We release the benchmark and codebase to support future research. contribution 0.80 0.85 0.02 asserted 0.91

What kind of paper the classifier reports this is: dataset or benchmark 0.84 · evidence from observation or experiment 0.94

What was examined, and what was not

Sources read

  • arXiv listing abstract v1

Not examined

  • full text
  • figures and tables
  • references and citing sentences
  • relations to other papers

Levels run

  • P abstract

Not run

  • S sections
  • T body sentences
  • C citations
  • R cross-paper relations

How the labels were made

Compiled by classifier-1.13.0 via TypeSafe API, question bank abstract-p1-20260923, 2026-10-14: 1 request, 10,628 input tokens.

Every quoted sentence on this record is the authors'. Every label is from Pith's closed vocabulary, with the probability shown beside it. This record as JSON

Figures (21)
Figure 1 of the paper
Figure 1. SEAR is a framework and benchmark for self-evolving agentic robotics: frontier language￾model agents that can improve robot performance from their own experience. We disentangle real progress across (top) 330 simulated tasks, two physics engines, and (middle) five physical tasks on a two-arm rig, tested on human-designed scenes. Bottom: the same harness on a dexterous hand. Frontier language-model agents can now improve robots on their own: they write control code, test it, and keep what works as skills and memories for the next task. We introduce SEAR (Self-Evolving Agentic…
Figure 2 of the paper
Figure 2. The self-evolving loop, and what it looks like on a real robot.
Figure 3 of the paper
Figure 3. The testbed. Five simulated task families (top) and five physical tasks (bottom): Banana into Bowl, Push-T, Cup Upright, Block Stacking, and Spell “SEAR”. • robosuite (Zhu et al., 2020), in MuJoCo (Todorov et al., 2012): 7 tasks, from lifting a cube to a two-arm handover. • LIBERO (Liu et al., 2023), in MuJoCo: 40 language-conditioned household tasks. • LIBERO-PRO (Zhou et al., 2025): 160 tasks made by changing the wording, objects, layout, or goal of the 40 LIBERO tasks. The changes also move tasks away from instances a model may have seen in pretraining. • LIBERO-RIG (Cui et al.,…
Figure 4 of the paper
Figure 4. Five models over development time. Validation success, averaged over the five simulated families with equal weight (
Figure 5 of the paper
Figure 5. One long session wins when a short attempt would end before the first success.
Figure 6 of the paper
Figure 6. The library helps most in the middle. One point per model and family: TEST success without a library (Vanilla) against the library’s gain (Lineage − Vanilla), from
Figure 7 of the paper
Figure 7. What an inherited skill changes. Kimi-k3 on the RoboLab task “put the Rubik’s cube and the banana in the bowl”, same scene. Without a library (left), the policy copies the gripper’s yaw from the object pose, the fingers land on the banana, and all ten TEST states fail. With the library (right), it calls an inherited skill, written on an earlier banana task, that finds the banana’s ridge and long axis and pinches across it; all ten TEST states succeed. model’s curves). Over the 25 model–family pairs of
Figure 8 of the paper
Figure 8. One task-general program serves an explored and a never-explored scene.
Figure 9 of the paper
Figure 9. Code and weights, side by side. (a) The TEST success of the VLA (the student) against the code policy (its teacher), for the VLA Lineage sessions of an Opus auto-research agent, pooled over the 305 tasks of LIBERO, LIBERO-PRO, and RoboLab; marker area counts tasks. The student beats its teacher on 15 tasks (orange), trails it on 126 (blue), and ties on 164 (grey). (b) LIBERO-practiced Opus policies re-graded on LIBERO-RIG, averaged over the fault axes in each group (per-axis values in
Figure 10 of the paper
Figure 10. Code against weights over accumulated session time.
Figure 11 of the paper
Figure 11. What hardware exposes that simulation misses.
Figure 12 of the paper
Figure 12. Dexterous-hand numpad typing. (a) The Sharpa Wave hand on a Flexiv Enlight arm at a physical numpad. (b) Accuracy (blue, left axis) and speed (purple, right axis) over 2.5 hours from first launch, including a restart. Dots are scored evaluations, hollow squares the speeds of unscored attempts; curves are Gaussian￾smoothed with a 12-minute bandwidth. Shading marks phases annotated from the agent’s journals; the dashed line is the 5 keys/s speed gate. Completing the task also requires at least 90% accuracy over a sliding window of ten evaluations. robosuite LIBERO LIBERO￾PRO…
Figure 13 of the paper
Figure 13. Model scores across task families, without a library.
Figure 14 of the paper
Figure 14. The proposed co-evolving loop. Experience from each interaction round can update four components that return to the agent. Solid grey: what SEAR does today (the library carries over between tasks; a VLA is trained but stays with its task suite). Dashed blue: proposed (harness revision and training the LM on its own traces). its audit, the shortcuts of Section A.9 would count as skill. Without the tiers, a gain on held-out states would pass for transfer. Section A.7 turns these findings into design rules for anyone building such a system, and Section A.8 lists the limitations of…
Figure 15 of the paper
Figure 15. Where a gain comes from and how far it reaches, for Opus.
Figure 16 of the paper
Figure 16. A task-blind lineage graded on scenes it has not yet reached.
Figure 17 of the paper
Figure 17. Opus policies under LIBERO-RIG faults, with and without a library.
Figure 19 of the paper
Figure 19. Self-evolution on hardware, every agent on every task (continued in Figures 20 and 21). Each block is one task, with one row per agent and four attempts per row, numbered in session order: the overview camera at the end of the attempt, with the elapsed session time, the attempt kind, and the judge’s verdict (green success, red failure). The axis under each block places the four numbered attempts of each agent at their session times, so the rows read as time lapses of 12-hour sessions. For Opus 5 on Banana into Bowl and Fable 5.1 on the other tasks the attempts are chosen to show…
Figure 20 of the paper
Figure 20. Self-evolution on hardware, continued from
Figure 21 of the paper
Figure 21. Self-evolution on hardware, continued from

Showing 20 of the paper's 21 figures. Figures are the authors', reproduced from the paper.

References (50)
  1. [1] Artificial Analysis . Artificial Analysis Intelligence Index . https://artificialanalysis.ai/methodology/intelligence-benchmarking, 2026. Version 4.3.2; scores accessed October 2026 on Pith
  2. [2] BinceQu . RoboHarness : A simple harness outperforms VLA and world action models. https://github.com/BinceQu/RoboHarness, 2026. GitHub repository, accessed October 2026 on Pith
  3. [3] Kevin Black, Noah Brown, James Darpinian, Karan Dhabalia, Danny Driess, Adnan Esmail, Michael Equi, Chelsea Finn, Niccolo Fusai, Manuel Y. Galliker, Dibya Ghosh, Lachy Groom, Karol Hausman, Brian Ichter, Szymon Jakubczak, Tim Jones, Liyiming Ke, Devin LeBlanc, Sergey Levine, A... on PithPith's reviewarXiv
  4. [4] Alexander Bondarenko, Denis Volk, Dmitrii Volkov, and Jeffrey Ladish. Demonstrating specification gaming in reasoning models. arXiv preprint arXiv:2502.13295, 2025 on PithPith's reviewarXiv
  5. [5] Bradley Brown, Jordan Juravsky, Ryan Ehrlich, Ronald Clark, Quoc V. Le, Christopher R \'e , and Azalia Mirhoseini. Large language monkeys: Scaling inference compute with repeated sampling. arXiv preprint arXiv:2407.21787, 2024 on PithPith's reviewarXiv
  6. [6] Nicolas Carion, Laura Gustafson, Yuan-Ting Hu, Shoubhik Debnath, Ronghang Hu, Didac Suris Coll-Vinent, Chaitanya Ryali, Kalyan Vasudev Alwala, Haitham Khedr, Andrew Huang, Jie Lei, Tengyu Ma, Baishan Guo, Arpit Kalla, Markus Marks, Joseph Greer, Meng Wang, Peize Sun, Roman R \... on Pith
  7. [7] Jun Shern Chan et al. MLE-bench : Evaluating machine learning agents on machine learning engineering. arXiv preprint arXiv:2410.07095, 2024 on PithPith's reviewarXiv
  8. [8] Andrea Yaoyun Cui, Hanyu Wang, Xinyi Wang, Bangzheng Li, Lin Zhang, Zhenggang Tang, Shyamal Buch, Dejia Xu, and Matthew Tancik. LIBERO-RIG : A hardware-setup robustness benchmark for vision-language-action models, 2026. To appear on arXiv on Pith
  9. [9] Mengda Fu, Jieyu Yu, Kareem El-Refai, Eric Kou, Han Xue, Hao Huang, William Xiao, et al. CaP-X : A framework for benchmarking and improving coding agents for robot manipulation. arXiv preprint arXiv:2603.22435, 2026 on PithPith's reviewarXiv
  10. [10] Liane Galanti, Dhruv Shah, and Tri Dao. Addressing the orchestration gap in generalist robots via physical agency. arXiv preprint arXiv:2607.21725, 2026 on PithPith's reviewarXiv
  11. [11] Huan-ang Gao, Jiayi Geng, Wenyue Hua, Mengkang Hu, Xinzhe Juan, Hongzhang Liu, Shilong Liu, Jiahao Qiu, Xuan Qi, Qihan Ren, Yiran Wu, Hongru Wang, Han Xiao, Yuhang Zhou, Shaokun Zhang, Jiayi Zhang, Jinyu Xiang, Yixiong Fang, Qiwen Zhao, Dongrui Liu, Cheng Qian, Zhenhailong Wan... on Pith
  12. [12] Shijia Ge, Alex Zhou, Jianshu Zeng, Yexing Wan, Di Wu, Zelin Zheng, Yazhe Wang, Zhiqi Jia, Xuan Shangguan, Jay Zhu, Yijun Liu, Lingyu He, Sihang Wu, Xiao He, and Hongcheng Gao. Make code as policy great again: Frontier agents write, call, and evolve robot tools. arXiv preprint... on PithPith's reviewarXiv
  13. [13] Jiaheng Hu, Roberto Mart \' n-Mart \' n, Peter Stone, Rocky Duan, Zhenyu Jiang, and Guanya Shi. SimEX : Simulation-integrated robotics AutoResearch . arXiv preprint arXiv:2609.38982, 2026 a on PithPith's reviewarXiv
  14. [14] Kairui Hu, Siyuan Hu, Fangzhou Hong, Zhaoxi Chen, and Ziwei Liu. Embodied Turing machines: Stateful code for robot recursive self-improvement. arXiv preprint arXiv:2610.12369, 2026 b on PithPith's reviewarXiv
  15. [15] So Kuroki and Yujin Tang. Learning and transferring closed-loop robot software. arXiv preprint arXiv:2609.19906, 2026 on PithPith's reviewarXiv
  16. [16] Zexi Li, Yehang Zhang, Haojian Huang, Bohan Zhou, Wenqian Li, Chenxu Wang, Yifan Chang, Yangkai Wei, Tianyi Zhang, Ying-Cong Chen, Kaiwen Zhou, Yinchuan Li, and James Cheng. RACaP : Agentic reasoning, acting, and coding as policies for evolvable robot learning. arXiv preprint ... on PithPith's reviewarXiv
  17. [17] Jacky Liang, Wenlong Huang, Fei Xia, Peng Xu, Karol Hausman, Brian Ichter, Pete Florence, and Andy Zeng. Code as policies: Language model programs for embodied control. In IEEE International Conference on Robotics and Automation (ICRA), 2023 on Pith
  18. [18] Jiayi Lin, Sitong Liu, Chenhao Pan, Ling Lin, Shichao Dou, Zhihong Xi, Xiao Huang, et al. Agentic harness engineering: Observability-driven automatic evolution of coding-agent harnesses. arXiv preprint arXiv:2604.25850, 2026 a on PithPith's reviewarXiv
  19. [19] Mingyang Lin, Jiayi Wu, Zihe Wang, Zhiyu Shi, Yi Sang, Bo He, Zhiyuan Liu, et al. Harness updating is not harness benefit: Disentangling evolution capabilities in self-evolving LLM agents. arXiv preprint arXiv:2605.30621, 2026 b on PithPith's reviewarXiv
  20. [20] Bo Liu, Yifeng Zhu, Chongkai Gao, Yihao Feng, Qiang Liu, Yuke Zhu, and Peter Stone. LIBERO : Benchmarking knowledge transfer for lifelong robot learning. In Advances in Neural Information Processing Systems, volume 36, pp.\ 44776--44791, 2023 on Pith
show 30 more
  1. [21] Runze Lu, Yizhi Wu, Eric Kou, Letian Fu, William Xiao, Ajay Mandlekar, Yuqian Xu, Guanya Shi, Ken Goldberg, Annie Chen, Monica Chowdhury, Yuke Zhu, Linxi Fan, and Guanzhi Wang. ASPIRE : Agentic skills discovery for robotics. arXiv preprint arXiv:2607.00272, 2026 on PithPith's reviewarXiv
  2. [22] Haitong Ma, Chenxiao Gao, Rushi Qiang, Bo Dai, and Na Li. RLE-Bench : A qualifying exam for coding agents as robot learning engineers. arXiv preprint arXiv:2609.34210, 2026 on PithPith's reviewarXiv
  3. [23] Yecheng Jason Ma, William Liang, Guanzhi Wang, De-An Huang, Osbert Bastani, Dinesh Jayaraman, Yuke Zhu, Linxi Fan, and Anima Anandkumar. Eureka: Human-level reward design via coding large language models. In International Conference on Learning Representations (ICLR), 2024 on Pith
  4. [24] NVIDIA . Isaac Sim , 2025. URL https://github.com/isaac-sim/IsaacSim on Pith
  5. [25] Benedikt Rank, Harsh Bhatnagar, Ameya Prabhu, Samuel Eisenberg, Khoi Nguyen, Matthias Bethge, and Maksym Andriushchenko. PostTrainBench : Can LLM agents automate LLM post-training? arXiv preprint arXiv:2603.08640, 2026. Leaderboard: https://posttrainbench.com, retrieved September 2026 on PitharXiv
  6. [26] Zeyu Shen, Haoxiang You, Yilang Liu, Zhicheng Zheng, Lihan Zha, Kashu Yamazaki, Mingtong Zhang, Suning Huang, Jiankai Sun, Qianzhong Chen, Lucy He, Kaiyuan Liu, Haoran Chang, Katerina Fragkiadaki, Dhruv Shah, Mac Schwager, Peter Henderson, Ian Abraham, and Canwen Xu. EmbodiedS... on PithPith's reviewarXiv
  7. [27] Noah Shinn, Federico Cassano, Edward Berman, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement learning. In Advances in Neural Information Processing Systems (NeurIPS), 2023 on Pith
  8. [28] Ruiyang Si, Jianxin Bi, Shunyu Yang, Rui Ni, Wenbo Huang, Qiang Wang, Shulong Jiang, Duomin Wang, Xiuyu Li, Haiwen Feng, Zhen Dong, and Daquan Zhou. Fewer tokens, better action: GPT-6 Astra robot agents with 14\ higher success rate but 65\ arXiv preprint arXiv:2610.01939, 2026 on PithPith's reviewarXiv
  9. [29] Ishika Singh, Valts Blukis, Arsalan Mousavian, Ankit Goyal, Danfei Xu, Jonathan Tremblay, Dieter Fox, Jesse Thomason, and Animesh Garg. ProgPrompt : Generating situated robot task plans using large language models. In IEEE International Conference on Robotics and Automation (I... on Pith
  10. [30] Giulio Starace et al. PaperBench : Evaluating AI 's ability to replicate AI research. In International Conference on Machine Learning (ICML), 2025 on Pith
  11. [31] Emanuel Todorov, Tom Erez, and Yuval Tassa. MuJoCo : A physics engine for model-based control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp.\ 5026--5033. IEEE, 2012. doi:10.1109/IROS.2012.6386109 on Pithdoi:10.1109/iros.2012.6386109
  12. [32] Waddle Labs . Introducing waddle: Agents that control robots. Waddle Labs blog, https://waddlelabs.ai/research/introducing-waddle, July 2026 on Pith
  13. [33] Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandkumar. Voyager: An open-ended embodied agent with large language models. arXiv preprint arXiv:2305.16291, 2023 on PithPith's reviewarXiv
  14. [34] Jiaming Wang. What stops recursive self-improvement in robotics? lessons from 123 rounds of agentic skill discovery. arXiv preprint arXiv:2609.31760, 2026 on PithPith's reviewarXiv
  15. [35] Yen-Jen Wang, Haozhe Jiang, Shuying Deng, Haoru Xue, Weirui Ye, Rocky Duan, Nika Haghtalab, S. Shankar Sastry, Pieter Abbeel, and Haozhi Qi. Reconstruct, practice, go real: Guided self-improvement for embodied agents. arXiv preprint arXiv:2610.02204, 2026 on PithPith's reviewarXiv
  16. [36] Yufei Wang, Zhou Xian, Feng Chen, Tsun-Hsuan Wang, Yian Wang, Katerina Fragkiadaki, Zackory Erickson, David Held, and Chuang Gan. RoboGen : Towards unleashing infinite data for automated robot learning via generative simulation. In International Conference on Machine Learning ... on Pith
  17. [37] Zhijie Wei, Ferris Tan, and Jinghui Wang. Scale and selection: What makes automatic harness evolution work for visual-interface robot agents. arXiv preprint arXiv:2609.39304, 2026 on PithPith's reviewarXiv
  18. [38] Zimo Wen, Yijin Chen, Yuxuan Cao, Wendi Chen, Yanwen Zou, Wenye Yu, Fuhang Kuang, Han Xue, Jun Lv, Chuan Wen, and Cewu Lu. RoboRSI : Stable, efficient, and reusable robot self-evolution in complex real-world environments. arXiv preprint arXiv:2610.12424, 2026 on PithPith's reviewarXiv
  19. [39] Lilian Weng. Harness engineering for self-improvement. https://lilianweng.github.io, July 2026 on Pith
  20. [40] Hjalmar Wijk et al. RE-Bench : Evaluating frontier AI R&D capabilities of language model agents. In International Conference on Machine Learning (ICML), 2025 on Pith
  21. [41] William Xiao, Jinghan Xie, Tony Zhang, Haoran Lin, Letian Fu, Han Xue, Jianlan Lu, Yifeng Yang, Chuyuan Dai, Zhiao Wang, Jiajun Wu, Guanzhi Wang, Shankar S. Sastry, Ken Goldberg, Linxi Fan, Yuke Zhu, and Guanya Shi. ENPIRE : Agentic robot policy self-improvement in the real wo... on PithPith's reviewarXiv
  22. [42] XPolicyLab . XPolicyLab : A unified standard and open ecosystem for robot policy evaluation and deployment. https://github.com/XPolicyLab/XPolicyLab, 2026 on Pith
  23. [43] Xingyuan Yang, Rishik Dagli, Alexander Zook, Harry Hadfield, Ankit Goyal, Stan Birchfield, Fabio Ramos, and Jonathan Tremblay. RoboLab : A high-fidelity simulation benchmark for analysis of task generalist policies. arXiv preprint arXiv:2604.09860, 2026 a on PithPith's reviewarXiv
  24. [44] Zhiqin Yang, Chenxin Li, Xiaomeng Hu, Yibin Liu, Weidong Huang, Jiankai Sun, Haitao Li, Zijian Wu, Yuzhi Huang, Fanding Huang, Hanwen Sun, Jiashun Liu, Jingqi Tong, Mingxin Huang, Shaoli Hu, Shijue Huang, Tianyi Bai, Xinyuan Wang, Yunlong Lin, Zhengyang Tang, Zhexin Zhang, Zhu... on PithPith's reviewarXiv
  25. [45] Eric Zelikman, Eliana Lorch, Lester Mackey, and Adam Tauman Kalai. Self-taught optimizer ( STOP ): Recursively self-improving code generation. In Conference on Language Modeling (COLM), 2024 on Pith
  26. [46] Jesse Zhang, Jiahui Ge, Hyojun Yoo, Letian Fu, Zhenjia Yang, Yilun Liu, Raghav Saravanan, et al. Playful agentic robot learning. arXiv preprint arXiv:2606.19419, 2026 a on PithPith's reviewarXiv
  27. [47] Xilun Zhang, Maggie Wang, Erik Bauer, Hong-Xing Yu, Huang Huang, Jiajun Wu, and Marco Pavone. Co-evolving robot orchestrators and policies through deployment. arXiv preprint arXiv:2610.09228, 2026 b on PithPith's reviewarXiv
  28. [48] Xinyi Zhou, Yufei Xu, Guanhua Tie, Yuxiao Chen, Ge Zhang, Dayuan Chu, Pan Zhou, and Lin Sun. LIBERO-PRO : Towards robust and fair evaluation of vision-language-action models beyond memorization. arXiv preprint arXiv:2510.03827, 2025 on PithPith's reviewarXiv
  29. [49] Deyao Zhu, Xin Zhou, Shengling Qin, Xuekai Zhu, Hangliang Ding, Shu Zhong, Zixin Wen, Zhonglin Xie, et al. EdgeBench : Unveiling scaling laws of learning from real-world environments. arXiv preprint arXiv:2607.05155, 2026. Harness (SForge) released under Apache-2.0, tasks unde... on PithPith's reviewarXiv
  30. [50] Yuke Zhu, Josiah Wong, Ajay Mandlekar, Roberto Mart \' n-Mart \' n, Abhishek Joshi, Soroush Nasiriany, and Yifeng Zhu. robosuite: A modular simulation framework and benchmark for robot learning. arXiv preprint arXiv:2009.12293, 2020 on PithPith's reviewarXiv
Cited by 6 Pith papers

Challenges and discussion

Disagree with the review, or know something it missed? Leave a comment, or challenge this review formally: a challenge becomes a permanent entry in this paper's record, and its decision is published here.

No comments yet.

Sign in to comment