Pith.

Robotics · arXiv:2611.12177 v1 · submitted 9 Nov 2026 · 204 references

Embodied First Exam

Kangrui Wang, Qineng Wang, Changhe Chen, Pingyue Zhang, Chi Gui, Feishi Wang, Canyu Chen, Wenli Xiao

show 22 more authors

Xiaoyu Huang, Tonghe Zhang, Linjie Li, Zihan Wang, Minyeong Kim, Yiyou Sun, Mingkai Deng, Jinyu Hou, Qi Zhu, Han Liu, Eric P. Xing, Huaxiu Yao, Zhiwen Fan, Yuejiang Liu, Dawn Song, Pieter Abbeel, Li Fei-Fei, Jiajun Wu, Yuke Zhu, Guanya Shi, Ruohan Zhang, Manling Li

The paper says

“Early successes on agentic robotics point to a potential paradigm shift toward using coding agents to control robots.”

more of the abstract

Existing robotics benchmarks are built around visuomotor policy learning, leaving it open whether coding agents can reliably control diverse robots under physical-world constraints. We therefore introduce the Embodied First Exam, a unified, comprehensive, and agent-native evaluation framework where coding agents can choose their own control approach: direct control, code to control, or train a policy to control. We hope it serves as a first exam for coding agents before they enter the physical world, with 1,156 tasks in 35 suites on 47 robots, including 9 new suites that go beyond pick-and-place to spatial puzzles, dexterous manipulation, multi-robot interaction, dynamic whole-body control, and sim-to-sim engineering. Each task can be run in up to three evaluation settings. The Standard Setting mimics the real world: the agent receives only sensor observations and, by default, cannot reset the environment. Its Realtime variant keeps the environment running while the agent reasons. The Privileged Setting gives the agent access to the full simulator state and unlimited resets, allowing it to search for an oracle solution for training purposes. Success rates in this setting serve as an approximate upper bound. On the 456 core tasks, the strongest of eight agents, GPT-6 Astra in Codex, achieves a success rate of 34% in the Standard Setting, leaving much room for improvement. To see where agents fail, we label the stage at which failed attempts go wrong and compare failures across agents, settings, and stage categories.

from the abstract

Pith says

Machine review · deepseek-v4-flash · 10 Nov 2026 · what was read was not recorded · not checked by a person

It can be wrong. Challenge this review or reply to it.

The passes disagreed. Reader: holds up. Skeptic: agrees with the reader. Referee: major revision.

A massive, well-executed benchmark for coding agents in robotics. The three-setting design and failure-stage analysis are genuinely new. The main limitation is simulation-only results and single-attempt evaluations, but the paper is transparent about both. Worth engaging with. This line is wrong

5 major objections · 10 minor

How often objections from reviews like this one are right: not yet measured

Are you an author? What you can do.

Not checked: figures, code, what the 204 references say.

The paper says, at §2.3, §3.1, §J

The review did not quote this passage.

A classifier read this sentence of the paper as acknowledging the concern (probability 0.95):

“Because GPT-6 Luna belongs to the same model family as GPT-6 Astra, the screening may bias the core set against tasks that this family solves.”

Pith says: Objection 1 of 5 · major

The core set is screened by GPT-6 Luna, which is from the same family as the strongest evaluated agent (GPT-6 Astra). The authors acknowledge this bias (b109), but the potential impact is not quantified. If the screening removes tasks that GPT-6 Astra would solve, the reported success rates may be artificially low for that family, and the gap between GPT-6 Astra and other agents may be exaggerated.

more of objection 1

The authors should either re-run the screening with a model from a different family, or provide an analysis of how many tasks were removed by the screening and what fraction of those are solved by GPT-6 Astra in the full set.

The paper says, at §3.2, Fig. 12a

The review did not quote this passage.

Pith says: Objection 2 of 5 · major

The controlled runs add features (state query, resets) to the Standard interface, but the baseline run already has the success signal, which is not present in the Privileged setting. The 'no success signal' run is used as a control, but it changes the agent's behavior (it must decide when to stop).

more of objection 2

The comparison between the state-only and reset-only runs is informative, but the claim that 'resets help GPT-6 Astra more than the exact object state does' (b51) is based on a 53% vs 45% difference. The paper should report whether this difference is statistically significant (e.g., via a paired test), and should discuss the possibility that the state query is less useful because the agent does not always use it effectively (the paper notes that the agent did call it, but not how often or how accurately it was used).

The paper says, at §3.3, Fig. 14a

The review did not quote this passage.

Pith says: Objection 3 of 5 · major

The concentration of failure stages is measured using the Simpson index, and compared to a baseline where failure stages are drawn at random with probabilities derived from the relative position of stages in other tasks. This baseline is reasonable but the paper should also report the simpler uniform baseline (each stage equally likely) to give a sense of the effect size.

more of objection 3

The current baseline already accounts for the fact that early stages fail more often, which is the main confound. The conclusion that failures concentrate more than expected is supported, but the magnitude of the effect should be clearer.

The paper says, at §3.4, Fig. 16c

The review did not quote this passage.

Pith says: Objection 4 of 5 · major

The harness comparison shows that GPT-6 Astra performs worse in Claude Code than in Codex (34% vs 22%). The paper mentions that cross-provider runs go through an adapter, which may introduce differences. The authors should discuss whether the adapter could degrade performance (e.g., by losing information or changing prompt formatting).

more of objection 4

The claim that 'switching harnesses can change a model's success rate by up to 12 percentage points' is based on this comparison, but the adapter is a potential confound. The paper should either provide evidence that the adapter is neutral, or temper the claim.

The paper says, at §3.5, Fig. 18b

The review did not quote this passage.

A classifier read this sentence of the paper as acknowledging the concern (probability 0.93):

“The two versions differ at the same time in the GPU, the library, the training copies of the match and the instruction, so the comparison cannot separate their effects.”

Pith says: Objection 5 of 5 · major

The soccer task comparison between GPU and CPU versions is interesting, but the two versions differ in multiple ways (GPU, library, training copies, instruction). The paper acknowledges this (b248, b662), but still draws a strong conclusion: 'agents trained a policy when given a GPU and a training library, and only trained policies won.'

more of objection 5

This is a descriptive statement about the observed behavior, not a causal claim, so it is acceptable. However, the paper should be clearer that this is a single task with confounded variables, and that the result may not generalize.

10 minor objections

The paper says, at §1, Fig. 1

Pith says: minor objection 1 of 10

The figure caption says 'one command runs any coding agent', but the paper later notes that some harnesses require adapters. The caption should be qualified.

The paper says, at §2.1, Fig. 3b

Pith says: minor objection 2 of 10

The diagram shows 'Container 1' and 'Container 2' but the labels are small and the distinction between Privileged and Standard/Realtime is not immediately clear. Consider adding a brief explanation in the caption.

show 8 more

The paper says, at §3.1, Fig. 8

Pith says: minor objection 3 of 10

The figure uses a non-standard notation for percentages (e.g., '34.7%' appears as '34.7%' but the font is unusual). Ensure that the figure is legible in print.

The paper says, at §3.2, Fig. 12b

Pith says: minor objection 4 of 10

The y-axis labels are rotated and hard to read. Consider using horizontal labels or a different layout.

The paper says, at §3.3, Fig. 14b

Pith says: minor objection 5 of 10

The figure shows failure rates by category, but the categories are not independent (e.g., Material stages occur mostly in RoboPaint). The paper acknowledges this in the text, but the figure caption should also note that the rates are pooled across suites and may reflect suite-specific factors.

The paper says, at §3.4, Fig. 16a

Pith says: minor objection 6 of 10

The 'zero progress' row is defined as 'the attempt reached no stage', but the denominator is not immediately clear. The caption should specify that this is the share of failed attempts in Standard and Realtime settings.

The paper says, at §H.1

Pith says: minor objection 7 of 10

The labeling procedure uses GPT-6.1 Sol and Claude Sonnet 5.5. The paper should state whether these models had access to the task instructions and verifier code, and whether they were given examples of correct labels. This is already described, but it would be helpful to summarize the prompt design in a few sentences.

The paper says, at §J

Pith says: minor objection 8 of 10

The limitations section is thorough, but it could be expanded to discuss the potential for overfitting to the specific simulation environments (e.g., agents may learn to exploit simulator quirks). The paper notes that realism rules are applied, but a brief discussion of the generalizability to real robots would be useful.

The paper says, at §A.1

Pith says: minor objection 9 of 10

The description of RoboEngineer mentions that it uses learned dynamics models (Neural Lander, Neural-Fly). It would be helpful to clarify whether these models are pre-trained or trained during the task.

The paper says, at §C.2

Pith says: minor objection 10 of 10

The grading section mentions that the verifier replays the trajectory in a fresh process. For non-deterministic simulators, it replays up to three times. The paper should state what happens if the three replays give different results (e.g., majority vote, or any success counts).

What would settle it

Run a subset of 50 core Standard tests with 3 repeated attempts per agent to quantify the variance from single-attempt evaluation; if the rank order of agents changes, the single-attempt leaderboard is unreliable.

from the skeptic's reading

How the review went

4 readings by deepseek-v4-flash · 10 Nov 2026

  1. Reader holds up confidence high
  2. Skeptic agrees 12 s · 69,320 tokens read, 343 written
  3. Editor wrote the verdict above 12 s · 70,082 tokens read, 569 written
  4. Referee major revision: 5 major objections, 10 minor 21 s · 69,503 tokens read, 1,822 written

Not run on this paper: the simulated authors' defence.

Pith did not check this review's objections against the paper.

The review in full

The review in brief, machine-written

This is a serious piece of work. The Embodied First Exam is the largest and most systematic evaluation of coding agents on robotics tasks I've seen. The three-setting design (Privileged, Standard, Realtime) is a real contribution—it cleanly separates what an agent can do with full simulator access from what it can do under real-robot constraints. The failure-stage analysis, while generated by LLMs, provides a useful taxonomy of where agents go wrong. The scale is impressive: 1,156 tasks, 47 robots, 35 suites, 8 agents. The paper is also unusually transparent about its limitations—single attempts, agent-generated labels, simulation-only results—which I appreciate.

The main finding—that the strongest agent (GPT-6 Astra) solves only 34% of Standard tests and 77% of Privileged tests—is sobering and credible. The ablation showing resets help more than exact state, and that precision stages are the bottleneck, is informative. The soccer task result (agents train policies only when given a GPU) is a nice concrete illustration.

Soft spots: (1) All results are in simulation. The paper acknowledges this and shows two real-robot demos, but the central claim about "physical-world constraints" is tested only in simulation. (2) Single attempt per test means the reported success rates have unknown variance. The authors acknowledge this and promise multi-run evaluations in future work. (3) The stage labels are written by GPT-6.1 Sol, same family as GPT-6 Astra, which could bias the failure analysis. The authors flag this. (4) The core set was screened by GPT-6 Luna, also same family, potentially biasing the task selection against tasks that GPT-6 Astra would solve. Again, acknowledged.

Who this is for: anyone working on coding agents for robotics, benchmark designers, and researchers evaluating LLM-based robot control. The paper deserves a serious referee. I would cite it and bring it to a reading group.

Recommendation: accept with minor revisions. The main request would be to add confidence intervals or multi-run results for at least a subset of tasks, and to clarify how the simulation-only results relate to the claim about "physical-world constraints."

The strongest objection

The paper is a thorough, well-executed benchmark study. The central claim is supported by a large evaluation with objective verifiers and detailed analysis. Limitations are disclosed transparently, including single attempts, agent-generated labels, and simulation-only results. These do not undermine the central success-rate findings. The benchmark is novel and reproducible, and the analysis provides useful insights for the field. I found no fatal flaws or reason to reject.

The reader's weakest_assumption is that simulated physics and sensor interfaces adequately capture real-robot constraints. The paper acknowledges this limitation explicitly (Section I, Limitations). The central claim is about coding agent performance in simulation, not about real-robot transfer. The simulation-only nature is a scope condition, not a flaw. The paper's own limitations section and the two physical demonstrations show awareness. The reader's verdict ACCEPT is appropriate. I find no load-bearing concern that would change the verdict. The single-attempt evaluation is a limitation but not a fatal flaw; the paper acknowledges it and plans multiple runs in future work. The stage-level labels from agents are a potential bias, but the paper discusses this and uses human oversight. The core results are robust to these limitations.

The referee's report

If the results hold, EFE would be a valuable resource for the community: it is the most comprehensive benchmark for coding agents in robotics to date, with a principled separation of evaluation settings that isolates the impact of physical-world constraints. The analysis of failure stages and the controlled experiments on resets vs. state are informative. The benchmark is released with open infrastructure, which supports reproducibility and extension.

Pith's reading of the paper (a machine's summary, not the authors')

The paper introduces the Embodied First Exam (EFE), a large-scale evaluation framework designed to test whether general-purpose coding agents (LLM-based agents that write code) can control robots under realistic physical constraints. It comprises 1,156 tasks across 47 robots and 35 suites, with nine new suites that go beyond simple pick-and-place to include spatial puzzles, dexterous manipulation, multi-robot coordination, dynamic whole-body control, and sim-to-sim engineering. The central finding is that the strongest agent tested, GPT-6 Astra in Codex, achieves only a 34% success rate on the 456 core tasks in the Standard Setting, which mimics real-world conditions (sensor observations only, no environment resets). This leaves substantial room for improvement and establishes a baseline for a new paradigm: using coding agents for robot control.

What carries the argument. The key machinery is the Embodied First Exam (EFE) framework itself, which provides a unified, containerized infrastructure (based on Harbor) to evaluate any coding agent on any supported robot simulator. It defines three evaluation settings (Privileged, Standard, Realtime) that systematically vary the agent's access to simulator state, resets, and real-time constraints. The framework also includes a stage-level task-graph analysis, where each task is decomposed into stages with types (e.g., Relation, Pose, Dynamic, Mechanism, Material, Body) to pinpoint where and why agents fail. The comparison of success rates across these settings and the stage-level failure labels constitute the core analytical machinery.

Core claim. The paper establishes that current state-of-the-art coding agents, despite their success in digital software engineering tasks, struggle to reliably control diverse simulated robots under physical-world constraints. The strongest agent, GPT-6 Astra in Codex, solves only 34% of the core Standard Setting tasks (456 tasks) and 30% of the same 391 tasks that also have a Privileged test (where it reaches 77% success). The drop is primarily due to the lack of exact state information and unlimited resets in the Standard Setting. Resets are shown to help more than exact state, especially for tasks requiring precision. The paper also introduces a detailed failure analysis by breaking tasks into stages, revealing that failures tend to concentrate at the same stage across different agents, often the first stage, and that weaker agents are much more likely to fail before making the required contact.

The premise it rests on. The simulated physics and sensor interfaces across 35 suites adequately capture the physical-world constraints of real robots, so that the Standard-setting success rates are meaningful predictors of real-robot performance.

If the paper is right (Pith's inferences, not the authors')

If the EFE framework is adopted as a standard benchmark, the 34% success rate on Standard tasks will serve as a clear baseline for future progress in coding-agent-based robotics.

The finding that resets help more than exact state suggests that improving an agent's ability to recover from mistakes (e.g., through better planning or self-correction) may be more impactful than providing perfect perception.

The stage-level failure analysis implies that future work should focus on solving the first stage of tasks more reliably, as failures concentrate there, and on improving performance on precise manipulation and material-handling stages.

The demonstration that agents trained policies only when given a GPU and RL library, and that only trained policies won the soccer match, indicates that providing appropriate computational resources can fundamentally change the control approach an agent uses, with significant performance implications.

The EFE framework could be extended to real-robot evaluation using the same Standard Setting interface, as the authors note (Section I), which would validate whether simulation results transfer to physical hardware and reveal sim-to-real gaps for coding agents.

The concentration of failures at early task stages suggests a potential 'cold-start' problem for coding agents: they struggle to initiate the first meaningful interaction with the environment. This could be addressed by providing few-shot demonstrations or better initial exploration strategies.

The large gap between Privileged (77%) and Standard (30%) success rates implies that current coding agents are heavily reliant on perfect information and unlimited trial-and-error, which are unrealistic in most real-world deployments. Bridging this gap will require advances in perception, state estimation, and robust planning under uncertainty.

Check this review

Pith stored no record of this review's passes and took no fingerprint of it. Reviews made by code from 13 Nov 2026 on carry both.

How often reviews like this one are right

Reviews like this one: written by deepseek-v4-flash, where what was read was not recorded.

How often objections from reviews like this one are right: not yet measured

Every value above, in the review's public record (JSON)

Reviews by people

This review was written by a machine. No person has reviewed this paper on Pith yet. A scientist listed on Pith can review it for a fee, and their review is published here whatever it concludes.

Request a human review

How Pith checks papers · The full audit of this paper

The paper

Abstract

Early successes on agentic robotics point to a potential paradigm shift toward using coding agents to control robots. Existing robotics benchmarks are built around visuomotor policy learning, leaving it open whether coding agents can reliably control diverse robots under physical-world constraints. We therefore introduce the Embodied First Exam, a unified, comprehensive, and agent-native evaluation framework where coding agents can choose their own control approach: direct control, code to control, or train a policy to control. We hope it serves as a first exam for coding agents before they enter the physical world, with 1,156 tasks in 35 suites on 47 robots, including 9 new suites that go beyond pick-and-place to spatial puzzles, dexterous manipulation, multi-robot interaction, dynamic whole-body control, and sim-to-sim engineering. Each task can be run in up to three evaluation settings. The Standard Setting mimics the real world: the agent receives only sensor observations and, by default, cannot reset the environment. Its Realtime variant keeps the environment running while the agent reasons. The Privileged Setting gives the agent access to the full simulator state and unlimited resets, allowing it to search for an oracle solution for training purposes. Success rates in this setting serve as an approximate upper bound. On the 456 core tasks, the strongest of eight agents, GPT-6 Astra in Codex, achieves a success rate of 34% in the Standard Setting, leaving much room for improvement. To see where agents fail, we label the stage at which failed attempts go wrong and compare failures across agents, settings, and stage categories.

The paper's own sentences, arranged, kept beside the review and separate from it. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. No sentence in this record was written by a model.

abstract compiled · 2026-11-10

The authors' main claim, as the classifier picked it out 0.98

“On the 456 core tasks, the strongest of eight agents, GPT-6 Astra in Codex, achieves a success rate of 34% in the Standard Setting, leaving much room for improvement.”

How the main claim is stated: asserted 1.00

What the abstract itself states, as probabilities: claims first 0.37 · states limitation 0.87 · human participants 0.03 · controlled comparison 0.19 · randomized assignment 0.02 · states quantitative result 0.98 · compares with prior results 0.50 · mentions code or data availability 0.06

The abstract, sentence by sentence

This table labels the abstract only (arXiv listing abstract v1). The number is the probability the classifier gave the label.

SentenceRoleThe authors' own workStates the main conclusionStrength
Early successes on agentic robotics point to a potential paradigm shift toward using coding agents to control robots. background 1.00 0.05 0.06 hedged possibility 0.90
Existing robotics benchmarks are built around visuomotor policy learning, leaving it open whether coding agents can reliably control diverse robots under physical-world constraints. gap 0.99 0.06 0.03 asserted 0.91
We therefore introduce the Embodied First Exam, a unified, comprehensive, and agent-native evaluation framework where coding agents can choose their own control approach: direct control, code to control, or train a policy to control. contribution 0.99 0.88 0.10 asserted 0.99
We hope it serves as a first exam for coding agents before they enter the physical world, with 1,156 tasks in 35 suites on 47 robots, including 9 new suites that go beyond pick-and-place to spatial puzzles, dexterous manipulation, multi-robot interaction, dynamic whole-body control, and sim-to-sim engineering. contribution 0.73 0.84 0.08 asserted 0.87
Each task can be run in up to three evaluation settings. method 0.95 0.67 0.03 asserted 0.84
The Standard Setting mimics the real world: the agent receives only sensor observations and, by default, cannot reset the environment. method 0.98 0.54 0.03 asserted 1.00
Its Realtime variant keeps the environment running while the agent reasons. method 0.98 0.52 0.02 asserted 1.00
The Privileged Setting gives the agent access to the full simulator state and unlimited resets, allowing it to search for an oracle solution for training purposes. method 0.98 0.47 0.03 asserted 0.99
Success rates in this setting serve as an approximate upper bound. interpretation 0.63 0.65 0.05 asserted 0.67
On the 456 core tasks, the strongest of eight agents, GPT-6 Astra in Codex, achieves a success rate of 34% in the Standard Setting, leaving much room for improvement. result 1.00 0.93 0.31 asserted 1.00
To see where agents fail, we label the stage at which failed attempts go wrong and compare failures across agents, settings, and stage categories. method 0.82 0.94 0.07 asserted 0.93

What kind of paper the classifier reports this is: dataset or benchmark 1.00 · evidence from observation or experiment 0.83

What was examined, and what was not

Sources read

  • arXiv listing abstract v1

Not examined

  • full text
  • figures and tables
  • references and citing sentences
  • relations to other papers

Levels run

  • P abstract

Not run

  • S sections
  • T body sentences
  • C citations
  • R cross-paper relations

How the labels were made

Compiled by classifier-1.13.0 via TypeSafe API, question bank abstract-p1-20260923, 2026-11-10: 1 request, 8,224 input tokens.

Every quoted sentence on this record is the authors'. Every label is from Pith's closed vocabulary, with the probability shown beside it. This record as JSON

Figures (34)
Figure 1 of the paper
Figure 1. Overview of the Embodied First Exam, a comprehensive evaluation framework for coding agents on robotics tasks. It is large and diverse, spanning 1,156 tasks, 35 suites and 47 robots, and easy to run and evaluate: one command runs any coding agent, and each task can be graded in up to three evaluation settings. Website: embodied-first-exam.ai Code: github.com/mll-lab-nu/embodied-first-exam Trajectory Data: huggingface.co/datasets/MLL-Lab/embodied-exam-data Abstract Early successes on agentic robotics point to a potential paradigm shift toward using coding agents to control robots.…
Figure 2 of the paper
Figure 2. Examples of the control approaches, each with an excerpt of the agent’s commands or code. Dark tabs mark the approaches used (model is direct control), and the framed line is a policy. setting, and a verifier outside the agent’s control grades every attempt [196, 203]. We evaluate eight coding agents on a core set of 456 hard tasks. The strongest agent, GPT-6 Astra [116] in Codex, solves 77% of the Privileged tests but only 30% of the Standard tests of the same 391 tasks. Apart from the three strongest agents, GPT-6 Astra, Claude Opus 5.5 and Gemini 4 Argon, the best result comes…
Figure 3 of the paper
Figure 3. The EFE framework and evaluation protocol. (a) Our unified framework supports different models, harnesses, control approaches, and simulators. (b) In Privileged, the agent and simulator share a container, and the verifier replays the submitted answer separately. In Standard and Realtime, the simulator and verifier are separated from the agent and grade the live episode. 2. The Embodied First Exam: A Unified Evaluation Framework for Coding Agents in Robotics In this section, we describe how we evaluate coding agents on robotics tasks (Section 2.1), the resource constraints of each…
Figure 4 of the paper
Figure 4. Resource constraints across the exam. The large number and the tick give the median over tests (for GPU memory, over the tests that reserve a GPU), the line spans the range, and the dark box spans the middle half of the tests. (Section D). For a given task, all settings share the same initial scene, success rule, and time limit, so their results are paired. They differ in what the agent observes, whether it can reset, whether the world waits, and where the simulator and the verifier run, never in the control approach the agent may choose. We ablate the first two differences in…
Figure 5 of the paper
Figure 5. The tasks of the exam: the 9 suites built for the exam, each with a frame and its number of tasks, in the three groups of Section 2.3, and a frame from each of the 26 adapted suites, ordered by their main robot embodiment, from single arms to aerial robots. provide a GPU, agents can train a policy to control the robot (
Figure 6 of the paper
Figure 6. The core tasks by the stage categories of their task graphs, with an example task of each category. Each seat is one task in one category, and the angle of a category is proportional to its number of tasks. A task counts once in each category its task graph contains. The centre counts all 456 core tasks, and the seats cover the 427 with a task graph; no task graph has a Safety or Information stage, since the graphs of Embodied Safety Exam give its safety rules as conditions of their stages (Section H.1). rules of the original benchmark, in some suites with a stricter success rule.…
Figure 7 of the paper
Figure 7. Task graph and progress maps of carrying in groceries (BEHAVIOR-1K). Panel (a) gives the stages, the order in which they must be met, and the type and contact of each stage. Panel (b) gives the share of the stages whose conditions hold at each moment of the graded episode, for three attempts. Red lines mark failure stages and their failure phenomena (Section H.6). 8
Figure 8 of the paper
Figure 8. Success rate of each agent on the core set, with solved tests in parentheses. The harness dsh is DeepSeek Harness, DeepSeek’s own harness. Three agents each solve more than twice as many Standard tests as any of the other five. GPT-6 Astra, Claude Opus 5.5 and Gemini 4 Argon solve 24.1% to 33.6% of the 456 Standard tests (
Figure 10 of the paper
Figure 10. Success rates by suite, for suites with at least five core tests in the setting. Dots give the success rate of each of the three strongest agents, and bars the share of tests that none of them solves. The IKEA assembly tasks of RoboARC count as their own suite. nearly every test that any agent solves. The five weaker agents add only a few tests in either setting. Section 3.4 examines which tests each agent solves. In most suites with at least five Standard tests, at least half of these tests are solved by none of the three strongest agents. This holds in 14 of these 22 suites. Of…
Figure 11 of the paper
Figure 11. The 391 core tasks that have both a Privileged and a Standard test, split for each agent by the settings in which it solves them. Success rate (%) 0 25 50 75 100 Standard interface 28 + exact state 45 + unlimited resets 53 + both 68 Privileged Setting 77 (a) Failure rate of a stage type (%) Privileged Standard rise Precise Coarse Other 0 25 50 75 100 Assemble +46 Strike +44 Adjust pose +41 Catch +39 Insert +36 Thread/hang +25 Follow path +24 Put in +20 Pick up +18 Open +18 Switch on/off +15 Place +12 Close +4 Press +1 Clean +51 Jump +30 Traverse +27 Stack +26 Disassemble +26 (b)…
Figure 12 of the paper
Figure 12. (a) Success rate of GPT-6 Astra in Codex on the 391 core tasks that have both settings, as features of the Privileged Setting are added to the Standard interface. (b) Failure rate of each stage type compared in at least two suites, eight agents. (c) Share of GPT-6 Astra’s failures at precise and coarse stages that each run solves. Astra in Codex, we add one or both of two features to the Standard interface: a query for the exact state of every object, and unlimited resets with saved states (Figure 12a, Section G). On the 391 core tasks that have both settings, the success rate…
Figure 13 of the paper
Figure 13. Claude Opus 5.5 installing the front guard of a fan (DexHandExam) in the Privileged and the Standard Setting. Red marks where the Standard attempt goes wrong. Times are seconds on the replay of the Standard episode. and placing from 23% to 35%. If we count only stages that the same agent reaches on the same task in both settings, the difference between the two groups remains significant: precise stages rise by 25 points and coarse stages by 15 (Section E.1)
Figure 14 of the paper
Figure 14. (a) Concentration of failure stages, mean over tests, and share of tasks with the same most common failure stage in both settings, without the tasks where it is the first stage in both. Gray bars: random failure stages; black lines: 95% intervals. (b) Failure rate of a stage once reached, by category, both settings pooled (
Figure 15 of the paper
Figure 15. Catching a ball twice (RoboDynamicExam): the stages that the attempt of each agent reached and the stage at which it failed, in each setting, and Standard replay frames at three failures at stage 2. (Section H.5). Over the eight agents, the concentration is 61% in the Standard Setting, against 37% for the baseline. The five weaker agents often fail before they reach any stage of a task (Section 3.4), so much of this agreement could come from failures at the first stage. We therefore check the three strongest agents separately. Over the 217 Standard tests with at least two located…
Figure 16 of the paper
Figure 16. (a) Failure profile of each agent; zero progress means that the attempt reached no stage. (b) Share of the row agent’s solved Standard tests (first column) that the column agent also solves; the last column uses any agent above it in
Figure 17 of the paper
Figure 17. Preparing a lunch box (BEHAVIOR-1K, Standard Setting): a head-camera frame of four failed attempts, with the failure circled. The five weaker agents more often fail without making the required contact. In the failure profiles of Figure 16a, this happens in 17% to 24% of their failed Standard and Realtime attempts, against 7% to 10% for the three strongest agents. Of their failed Standard attempts on tasks with at least two stages, they reach no stage at all in 44% to 64%, against 27% to 36% (
Figure 18 of the paper
Figure 18. (a) Share of each approach by setting in 1,489 sampled attempts. Mixed attempts switched approach midway or used two approaches equally. (b) Soccer attempts in the Standard and Realtime settings that trained a policy, scored or won. 3.5. Which Control Approach Do Agents Use?  Key Takeaways: How Agents Control the Robot • In the Standard and Realtime settings of a soccer task, agents trained a policy when given a GPU and a training library, and only trained policies won. • In the Standard Setting, agents mostly control the robot directly. They do so in 61% of the sampled Standard…
Figure 19 of the paper
Figure 19. GPT-6 Astra in Codex on the two soccer versions in the Standard Setting. Each row gives the approach and result, two replay frames and the goals over match time. 4. Related Work This section reviews the closest lines of work. Section K gives an extended review. Benchmarks for coding agents. Coding agents and other language-model agents are evaluated mostly on digital tasks, such as resolving issues in real repositories [64, 104], completing tasks in a terminal [102], machine learning engineering and research [14, 143], and operating websites and desktop applications [169, 198].…
Figure 20 of the paper
Figure 20. Success rate of each agent on the core set. Each bar gives the share of the core tests of one setting that one agent solves. attachments or scripted object motion, so objects move only through contact and gravity, and in the furniture assembly tasks of RoboARC, the robot holds parts only by finger friction. Where a suite keeps a grasp aid, its documentation states it. In the knot, ring and rope tasks of RoboARC, a rigid attachment holds a grasped object and slips when the pull exceeds 60 to 65 N for 20 ms. Adapted suites keep the grasp model of the original benchmark, such as the…
Figure 21 of the paper
Figure 21. Success rate of each agent against the mean list-price cost of its attempts. Success rate of each agent
Figure 22 of the paper
Figure 22. Share of the core tasks of each type whose Standard test each agent solves, with a gray line joining the three strongest agents. The number of core tasks of the type is in parentheses, and a task can have several types. Results by task type. A task type describes a core task as a whole, whereas the stage-level failure labels of Section 3 assign a type to each stage of a task. By task type (
Figure 23 of the paper
Figure 23. Share of the Privileged tests solved by the row agent that the column agent also solves, as in Figure 16b, with the agents in the order of
Figure 24 of the paper
Figure 24. First and last frames of the graded episodes of two Standard tests that break the order of
Figure 25 of the paper
Figure 25. Core tests of each setting, grouped by which of the three strongest agents solve them. E.4. Control Approaches A separate coding agent read the logs of 1,489 attempts, sampled at random, and labeled each attempt with the approach that produced its graded answer (Section B). Many Privileged tests give the agent a GPU to run the simulator, and their instructions name no training library. Apart from these, only eight tests give the agent a GPU: the GPU versions of the soccer match below and of a table tennis match, in each setting, and two RoboEngineer tests that ask the agent to…
Figure 27 of the paper
Figure 27. Failures by robot body, agent and setting, in the same format as
Figure 28 of the paper
Figure 28. Failure rate of a stage once reached, by stage category and by the robot part that makes contact, with the Privileged and Standard settings pooled (
Figure 29 of the paper
Figure 29. Task graph of a DexHandExam task, installing the front guard of a fan, and the stages that three attempts met over time. A bar shows when a stage’s condition held, and a cross marks the failure stage. 1 Apple half 1 2 Apple half 2 3 Sandwich 4 Cookie 5 Open fridge 6 Tea bottle 7 Fridge closed met at the start 0/7 1/7 2/7 3/7 4/7 5/7 6/7 7/7 0 100 200 300 400 500 Time in the graded episode (s) S t a g e s m e t Qwen3.8 Max runs on to 845 s GPT-6 Astra 5/7: packed four foods, then fell on the way to the fridge Gemini 4 Argon 5/7: packed four foods, then opened a cabinet that it…
Figure 32 of the paper
Figure 32. BEHAVIOR-1K, disposing of batteries, Standard Setting: the task graph and the stages of each agent, where the wastebasket on the floor holds from the start, and head-camera frames as the robot tips over. in the wrong direction. Kimi 3.1 Pro threaded the first grommet and the first ring by 395 s, and at the second grommet the gripper closed without catching the rope. Fetching cereal and a bowl (Section 3.4, Figure 31b). In this RoboCasa365 task a one-armed robot on a mobile base must open a cabinet, put a cereal box and a bowl from it on the counter, and close the cabinet. The…
Figure 33 of the paper
Figure 33. Privileged and Standard settings. (a) RoboTwin 2.0, hanging a mug: progress maps of GPT-6 Astra. (b) Embodied Olympics - Fun, heading a ball ten times: the headers of each agent in the Privileged (P) and Standard (S) settings. 62
Figure 34 of the paper
Figure 34. Buzz wire game on a physical robot arm. (a) Four frames in time order of one trial of GPT-6 Astra and one of Claude Opus 5.5, with a close-up of the light and the buzzer and the command being executed. In the last frame of Claude Opus 5.5’s trial, the loop has touched the wire and the light is on. (b) The plot gives the progress score of each agent in each of its eight trials. Figure 34a, the pad at the lower left highlights the command being executed, and the inset shows a close-up of the light and the buzzer, so a contact is visible in the video. Agents and trials. The two…

Showing 30 of the paper's 34 figures. Figures are the authors', reproduced from the paper.

References (204)
  1. [1] Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebotar, Omar Cortes, Byron David, Chelsea Finn, Chuyuan Fu, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, Daniel Ho, et al. Do As I Can, Not As I Say: Grounding Language in Robotic Affordances.arXiv preprint arXiv:2204.... on PithPith's reviewarXiv
  2. [2] Michael Ahn, Debidatta Dwibedi, Chelsea Finn, Montse Gonzalez Arenas, Keerthana Gopalakrish- nan, Karol Hausman, Brian Ichter, Alex Irpan, Nikhil Joshi, Ryan Julian, Sean Kirmani, Isabel Leal, Edward Lee, Sergey Levine, Yao Lu, Isabel Leal, Sharath Maddineni, Kanishka Rao, Dor... on PitharXiv
  3. [3] Firas Al-Hafez, Guoping Zhao, Jan Peters, and Davide Tateo. LocoMuJoCo: A Comprehensive Imitation Learning Benchmark for Locomotion.arXiv preprint arXiv:2311.02496, 2023. URL https: //arxiv.org/abs/2311.02496 on PithPith's reviewarXiv
  4. [4] Arthur Allshire, Himanshu Gaurav Singh, Ritvik Singh, Adam Rashid, Hongsuk Choi, David McAllister, Justin Yu, Yiyuan Chen, Huang Huang, Pieter Abbeel, Xi Chen, Rocky Duan, Phillip Isola, Jitendra Malik, Fred Shentu, Guanya Shi, Philipp Wu, and Angjoo Kanazawa. Scalable Behavio... on PithPith's reviewarXiv
  5. [5] Anthropic. Claude Code.https://github.com/anthropics/claude-code, 2025 on Pith
  6. [6] Anthropic. Claude Fable 5 and Claude Mythos 5. https://www.anthropic.com/news/claud e-fable-5-mythos-5, 2026 on Pith
  7. [7] Anthropic. Claude Opus 5.5.https://www.anthropic.com/claude-opus-5-5, 2026 on Pith
  8. [8] ARISE Initiative. YAMLab.https://github.com/ARISE-Initiative/yamlab, 2026 on Pith
  9. [10] Kevin Black, Noah Brown, Danny Driess, Adnan Esmail, Michael Equi, Chelsea Finn, Niccolo Fusai, Lachy Groom, Karol Hausman, Brian Ichter, Szymon Jakubczak, Tim Jones, Liyiming Ke, Sergey Levine, Adrian Li-Bell, Mohith Mothukuri, Suraj Nair, Karl Pertsch, Lucy Xiaoyang Shi, Jam... on PithPith's reviewarXiv
  10. [12] Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Xi Chen, Krzysztof Choro- manski, Tianli Ding, Danny Driess, Avinava Dubey, Chelsea Finn, Pete Florence, Chuyuan Fu, Montse Gonzalez Arenas, Keerthana Gopalakrishnan, Kehang Han, Karol Hausman, Alexander Herzog, Ja... on PithPith's reviewarXiv
  11. [13] Vittorio Caggiano, Huawei Wang, Guillaume Durandau, Massimo Sartori, and Vikash Kumar. MyoSuite – A contact-rich simulation suite for musculoskeletal motor control.arXiv preprint arXiv:2205.13600, 2022. URLhttps://arxiv.org/abs/2205.13600 on PithPith's reviewarXiv
  12. [14] Jun Shern Chan, Neil Chowdhury, Oliver Jaffe, James Aung, Dane Sherburn, Evan Mays, Giulio Starace, Kevin Liu, Leon Maksin, Tejal Patwardhan, Lilian Weng, and Aleksander M ˛ adry. MLE- bench: Evaluating Machine Learning Agents on Machine Learning Engineering.arXiv preprint arX... on PithPith's reviewarXiv
  13. [15] Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, et al. Evaluating Large Language Models Trained on Code.arXiv preprint arXiv:2107.03374, 2021. URL ... on PithPith's reviewarXiv
  14. [16] Tianxing Chen, Zanxin Chen, Baijun Chen, Zijian Cai, Yibin Liu, Zixuan Li, Qiwei Liang, Xianliang Lin, Yiheng Ge, Zhenyu Gu, Weiliang Deng, Yubin Guo, Tian Nian, Xuanbing Xie, Qiangyu Chen, Kailun Su, Tianling Xu, Guodong Liu, Mengkang Hu, Huan ang Gao, Kaixuan Wang, Zhixuan L... on PithPith's reviewarXiv
  15. [17] Tianxing Chen, Yue Chen, Zixuan Li, Junyuan Tang, Kailun Su, Haoran Lu, Weijie Wan, Baijun Chen, Songling Liu, Haowen Yan, Honghao Su, Zhiyang Dou, Kaixuan Wang, Dandan Zhang, Yunze Liu, Yan Qin, Qiwei Liang, Qiwei Wu, Zijian Lin, Wenwei Lin, Yuran Wang, Minghua He, Tianshu Wu... on PithPith's reviewarXiv
  16. [18] Yanzhe Chen, Zechen Bai, Zhijun Cao, Wenzheng Zeng, Kevin Qinghong Lin, Yiqi Lin, Guoqiang Liang, Kevin Yuchen Ma, Qiming Huang, and Mike Zheng Shou. Show-Harness: Just a VLM Agent Can Play Robots.arXiv preprint arXiv:2609.10522, 2026. URL https://arxiv.org/abs/ 2609.10522 on PithPith's reviewarXiv
  17. [19] Yanzhe Chen, Ziyi Yang, Jifeng Zhu, Qiming Huang, Ruihe An, Peiyao Xu, Hesen Yang, Runda Liu, Chang Gong, Zhijun Cao, Zechen Bai, Wenzheng Zeng, et al. Multimodal Embodied Agents: A Unified Capability-centric Perspective from Computer-Use to Robot-Use.arXiv preprint arXiv:2610... on PithPith's reviewarXiv
  18. [20] Yuanpei Chen, Tianhao Wu, Shengjie Wang, Xidong Feng, Jiechuang Jiang, Stephen Marcus McAleer, Yiran Geng, Hao Dong, Zongqing Lu, Song-Chun Zhu, and Yaodong Yang. Towards Human-Level Bimanual Dexterous Manipulation with Reinforcement Learning.arXiv preprint arXiv:2206.08686, 2... on PithPith's reviewarXiv
  19. [21] Dongzhou Cheng, Taoran Yi, Ye Fang, Xingwu Zhang, Fan Feng, Yixuan Li, Gengxiong Zhuang, Rongze Wang, Shuai Yang, Wei Song, Weizhi Xue, Minyan Wu, Jie Gui, Jiaqi Wang, and Tong Wu. In-Context Robot Learning with VLM Agents.arXiv preprint arXiv:2609.19138, 2026. URL https://arx... on PithPith's reviewarXiv
  20. [22] Cheng Chi, Siyuan Feng, Yilun Du, Zhenjia Xu, Eric Cousineau, Benjamin Burchfiel, and Shu- ran Song. Diffusion Policy: Visuomotor Policy Learning via Action Diffusion.arXiv preprint arXiv:2303.04137, 2023. URLhttps://arxiv.org/abs/2303.04137. RSS 2023. 71 on PithPith's reviewarXiv
show 80 more
  1. [23] Zhaoyang Chu, Earl T. Barr, Claire Le Goues, Peter O’Hearn, Mark Harman, Federica Sarro, and He Ye. OpenRUA: Robot-Use Agents Are Zero-Shot Visuomotor Policies.arXiv preprint arXiv:2610.02459, 2026. URLhttps://arxiv.org/abs/2610.02459 on PithPith's reviewarXiv
  2. [24] David B. D’Ambrosio, Saminda Abeyruwan, Laura Graesser, Atil Iscen, Heni Ben Amor, Alex Bewley, Barney J. Reed, Krista Reymann, Leila Takayama, Yuval Tassa, et al. Achieving Human Level Competitive Robot Table Tennis.arXiv preprint arXiv:2408.03906, 2024. URL https://arxi v.or... on PithPith's reviewarXiv
  3. [25] Danny Driess, Fei Xia, Mehdi S. M. Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, Wenlong Huang, Yevgen Chebotar, Pierre Sermanet, Daniel Duckworth, Sergey Levine, Vincent Vanhoucke, Karol Hausman, Marc Toussaint... on PithPith's reviewarXiv
  4. [26] Yilun Du, Mengjiao Yang, Bo Dai, Hanjun Dai, Ofir Nachum, Joshua B. Tenenbaum, Dale Schuur- mans, and Pieter Abbeel. Learning Universal Policies via Text-Guided Video Generation.arXiv preprint arXiv:2302.00111, 2023. URLhttps://arxiv.org/abs/2302.00111. NeurIPS 2023 on PithPith's reviewarXiv
  5. [27] Karim Elmaaroufi, Justin Svegliato, Sarunas Kalade, Graham Schelle, Sanjit A. Seshia, and Matei Zaharia. RHO: Your Coding Agent is Secretly a Roboticist.arXiv preprint arXiv:2606.16458, 2026. URLhttps://arxiv.org/abs/2606.16458 on PitharXiv
  6. [28] Linxi Fan, Guanzhi Wang, Yunfan Jiang, Ajay Mandlekar, Yuncong Yang, Haoyi Zhu, Andrew Tang, De-An Huang, Yuke Zhu, and Anima Anandkumar. MineDojo: Building Open-Ended Embodied Agents with Internet-Scale Knowledge.arXiv preprint arXiv:2206.08853, 2022. URL https://arxiv.org/ab... on PithPith's reviewarXiv
  7. [29] Kuan Fang, Fangchen Liu, Pieter Abbeel, and Sergey Levine. MOKA: Open-World Robotic Manipulation through Mark-Based Visual Prompting.arXiv preprint arXiv:2403.03174, 2024. URL https://arxiv.org/abs/2403.03174. RSS 2024 on PithPith's reviewarXiv
  8. [30] Zhirui Fang, Qingchi Yu, Ziyang Chen, Longfei Li, Haoran Ma, Keru Zhou, Xinrun Xu, Samith Va, Yuxuan Hu, Peixuan Song, Qiang Du, Bin Qian, Yongkang Deng, Xin Li, Yezhen Wang, Zhe Li, Hao Luo, Shuyan Li, Ziwei Wang, Weijian Deng, and Xiu Li. EMERGE-Policy: A Robot Mind Emerges ... on PithPith's reviewarXiv
  9. [31] Frontier Red Team. Project Fetch: Can Claude train a robot dog? https://www.anthropic.co m/research/project-fetch-robot-dog, 2025 on Pith
  10. [32] Letian Fu, Justin Yu, Karim El-Refai, Ethan Kou, Haoru Xue, Huang Huang, Wenli Xiao, Fei-Fei Li, Guanya Shi, Jiajun Wu, Shankar Sastry, Yuke Zhu, Ken Goldberg, and Linxi “Jim” Fan. CaP- X: A Framework for Benchmarking and Improving Coding Agents for Robot Manipulation. In Proc... on Pith
  11. [33] Liane Galanti, Dhruv Shah, and Tri Dao. Addressing the Orchestration Gap in Generalist Robots via Physical Agency.arXiv preprint arXiv:2607.21725, 2026. URL https://arxiv.org/abs/2607 .21725 on PithPith's reviewarXiv
  12. [34] Hongcheng Gao, Jianshu Zeng, Alex Zhou, Shijia Ge, Xuan Shangguan, Yazhe Wang, Yexing Wan, Zelin Zheng, Jay Zhu, Xiping Li, Di Wu, Yongli Wang, et al. HexaShell: A Universal Harness for Robot Coding Agents in Simulation and the Real World.arXiv preprint arXiv:2610.23479, 2026.... on PithPith's reviewarXiv
  13. [35] Hongcheng Gao, Jingjing Zhou, Zelin Zheng, Shijia Ge, Jay Zhu, Yazhe Wang, Jianshu Zeng, Xuan Shangguan, Di Wu, Lingyu He, Zhiqi Jia, Sihang Wu, and Xiao He. Self-Evolving Coding Agents: 72 From Digital Programs to Physical-World Intelligence.arXiv preprint arXiv:2609.35432, 2... on PithPith's reviewarXiv
  14. [36] Gemini Robotics Team. Gemini Robotics: Bringing AI into the Physical World.arXiv preprint arXiv:2503.20020, 2025. URLhttps://arxiv.org/abs/2503.20020 on PithPith's reviewarXiv
  15. [38] Gemini Robotics Team, Krzysztof Choromanski, Coline Devin, Yilun Du, Debidatta Dwibedi, Ruiqi Gao, Abhishek Jindal, Thomas Kipf, et al. Evaluating Gemini Robotics Policies in a Veo World Simulator.arXiv preprint arXiv:2512.10675, 2025. URLhttps://arxiv.org/abs/2512.10675 on PitharXiv
  16. [39] Haoran Geng, Feishi Wang, Songlin Wei, Yuyang Li, Bangjun Wang, Boshi An, Charlie Tianyue Cheng, Haozhe Lou, Peihao Li, Yen-Jen Wang, Yutong Liang, Dylan Goetting, et al. RoboVerse: Towards a Unified Platform, Dataset and Benchmark for Scalable and Generalizable Robot Learning... on PithPith's reviewarXiv
  17. [40] Ken Goldberg. Good old-fashioned engineering can close the 100,000-year “data gap” in robotics. Science Robotics, 10(105):eaea7390, 2025. doi: 10.1126/scirobotics.aea7390 on Pithdoi:10.1126/scirobotics.aea7390
  18. [41] URLhttps://arxiv.org/abs/2510.03342 on PithPith's reviewarXiv
  19. [42] Yanjiang Guo, Lucy Xiaoyang Shi, Jianyu Chen, and Chelsea Finn. Ctrl-World: A Controllable Generative World Model for Robot Manipulation.arXiv preprint arXiv:2510.10125, 2025. URL https://arxiv.org/abs/2510.10125. ICLR 2026 on PithPith's reviewarXiv
  20. [44] Harbor Framework Team. Harbor: A framework for evaluating and optimizing agents and models in container environments.https://doi.org/10.5281/zenodo.20953922, 2026 on PithPith's reviewdoi:10.5281/zenodo.20953922
  21. [45] Jiayuan Gu, Fanbo Xiang, Xuanlin Li, Zhan Ling, Xiqiang Liu, Tongzhou Mu, Yihe Tang, Stone Tao, Xinyue Wei, Yunchao Yao, Xiaodi Yuan, Pengwei Xie, Zhiao Huang, Rui Chen, and Hao Su. ManiSkill2: A Unified Benchmark for Generalizable Manipulation Skills.arXiv preprint arXiv:2302... on PithPith's reviewarXiv
  22. [46] Tairan He, Jiawei Gao, Wenli Xiao, Yuanhang Zhang, Zi Wang, Jiashun Wang, Zhengyi Luo, Guanqi He, Nikhil Sobanbabu, Chaoyi Pan, Zeji Yi, Guannan Qu, Kris Kitani, Jessica Hodgins, Linxi Fan, Yuke Zhu, Changliu Liu, and Guanya Shi. ASAP: Aligning Simulation and Real-World Physic... on PithPith's reviewarXiv
  23. [47] Hengyuan Hu, Jensen Gao, Priya Sundaresan, Satvik Sharma, Jeannette Bohg, and Dorsa Sadigh. VIA: Visual Interface Agent for Robot Control.arXiv preprint arXiv:2607.11119, 2026. URL https: //arxiv.org/abs/2607.11119 on PithPith's reviewarXiv
  24. [48] Jiaheng Hu, Roberto Martin-Martin, Peter Stone, Rocky Duan, Zhenyu Jiang, and Guanya Shi. SimEX: Simulation-Integrated Robotics AutoResearch.arXiv preprint arXiv:2609.38982, 2026. URL https://arxiv.org/abs/2609.38982 on PithPith's reviewarXiv
  25. [49] Kairui Hu, Siyuan Hu, Fangzhou Hong, Zhaoxi Chen, and Ziwei Liu. Embodied Turing Machines: Stateful Code for Robot Recursive Self-Improvement.arXiv preprint arXiv:2610.12369, 2026. URL https://arxiv.org/abs/2610.12369. 73 on PithPith's reviewarXiv
  26. [50] Sida He, Lingxi Xie, Yunning Cao, Pengfei Chen, Kaiwen Duan, Jiannan Ge, Xinyue Huo, Jiacheng Shao, and Qi Tian. Robot Manipulation with GPT-6-Astra: Body Knowledge, Experience Reuse, Emergent Skills, and Sim2Real Transfer.arXiv preprint arXiv:2609.31770, 2026. URL https: //ar... on PithPith's reviewarXiv
  27. [51] Siyuan Huang, Zhengkai Jiang, Hao Dong, Yu Qiao, Peng Gao, and Hongsheng Li. Instruct2Act: Mapping Multi-modality Instructions to Robotic Actions with Large Language Model.arXiv preprint arXiv:2305.11176, 2023. URLhttps://arxiv.org/abs/2305.11176 on PithPith's reviewarXiv
  28. [52] Wenlong Huang, Fei Xia, Ted Xiao, Harris Chan, Jacky Liang, Pete Florence, Andy Zeng, Jonathan Tompson, Igor Mordatch, Yevgen Chebotar, Pierre Sermanet, Noah Brown, Tomas Jackson, Linda Luu, Sergey Levine, Karol Hausman, and Brian Ichter. Inner Monologue: Embodied Reasoning th... on PithPith's reviewarXiv
  29. [53] Wenlong Huang, Chen Wang, Ruohan Zhang, Yunzhu Li, Jiajun Wu, and Li Fei-Fei. VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Models. InConference on Robot Learning (CoRL), 2023 on Pith
  30. [54] Wenlong Huang, Chen Wang, Yunzhu Li, Ruohan Zhang, and Li Fei-Fei. ReKep: Spatio- Temporal Reasoning of Relational Keypoint Constraints for Robotic Manipulation.arXiv preprint arXiv:2409.01652, 2024. URLhttps://arxiv.org/abs/2409.01652. CoRL 2024 on PithPith's reviewarXiv
  31. [55] Runhan Huang, Xiangcheng Zhang, Haoyu Chen, Yitong Li, Yen-Jen Wang, Bohan Lyu, Haotian Ye, Chuye Hong, Yixuan Wang, Chloe Su, Haoyu Xiong, Yangzhen Wu, et al. Scaling Online Exploration for Robot Agents.arXiv preprint arXiv:2610.22496, 2026. URLhttps://arxiv.org/ abs/2610.22496 on PithPith's reviewarXiv
  32. [56] Michael Ilie, C. Daniel Freeman, and Kevin K. Troy. Project Fetch: Phase two. https://www.an thropic.com/research/project-fetch-phase-two, 2026 on Pith
  33. [57] Yuki Imajuku, Kohki Horie, Yoichi Iwata, Kensho Aoki, Naohiro Takahashi, and Takuya Akiba. ALE-Bench: A Benchmark for Long-Horizon Objective-Driven Algorithm Engineering.arXiv preprint arXiv:2506.09050, 2025. URL https://arxiv.org/abs/2506.09050 . NeurIPS 2025 Datasets and Ben... on PitharXiv
  34. [58] Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida I. Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica. LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code.arXiv preprint arXiv:2403.07974, 2024. URL https:... on PithPith's reviewarXiv
  35. [59] Stephen James, Zicong Ma, David Rovick Arrojo, and Andrew J. Davison. RLBench: The Robot Learning Benchmark & Learning Environment.arXiv preprint arXiv:1909.12271, 2019. URL https: //arxiv.org/abs/1909.12271. IEEE RA-L 2020 on PithPith's reviewarXiv
  36. [60] Yixuan Huang, Bowen Li, Vaibhav Saxena, Yichao Liang, Utkarsh A. Mishra, Liang Ji, Lihan Zha, Jimmy Wu, Nishanth Kumar, Sebastian Scherer, Danfei Xu, and Tom Silver. KinDER: A Physical Reasoning Benchmark for Robot Learning and Planning.arXiv preprint arXiv:2604.25788, 2026. U... on PithPith's reviewarXiv
  37. [61] Jack B. Jedlicki, Amish Sethi, and Heng Yang. Can a Frontier VLM Serve as a Robot Policy for Memory-Dependent Manipulation?arXiv preprint arXiv:2610.17350, 2026. URL https: //arxiv.org/abs/2610.17350 on PithPith's reviewarXiv
  38. [62] Mengzhao Jia, Yang Lin, Xixin Zhang, Zhihan Zhang, Xiaobai Liu, and Meng Jiang. Agent as Policy for Robotic Manipulation.arXiv preprint arXiv:2609.12541, 2026. URL https://arxiv.org/abs/ 2609.12541 on PithPith's reviewarXiv
  39. [63] Yunfan Jiang, Agrim Gupta, Zichen Zhang, Guanzhi Wang, Yongqiang Dou, Yanjun Chen, Li Fei- Fei, Anima Anandkumar, Yuke Zhu, and Linxi Fan. VIMA: Robot Manipulation with Multimodal Prompts.arXiv preprint arXiv:2210.03094, 2022. URL https://arxiv.org/abs/2210.03094 . ICML 2023. 74 on PithPith's reviewarXiv
  40. [64] Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. SWE-bench: Can Language Models Resolve Real-World GitHub Issues?arXiv preprint arXiv:2310.06770, 2023. URLhttps://arxiv.org/abs/2310.06770. ICLR 2024 on PithPith's reviewarXiv
  41. [65] Joel Jang, Seonghyeon Ye, Zongyu Lin, Jiannan Xiang, Johan Bjorck, Yu Fang, Fengyuan Hu, Spencer Huang, et al. DreamGen: Unlocking Generalization in Robot Learning through Video World Models. arXiv preprint arXiv:2505.12705, 2025. URLhttps://arxiv.org/abs/2505.12705 on PithPith's reviewarXiv
  42. [66] Alexander Khazatsky, Karl Pertsch, Suraj Nair, Ashwin Balakrishna, Sudeep Dasari, Siddharth Karamcheti, Soroush Nasiriany, Mohan Kumar Srirama, Lawrence Yunliang Chen, Kirsty Ellis, Peter David Fagan, Joey Hejna, et al. DROID: A Large-Scale In-The-Wild Robot Manipulation Datas... on PithPith's reviewarXiv
  43. [67] Moo Jin Kim, Karl Pertsch, Siddharth Karamcheti, Ted Xiao, Ashwin Balakrishna, Suraj Nair, Rafael Rafailov, Ethan Foster, Pannag Sanketi, Quan Vuong, Thomas Kollar, Benjamin Burchfiel, Russ Tedrake, Dorsa Sadigh, Sergey Levine, Percy Liang, and Chelsea Finn. OpenVLA: An Open-S... on PithPith's reviewarXiv
  44. [68] Moo Jin Kim, Chelsea Finn, and Percy Liang. Fine-Tuning Vision-Language-Action Models: Optimizing Speed and Success.arXiv preprint arXiv:2502.19645, 2025. URL https://arxiv.org/ abs/2502.19645. RSS 2025 on PithPith's reviewarXiv
  45. [69] Moo Jin Kim, Yihuai Gao, Tsung-Yi Lin, Yen-Chen Lin, Yunhao Ge, Grace Lam, Percy Liang, Shuran Song, et al. Cosmos Policy: Fine-Tuning Video Models for Visuomotor Control and Planning.arXiv preprint arXiv:2601.16163, 2026. URLhttps://arxiv.org/abs/2601.16163 on PithPith's reviewarXiv
  46. [70] Kushal Kedia, Tyler Ga Wei Lum, Jeannette Bohg, and C. Karen Liu. SimToolReal: An Object- Centric Policy for Zero-Shot Dexterous Tool Manipulation.arXiv preprint arXiv:2602.16863, 2026. URLhttps://arxiv.org/abs/2602.16863 on PithPith's reviewarXiv
  47. [71] Yejin Kim, Wilbert Pumacay, Omar Rayyan, Max Argus, Winson Han, Eli VanderBilt, Jordi Salvador, Abhay Deshpande, Rose Hendrix, Snehal Jauhri, Shuo Liu, Nur Muhammad Mahi Shafiullah, Maya Guru, Ainaz Eftekhar, Karen Farley, Donovan Clay, Jiafei Duan, Arjun Guru, Piper Wolters, ... on PitharXiv
  48. [72] So Kuroki, Kento Kawaharazuka, and Yujin Tang. Learning and Transferring Closed-Loop Robot Software.arXiv preprint arXiv:2609.19906, 2026. URLhttps://arxiv.org/abs/2609.19906 on PithPith's reviewarXiv
  49. [74] Zhiqian Lan, Yuxuan Jiang, Ruiqi Wang, Xuanbing Xie, Rongkui Zhang, Yicheng Zhu, Peihang Li, Tianshuo Yang, Tianxing Chen, Haoyu Gao, Xiaokang Yang, Xuelong Li, Hongyuan Zhang, Yao Mu, and Ping Luo. AutoBio: A Simulation and Benchmark for Robotic Automation in Digital Biology ... on PithPith's reviewarXiv
  50. [75] Seungyeon Kim, Junhoo Lee, Minkyu Kim, Baekseung Kim, and Nojun Kwak. Recursive Harness Distillation across Agents for Robot Manipulation.arXiv preprint arXiv:2609.33378, 2026. URL https://arxiv.org/abs/2609.33378 on PithPith's reviewarXiv
  51. [76] Chengshu Li, Ruohan Zhang, Josiah Wong, Cem Gokmen, Sanjana Srivastava, Roberto Martín- Martín, Chen Wang, Gabrael Levine, Wensi Ai, Benjamin Martinez, Hang Yin, Michael Lingelbach, Minjune Hwang, Ayano Hiranaka, Sujay Garlanka, Arman Aydin, Sharon Lee, Jiankai Sun, Mona Anvar... on PithPith's reviewarXiv
  52. [77] Long Li, Qichao Zhao, Yue Yang, Fan Xu, Zhe Wang, Alan Wee-Chung Liew, Chao Qu, Heng Tao Shen, and Shirui Pan. Spotter: Let the Embodied Model Lead, and the VLM Reflect for It.arXiv preprint arXiv:2609.36808, 2026. URLhttps://arxiv.org/abs/2609.36808 on PithPith's reviewarXiv
  53. [78] Manling Li, Shiyu Zhao, Qineng Wang, Kangrui Wang, Yu Zhou, Sanjana Srivastava, Cem Gokmen, Tony Lee, Li Erran Li, Ruohan Zhang, Weiyu Liu, Percy Liang, Li Fei-Fei, Jiayuan Mao, and Jiajun Wu. Embodied Agent Interface: Benchmarking LLMs for Embodied Decision Making.arXiv prepr... on PithPith's reviewarXiv
  54. [79] URLhttps://arxiv.org/abs/2503.14499 on PithPith's reviewarXiv
  55. [80] Ruiying Li, Yunlang Zhou, YuYao Zhu, Kylin Chen, Jingyuan Wang, Sukai Wang, Kongtao Hu, Minhui Yu, Bowen Jiang, Zhan Su, Jiayao Ma, Xin He, Yongjian Shen, Yang Yang, Guanghui Ren, Maoqing Yao, Wenhao Wang, and Yao Mu. RoboClaw: An Agentic Framework for Scalable Long-Horizon Ro... on PitharXiv
  56. [81] Hojoon Lee, Ajay Subramanian, Ben Abbatematteo, Vijay Veerabadran, Pedro Matias, Karl Ridge- way, and Nitin Kamra. Reward Design Agent for Reinforcement Learning.arXiv preprint arXiv:2606.01672, 2026. URLhttps://arxiv.org/abs/2606.01672. RLC 2026. 75 on PithPith's reviewarXiv
  57. [82] Xuanlin Li, Kyle Hsu, Jiayuan Gu, Karl Pertsch, Oier Mees, Homer Rich Walke, Chuyuan Fu, Ishikaa Lunawat, Isabel Sieh, Sean Kirmani, Sergey Levine, Jiajun Wu, Chelsea Finn, Hao Su, Quan Vuong, and Ted Xiao. Evaluating Real-World Robot Manipulation Policies in Simulation.arXiv ... on PithPith's reviewarXiv
  58. [83] Zechu Li, Yufeng Jin, Xiaoyang Liu, Puze Liu, Vignesh Prasad, Carlo D’Eramo, and Georgia Chalvatzaki. HARBOR: A Harness Framework for Agentic Robot Reinforcement Learning.arXiv preprint arXiv:2606.08610, 2026. URLhttps://arxiv.org/abs/2606.08610 on PithPith's reviewarXiv
  59. [84] Jacky Liang, Wenlong Huang, Fei Xia, Peng Xu, Karol Hausman, Brian Ichter, Pete Florence, and Andy Zeng. Code as Policies: Language Model Programs for Embodied Control.arXiv preprint arXiv:2209.07753, 2022. URLhttps://arxiv.org/abs/2209.07753. ICRA 2023 on PithPith's reviewarXiv
  60. [85] Rui Li, Zixuan Hu, Wenxi Qu, Jinouwen Zhang, Zhenfei Yin, Sha Zhang, Xuantuo Huang, Hanqing Wang, Tai Wang, Jiangmiao Pang, Wanli Ouyang, Lei Bai, Wangmeng Zuo, Ling-Yu Duan, Dongzhan Zhou, and Shixiang Tang. LabUtopia: High-Fidelity Simulation and Hierar- chical Benchmark for... on PitharXiv
  61. [86] Kevin Lin, Christopher Agia, Toki Migimatsu, Marco Pavone, and Jeannette Bohg. Text2Motion: From Natural Language Instructions to Feasible Plans.arXiv preprint arXiv:2303.12153, 2023. URL https://arxiv.org/abs/2303.12153. Autonomous Robots 2023 on PithPith's reviewarXiv
  62. [87] Shuang Li, Yihuai Gao, Dorsa Sadigh, and Shuran Song. Unified Video Action Model.arXiv preprint arXiv:2503.00200, 2025. URLhttps://arxiv.org/abs/2503.00200. RSS 2025 on PithPith's reviewarXiv
  63. [88] Fangcheng Liu, Yeqing Shen, Anda Cheng, Weishi Mi, Chao Tang, Chenyuan Liu, Yushun Xiang, Tingguang Li, Yong-Lu Li, and Yehui Tang. RoboICL: Embodied In-Context Learning with GPT-6 Astra.arXiv preprint arXiv:2609.34261, 2026. URLhttps://arxiv.org/abs/2609.34261 on PithPith's reviewarXiv
  64. [89] Renhang Liu, Navonil Majumder, Tej Deep Pala, and Soujanya Poria. RoboQuest: Generalist Physical Agents that Search, Inspect and Test.arXiv preprint arXiv:2610.10388, 2026. URL https: //arxiv.org/abs/2610.10388 on PithPith's reviewarXiv
  65. [90] Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding, Kaiwen Men, Kejuan Yang, Shudan Zhang, Xiang Deng, Aohan Zeng, Zhengxiao Du, Chenhui Zhang, Sheng Shen, Tianjun Zhang, Yu Su, Huan Sun, Minlie Huang, Yuxiao Dong, and Jie Tang. AgentBench:... on PithPith's reviewarXiv
  66. [91] Chunru Lin, Hongxin Zhang, Fenghao Yu, Zhehuan Chen, Thomas L. Griffiths, Yejin Choi, David Held, and Chuang Gan. RoboWits: Unexpected Challenges for Robotic Creative Problem Solving. arXiv preprint arXiv:2605.30326, 2026. URLhttps://arxiv.org/abs/2605.30326 on PithPith's reviewarXiv
  67. [92] Yuyao Liu, Jiayuan Mao, David Hsu, Leslie Pack Kaelbling, and Tomás Lozano-Pérez. RAPID: Robot Agentic Programming from Demonstrations.arXiv preprint arXiv:2609.30249, 2026. URL https://arxiv.org/abs/2609.30249 on PithPith's reviewarXiv
  68. [94] URL https://arxiv.org/abs/2306.03310 . NeurIPS 2023 Datasets and Benchmarks Track. 76 on PithPith's reviewarXiv
  69. [95] Yecheng Jason Ma, William Liang, Guanzhi Wang, De-An Huang, Osbert Bastani, Dinesh Jayara- man, Yuke Zhu, Linxi “Jim” Fan, and Anima Anandkumar. Eureka: Human-Level Reward Design via Coding Large Language Models. InInternational Conference on Learning Representations (ICLR), 2024 on Pith
  70. [96] Yecheng Jason Ma, William Liang, Hung-Ju Wang, Yuke Zhu, Linxi “Jim” Fan, Osbert Bastani, and Dinesh Jayaraman. DrEureka: Language Model Guided Sim-To-Real Transfer.arXiv preprint arXiv:2406.01967, 2024. URLhttps://arxiv.org/abs/2406.01967. RSS 2024 on PithPith's reviewarXiv
  71. [97] Yuntao Ma, Andrei Cramariuc, Farbod Farshidian, and Marco Hutter. Learning coordinated badminton skills for legged manipulators.Science Robotics, 10(102):eadu3922, 2025. URL https: //arxiv.org/abs/2505.22974 on PitharXiv
  72. [98] Xiao Liu, Tianjie Zhang, Yu Gu, Iat Long Iong, Yifan Xu, Xixuan Song, Shudan Zhang, Hanyu Lai, Xinyi Liu, Hanlin Zhao, Jiadai Sun, Xinyue Yang, Yu Yang, Zehan Qi, Shuntian Yao, Xueqiao Sun, Siyi Cheng, Qinkai Zheng, Hao Yu, Hanchen Zhang, Wenyi Hong, Ming Ding, Lihang Pan, Xia... on PithPith's reviewarXiv
  73. [101] URLhttps://arxiv.org/abs/2607.00272 on PithPith's reviewarXiv
  74. [102] Zuxing Lu, Hongjia Zhai, Guanzhi Wang, Huajian Zeng, Jiaqi Yang, Jingyu Liu, Lei Cheng, Yuantai Zhang, et al. World-Action Models for Robot Learning and Control: A Survey.arXiv preprint arXiv:2609.16074, 2026. URLhttps://arxiv.org/abs/2609.16074 on PithPith's reviewarXiv
  75. [103] Grégoire Mialon, Clémentine Fourrier, Craig Swift, Thomas Wolf, Yann LeCun, and Thomas Scialom. GAIA: a benchmark for General AI Assistants.arXiv preprint arXiv:2311.12983, 2023. URL https://arxiv.org/abs/2311.12983. ICLR 2024 on PithPith's reviewarXiv
  76. [104] Samuel Miserendino, Michele Wang, Tejal Patwardhan, and Johannes Heidecke. SWE-Lancer: Can Frontier LLMs Earn $1 Million from Real-World Freelance Software Engineering?arXiv preprint arXiv:2502.12115, 2025. URLhttps://arxiv.org/abs/2502.12115 on PithPith's reviewarXiv
  77. [105] Mayank Mittal, Calvin Yu, Qinxi Yu, Jingzhou Liu, Nikita Rudin, David Hoeller, Jia Lin Yuan, Ritvik Singh, Yunrong Guo, Hammad Mazhar, Ajay Mandlekar, Buck Babich, Gavriel State, Marco Hutter, and Animesh Garg. Orbit: A Unified Simulation Framework for Interactive Robot Learni... on PithPith's reviewarXiv
  78. [107] Soroush Nasiriany, Sepehr Nasiriany, Abhiram Maddukuri, and Yuke Zhu. RoboCasa365: A Large- Scale Simulation Framework for Training and Benchmarking Generalist Robots.arXiv preprint arXiv:2603.04356, 2026. URLhttps://arxiv.org/abs/2603.04356. ICLR 2026 on PitharXiv
  79. [108] Chang Nie, Zhe Liu, and Hesheng Wang. Teach and Grow: An Agent-Centered Architecture for General Robot Learning.arXiv preprint arXiv:2608.17209, 2026. URL https://arxiv.org/abs/ 2608.17209 on PithPith's reviewarXiv
  80. [109] URL https://arxiv.org/abs/2505.11289 . NeurIPS 2025 Datasets and Benchmarks Track. 77 on PitharXiv

The first 100 of 204 references are listed here. All of them are in this paper's citation record.

Challenges and discussion

Disagree with the review, or know something it missed? Leave a comment, or challenge this review formally: a challenge becomes a permanent entry in this paper's record, and its decision is published here.

No comments yet.

Sign in to comment