Pith.

Robotics · arXiv:2610.30136 v1 · submitted 21 Oct 2026 · 15 references

Agentic World Action Models

Zhuoyang Liu, Dantong Niu, Botong Zhang, Boning Shao, Jing Wang, Ruijie Zheng, Yuvan Sharma, Runyu Lu

show 10 more authors

Jiaming Liu, Renrui Zhang, Roei Herzig, Yutong Bai, Ken Goldberg, Danfei Xu, Yuke Zhu, Guanzhi Wang, Linxi "Jim" Fan, Trevor Darrell

The paper says

“Recent world action models provide visual foresight and future prediction for robot manipulation, but lack explicit mechanisms for compositional skill reuse. Agentic robotics combines code-based planning and reasoning with reusable skills, but often lacks predictive grounding in physical dynamics.”

more of the abstract

We introduce AW-0, an agentic world action model framework that integrates code-based planning with joint video and action prediction. We jointly model code, future video, and robot actions as three coupled generative streams within a unified mixture-of-transformers architecture. Within this architecture a code expert generates parameterized skill routines and control logic while video and action experts generate future visual states and low-level motor commands. Code serves two complementary roles: it structures multi-step prediction and control within the world model and represents skills as reusable programs for subsequent agentic execution. We further introduce an automated curation pipeline that labels robot and human video trajectories with aligned programs and consolidates recurring routines into reusable skill libraries. Evaluations on RoboTwin 2.0 and physical bimanual manipulation tasks show improved task performance over VLA and WAM baselines, alongside gains in language following, semantic reasoning and robustness, demonstrating that agentic code enables connection between planning, prediction and execution within WAMs.

from the abstract

Pith says

Machine review · deepseek-v4-flash · 22 Oct 2026 · what was read was not recorded · not checked by a person

It can be wrong. Challenge this review or reply to it.

Holds up, with conditions. The joint code-video-action denoising is genuinely new against the cited WAM literature and the ablations are unusually honest, but the causal claim that code semantics drive the gains stays conditional because label fidelity is measured, reported as modest, and never experimentally isolated. This line is wrong

2 major objections · 7 minor

How often objections from reviews like this one are right: not yet measured

Are you an author? What you can do.

Not checked: figures, code, what the 15 references say.

The paper says, at §2.3, §3.3, Eq. (2)

The review did not quote this passage.

A classifier read this sentence of the paper as acknowledging the concern (probability 0.78):

“Generated programs can select an appropriate skill yet contain incorrect arguments, as illustrated in Appendix F.1; syntactic validity therefore does not ensure correct exe- cution.”

Pith says: Objection 1 of 2 · major

The central claim (Section 1: 'our primary contribution is the conceptualization and realization of agentic robotic skills within a world action model framework, showing that code generation improves action prediction') is causal: code improves action prediction because it supplies grounded semantic plans.

more of objection 1

The Q1 ablations control for parameter count and attention placement—the V→C→A variant retains the full code expert with identical data, budget, and settings and still underperforms C→V→A (Figure 5)—and the Q2 ablations show that comments and skill calls matter (Figure 7). What is not controlled is the fidelity of the code labels themselves: Appendix K.2 reports that validation is static (syntax, function existence, argument binding, vocabulary) and never checks physical correctness against the video, and Table 10 reports modest agreement (Entity F1 0.534, Primitive F1 0.365). The authors acknowledge the consequence in Appendix L ('syntactic validity therefore does not ensure correct execution'), but no experiment separates 'a code stream whose content is semantically correct' from 'a code stream of the same form whose labels are arbitrarily assigned.' I recommend a scrambled-label variant: train with the same code stream, canvas length, and budget, but with program labels shuffled across episodes of the same task (or with argument values corrupted), and compare against the full model; a human audit of, say, 100 sampled labels per dataset for physical correctness would also quantify label quality. The YAM constraint that programs must reproduce the sensor-derived gripper-event list (Appendix K.2) partially grounds the contact structure of the YAM labels, but object names, motion arguments, and the entire egocentric-human half of the corpus are ungrounded; the mitigation should cover both datasets.

The paper says, at Tables 1, 2, 4; Figures 5–7

The review did not quote this passage.

Pith says: Objection 2 of 2 · major

None of the quantitative tables reports variance, confidence intervals, or significance tests. The headline comparisons—50.7% vs 41.3% on RoboTwin 2.0 and 71.1% vs 46.0% on the real-world tasks—are averages over 50 rollouts per simulated task and 3 seeds × 16 rollouts per real-world task.

more of objection 2

With binomial sampling, a per-task real-world gap of 5–10 points (e.g., Replace Lightbulb 75.0 vs 67.4 in Table 2) is plausibly within noise, and the RoboTwin per-task table shows wide dispersion with many near-zero entries. The 29.2% vs 27.7% hybrid gain in the random setting is a 1.5-point margin and should be treated with particular care. I ask for seed-level spreads or binomial confidence intervals on the headline averages and a paired (per-task) or permutation test for the main AW-0-versus-best-baseline differences, so the reader can assess whether the claimed gains are statistically distinguishable.

7 minor objections

The paper says, at Table 11

Pith says: minor objection 1 of 7

The per-task RoboTwin results in Table 11 appear misaligned in the manuscript text, with entries running together (several rows read as single long numbers); please verify the table rendering in the camera-ready version.

The paper says, at §3.3, Figure 7

Pith says: minor objection 2 of 7

The 'Subgoal' variant in Figure 7 is not fully specified: please state whether the natural-language subgoals use the same tokenizer, canvas length, masking procedure, and loss as the code stream, so that the comparison is controlled.

show 5 more

The paper says, at §3.1, Appendix J.2

Pith says: minor objection 3 of 7

The hybrid-versus-default comparison is confounded with the switch to CaP-collected demonstrations, since the two configurations differ in both data collection and execution mode; the authors disclose this in Appendix J.2, but the main text should state the confound explicitly next to the robustness claim based on Table 1.

The paper says, at Appendix G.1, G.2

Pith says: minor objection 4 of 7

The qualitative video-prediction comparisons (Figures 33–34) use non-identical initial observations from separate rollouts; a controlled comparison with matched initial states, or a small quantitative video metric, would strengthen the language-following and skill-robustness discussion.

The paper says, at §2.2, Eq. (6)

Pith says: minor objection 5 of 7

The generation hyperparameters α=0.5, γ_v=γ_a=5, and the loss weights (1, 0.5, 1) are presented as fixed choices without a sensitivity analysis; please report at least one sensitivity run or state explicitly that these values were fixed a priori.

The paper says, at Appendix I, F.2

Pith says: minor objection 6 of 7

Because the hybrid arbiter accepts slices on validity alone, it would be informative to report the fraction of generated slices that pass the arbiter's gates during deployment; the malformed-comment example in F.2 shows this pass rate can be below 100%, which is directly relevant to the agentic-reuse claim.

The paper says, at References

Pith says: minor objection 7 of 7

There are several typographical errors in the reference list (e.g., 'Huan ang Gao' in the RoboTwin 2.0 entry) and inconsistent spacing around some baseline names in the main text; please copy-edit.

What would settle it

Run a matched-budget post-training comparison on the eight real-world tasks (or a RoboTwin subset) among: (A) standard annotated code; (B) scrambled-semantics code, where skill names and object arguments within each task's program slices are permuted across demonstrations while keeping syntax and token statistics fixed; and (C) the existing V→A w/o-code baseline, using the same C→V→A mixture-of-transformers architecture wherever a code stream is present. If (B) retains most of the (A) vs (C) gap, the code stream's benefit is not semantic planning and the central claim needs to be narrowed; if (B) collapses toward (C) while (A) keeps its gain, label content is causal and the claim survives this test.

from the skeptic's reading

How the review went

4 readings by deepseek-v4-flash · 22 Oct 2026

  1. Reader holds up, with conditions confidence high
  2. Skeptic agrees 50 s · 44,593 tokens read, 6,345 written
  3. Editor wrote the verdict above 35 s · 45,674 tokens read, 4,436 written
  4. Referee major revision: 2 major objections, 7 minor 1 min 23 s · 44,931 tokens read, 12,135 written

Not run on this paper: the simulated authors' defence.

Pith did not check this review's objections against the paper.

The review in full

The review in brief, machine-written

Two things to know. First, AW-0's core move is genuinely new relative to the cited literature: no prior WAM jointly denoises a discrete code stream with video and action under blockwise-causal C-to-V-to-A attention. WLA generates text subtasks autoregressively, DreamZero and Cosmos-Policy do video and action without a code stream, and CaP/ASPIRE execute code outside a trained policy. Second, hold the causal claim from Section 1 at arm's length: the paper asserts that code generation improves action prediction and that the code is reusable agentically. The ablations support the first half, but the one experiment that would settle it is missing.

What I credit. The ablation suite is the strongest part. The w/o-code and attention-order variants share pretraining data, budget, and optimization settings, and the content ablations (no comments, no skills, subgoal text instead of code) degrade performance across all 8 shared real-world tasks. Removing comments or skills changes label content while keeping the code stream present, so the crude "any code-shaped token stream helps" hypothesis is partly ruled out. The paper is also transparent on its weak spots: it admits the simulation hybrid comparison is confounded with a separate CaP-based data-collection pipeline (J.2), that syntactic validity does not ensure correct execution (Appendix L; the malformed-comment and bad-argument examples in F.1 and F.2), and that the random-setting RoboTwin result (20.6%) sits below pi0.5 (27.7%). That honesty is real and not routine.

The soft spots, in proportion. Reporting is the minor one: no variance or confidence intervals anywhere, despite 3 seeds times 16 rollouts per real task being claimed. Fixable, but it should be there. Mechanism is the real one: the annotation pipeline validates syntax, vocabulary, and argument binding only (K.2), never physical correctness against the video, and the pretrained code expert's agreement with reference programs is modest (Entity F1 0.534, Primitive F1 0.365). The authors flag the label-quality limitation explicitly, so this is not a concealed flaw; what is unresolved is whether the measured gains come from the planning semantics or from the added structure and reduced task ambiguity. A scrambled-label or corrupted-label ablation with the same budget would close that gap, and its absence is the load-bearing weakness given the headline claim is causal.

Who this is for: anyone working on world action models, VLA-planning integration, or programmatic skill discovery from video. It deserves a serious referee, not a desk reject. The architecture is worth engaging with, the ablations are well designed, and the failure analysis is instructive. A revision adding the label-fidelity control and per-cell variance would let the empirical and causal claims line up.

The strongest objection

The paper's central claim (Section 1) is causal: code generated inside the world model improves action prediction and can be reused agentically. The training signal for that code stream is the annotation pipeline of Section 2.3, and the paper's own appendices show the labels are validated syntactically, not physically: static checks cover syntax, function existence, argument binding, and vocabulary (Appendix K.2), while the model's agreement with the reference programs is modest (Entity F1 0.534, Primitive F1 0.365, 87.5% parse rate), and even strong generations can contain wrong argument counts or malformed comments (Appendices F.1, F.2). The ablations in Figure 7 and Table 4 show that code, comments, and skills matter relative to a model with no code stream, but they do not control for label fidelity: a code expert trained on plausible-but-physically-wrong programs could still help through extra parameters, reduced task ambiguity, or the C→V→A attention ordering, without encoding the reusable skills the paper claims. The second half of the primary claim is exposed too, because agentic reuse requires executability, and the paper's own deployment examples include raw output that would not run as generated (F.2). The authors are transparent about these limitations in Appendix L, but the causal attribution in the headline claim has not yet been separated from annotation quality.

I read AW-0 in good faith. The framework is genuinely novel: jointly denoising code, video, and actions with a code expert is not present in the cited WAM or agentic literatures, the ablation suite is extensive, the real-world gains are large, and the paper is unusually transparent about its failure cases and confounds (Appendix J.2). The reader's conditional verdict is well calibrated. The remaining soft spot is not that the labels are imperfect—the paper says so—but that the central causal claim has not been separated from label fidelity. The reader's weakest assumption identifies exactly this issue, and I agree with it. Existing ablations compare code vs. no code, and full code vs. degraded code content, but no variant holds the code stream fixed while breaking the semantic link between labels and demonstrated motion. Without that control, the measured gains could in principle come from a code-shaped conditioning stream rather than from accurate skill-level planning. This is a real concern, but it is not a demonstrated failure: the code-content ablations (comments, skills, subgoals) lean in the paper's favor, and the authors explicitly acknowledge that syntactic validity does not imply executability. The paper should therefore remain CONDITIONAL rather than being rejected or marked unverdictable, and the reader's verdict does not need to change. A scrambled-label control is the single experiment that would decide whether the central claim should be re-expressed from 'code improves action prediction' to 'a code-shaped stream improves action prediction.'

The referee's report

AW-0 is a well-engineered integration of two active lines of work—world action models and code-based agentic policies—and, if the mechanism holds, provides a useful template for inserting structured planning into predictive world-action models. The evaluation is externally grounded: RoboTwin success rates and additive real-world rubrics are measured against baselines rather than against quantities the model was fit to reproduce. The ablations are a genuine strength: the w/o-code and V→C→A variants in Figure 5 share pretraining data, training budget, and optimization settings with the full model, ruling out the crudest 'added parameters' explanation, and the content ablations in Figure 7 show that comments and skill calls contribute beyond raw program structure. The paper is also unusually transparent: Appendix K.2 states that label validation is static, Appendices F.1 and F.2 show generated programs with wrong argument counts and non-parsing output, Appendix J.2 discloses the CaP-collection confound in the hybrid comparison, and Appendix L acknowledges that syntactic validity does not ensure correct execution.

Pith's reading of the paper (a machine's summary, not the authors')

AW-0 sets out to give robot world models an explicit planning layer. The central claim is that adding a code-generation stream to a world action model — so that skill programs, future video, and motor commands are generated together, with the program conditioning the video and the video conditioning the actions — improves task success over both vision-language-action and world-action baselines. On RoboTwin 2.0 the model averages 50.7% success across clean and randomized settings (80.8% clean), and on eight real bimanual and dexterous-hand tasks it averages 71.1% rubric score versus 46.0% for the strongest baseline. The paper further shows the generated code can be executed directly as a policy, with a hybrid controller that improves robustness under visual randomization. If the claim holds, writing reusable skill programs inside the model rather than outside it is a practical way to connect planning, prediction, and control.

What carries the argument. The load-bearing object is the mixture-of-transformers backbone with three experts — code, video, and action — that communicate through masked joint attention with a blockwise causal order of code before video before action ($C \to V \to A$). The code expert runs discrete absorbing-mask diffusion over program tokens while the video and action experts run continuous flow matching, and all three streams are denoised in the same loop (Eq. 2), so a partially decoded program conditions future-frame prediction, which in turn conditions action generation. Two supporting mechanisms carry the rest: an automated curation pipeline that annotates roughly 2,900 hours of robot and human demonstrations with aligned programs and distills a frozen library of 50 reusable skills, and a hybrid execution controller that arbitrates between executing the decoded program and playing the predicted action chunk.

Core claim. The paper claims that agentic code belongs inside a world action model, not outside it. Concretely, AW-0 is a mixture-of-transformers with three coupled generative streams — a code expert that writes parameterized skill routines, a video expert that predicts future frames, and an action expert that emits motor commands — denoised jointly, with blockwise causal attention in the order code → video → action ($C \to V \to A$) so the program shapes both future prediction and control. Trained on about 2,900 hours of robot and human demonstrations labeled by an automated pipeline with temporally aligned program slices, the model reaches 50.7% average success on RoboTwin 2.0 and 71.1% average rubric score across eight real-world tasks, and its ablations attribute the gains to the code stream, to its $C \to V \to A$ placement, and to the skill calls inside the programs. A hybrid execution mode that runs the decoded program when it passes validity gates improves robustness under distribution shift (29.2% vs 20.6% under randomization) at a cost in the clean setting (63.1% vs 80.8%). The authors report that generated code is only partially grounded — teacher-forced token accuracy is high but Entity F1 is 0.534 and Primitive F1 is 0.365 — so the claim concerns planning as a conditioning structure as much as executable output.

The premise it rests on. The load-bearing premise is that the automatically written programs and the frozen 50-skill library genuinely describe the motions in the demonstrations, since the labels are checked only for syntax, vocabulary, and argument binding and never for physical correctness against the video.

If the paper is right (Pith's inferences, not the authors')

Code generation inside the world model improves action prediction: the $C \to V \to A$ ordering outperforms both the no-code and $V \to C \to A$ variants on all eight real-world tasks, so task plans written as programs are a workable conditioning signal for future video and control.

The 50-skill libraries extracted from roughly 2,900 hours of robot and human demonstrations give pretraining a shareable planning vocabulary: removing skill calls from programs drops performance on all eight tasks, most sharply where skills matter (Deal Poker, 58.3% vs 42.0%).

Generated code is reusable at execution time: hybrid arbitration between decoded programs and direct action chunks improves robustness under visual distribution shift (29.2% vs 20.6% on RoboTwin random) while retaining 46% of clean-setting success versus 25% without code execution.

Joint denoising is both more accurate and cheaper than autoregressive program generation (0.23 s vs 0.55 s inference, better on 7 of 8 tasks), so planning can be folded into the generative loop instead of serialized before it.

Language following and semantic reasoning in world models improve when semantic plans shape video prediction, with gains concentrating on instruction-sensitive tasks such as Arrange Fruits and Flip Scoreboard.

If the code stream is the genuine driver, then physically grounding the program labels should raise the ceiling further: the paper's own appendix reports Entity F1 of 0.534 and Primitive F1 of 0.365 for generated code, so correcting argument-level errors against the video is a direct, testable next step.

A learned arbiter that predicts whether code execution will beat the action chunk could resolve the clean/random trade-off, since the current controller admits code on validity checks alone and concedes 17.7 points in the clean setting.

The skill-library framing suggests a scaling claim worth testing: because the 50 skills recur across 606 pretraining tasks, success on a new task should track whether its motions are covered by the frozen library, and adding skills should transfer to new tasks without retraining.

Check this review

Pith stored no record of this review's passes and took no fingerprint of it. Reviews made by code from 13 Nov 2026 on carry both.

How often reviews like this one are right

Reviews like this one: written by deepseek-v4-flash, where what was read was not recorded.

How often objections from reviews like this one are right: not yet measured

Every value above, in the review's public record (JSON)

Reviews by people

This review was written by a machine. No person has reviewed this paper on Pith yet. A scientist listed on Pith can review it for a fee, and their review is published here whatever it concludes.

Request a human review

How Pith checks papers · The full audit of this paper

The paper

Abstract

Recent world action models provide visual foresight and future prediction for robot manipulation, but lack explicit mechanisms for compositional skill reuse. Agentic robotics combines code-based planning and reasoning with reusable skills, but often lacks predictive grounding in physical dynamics. We introduce AW-0, an agentic world action model framework that integrates code-based planning with joint video and action prediction. We jointly model code, future video, and robot actions as three coupled generative streams within a unified mixture-of-transformers architecture. Within this architecture a code expert generates parameterized skill routines and control logic while video and action experts generate future visual states and low-level motor commands. Code serves two complementary roles: it structures multi-step prediction and control within the world model and represents skills as reusable programs for subsequent agentic execution. We further introduce an automated curation pipeline that labels robot and human video trajectories with aligned programs and consolidates recurring routines into reusable skill libraries. Evaluations on RoboTwin 2.0 and physical bimanual manipulation tasks show improved task performance over VLA and WAM baselines, alongside gains in language following, semantic reasoning and robustness, demonstrating that agentic code enables connection between planning, prediction and execution within WAMs.

The paper's own sentences, arranged, kept beside the review and separate from it. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. No sentence in this record was written by a model.

abstract compiled · 2026-10-22

The authors' main claim, as the classifier picked it out 1.00

“Evaluations on RoboTwin 2.0 and physical bimanual manipulation tasks show improved task performance over VLA and WAM baselines, alongside gains in language following, semantic reasoning and robustness, demonstrating that agentic code enables connection between planning, prediction and execution within WAMs.”

How the main claim is stated: asserted 1.00

What the abstract itself states, as probabilities: claims first 0.09 · states limitation 0.09 · human participants 0.35 · controlled comparison 0.92 · randomized assignment 0.01 · states quantitative result 0.06 · compares with prior results 0.97 · mentions code or data availability 0.05

The abstract, sentence by sentence

This table labels the abstract only (arXiv listing abstract v1). The number is the probability the classifier gave the label.

SentenceRoleThe authors' own workStates the main conclusionStrength
Recent world action models provide visual foresight and future prediction for robot manipulation, but lack explicit mechanisms for compositional skill reuse. gap 0.75 0.05 0.02 asserted 1.00
Agentic robotics combines code-based planning and reasoning with reusable skills, but often lacks predictive grounding in physical dynamics. background 0.52 0.04 0.02 asserted 0.95
We introduce AW-0, an agentic world action model framework that integrates code-based planning with joint video and action prediction. contribution 1.00 0.93 0.08 asserted 1.00
We jointly model code, future video, and robot actions as three coupled generative streams within a unified mixture-of-transformers architecture. method 0.96 0.89 0.04 asserted 0.99
Within this architecture a code expert generates parameterized skill routines and control logic while video and action experts generate future visual states and low-level motor commands. method 0.99 0.75 0.04 asserted 1.00
Code serves two complementary roles: it structures multi-step prediction and control within the world model and represents skills as reusable programs for subsequent agentic execution. method 0.87 0.65 0.15 asserted 1.00
We further introduce an automated curation pipeline that labels robot and human video trajectories with aligned programs and consolidates recurring routines into reusable skill libraries. method 0.75 0.95 0.04 asserted 1.00
Evaluations on RoboTwin 2.0 and physical bimanual manipulation tasks show improved task performance over VLA and WAM baselines, alongside gains in language following, semantic reasoning and robustness, demonstrating that agentic code enables connection between planning, prediction and execution within WAMs. result 0.97 0.96 0.90 asserted 1.00

What kind of paper the classifier reports this is: method or tool 1.00 · evidence from observation or experiment 0.93

What was examined, and what was not

Sources read

  • arXiv listing abstract v1

Not examined

  • full text
  • figures and tables
  • references and citing sentences
  • relations to other papers

Levels run

  • P abstract

Not run

  • S sections
  • T body sentences
  • C citations
  • R cross-paper relations

How the labels were made

Compiled by classifier-1.13.0 via TypeSafe API, question bank abstract-p1-20260923, 2026-10-22: 1 request, 6,258 input tokens.

Every quoted sentence on this record is the authors'. Every label is from Pith's closed vocabulary, with the probability shown beside it. This record as JSON

Figures (38)
Figure 1 of the paper
Figure 1. Overview of AW-0. AW-0 is an agentic world action model that unifies code-based planning with world action modeling by jointly denoising skill programs, future video, and control trajectories. An agentic pipeline discovers reusable skills and annotates robot and human demonstra￾tions with programs for both pretraining and downstream fine-tuning. The resulting policy achieves strong performance on both parallel-gripper and dexterous-hand tasks. ABSTRACT Recent world action models provide visual foresight and future prediction for robot manipulation, but lack explicit mechanisms for…
Figure 2 of the paper
Figure 2. AW-0 Framework. AW-0 includes a Mixture-of-Transformers backbone with three ex￾perts generating code, video and action, respectively, which are denoised jointly, with video and action under continuous flow matching and the code under discrete masked diffusion. 2 AGENTIC WORLD ACTION MODEL We formulate AW-0 as a unified conditional generative model over code, video, and actions, which respectively describe structured skill plans, future scene dynamics, and continuous robot motion. We represent a skill program as a discrete token sequence C0 = (c1, . . . , cNc ), future observations…
Figure 3 of the paper
Figure 3. Automatic Annotation Pipeline. Coding agents generate subtask-level programs using predefined primitive APIs and available skills, while a coordinator distills recurring patterns into a library of reusable skills across tasks. With the library frozen, we annotate the corpus by reusing cached code when contact and motion evidence match and generating new labels otherwise. Code Usage in Model. Annotated programs define the interface through which the code expert plans. During training, the code stream concatenates a conditioning prompt and the active slice, a short block of primitive…
Figure 4 of the paper
Figure 4. Average Scores from
Figure 5 of the paper
Figure 5. Impact of Code Expert and Attention Structure. Avg. rubric score (%) over 3 × 16 rollouts/task. C, V , and A denote the code, video, and action streams
Figure 6 of the paper
Figure 6. Performance and Inference La￾tency. Average rubric score over the 8 real￾world tasks (bars) and inference latency on an NVIDIA RTX 5090 (red line). contribute to task performance, and 3) how joint denoising affects task performance and inference efficiency. All the variants are post-trained and evaluated under the protocol of Section 3.2. Q1: How should the code expert be integrated into a WAM? We first compare three blockwise causal attention orders during joint denoising: V → A (w/o code), V → C → A, and C → V → A (
Figure 7 of the paper
Figure 7. Impact of Code Label Content. Average rubric score (%) over 3 × 16 rollouts per task. Q2: What makes code an effective planning representation? To assess the planning content produced by our annotation pipeline, we compare (1) full annotated programs, (2) code without comments, (3) code without skills, and (4) natural-language subgoals in
Figure 8 of the paper
Figure 8. Key frames of Skill I, fold edge over. Skill definition: def fold_edge_over(robot, item_name, left_delta, right_delta): # pinch the edge with both arms robot.grasp(item_name, ’left’) robot.grasp(item_name, ’right’) # carry the pinched edge across the item robot.servo_ee_delta(’left’, left_delta) robot.servo_ee_delta(’right’, right_delta) # let the fold lie robot.release(’left’) robot.release(’right’) Program slice for this episode: fold_edge_over(robot, ’blanket’, [0.15, 0.0, 0.0], [0.15, 0.0, 0.0]) 21
Figure 9 of the paper
Figure 9. Key frames of Skill II, alternating smoothing pats. Skill definition: def alternating_smoothing_pats(robot, fabric_name, first_side, second_side): # pat 1 -- press the fabric flat, then come off robot.grasp(fabric_name, first_side) robot.release(first_side) # pat 2 -- the other hand, on the other half of the sheet robot.grasp(fabric_name, second_side) robot.release(second_side) Program slice for this episode: # Smooth out the towel with alternating pats alternating_smoothing_pats(robot, ’towel’, ’left’, ’right’) Skill III: flip object over Task: “fold napkin into a case and place…
Figure 10 of the paper
Figure 10. Key frames of Skill III, flip object over. Skill definition: def flip_object_over(robot, object_name, side, lift_delta, turn_deg=180.0): # lift just clear so the object can swing robot.servo_ee_delta(side, lift_delta) # roll the wrist -- this is the flip robot.rotate(object_name, turn_deg, side) # let it fall the last centimetre, landing inverted robot.release(side) # pin the flipped object down again for the next slice robot.grasp(object_name, side) Program slice for this episode: # Lift, roll the wrist to flip the spoon, drop it, and pin it flip_object_over(robot, ’spoon’, ’lef
Figure 11 of the paper
Figure 11. Key frames of Skill IV, adjust in place. Skill definition: def adjust_in_place(robot, object_name, side, nudge_delta): # take hold where it sits -- no lift robot.grasp(object_name, side) # slide it by a small bounded amount robot.servo_ee_delta(side, nudge_delta) # let it settle robot.release(side) Program slice for this episode: adjust_in_place(robot, ’bookmark’, ’right’, [0.05, 0.0, 0.0]) B.4 VISUALIZATIONS OF EGOCENTRIC HUMAN SKILLS We visualize six of the 27 egocentric human skills. For each, we select one representative slice and show a strip of key frames, the corresponding…
Figure 12 of the paper
Figure 12. Key frames of Skill I, open closure. Skill definition: def open_closure(robot, container_name, closure_name, side, brace_side, axis, angle_deg): # Hold the container body down so the closure, not the container, moves. robot.grasp(container_name, brace_side) # Take the closure where it sits. robot.grasp(closure_name, side) # Swing it open about its hinge. robot.rotate(closure_name, axis, angle_deg, side) # Come off the closure; it stays open. robot.release(side) Program slice for this episode: open_closure(robot, ’photo album’, ’photo album cover’, ’left’, ’right’, ’object_long’,
Figure 13 of the paper
Figure 13. Key frames of Skill II, pour into target. Skill definition: def pour_into_target(robot, vessel_name, target_name, side, axis, angle_deg): # Bring the held vessel’s mouth over the receiving target. robot.goto_relative(side, target_name, ’above’) # Tip it until it pours. robot.rotate(vessel_name, axis, angle_deg, side) # Bring it back upright; the pour ends. robot.rotate(vessel_name, axis, -angle_deg, side) Program slice for this episode: pour_into_target(robot, ’spoon’, ’bowl’, ’right’, ’object_long’, 110.0) Skill III: stir contents Task: “mixing solutions with stirring rod”…
Figure 14 of the paper
Figure 14. Key frames of Skill III, stir contents. Skill definition: def stir_contents(robot, tool_name, vessel_name, side): # Bring the held tool down inside the vessel, among the contents. robot.goto_relative(side, vessel_name, ’inside’) # Sweep it round the vessel; the grip never changes, nothing is let go. robot.rotate(tool_name, ’vertical’, 360.0, side) robot.rotate(tool_name, ’vertical’, 360.0, side) Program slice for this episode: brace_and_take_tool(robot, ’beaker’, ’left’, ’stirring rod’, ’right’) stir_contents(robot, ’stirring rod’, ’beaker’, ’right’) 24
Figure 15 of the paper
Figure 15. Key frames of Skill IV, wipe surface back and forth. Skill definition: def wipe_surface_back_and_forth(robot, surface_name, relation, stroke_direction, return_direction, side): # Park the cloth on the surface. robot.goto_relative(side, surface_name, relation) # Out and back. robot.move_arm(side, stroke_direction) robot.move_arm(side, return_direction) # And again. robot.move_arm(side, stroke_direction) robot.move_arm(side, return_direction) Program slice for this episode: # Roll the lint roller back and forth over the pants (4 pairs of strokes) for _ in range(2):…
Figure 16 of the paper
Figure 16. Key frames of Skill V, fold over and crease. Skill definition: def fold_over_and_crease(robot, item_name, side, brace_side, direction, fold_direction): # Hold the item down so the fold, not the whole item, travels. robot.grasp(item_name, brace_side) # Take the free edge with the working hand. robot.grasp(item_name, side) # Carry the edge across the item ... robot.move_arm(side, direction) # ... and land it on the far side. robot.move_arm(side, fold_direction) # Let the folded edge down. robot.release(side) # Press the crease flat. robot.press(item_name, side) Program slice for
Figure 17 of the paper
Figure 17. Key frames of Skill VI, wrap around target. Skill definition: def wrap_around_target(robot, target_name, side): # Start over the top of the target. robot.goto_relative(side, target_name, ’above’) # Over the far side. robot.move_arm(side, ’away_from_body’) # Down its far face. robot.move_arm(side, ’down’) # Back underneath, toward the body. robot.move_arm(side, ’toward_body’) # And up to close the circuit. robot.move_arm(side, ’up’) Program slice for this episode: # Wrap the hair around the curling iron twice. for _ in range(2): wrap_around_target(robot, ’curling iron’, ’left’) B.5…
Figure 18 of the paper
Figure 18. Skills of the YAM Skill Library in the Embedding Space. Close the lid of the portable gas stove. Open the photo album. Remove the key from the mailbox. Place a stack of documents secured with a binder clip. Press the decorative sticker on the paper-cut decoration. Pour the oil into the funnel. Pick up the hanger from a piece of clothing. Secure the container with rice. Brush off the clump of debris from the keyboard
Figure 19 of the paper
Figure 19. Skills of the Egocentric Human Skill Library in the Embedding Space. C BASELINE IMPLEMENTATION DETAILS In Sections 3.1 and 3.2, we compare AW-0 with four baselines: π0.5, DreamZero, EgoScale, and WLA. Here, we describe how each is reproduced and adapted. On RoboTwin 2.0, we train a single multi-task policy per baseline on 2,500 demo clean demonstrations (50 tasks × 50 demonstra￾tions). For each of the 8 real-world tasks, we train a separate policy per baseline using task-specific demonstrations sampled at 30 Hz. Every baseline receives a language instruction, the robot state, and…
Figure 20 of the paper
Figure 20. Real-world Robot Setups. (a) The YAM station: two arms with linear grippers, with a RealSense D405 camera overhead and one on each wrist. The bottom camera is not used. (b) The Tianji–Sharpa station: a Tianji torso with two Sharpa Wave dexterous hands, observed by an SHW5G camera on the head and one on each wrist. targets for the two Tianji arms. In parallel, finger-motion measurements from the Manus gloves are retargeted to 22-dimensional joint-position targets for each Sharpa hand. At each teleoperation step, the arm-pose and hand-joint targets are passed to the corresponding…
Figure 21 of the paper
Figure 21. Key frames and code label of Arrange Fruits. Grading rubric (additive): • +0.3: (a) Moves each fruit not named in the instruction from the left bowl into the right basket (or none to remove). • +0.3: (b) Moves each named fruit not yet in the left bowl from the right basket into the bowl (or none to fetch). • +0.4: (c) Final state matches the instruction and no fruit is dropped. 30
Figure 22 of the paper
Figure 22. Key frames and code label of Pack Backpack. Grading rubric (additive): • +0.15: (a) Left gripper grasps the zipper pull at the left end and slides it to the middle. • +0.15: (b) Right gripper re-grasps the pull at the middle and slides it to the right end. • +0.2: (c) Open the backpack with the left gripper. • +0.2: (d) Picks up the item from the table and places it inside the open backpack. • +0.15: (e) Right gripper grasps the pull at the right end and slides it back to the middle. • +0.15: (f) Left gripper slides the pull from the middle to the left end; the backpack is fully…
Figure 23 of the paper
Figure 23. Key frames and code label of Replace Lightbulb. Condition A – dead bulb, replacement required. Grading rubric (additive): • +0.1: (a) Left gripper presses the inline switch and the bulb stays dark. • +0.1: (b) Left gripper braces the lamp base. • +0.2: (c) Right gripper unscrews the dead bulb and lifts it out of the socket. • +0.1: (d) Dead bulb is set down in the bulb tray. • +0.1: (e) Right gripper picks up the spare bulb. • +0.3: (f) Spare bulb is seated in the socket and screwed in clockwise until lit. • +0.1: (g) When the bulb is lit, release both arms to their home positions.…
Figure 24 of the paper
Figure 24. Key frames and code label of Fold and Stack Towels. Grading rubric (additive): • +0.2: (a) Grasps a towel edge and completes one fold. • +0.2: (b) Each of the three towels is folded flat. • +0.1: (c) Picks a folded towel up and lays it on the pile. • +0.2: (d) All three folded towels end up in a single pile, none dropped or left behind. • +0.3: (e) The pile reads, bottom to top, in the colour order the instruction names. E.2 TIANJI–SHARPA TASKS 33
Figure 25 of the paper
Figure 25. Key frames and code label of Arrange Fruits with Tongs. Grading rubric (additive): • +0.2: (a) Right hand takes the tongs off their stand and keeps hold of them. • +0.25: (b) Moves each fruit not named in the instruction from the left bowl into the tray (or none to remove). • +0.25: (c) Moves each named fruit not yet in the left bowl from the tray into the bowl (or none to fetch). • +0.2: (d) Final state matches the instruction and no fruit is dropped. • +0.1: (e) Tongs are put back on their stand. 34
Figure 26 of the paper
Figure 26. Key frames and code label of Deal Poker. Grading rubric (additive): • +0.1: (a) Right hand grasps the card stack from above. • +0.2: (b) Right hand transfers the stack to the left hand (handover). • +0.3: (c) Right thumb flicks and adjusts until exactly one card protrudes from the top. • +0.3: (d) Right hand grasps the single protruding card. • +0.1: (e) Right hand inserts the card vertically into the card slot. Task III: Flip Scoreboard. 1 # add 2 to the scoreboard then press the button 2 current = robot.read_number('scoreboard') # the board shows 3 3 target = current + 2 # -> 5: w
Figure 27 of the paper
Figure 27. Key frames and code label of Flip Scoreboard. Grading rubric (additive): • +0.1: (a) Left hand holds the scoreboard steady. • +0.2: (b) Right hand flips a digit card over (at least one complete flip). • +0.4: (c) Flips exactly k times, so the board shows the initial score plus k. • +0.1: (d) Left hand releases the board. • +0.2: (e) Right hand presses the red button while the target score is showing. Task IV: Solve Rubik’s Cube. 1 # both hands move in over the cube, fingers still open 2 cube = robot.detect_objects_oneshot('rubiks cube') 3 reach_toward(robot, 'rubiks cube', 'both')…
Figure 28 of the paper
Figure 28. Key frames and code label of Solve Rubik’s Cube. For each rollout, one randomly selected face is scrambled clockwise or counterclockwise; the visible face colors and placement angle are fully randomized, with no fixed layer or turn sequence. Grading rubric (additive): • +0.2: (a) Right hand grasps the cube and lifts it off the table. • +0.3: (b) Fingers turn the scrambled layer a full quarter turn, leaving no layer half-turned. • +0.3: (c) The cube is solved: every face shows a single colour. • +0.2: (d) The cube is put back on the table and both hands let go. F ANALYSIS ON CODE…
Figure 29 of the paper
Figure 29. Pretrained Code Generation on a YAM Validation Example. The generated program is compared with a reference program from another labeled episode of the same task. Egocentric Human Demonstrations
Figure 30 of the paper
Figure 30. Pretrained Code Generation on an Egocentric Human Validation Example. Under the instruction “Squeeze toothpaste,” the model generates the cap-removal subtask, matching the annotated program for the held-out window. F.2 POST-TRAINED CODE GENERATION QUALITY We examine code generated online by the post-trained model during real-world deployment on the YAM and Tianji–Sharpa platforms. Figures 31 and 32 show examples from Pack Backpack and Solve Rubik’s Cube, respectively. Each example pairs the observation at a replanning point with the generated program and subsequent recorded frames…
Figure 31 of the paper
Figure 31. Online Code Generation during Real-world Deployment on Pack Backpack. The panel labeled “Annotated Program” shows a reference program from a separate demonstration of the same task and subtask. “Generated Program” is the model output recorded online during de￾ployment. The comparison illustrates semantic correspondence in skill selection and arguments. Tianji–Sharpa Deployment Rollout
Figure 32 of the paper
Figure 32. Online Code Generation during Real-world Deployment on Solve Rubik’s Cube. The panel labeled “Annotated Program” shows a reference program from a separate demonstration of the same task and subtask. “Generated Program” is the model output recorded online during deployment. The comparison illustrates semantic correspondence in skill selection and arguments. 39
Figure 33 of the paper
Figure 33. Future-frame Prediction on Arrange Fruits for Three Models. G ADDITIONAL ANALYSIS ON REAL-WORLD RESULTS G.1 IMPACT OF CODE EXPERT ON LANGUAGE FOLLOWING CAPABILITY We further examine whether code-based planning improves the alignment of predicted visual dy￾namics with task semantics. As shown in
Figure 34 of the paper
Figure 34. Future-frame Prediction on Deal Poker for Three Models. Code exec Direct actions Code exec Direct actions Code exec Direct actions
Figure 35 of the paper
Figure 35. Qualitative Example of Hybrid Actions for the “Fold and Stack Towels” Task. A single episode continues from the upper to the lower row, with green and purple indicating code execution and direct actions, respectively. ing motions rely primarily on direct actions from the action expert of AW-0, while code execution handles more structured motions, such as approaching and stacking the towels, which provide direct guidance on the stacking order and motion direction, improving the semantic reasoning capability for the whole system. H FAILURE CASE ANALYSIS
Figure 36 of the paper
Figure 36. Failure Case Analysis. Each row shows one representative failure case. Frames are ordered from left to right by execution progress. The red box on the last frame marks the region enlarged in the rightmost panel, where the failure occurs. could adjust its grasp as the bag deforms, while the pulling hand could slow down or pause when the bag starts to shift or tip. 3) Imprecise Alignment. In Replace Lightbulb in the third row, the robot grasps the bulb and brings it above the socket, but the bulb base remains misaligned with the socket opening and is not screwed in. Completing this…
Figure 37 of the paper
Figure 37. Qualitative examples of hybrid actions for RoboTwin 2.0 tasks. Each row shows one successful episode from the hybrid evaluation in
Figure 38 of the paper
Figure 38. Declaration-only APIs for parallel grippers and dexterous hands. System prompt for egocentric human demonstrations Setting Two hands, ‘left’ and ‘right’, seen from a head-mounted camera. No contact-event list is given; infer every contact from the frames and the annotation, judging object motion relative to the scene. Contact semantics pick/grasp close a hand (pick lifts, grasp holds in place); place/release/insert/attach open it; regrasp opens then closes; press closes then opens; handover closes the receiver before opening the giver. Motion and naming Motion is purely semantic: mo

Figures are the authors', reproduced from the paper.

References (15)
  1. [1] Container Grasp.InArrange Fruitsin the first row, the target fruit lies close to the basket rim. As the gripper approaches, its left jaw moves outside the basket, causing the closing motion to grasp the basket wall rather than only the fruit. This case highlights the need to a... on Pith
  2. [2] Object Dragging.InPack Backpackin the second row, when the robot pulls the zipper, the backpack moves and tips over rather than remaining stable. The visible failure suggests insufficient stabilization of the deformable object during manipulation. A potential improvement is to... on Pith
  3. [3] Imprecise Alignment.InReplace Lightbulbin the third row, the robot grasps the bulb and brings it above the socket, but the bulb base remains misaligned with the socket opening and is not screwed in. Completing this step requires both positional and angular alignment before the... on Pith
  4. [4] Tool Collision.InArrange Fruits with Tongsin the fourth row, the tongs approach the pear with their tips closed and poke it rather than opening around it to establish a grasp. Successful tool use requires coordinating the tool’s opening with its approach trajectory, in additio... on Pith
  5. [5] arXiv:2502.19417. 13 Ishika Singh, Valts Blukis, Arsalan Mousavian, Ankit Goyal, Danfei Xu, Jonathan Tremblay, Dieter Fox, Jesse Thomason, and Animesh Garg. ProgPrompt: Generating situated robot task plans using large language models. InICRA, 2023. Martin Sundermeyer, Arsalan ... on PithPith's reviewarXiv
  6. [6] URLhttps://arxiv.org/abs/2503.20314. Xingyao Wang, Yangyi Chen, Lifan Yuan, Yizhe Peng, Hao Li, Hao Peng, and Heng Ji. Executable code actions elicit better LLM agents. InInternational Conference on Machine Learning, 2024. arXiv:2402.01030. Wenli Xiao, Jia Xie, Tonghe Zhang, H... on PithPith's reviewarXiv
  7. [7] URLhttps://arxiv.org/abs/2602.15922. Tianyuan Yuan et al. Fast-W AM: Do world action models need test-time future imagination?, 2026. URLhttps://arxiv.org/abs/2603.16666. 14 Junyi Zhang, Jiaxin Ge, Hanjun Yoo, Letian Fu, Zihan Yang, Yaowei Liu, Raj Saravanan, Shaofeng Yin, Jus... on PithPith's reviewarXiv
  8. [8] URLhttps://arxiv.org/abs/2602.16710. Chuning Zhu, Raymond Yu, Siyuan Feng, Benjamin Burchfiel, Paarth Shah, and Abhishek Gupta. Unified world models: Coupling video and action diffusion for pretraining on large robotic datasets, 2025. URLhttps://arxiv.org/abs/2504.02792. Brian... on PitharXiv
  9. [14] Slipping Off.InDeal Pokerin the fifth row, the left hand fails to secure the card stack during handover from the right hand, causing the cards to slip. A successful handover requires the receiving hand to establish a stable grasp before the giving hand releases the stack. This... on Pith
  10. [15] Contact Loss.InFlip Scoreboardin the sixth row, the fingertip reaches the edge of a score card but slides off before turning it over, leaving the displayed score unchanged. Unlike a brief touch, a successful flip requires sustained contact along the card’s rotation about the b... on Pith
  11. [40] PMLR, 2025. URLhttps://proceedings.mlr.press/v305/black25a.html. Junhao Cai, Zetao Cai, Jiafei Cao, Yilun Chen, Zeyu He, Lei Jiang, Hang Li, Hengjie Li, Yang Li, Yufei Liu, Yanan Lu, Qi Lv, Haoxiang Ma, Jiangmiao Pang, Yu Qiao, Zherui Qiu, Yanqing Shen, Xu Shi, Yang Tian, Bolu... on PitharXiv
  12. [256] We use a peak learning rate of5×10 −5 with cosine decay to5×10 −6 and 2,000 warmup steps. DreamZero(Ye et al., 2026) is a world-action model that jointly generates future video and actions using a pretrained video diffusion backbone. We follow the official implementation 3 and... on Pith
  13. [2024] URLhttps://arxiv.org/abs/2406.11815. Dantong Niu, Yuvan Sharma, Haoru Xue, Giscard Biamby, Junyi Zhang, Ziteng Ji, Trevor Darrell, and Roei Herzig. Pre-training auto-regressive robotic models with 4d representations. InForty- second International Conference on Machine Learning... on PithPith's reviewarXiv
  14. [2025] URLhttps://arxiv.org/abs/2505.12705. Moo Jin Kim, Karl Pertsch, Siddharth Karamcheti, Ted Xiao, Ashwin Balakrishna, Suraj Nair, Rafael Rafailov, Ethan Foster, Grace Lam, Pannag Sanketi, Quan Vuong, Thomas Kollar, Ben- jamin Burchfiel, Russ Tedrake, Dorsa Sadigh, Sergey Levine,... on PithPith's reviewarXiv
  15. [2026] URLhttps://arxiv.org/abs/2601.16163. Sergey Levine, Chelsea Finn, Trevor Darrell, and Pieter Abbeel. End-to-end training of deep visuo- motor policies, 2016. URLhttps://arxiv.org/abs/1504.00702. Ruiying Li, Yunlang Zhou, YuYao Zhu, Kylin Chen, Jingyuan Wang, Sukai Wang, Kongta... on PithPith's reviewarXiv
Cited by 2 Pith papers

Challenges and discussion

Disagree with the review, or know something it missed? Leave a comment, or challenge this review formally: a challenge becomes a permanent entry in this paper's record, and its decision is published here.

No comments yet.

Sign in to comment