Pith.

Computer Vision and Pattern Recognition · arXiv:2611.06817 v3 · submitted 4 Nov 2026

Open-HumanWAM: An Open Recipe for Scaling Human and Robot Experience for Embodied Foundation Models

Juncheng Ma, Jiaxin Li, Jiankai Tu, Bo Liang, Guodong Lv, Yufan Deng, Junzhe Tang, Liangzhu Yang

show 27 more authors

Chen Bian, Shukai Gong, Ye Huang, Yintianrun Zhang, Zicong Cao, Baohao Zhu, Jianxin Bi, Tengfei Yan, Yinyuan Zhang, Kewei Zhang, Xinglong Chen, Kaiqi Chen, Duomin Wang, Xiaotian Tang, Liang Tan, Li Li, Yuan Xu, Haosen Yang, Yuqi Wang, Jingfei Xia, Songyan Huang, Zhiyang Dou, Zhen Dong, Enze Xie, Wojciech Matusik, Tat-Seng Chua, Daquan Zhou

The paper says

“Egocentric human video offers a scalable source of manipulation priors for robot learning, yet translating raw data scale into generalizable robot policies demands rigorous data curation and training recipes.”

more of the abstract

We introduce Open-HumanWAM, the first fully open recipe for scaling egocentric pretraining to 50,000 hours, anchored by Open-HumanWAM-50K—a curated egocentric video corpus densely annotated with hand motion and natural language. Leveraging this corpus, we systematically investigate pure human pretraining via closed-loop simulation evaluations and uncover an empirical log-linear scaling law linking pretraining data hours to downstream robot success rates. Grounded in these findings, we develop HumanWAM-Vanilla, a world–action model pretrained exclusively on 50k hours of egocentric human video. Upon robot post-training, HumanWAM-Vanilla reaches 92.08% success on RoboTwin 2.0 Clean2Clean and 68.88% on Clean2Random, alongside a 14.17% mean success rate and a 19.00 mean progress score on RoboDojo. To push the capability frontier further, we explore joint human–robot pretraining with HumanWAM-Goal, a visual-goal-conditioned world–action model. Hybrid data scaling unlocks broad zero-shot generalization across both simulation and physical benchmarks, propelling post-trained RoboTwin 2.0 performance to new state-of-the-art levels: 92.70% on Clean2Clean and a substantial leap to 84.60% on Clean2Random. On physical robots, HumanWAM-Goal further exhibits emergent embodied in-context learning, dynamically adapting its manipulation behavior to unseen visual goals without parameter updates. To support reproducible scaling research in embodied AI, we open-source the Open-HumanWAM-50K corpus, data curation and pretraining pipelines, alongside the pretrained and post-trained HumanWAM checkpoints, inference code, and simulation evaluation suites.

from the abstract

Pith says

Machine review · deepseek-v4-flash · 6 Nov 2026 · of version 2 of the paper; version 3 is newer · what was read was not recorded · not checked by a person

It can be wrong. Challenge this review or reply to it.

Holds up. A thorough, open-scaling study of egocentric pretraining for robot learning, with a legitimate but not fatal gap in the generality of the scaling law across post-training budgets. This line is wrong

2 major objections · 6 minor

How often objections from reviews like this one are right: not yet measured

Are you an author? What you can do.

Not checked: figures, code, what the references say.

The paper says, at §3.2, Figs. 5 and 6

The review did not quote this passage.

Pith says: Objection 1 of 2 · major

The claimed log-linear scaling law (SR = a + b log₁₀ H) is established under a single post-training condition: 50 clean demonstrations per task on RoboTwin 2.0. The paper does not examine whether the functional form or slope persists when the demonstration budget changes (e.g., 5, 10, or 25 demos).

more of objection 1

Section 3.4 studies sample efficiency at different budgets but does not test whether the log-linear relationship itself is stable. Since the paper presents this as a central empirical finding, the lack of budget variation limits its generality. The revised manuscript should either add such experiments or explicitly state the condition (50 demos) and discuss how the law might depend on post-training resources.

The paper says, at §3.1.2

The review did not quote this passage.

Pith says: Objection 2 of 2 · major

The paper adjusts the global batch size according to corpus size to avoid over-training small corpora, but this couples the scaling variable (data hours) with training hyperparameters. The analysis would benefit from a control experiment that holds batch size constant across scales and verifies that the scaling trend is not an artifact of the batch-size schedule. At minimum, the potential confounding should be discussed.

6 minor objections

The paper says, at §1 and §6.3

Pith says: minor objection 1 of 6

The term 'emergent embodied in-context learning' is used to describe the model's ability to adapt to visual goals without parameter updates. Since goal-image conditioning is a designed architectural feature (explicitly trained, §5.3), describing the behavior as 'emergent' overstates the novelty. The authors could more precisely say 'visually conditioned in-context execution' or acknowledge that the training explicitly enables this capability.

The paper says, at §3.5, Fig. 9

Pith says: minor objection 2 of 6

The comparison between Aligned and Raw action representations uses a 1,000-hour mixture from four sources. The claim that alignment 'substantially' improves performance would be strengthened by reporting whether the differences are statistically significant (e.g., confidence intervals across multiple random seeds). As presented, only a single run per setting is shown.

show 4 more

The paper says, at §4.2, Table 8

Pith says: minor objection 3 of 6

The RoboDojo results for HumanWAM-Vanilla are reported alongside baselines that include agent-based systems (DM0.5). A brief statement clarifying which baselines use additional data or architectural components beyond world–action modeling would help the reader interpret the comparison.

The paper says, at §A.4, Table 15

Pith says: minor objection 4 of 6

The goal-image review pipeline reports a 90.7% first-pass pass rate. The impact of the 7.7% failures and 1.6% uncertain cases on downstream evaluation results is not quantified. The authors should confirm that repaired images do not introduce systematic bias (e.g., by comparing success rates on a subset with and without repair).

The paper says, at §3.7, Fig. 12

Pith says: minor objection 5 of 6

The robot pretraining corpus at 5,000 hours includes Cobot Magic data in addition to AgiBot Beta and ABC-130k, while smaller scales do not. This confounds scale with composition. The authors should either match the sources across scales or discuss the potential effect of data composition on the comparison.

The paper says, at General

Pith says: minor objection 6 of 6

Several references are to technical reports or preprints from 2026 (e.g., [14], [39], [40]) that may not be peer-reviewed. The paper should note the review status of key cited results, particularly the scaling laws in Dyna-2 [14] and the performance of GO-3 Nano Preview, which is listed as a baseline leaderboard entry.

What would settle it

Using the authors' existing or easily recovered checkpoints and evaluation seeds, recompute the log-linear scaling fit (SR = a + b log10 H) for pretraining hours vs. success rate at each post-training demonstration budget tested in Figure 8 (5, 10, 25, 50 demos per task). To enable a meaningful fit, evaluate additional intermediate pretraining scales (e.g., 100, 1,000, 10,000 hours) at each budget. If the R-squared for the log-linear fit at 5 or 10 demos falls below 0.7, or if the slope at any budget differs by more than a factor of 2 from the slope at 50 demos, then the claimed scaling law is contingent on the post-training budget and should be reported with the budget as a qualifier.

from the skeptic's reading

How the review went

4 readings by deepseek-v4-flash · 6 Nov 2026

  1. Reader holds up confidence high
  2. Skeptic partly agrees 1 min 6 s · 51,442 tokens read, 6,239 written
  3. Editor wrote the verdict above 12 s · 52,397 tokens read, 1,188 written
  4. Referee minor revision: 2 major objections, 6 minor 28 s · 51,797 tokens read, 2,535 written

Not run on this paper: the simulated authors' defence.

Pith did not check this review's objections against the paper.

The review in full

The review in brief, machine-written

This paper does what the title says: it provides an open recipe for scaling egocentric human video pretraining up to 50k hours, and it backs the recipe with a systematic empirical study. The main contributions are solid. First, they assemble and release a curated 50k-hour egocentric corpus with hand-motion and language annotations. Second, they demonstrate a log-linear scaling law between pretraining hours and downstream closed-loop task success on RoboTwin, observed under two attention configurations. Third, they show that data diversity (task/scene coverage) is a separate scaling dimension that yields steeper gains than duration alone. Fourth, they extend to hybrid human–robot pretraining with visual-goal conditioning and achieve state-of-the-art results on RoboTwin 2.0 after post-training. The paper is well-executed, the experiments are extensive, and the open-source release is a genuine asset to the community. Credit where due: this is careful, reproducible work that advances the empirical understanding of what scales in embodied pretraining.

The soft spots are real but in proportion. The stress-test concern lands: the central scaling law is established under a single post-training budget (50 demos per task). The paper shows that larger pretraining improves sample efficiency (Figure 8), but it does not verify whether the functional form or slope of the log-linear law persists at, say, 10 or 5 demos per task. The law is presented as a property of pretraining data alone, but it may actually be an interactive effect with the post-training budget. That is a gap in the generality claim. The authors are honest about other limitations—architectural simplicity limiting RoboDojo performance, the approximation in human-to-robot action alignment—but they do not flag this particular condition-dependence. A second concern is that the hybrid scaling results (HumanWAM-Goal) use a different base model (Cosmos3-Nano-16B) and a separate training recipe, so the two halves of the paper feel somewhat disconnected. The hybrid part is impressive on its own terms, but it does not directly extend the scaling law from the pure egocentric study.

This paper is for researchers working on robot foundation models, scaling laws for pretraining, and open benchmarks. It deserves serious peer review. I would recommend acceptance with a request for one additional demonstration budget (e.g., 10 demos) to check whether the log-linear scaling law holds at that budget, or at least a clear discussion of this limitation. The open release and the empirical rigor outweigh this gap.

The strongest objection

The central claim of a log-linear scaling law between egocentric pretraining hours and downstream robot success rates (Figures 5 and 6) is established under a single post-training condition: 50 clean demonstrations per task on RoboTwin 2.0. While the paper shows that larger pretraining scale improves sample efficiency (Figure 8), it does not examine whether the functional form (log-linear) or the slope of the scaling law persists when the number of post-training demonstrations changes. If the scaling relationship is sensitive to the demonstration budget—for instance, appearing log-linear only at 50 demos but flattening or shifting at 10 or 5 demos—then the claimed law is not a robust property of the pretraining data alone but an interactive effect between pretraining and post-training resources. Since practical deployment often has limited demonstrations, the generality of the law across realistic adaptation budgets remains untested. The paper itself acknowledges that smaller corpora can degrade with further pretraining (Section 3.1.2), so the interplay between pretraining scale and post-training budget is a natural concern that the current analysis does not address.

The paper provides a thorough empirical study with a clear central claim (log-linear scaling of downstream success with egocentric pretraining hours). The reader's weakest assumption (hand annotation accuracy) is supported by the paper's own analysis of alignment, but the scaling law's dependence on the post-training demonstration budget is a more fundamental concern for the generality of the claim. The paper does not test whether the log-linear functional form holds across different demonstration budgets, and the sample efficiency analysis (Figure 8) uses only three pretraining scales, insufficient for a full scaling law fit. However, the paper is otherwise methodologically sound, open-sourced, and the scaling law is clearly presented within its stated experimental conditions. The concern does not invalidate the findings but suggests a qualification. Therefore, the reader's ACCEPT verdict remains unchanged, but the stress-test identifies a specific gap the authors could address in future work.

The referee's report

If the scaling law holds generally, the paper provides a principled justification for continued collection of egocentric human video at larger scales for robot foundation models. The open release of the 50k-hour corpus, training pipelines, and checkpoints is a substantial contribution to reproducible research in embodied AI. The hybrid co-training results and zero-shot real-robot experiments advance the practical deployment of world–action models.

Pith's reading of the paper (a machine's summary, not the authors')

Open-HumanWAM proposes a fully open recipe for using large-scale egocentric human video to pretrain embodied foundation models. The paper claims that pretraining on curated human video from 100 to 50,000 hours produces a log-linear scaling law in downstream robot task success rates, particularly in out-of-distribution settings. It further shows that joint co-training with human and robot data, combined with visual-goal conditioning, unlocks zero-shot generalization and emergent in-context learning on physical robots, achieving state-of-the-art results on the RoboTwin 2.0 benchmark.

What carries the argument. The central machinery is a unified bimanual end-effector action representation (16-D for pure egocentric, 49-D for hybrid human–robot) that aligns wrist poses, gripper openness, and camera frames across multiple heterogeneous human and robot datasets, enabling consistent action supervision during pretraining. This is coupled with a world–action model architecture (based on Wan2.2-TI2V-5B and Cosmos3-Nano-16B) that jointly predicts future video and actions, and for HumanWAM-Goal incorporates explicit visual-goal conditioning to specify task outcomes.

Core claim. The paper establishes an empirical log-linear scaling law relating the amount of egocentric human pretraining data (from 100 to 50,000 hours) to closed-loop robot success rates after post-training, observed across two different attention configurations (isolated and inverse-dynamics) with high R-squared values (0.877–0.971). Beyond data duration, it finds that expanding task and scene diversity alongside hours produces steeper scaling curves. It then demonstrates that a joint human–robot pretrained model with visual-goal conditioning (HumanWAM-Goal) achieves 92.70% on Clean2Clean and 84.60% on Clean2Random on RoboTwin 2.0 after post-training, and exhibits embodied in-context learning where a frozen policy adapts its behavior to different visual goals without parameter updates.

The premise it rests on. The hand motion annotations estimated from egocentric video (wrist poses, gripper openness) are accurate enough and transferable to robot bimanual control after the unified action alignment procedure, such that pretraining on human video consistently improves robot policy performance.

If the paper is right (Pith's inferences, not the authors')

Each tenfold increase in egocentric pretraining hours yields fitted gains of 4–16 percentage points in downstream robot success, with larger gains in out-of-distribution settings; further scaling beyond 50,000 hours is expected to continue improving generalization.

Joint human–robot co-training with visual-goal conditioning reduces the need for target-task demonstrations: the model can execute unseen tasks zero-shot and achieve higher post-trained performance at matched demo budgets.

The embodied in-context learning capability—adapting a frozen policy to different visual goals—provides a practical interface for task specification without parameter updates, which could simplify deployment of generalist robot policies.

Data diversity (task and scene coverage) is a separate scaling axis: expanding diversity alongside duration produces steeper scaling slopes than increasing duration at fixed coverage, guiding efficient corpus construction.

If the log-linear scaling law holds beyond 50,000 hours, it suggests that even modest further increases in human video data could yield meaningful gains in robot robustness, potentially surpassing the need for large robot-specific data collection.

The finding that egocentric human pretraining can outperform robot pretraining at matched scales (from 500 hours upward) implies that human interaction diversity provides richer manipulation priors than teleoperated robot data, which could shift data-collection strategies toward cheaper human video.

The emergent in-context learning via visual goals after joint pretraining hints that such models may be able to follow compositional task specifications from arbitrary goal images, including those from other embodiments (human hands), opening avenues for cross-embodiment policy transfer.

Check this review

Pith stored no record of this review's passes and took no fingerprint of it. Reviews made by code from 13 Nov 2026 on carry both.

How often reviews like this one are right

Reviews like this one: written by deepseek-v4-flash, where what was read was not recorded.

How often objections from reviews like this one are right: not yet measured

Every value above, in the review's public record (JSON)

Reviews by people

This review was written by a machine. No person has reviewed this paper on Pith yet. A scientist listed on Pith can review it for a fee, and their review is published here whatever it concludes.

Request a human review

How Pith checks papers · The full audit of this paper · Review history (2 versions)

The paper

Abstract

Egocentric human video offers a scalable source of manipulation priors for robot learning, yet translating raw data scale into generalizable robot policies demands rigorous data curation and training recipes. We introduce Open-HumanWAM, the first fully open recipe for scaling egocentric pretraining to 50,000 hours, anchored by Open-HumanWAM-50K—a curated egocentric video corpus densely annotated with hand motion and natural language. Leveraging this corpus, we systematically investigate pure human pretraining via closed-loop simulation evaluations and uncover an empirical log-linear scaling law linking pretraining data hours to downstream robot success rates. Grounded in these findings, we develop HumanWAM-Vanilla, a world–action model pretrained exclusively on 50k hours of egocentric human video. Upon robot post-training, HumanWAM-Vanilla reaches 92.08% success on RoboTwin 2.0 Clean2Clean and 68.88% on Clean2Random, alongside a 14.17% mean success rate and a 19.00 mean progress score on RoboDojo. To push the capability frontier further, we explore joint human–robot pretraining with HumanWAM-Goal, a visual-goal-conditioned world–action model. Hybrid data scaling unlocks broad zero-shot generalization across both simulation and physical benchmarks, propelling post-trained RoboTwin 2.0 performance to new state-of-the-art levels: 92.70% on Clean2Clean and a substantial leap to 84.60% on Clean2Random. On physical robots, HumanWAM-Goal further exhibits emergent embodied in-context learning, dynamically adapting its manipulation behavior to unseen visual goals without parameter updates. To support reproducible scaling research in embodied AI, we open-source the Open-HumanWAM-50K corpus, data curation and pretraining pipelines, alongside the pretrained and post-trained HumanWAM checkpoints, inference code, and simulation evaluation suites.

The paper's own sentences, arranged, kept beside the review and separate from it. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. No sentence in this record was written by a model.

abstract compiled · 2026-11-05

The authors' main claim, as the classifier picked it out 0.94

“Hybrid data scaling unlocks broad zero-shot generalization across both simulation and physical benchmarks, propelling post-trained RoboTwin 2.0 performance to new state-of-the-art levels: 92.70% on Clean2Clean and a substantial leap to 84.60% on Clean2Random.”

How the main claim is stated: asserted 1.00

What the abstract itself states, as probabilities: claims first 0.94 · states limitation 0.20 · human participants 0.39 · controlled comparison 0.53 · randomized assignment 0.02 · states quantitative result 0.99 · compares with prior results 0.58 · mentions code or data availability 0.99

The abstract, sentence by sentence

This table labels the abstract only (arXiv listing abstract v1). The number is the probability the classifier gave the label.

SentenceRoleThe authors' own workStates the main conclusionStrength
Egocentric human video offers a scalable source of manipulation priors for robot learning, yet translating raw data scale into generalizable robot policies demands rigorous data curation and training recipes. background 0.94 0.06 0.04 asserted 1.00
We introduce Open-HumanScale, the first fully open recipe for scaling egocentric pretraining to 50,000 hours, anchored by Open-HumanScale-50K---a curated egocentric video corpus densely annotated with hand motion and natural language. contribution 0.99 0.95 0.10 asserted 1.00
Leveraging this corpus, we systematically investigate pure human pretraining via closed-loop simulation evaluations and uncover an empirical log-linear scaling law linking pretraining data hours to downstream robot success rates. result 0.74 0.96 0.22 asserted 1.00
Grounded in these findings, we develop HumanWAM-Vanilla, a world--action model pretrained exclusively on 50k hours of egocentric human video. method 0.61 0.95 0.08 asserted 1.00
Upon robot post-training, HumanWAM-Vanilla reaches 92.08% success on RoboTwin 2.0 Clean2Clean and 68.88% on Clean2Random, alongside a 14.17% mean success rate and a 19.00 mean progress score on RoboDojo. result 1.00 0.96 0.14 asserted 1.00
To push the capability frontier further, we explore joint human--robot pretraining with HumanWAM-Goal, a visual-goal-conditioned world--action model. contribution 0.53 0.90 0.05 asserted 0.52
Hybrid data scaling unlocks broad zero-shot generalization across both simulation and physical benchmarks, propelling post-trained RoboTwin 2.0 performance to new state-of-the-art levels: 92.70% on Clean2Clean and a substantial leap to 84.60% on Clean2Random. result 1.00 0.95 0.37 asserted 1.00
On physical robots, HumanWAM-Goal further exhibits emergent embodied in-context learning, dynamically adapting its manipulation behavior to unseen visual goals without parameter updates. result 0.98 0.93 0.13 asserted 1.00
To support reproducible scaling research in embodied AI, we open-source the Open-HumanScale-50K corpus, data curation and pretraining pipelines, alongside the pretrained and post-trained HumanWAM checkpoints, inference code, and simulation evaluation suites. contribution 0.98 0.93 0.09 asserted 0.99

What kind of paper the classifier reports this is: method or tool 0.65 · evidence from observation or experiment 0.86

What was examined, and what was not

Sources read

  • arXiv listing abstract v1

Not examined

  • full text
  • figures and tables
  • references and citing sentences
  • relations to other papers

Levels run

  • P abstract

Not run

  • S sections
  • T body sentences
  • C citations
  • R cross-paper relations

How the labels were made

Compiled by classifier-1.13.0 via TypeSafe API, question bank abstract-p1-20260923, 2026-11-05: 1 request, 7,454 input tokens.

Every quoted sentence on this record is the authors'. Every label is from Pith's closed vocabulary, with the probability shown beside it. This record as JSON

Figures (28)
Figure 4 of the paper
Figure 4. RoboTwin evaluation setup for our egocentric pretraining exploration. We post-train policies exclusively on clean demonstrations and evaluate them under clean, scene-ood, and random conditions. rolled out on all 50 tasks with 50 episodes per task in each setting. Representative tasks and evaluation conditions are illustrated in
Figure 5 of the paper
Figure 5. shows that scaling egocentric pretraining with isolated attention consistently improves downstream policy performance on RoboTwin. On clean, success increases from 71.2% at 100 hours to 80.0% at 10,000 hours, compared with 69.5% without egocentric pretraining. These gains indicate improved in-distribution performance under the same post-training protocol. The benefits are larger under distribution shift. From 100 to 10,000 hours, scene-ood success rises from 25.2% to 58.1%, while random success increases from 8.4% 10
Figure 8 of the paper
Figure 8. Scaling egocentric pretraining improves the sample efficiency of robot adaptation. Success rates as a function of the number of robot demonstrations per task for different egocentric pretraining scales. Results
Figure 9 of the paper
Figure 9. Cross-source action alignment improves downstream policy performance. Aligned and Raw models are pretrained on the same four-source 1,000-hour corpus using aligned and native action representations. Each point reports the best success rate across post-trained checkpoints for the corresponding pretraining run and evaluation setting. Gray dashed lines denote the baseline without egocentric pretraining. Results
Figure 12 of the paper
Figure 12. Egocentric pretraining outperforms robot pretraining from 500 to 5,000 hours. Downstream RoboTwin success rates after pretraining on 100, 500, 1,000, and 5,000 hours of egocentric human or robot data. The egocentric series follows the scaled-diversity configuration in
Figure 13 of the paper
Figure 13. From egocentric pretraining to multi-view robot policies. Three view configurations and training routes with or without EgoPro mid-training. Multi-view configurations. We compare three configurations for visual conditioning and video prediction, illustrated in
Figure 15 of the paper
Figure 15. Robot pretraining data overview. The top row reports the overall robot pretraining budget. Word clouds summarize selected (a) action and (b) object terms, while the adjacent bars show label-level percentages for the eight most frequent terms in each group. Percentages use all 751 normalized task-collection labels as the denominator. Task diversity. The corpus spans activities such as organizing books and tools, folding clothing, preparing food, operating appliances, and inserting cables. To characterize this diversity, we extract action and object terms from the task-collection…
Figure 16 of the paper
Figure 16. HumanWAM-Goal architecture. (a) Conditioning. An explicit goal image augments language instructions, current and optional historical observations, and the 49-D robot state. (b) Joint rectified-flow sampling. With conditioning tokens held fixed, the shared video-action Generator iteratively predicts video and action velocities via UniPC updates. Visual goal and observation conditioning. To enable unambiguous task specification without modifying the backbone topology, all visual inputs are encoded into spatial latents via a frozen Wan VAE and mapped into the Generator through a…
Figure 18 of the paper
Figure 18. Task-level zero-shot scaling with human-robot hybrid training. Mean zero-shot success rates (averaged over Clean and Random) across six representative RoboTwin 2.0 tasks with robot data fixed at 14,618 hours and increasing egocentric human data. Task-level scaling analysis
Figure 19 of the paper
Figure 19. Post-trained performance across pretraining scales on RoboTwin 2.0. Random (88.65% Average), outperforming the previous leading entry (GO-3 Nano Preview) by 4.16 pp on Clean, 3.12 pp on Random, and 3.64 pp on average. Because this configuration incorporates post-training augmentations beyond the controlled scaling recipe, we report it separately from the scaling curves in
Figure 20 of the paper
Figure 20. Zero-shot real-robot performance stratified by task difficulty. Success rates across easy and hard tasks as egocentric human data are added to a fixed 14, 618 h robot corpus. The vertical dashed line at 60, 556 h highlights the additional gain on hard tasks from robot-balanced batch sampling (+8 pp). Scaling trends across task difficulty
Figure 23 of the paper
Figure 23. Visual goal conditioning in pretrained HumanWAM-Goal. Each panel pairs the inference-time goal image (left) with synchronized first-person and third-person keyframes (right). Without target-task post-training or parameter updates, changing the visual goal directs the policy to place the bottle (a) on the tabletop in front of the green block or (b) atop the large purple block. In-context learning behavior after post-training. We next evaluate whether goal-conditioned steerability is preserved—and extended to multi-step, compositional decisions—after adapting the co-trained…
Figure 24 of the paper
Figure 24. evaluates a post-trained policy on a multi-object tabletop organization task under three distinct visual goals. Rather than memorizing a single canonical sorting routine, the frozen policy parses multiple compositional constraints from the goal image within each rollout: it reverses category-to-bin assignments when the destinations of toy cars and plush toys are swapped (panel a vs. panel b), and it modifies both the stacking order (green vs. purple book on top) and the spatial placement of the book stack when prompted by panel c. Visual goal conditioning thus scales beyond…
Figure 25 of the paper
Figure 25. further tests paired goal variations in an office workspace involving articulated storage and multi-item sorting. In panels (a, b), the visual goal dictates which drawer of a cabinet holds a tape roll and tape measure; to satisfy the goal, the policy selectively pulls open either the upper or the middle drawer before inserting both items. In panels (c, d), swapping the visual arrangement of a glue stick and two pens between adjacent 36

Showing 14 of the paper's 28 figures. Figures are the authors', reproduced from the paper.

Challenges and discussion

Disagree with the review, or know something it missed? Leave a comment, or challenge this review formally: a challenge becomes a permanent entry in this paper's record, and its decision is published here.

No comments yet.

Sign in to comment