Pith. sign in

REVIEW 4 major objections 5 minor 117 references

HOST lets a robot learn a novel manipulation skill from a single human video in about 29 seconds, at 62% average success with no parameter updates, while retaining mastered skills.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

T0 review · deepseek-v4-flash

2026-08-01 10:58 UTC pith:DDDLMOTV

load-bearing objection Strong system contribution, well-executed study; the headline margin over fine-tuning is not statistically established, but the core inference-time claim is credible. the 4 major comments →

arxiv 2607.20033 v3 pith:DDDLMOTV submitted 2026-07-22 cs.RO

Robots Acquire Manipulation Skills in Seconds from a Single Human Video

classification cs.RO
keywords one-shot visual imitationinference-time skill acquisitionhuman video demonstrationtask progress alignmentself-grounded predictionskill retentionbimanual manipulationautoregressive flow matching
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper's central claim is that a robot can acquire a new manipulation skill at inference time from a single human video, with no gradient updates, in about 29 seconds, while keeping skills it already knows. The method, HOST, reaches 62% average success on 50 held-out manipulation tasks, outperforms zero-shot and one-shot visual-imitation baselines by 43–45 percentage points, and exceeds a strong baseline fine-tuned on 50 robot demonstrations per task while using 50 times fewer demonstrations and acquiring each skill 507 times faster. The reason this matters is that current skill acquisition requires collecting teleoperated demonstrations and hours of fine-tuning, which also erodes previously mastered skills. HOST instead treats the human video as an external specification that drives a cascade of prediction: where the robot is in the demonstrated procedure, what the robot's own next observations will look like, and what actions realize them.

Core claim

The discovery is that the structural mismatch between a human video and robot execution—different timing, viewpoint, embodiment, and appearance—can be resolved by coupling the prediction target to the video's future progression and then letting the policy generate its own future observations before actions. On a shared task-progress manifold, the robot and the human video are aligned frame-to-frame without manual annotation. At each robot timestep, the target is not a fixed temporal offset but the segment of the robot's own future trajectory that corresponds to the upcoming segment of the video. A frozen autoregressive flow-matching policy then outputs, in order, a localization scalar, futur

What carries the argument

The load-bearing object is a learned task-progress manifold with a monotonic frame correspondence between the human video and robot trajectories, recovered self-supervised via Smooth Dynamic Time Warping and temporal cycle-consistency. This correspondence redefines the robot's prediction target at each step, making the video the active driver of the prediction instead of passive context. Execution is then resolved by a causal cascade inside a single autoregressive diffusion transformer: a localization token estimates where the robot is in the video window, future-observation tokens translate the video's upcoming progression into the robot's own camera views, and action tokens decode those pr

Load-bearing premise

The method is evaluated on a single two-armed robot; if it cannot transfer to robots with substantially different bodies and grippers, the broad claim of learning manipulation skills from one human video is not yet established.

What would settle it

Run the trained HOST model, without any retraining, on a single-arm robot or a humanoid with different kinematics and camera views, giving it the same held-out human videos; if success on novel tasks falls well below the reported 62% average, the cross-embodiment claim fails. A cheaper in-domain check is to compare the localization head's predicted progress against human-annotated progress on videos with variable speed, repeated steps, or ambiguous sub-goals.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • A human video becomes a complete skill specification: acquiring a new skill takes roughly the time of recording the video (~29 s), not hours of teleoperation and fine-tuning.
  • Because policy weights stay frozen during acquisition, previously mastered skills are not overwritten; the paper measures retention near 100%, versus 20–43% for fine-tuned baselines.
  • Stored videos can be retrieved and reused for recurring tasks, so skills accumulate as an external library without repeated human involvement.
  • The specific design choices—progress-based target coupling and the self-grounded cascade—are each shown to be necessary: ablations drop success from 62% to 45% without progress-based windowing and to 34% when actions are predicted directly.
  • If the central claim holds, robotic deployment can shift from 'collect data and retrain per task' to 'record one video and execute,' which is the practical promise of one-shot visual imitation.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference — If cross-embodiment transfer proves weak, the same two-stage recipe could be retrained with human–robot pairs from other robot bodies; the paper explicitly leaves this untested, so the breadth of the claim rests on it.
  • Editorial inference — The causal predict-your-own-observations step suggests a diagnostic: on tasks with repeated or easily confused stages, the localization head should err first; measuring its predicted progress against human annotations on such tasks would isolate where HOST breaks.
  • Editorial inference — Since HOST stores skills as videos and retrieves them by instruction/scene similarity, its long-horizon limit is likely task discrimination, not execution; the paper's own discussion anticipates this as memory grows.
  • Editorial inference — If reproduced on other platforms, HOST's numbers imply that a small Stage-1 robot corpus plus a modest human-video set could be an economical route to 'watch once and do' policies, at the price of a large up-front pretraining cost.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces HOST, a framework that acquires manipulation skills from a single human video at inference time without parameter updates. The method couples prediction targets to the future progression of the video demonstration through a self-supervised temporal alignment module (SDTW + TCC) that maps robot trajectories and videos to a shared task-progress manifold, then resolves execution through a causal cascade: progress localization, future-observation prediction, and action prediction in an autoregressive flow-matching transformer. Training is two-stage: large-scale same-embodiment robot-robot pretraining (193,462 trajectories, 229 tasks) followed by adaptation with 5,847 human-robot pairs. On a bimanual ARX R5 platform, the authors report 62% success across 50 novel tasks from a single video, 43-45% gains over OSVI/zero-shot baselines, a 6-point margin over the best 50-demonstration fine-tuned baseline, retention of previously mastered skills, robustness to perturbations, ablations of each mechanism, and a retrieval-based persistence mechanism.

Significance. If sustained, HOST would be a significant step toward practical one-shot visual imitation: it replaces a collection-and-fine-tuning loop with an inference-time video condition, demonstrates large experimental breadth (50 novel tasks, 20 trials each, human evaluators), and provides ablations showing that target coupling and self-grounded prediction each contribute. The two-stage training scheme is a thoughtful response to the scarcity of human-robot paired data, and the persistence/retrieval module addresses a practical deployment concern. The strengths are real: the evaluation is substantial relative to much of the OSVI literature, and the large margins over zero-shot and OSVI baselines are not the main point of concern. However, the statistical support for one headline comparative claim and the internal validity of the localization metric require attention before the paper can be accepted.

major comments (4)
  1. [Sec. 2.5, Figs. 6A/C and Abstract] The claim that HOST 'even exceeds the baseline fine-tuned on 50 robot demonstrations per task' rests on a 62% vs 56% margin over Wall-OSS+SFT. No confidence intervals or significance tests are reported. If the comparison is on the eight tasks of Fig. 5 (as Fig. 6C suggests), n=160 per method and the pooled two-proportion standard error is approximately 5.5 points, so the 6-point margin is roughly one standard error (per-task clustering widens it further). The authors should report per-task success counts, CIs, and an appropriate paired/clustered test for this comparison, or soften the claim. The paper's own per-task qualifier ('this advantage holds across most of these tasks') concedes losses on some tasks, which is consistent with a noise-level margin.
  2. [Sec. 2.5 / Fig. 6 versus Sec. 2.3] The 62% average is defined over the full 50-task set (Sec. 2.3), while the fine-tuning comparison appears to use a smaller subset (the eight tasks of Fig. 5). The abstract and Sec. 2.1 present the fine-tuning comparison as if it applied to the full benchmark. Please state explicitly how many tasks are used in Fig. 6A/C, list them, and either provide full 50-task fine-tuning results or clearly scope the 'exceeds fine-tuned on 50 demos' claim to the subset. Without this, the headline comparison mixes two different evaluation sets.
  3. [Sec. 2.9, Eqs. (6)-(7), Fig. 10B] The localization accuracy evaluation uses 'ground-truth progress values provided by the alignment module' - the same module that constructs the training targets in Sec. 4.2. This is circular for an accuracy claim: it measures whether the localization head reproduces the alignment module's output, not whether the progress estimate is correct against true task progress. The manuscript already has an independent human-annotation protocol in Sec. 2.8 (Fig. 9D); apply that protocol to the localization head, or relabel the metric as 'alignment-module consistency' and discuss the interpretation accordingly.
  4. [Discussion / Sec. 2.2] All experiments use a single bimanual platform (ARX R5, 20-DoF); the Discussion correctly acknowledges that HOST 'remains untested on robots that differ substantially in embodiment.' Given that the title and abstract promise human-video skill acquisition generally, the cross-embodiment claim should be scoped to the tested platform, or the paper should include at least one additional embodiment or joint-level transfer check. This is a scope limitation rather than an internal inconsistency, but it is load-bearing for the generality of the central claim.
minor comments (5)
  1. [Sec. 2.4] Define 'zero-shot' as language-conditioned zero-shot and distinguish it from one-shot visual imitation; the 45% and 43% margins are easy to conflate.
  2. [Eq. 12] The '||' notation for token-sequence concatenation is not defined; add a sentence.
  3. [Abstract] The abstract mentions 'project website and GitHub repository' but no URLs appear in the text; include them.
  4. [Fig. 10E] Report retrieval accuracy and novelty recognition with confidence intervals or trial counts; the threshold-band claim would be easier to evaluate with error bars.
  5. [Sec. 2.9] Use 'alignment-derived labels' rather than 'ground-truth progress' when the alignment module's output is the reference.

Circularity Check

1 steps flagged

One supporting metric (localization accuracy) is self-referential, but the central skill-acquisition claims are externally benchmarked and do not reduce to the method's inputs.

specific steps
  1. fitted input called prediction [Sec. 2.9 (Fig. 10B), with labels/targets defined in Sec. 4.2 (Eq. 7) and trained by Eq. 15]
    "We evaluate localization accuracy on the novel tasks, using ground-truth progress values provided by the alignment module. The predicted progress p̂_t closely tracks the ground-truth p_t across all steps, as presented in Fig. 10B, with a mean absolute error of 0.013 in normalized progress."

    The 'ground-truth' progress values are produced by the same alignment module that constructs the scalar localization labels p_t and the coupled prediction targets T_t used to train the policy (Sec. 4.2: 'The current progress of the robot within W is recorded as a scalar localization label p_t'; Eq. 15 trains L_loc on y_p = E_p(p_t)). The reported 0.013 MAE therefore measures how well the localization head reproduces its own training-label generator, not agreement with an independent ground truth. This is a fitted-input-called-prediction loop. It is a supporting internal metric rather than the core claim: the alignment module itself is checked against human annotations in Fig. 9D, and the headline success/retention results are judged by external task success.

full rationale

The core derivation is an empirical system-building story, not a mathematical derivation. The headline claims—62% success on 50 novel tasks, 43–45% margins over OSVI/zero-shot baselines, and retention—are measured by externally judged task success, so they cannot reduce to the method's inputs by construction. The two mechanisms (target coupling and self-grounded prediction) are supported by ablations on held-out tasks, and the equations define a training objective rather than a self-referential prediction. The one genuinely circular evaluation I found is in Sec. 2.9: localization accuracy is scored against 'ground-truth progress values provided by the alignment module,' i.e., the same module that generated the p_t labels and coupled targets used to supervise the localization head. That makes the 0.013 MAE a fit-quality statistic with respect to the system's own label generator. This is not load-bearing for the central result because alignment is independently validated against human-annotated events (Fig. 9D) and the main success rates are externally judged. Separately, the 6-point margin over the best SFT baseline (62% vs 56%) lacks error bars or significance testing, which is a statistical robustness concern, not a circularity concern. On balance, the circular content is limited to a secondary self-referential metric, so the score is 3 rather than 0.

Axiom & Free-Parameter Ledger

8 free parameters · 5 axioms · 1 invented entities

The method's behavior hinges on a handful of tuned alignment and loss hyperparameters, and on the domain assumption that monotonic task-progress alignment between human video and robot execution is learnable from paired data on the same platform. The invented entity is a latent alignment manifold, which has partial external validation. No new physical entities or constants are introduced.

free parameters (8)
  • SDTW softmin temperature γ = 1.0
    Controls soft-min sharpness in alignment (Supp. Table 1). Chosen by the authors; affects matching probabilities and hence all coupled targets.
  • Column-normalization temperature γ_f = 0.1
    Sharpens cost matrix in Eq. 2 (Supp. Table 1). Directly shapes the alignment used to construct training targets.
  • Softmax temperature κ = 0.1
    Scales pairwise similarity in Eq. 2 (Supp. Table 1). Tuned, affects alignment quality.
  • TCC variance weight λ_var / DTW loss weight λ_DTW = 0.001 / 0.3
    Balance alignment losses in Eq. 9-10 (Supp. Table 1). Chosen by hand; if changed, alignment and therefore targets change.
  • Policy loss weights λ_o, λ_a, λ_p = 1.0 / 10.0 / 1.0
    Balance observation, action, and localization objectives in Eq. 16 (Supp. Table 3). Tuned; action weight is 10x to compensate token count.
  • NCP noise scale / probability = 0.5 / 0.5
    Noise added to clean progress token during training (Supp. B.2). Tuned for robust autoregressive conditioning.
  • Retrieval threshold δ★ and mixing weight ω = δ★ (exact value not stated); ω in Eq. 20
    Determines when a stored video is reused vs. a new video is requested (Sec. 2.11). The paper says δ★ lies within a wide band but does not give its value or ω.
  • Video window length L / action chunk H / downsample ratio = 192 frames / 32 steps / 8:1
    Design choices controlling context and target horizon (Supp. Table 4). Not derived from any principle.
axioms (5)
  • domain assumption A shared task-progress manifold exists that monotonically aligns any two executions of the same manipulation task.
    The entire target-coupling mechanism (Sec. 4.2, Eq. 3-7) assumes that Smooth DTW + temporal cycle-consistency can recover frame-level correspondence between a human video and a robot trajectory. This is empirically validated against human annotations in Fig. 9D, but remains an assumption for novel tasks and novel embodiments.
  • domain assumption Task progress is a scalar, monotonic quantity across both video demonstration and robot trajectory.
    The localization label p_t and window advancement Eq. 18 rely on strictly monotonic progress. Boundary clipping and monotonic path constraints in Eq. 3 bake in this assumption; tasks with loops or optional sub-steps would break it.
  • domain assumption Same-embodiment robot–robot pretraining transfers to human-video conditioning with only 5,847 human–robot pairs.
    The two-stage protocol (Sec. 4.4) assumes that Stage 1 builds a generic 'follow-a-demonstration' capability that Stage 2 can adapt from robot videos to human videos using a comparatively small paired set. This is plausible and ablated (Fig. 10D) but only on one platform.
  • domain assumption Human evaluators' success judgments against task-specific completion criteria are reliable and unbiased.
    All headline success rates depend on human judges (Sec. 2.2); no inter-evaluator agreement or blind evaluation protocol is reported.
  • ad hoc to paper The 50 novel tasks selected are representative of the manipulation skills the method claims to acquire.
    The task set is self-selected and spans varied objects/tools/primitives (Sec. 2.3), but no criteria are given for how they were chosen or whether failures on easier tasks were excluded.
invented entities (1)
  • Shared task progress manifold (learned embedding space for aligning human video and robot trajectory frames) independent evidence
    purpose: Provides frame-level correspondence for target coupling and window advancement (Sec. 4.2).
    The manifold is validated against human-annotated events in Fig. 9D (MAD 0.006 vs 0.079 for clock-time), giving it an external falsifiable handle within the paper. It is still a learned latent construct, not a physical entity.

pith-pipeline@v1.3.0-alltime-deepseek · 32028 in / 8896 out tokens · 81315 ms · 2026-08-01T10:58:30.042889+00:00 · methodology

0 comments
read the original abstract

The ability to acquire skills rapidly and effortlessly while retaining those already mastered is essential for robots. However, current methods still rely on a cumbersome training-time loop that is costly and slow, while eroding skills already mastered. In this paper, we introduce HOST (Human-to-robot One-Shot Skill AcquisiTion), a framework that enables a robot to acquire skills in seconds from a single human video while retaining previously mastered skills. HOST resolves skill acquisition through a cascade of self-grounded prediction. It first estimates the robot's progress within the demonstrated task, then translates the upcoming progression into the robot's own future observations, and finally derives actions from these predicted observations. This cascade is trained on targets coupled to the video demonstration, obtained by mapping the robot trajectory and the video demonstration onto a shared task progress manifold, then redefining each target to align with the future progression of the video. HOST thereby enables the robot to actively follow the demonstrated procedure and adapt it to the robot's embodiment. HOST acquires novel skills at inference time from a single human video in an average of 29 seconds and achieves a 62% average success rate. It exceeds the zero-shot baseline by 45% while retaining previously mastered skills. HOST even exceeds the baseline fine-tuned on 50 robot demonstrations per task while requiring 50 times fewer demonstrations and acquiring each skill 507 times faster. Additional information about HOST is available on the project website.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

117 extracted references · 7 canonical work pages

  1. [1]

    Learning agile and dynamic motor skills for legged robots.Science Robotics, 4(26):eaau5872, 2019

    Jemin Hwangbo, Joonho Lee, Alexey Dosovitskiy, Dario Bellicoso, Vassilios Tsounis, Vladlen Koltun, and Marco Hutter. Learning agile and dynamic motor skills for legged robots.Science Robotics, 4(26):eaau5872, 2019

  2. [2]

    Learning quadrupedal locomotion over challenging terrain.Science Robotics, 5(47):eabc5986, 2020

    Joonho Lee, Jemin Hwangbo, Lorenz Wellhausen, Vladlen Koltun, and Marco Hutter. Learning quadrupedal locomotion over challenging terrain.Science Robotics, 5(47):eabc5986, 2020

  3. [3]

    Visual dexterity: In-hand reorientation of novel and complex object shapes.Science Robotics, 8(84):eadc9244, 2023

    Tao Chen, Megha Tippur, Siyang Wu, Vikash Kumar, Edward Adelson, and Pulkit Agrawal. Visual dexterity: In-hand reorientation of novel and complex object shapes.Science Robotics, 8(84):eadc9244, 2023

  4. [4]

    Solving Rubik’s cube with a robot hand

    OpenAI, Ilge Akkaya, Marcin Andrychowicz, Maciek Chociej, Mateusz Litwin, Bob McGrew, Arthur Petron, Alex Paino, Matthias Plappert, Glenn Powell, Raphael Ribas, Jonas Schneider, Nikolas Tezak, Jerry Tworek, Peter Welinder, Lilian Weng, Qiming Yuan, Wojciech Zaremba, and Lei Zhang. Solving Rubik’s cube with a robot hand. arXiv preprint arXiv:1910.07113, 2019

  5. [5]

    QT-Opt: Scalable deep reinforcement learning for vision-based robotic manipulation

    Dmitry Kalashnikov, Alex Irpan, Peter Pastor, Julian Ibarz, Alexander Herzog, Eric Jang, Deirdre Quillen, Ethan Holly, Mrinal Kalakrishnan, Vincent Vanhoucke, and Sergey Levine. QT-Opt: Scalable deep reinforcement learning for vision-based robotic manipulation. InConference on Robot Learning, volume 87, pages 651–673. PMLR, 2018

  6. [6]

    Scaling up multi-task robotic reinforcement learning

    Dmitry Kalashnikov, Jake Varley, Yevgen Chebotar, Benjamin Swanson, Rico Jonschkowski, Chelsea Finn, Sergey Levine, and Karol Hausman. Scaling up multi-task robotic reinforcement learning. InProceedings of the 5th Conference on Robot Learning, volume 164 ofProceedings of Machine Learning Research, pages 557–575. PMLR,

  7. [7]

    A generalized path integral control approach to reinforce- ment learning.Journal of Machine Learning Research, 11:3137–3181, 2010

    Evangelos Theodorou, Jonas Buchli, and Stefan Schaal. A generalized path integral control approach to reinforce- ment learning.Journal of Machine Learning Research, 11:3137–3181, 2010

  8. [8]

    Path integral guided policy search

    Yevgen Chebotar, Mrinal Kalakrishnan, Ali Yahya, Adrian Li, Stefan Schaal, and Sergey Levine. Path integral guided policy search. In2017 IEEE International Conference on Robotics and Automation (ICRA), pages 3381–3388. IEEE,

  9. [9]

    Learning coordinated badminton skills for legged manipulators.Science Robotics, 10(102):eadu3922, 2025

    Yuntao Ma, Andrei Cramariuc, Farbod Farshidian, and Marco Hutter. Learning coordinated badminton skills for legged manipulators.Science Robotics, 10(102):eadu3922, 2025. doi: 10.1126/scirobotics.adu3922

  10. [10]

    Precise and dexterous robotic manipulation via human-in- the-loop reinforcement learning.Science Robotics, 10(105):eads5033, 2025

    Jianlan Luo, Charles Xu, Jeffrey Wu, and Sergey Levine. Precise and dexterous robotic manipulation via human-in- the-loop reinforcement learning.Science Robotics, 10(105):eads5033, 2025. doi: 10.1126/scirobotics.ads5033

  11. [11]

    Barreiros, Aykut Özgün Önol, Mengchao Zhang, Sam Creasey, Aimee Goncalves, Andrew Beaulieu, Aditya Bhat, Kate M

    Jose A. Barreiros, Aykut Özgün Önol, Mengchao Zhang, Sam Creasey, Aimee Goncalves, Andrew Beaulieu, Aditya Bhat, Kate M. Tsui, and Alex Alspach. Learning contact-rich whole-body manipulation with example-guided reinforcement learning.Science Robotics, 10(105):eads6790, 2025. doi: 10.1126/scirobotics.ads6790. 22

  12. [12]

    Is imitation learning the route to humanoid robots?Trends in Cognitive Sciences, 3(6):233–242, 1999

    Stefan Schaal. Is imitation learning the route to humanoid robots?Trends in Cognitive Sciences, 3(6):233–242, 1999

  13. [13]

    Sanketi, Grecia Salazar, Michael S

    Brianna Zitkovich, Tianhe Yu, Sichun Xu, Peng Xu, Ted Xiao, Fei Xia, Jialin Wu, Paul Wohlhart, Stefan Welker, Ayzaan Wahid, Quan Vuong, Vincent Vanhoucke, Huong Tran, Radu Soricut, Anikait Singh, Jaspiar Singh, Pierre Sermanet, Pannag R. Sanketi, Grecia Salazar, Michael S. Ryoo, et al. RT-2: Vision-language-action models transfer web knowledge to robotic ...

  14. [14]

    In Proceedings of Robotics: Science and Systems, Los Angeles, CA, USA, 6 2025

    Kevin Black, Noah Brown, Danny Driess, Adnan Esmail, Michael Robert Equi, Chelsea Finn, Niccolo Fusai, Lachy Groom, Karol Hausman, Brian Ichter, Szymon Jakubczak, Tim Jones, Liyiming Ke, Sergey Levine, Adrian Li-Bell, Mohith Mothukuri, Suraj Nair, Karl Pertsch, Lucy Xiaoyang Shi, Laura Smith, James Tanner, Quan Vuong, Anna Walling, Haohuan Wang, and Ury Z...

  15. [15]

    In9th Annual Conference on Robot Learning, 2025

    Kevin Black, Noah Brown, James Darpinian, Karan Dhabalia, Danny Driess, Adnan Esmail, Michael Robert Equi, Chelsea Finn, Niccolo Fusai, Manuel Y Galliker, et al.𝜋0.5: A vision-language-action model with open-world generalization. In9th Annual Conference on Robot Learning, 2025. arXiv:2504.16054

  16. [16]

    Toward next-generation learned robot manipulation.Science Robotics, 6(54):eabd9461,

    Jinda Cui and Jeff Trinkle. Toward next-generation learned robot manipulation.Science Robotics, 6(54):eabd9461,

  17. [17]

    Diffusion policy: Visuomotor policy learning via action diffusion

    Cheng Chi, Siyuan Feng, Yilun Du, Zhenjia Xu, Eric Cousineau, Benjamin Burchfiel, and Shuran Song. Diffusion policy: Visuomotor policy learning via action diffusion. InProceedings of Robotics: Science and Systems (RSS), 2023

  18. [18]

    Learning fine-grained bimanual manipulation with low-cost hardware

    Tony Z Zhao, Vikash Kumar, Sergey Levine, and Chelsea Finn. Learning fine-grained bimanual manipulation with low-cost hardware. InProceedings of Robotics: Science and Systems (RSS), 2023

  19. [19]

    RT-1: Robotics transformer for real-world control at scale

    Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Joseph Dabis, Chelsea Finn, Keerthana Gopalakr- ishnan, Karol Hausman, Alex Herzog, Jasmine Hsu, et al. RT-1: Robotics transformer for real-world control at scale. InProceedings of Robotics: Science and Systems (RSS), 2023

  20. [20]

    Open X-Embodiment: Robotic learning datasets and RT-X models

    Open X-Embodiment Collaboration. Open X-Embodiment: Robotic learning datasets and RT-X models. In2024 IEEE International Conference on Robotics and Automation (ICRA), pages 6892–6903. IEEE, 2024. doi: 10.1109/ ICRA57147.2024.10611477

  21. [21]

    Octo: An open-source generalist robot policy

    Dibya Ghosh, Homer Rich Walke, Karl Pertsch, Kevin Black, Oier Mees, Sudeep Dasari, Joey Hejna, Tobias Kreiman, Charles Xu, Jianlan Luo, You Liang Tan, Lawrence Yunliang Chen, Quan Vuong, Ted Xiao, Pannag R Sanketi, Dorsa Sadigh, Chelsea Finn, and Sergey Levine. Octo: An open-source generalist robot policy. InProceedings of Robotics: Science and Systems, ...

  22. [22]

    From intention to execution: Probing the generaliza- tion boundaries of vision-language-action models.arXiv preprint arXiv:2506.09930, 2025

    Irving Fang, Juexiao Zhang, Shengbang Tong, and Chen Feng. From intention to execution: Probing the generaliza- tion boundaries of vision-language-action models.arXiv preprint arXiv:2506.09930, 2025

  23. [23]

    Ricl: Adding in-context adaptability to pre-trained vision-language-action models

    Kaustubh Sridhar et al. Ricl: Adding in-context adaptability to pre-trained vision-language-action models. In Conference on Robot Learning, 2025

  24. [24]

    FLaRe: Achieving masterful and adaptive robot policies with large-scale reinforcement learning fine-tuning

    Jiaheng Hu, Rose Hendrix, Ali Farhadi, Aniruddha Kembhavi, Roberto Martin-Martin, Peter Stone, Kuo-Hao Zeng, and Kiana Ehsani. FLaRe: Achieving masterful and adaptive robot policies with large-scale reinforcement learning fine-tuning. In2025 IEEE International Conference on Robotics and Automation (ICRA), 2025

  25. [25]

    Qingwei Dong, Peng Zeng, Yunpeng He, Guangxi Wan, and Xiaoting Dong. Mitigating catastrophic forgetting in robot continual learning: A guided policy search approach enhanced with memory-aware synapses.IEEE Robotics and Automation Letters, 9(12):11242–11249, 2024. doi: 10.1109/LRA.2024.3487484

  26. [26]

    Learning by watching: Extracting reusable task knowledge from visual observation of human performance.IEEE Transactions on Robotics and Automation, 10(6):799–822, 1994

    Yasuo Kuniyoshi, Masayuki Inaba, and Hirochika Inoue. Learning by watching: Extracting reusable task knowledge from visual observation of human performance.IEEE Transactions on Robotics and Automation, 10(6):799–822, 1994

  27. [27]

    One-shot visual imitation learning via meta-learning

    Chelsea Finn, Tianhe Yu, Tianhao Zhang, Pieter Abbeel, and Sergey Levine. One-shot visual imitation learning via meta-learning. InConference on Robot Learning, pages 357–368. PMLR, 2017

  28. [28]

    One-shot imitation from observing humans via domain-adaptive meta-learning

    Tianhe Yu, Chelsea Finn, Annie Xie, Sudeep Dasari, Tianhao Zhang, Pieter Abbeel, and Sergey Levine. One-shot imitation from observing humans via domain-adaptive meta-learning. InProceedings of Robotics: Science and Systems (RSS), 2018

  29. [29]

    Towards more generalizable one-shot visual imitation learning

    Zhao Mandi, Fangchen Liu, Kimin Lee, and Pieter Abbeel. Towards more generalizable one-shot visual imitation learning. In2022 IEEE International Conference on Robotics and Automation (ICRA), pages 2434–2444. IEEE, 2022

  30. [30]

    Vid2Robot: 23 End-to-end Video-conditioned Policy Learning with Cross-Attention Transformers

    Vidhi Jain, Maria Attarian, Nikhil J Joshi, Ayzaan Wahid, Danny Driess, Quan Vuong, Pannag R Sanketi, Pierre Sermanet, Stefan Welker, Christine Chan, Igor Gilitschenski, Yonatan Bisk, and Debidatta Dwibedi. Vid2Robot: 23 End-to-end Video-conditioned Policy Learning with Cross-Attention Transformers. InProceedings of Robotics: Science and Systems, Delft, N...

  31. [31]

    Osvi-wm: One-shot visual imitation for unseen tasks using world-model-guided trajectory generation

    Raktim Goswami, Prashanth Krishnamurthy, Yann LeCun, and Farshad Khorrami. Osvi-wm: One-shot visual imitation for unseen tasks using world-model-guided trajectory generation. In D. Belgrave, C. Zhang, H. Lin, R. Pascanu, P. Koniusz, M. Ghassemi, and N. Chen, editors,Advances in Neural Information Processing Systems, volume 38, pages 54725–54745. Curran As...

  32. [32]

    Remembering the past to imagine the future: the prospective brain.Nature Reviews Neuroscience, 8(9):657–661, 2007

    Daniel L Schacter, Donna Rose Addis, and Randy L Buckner. Remembering the past to imagine the future: the prospective brain.Nature Reviews Neuroscience, 8(9):657–661, 2007

  33. [33]

    Infant imitation after a 1-week delay: long-term memory for novel acts and multiple stimuli

    Andrew N Meltzoff. Infant imitation after a 1-week delay: long-term memory for novel acts and multiple stimuli. Developmental Psychology, 24(4):470–476, 1988

  34. [34]

    Reinforcement learning and episodic memory in humans and animals: an integrative framework.Annual Review of Psychology, 68:101–128, 2017

    Samuel J Gershman and Nathaniel D Daw. Reinforcement learning and episodic memory in humans and animals: an integrative framework.Annual Review of Psychology, 68:101–128, 2017

  35. [35]

    Neural simulation of action: a unifying mechanism for motor cognition.NeuroImage, 14(1): S103–S109, 2001

    Marc Jeannerod. Neural simulation of action: a unifying mechanism for motor cognition.NeuroImage, 14(1): S103–S109, 2001

  36. [36]

    A unifying computational framework for motor control and social interaction.Philosophical Transactions of the Royal Society B: Biological Sciences, 358(1431):593–602, 2003

    Daniel M Wolpert, Kenji Doya, and Mitsuo Kawato. A unifying computational framework for motor control and social interaction.Philosophical Transactions of the Royal Society B: Biological Sciences, 358(1431):593–602, 2003

  37. [37]

    Neurophysiological mechanisms underlying the understanding and imitation of action.Nature Reviews Neuroscience, 2(9):661–670, 2001

    Giacomo Rizzolatti, Leonardo Fogassi, and Vittorio Gallese. Neurophysiological mechanisms underlying the understanding and imitation of action.Nature Reviews Neuroscience, 2(9):661–670, 2001

  38. [38]

    Neural circuits underlying imitation learning of hand actions: an event-related fmri study.Neuron, 42(2):323–334,

    Giovanni Buccino, Stefan Vogt, Afra Ritzl, Gereon R Fink, Karl Zilles, Hans-Joachim Freund, and Giacomo Rizzolatti. Neural circuits underlying imitation learning of hand actions: an event-related fmri study.Neuron, 42(2):323–334,

  39. [39]

    One-shot visual imitation via attributed waypoints and demonstration augmentation

    Matthew Chang and Saurabh Gupta. One-shot visual imitation via attributed waypoints and demonstration augmentation. In2023 IEEE International Conference on Robotics and Automation (ICRA), pages 5055–5062. IEEE, 2023

  40. [40]

    Igniting vlms toward the embodied space.arXiv preprint arXiv:2509.11766, 2025

    Andy Zhai, Brae Liu, Bruno Fang, Chalse Cai, Ellie Ma, Ethan Yin, Hao Wang, Hugo Zhou, James Wang, Lights Shi, et al. Igniting vlms toward the embodied space.arXiv preprint arXiv:2509.11766, 2025

  41. [41]

    Fast-wam: Do world action models need test-time future imagination?arXiv preprint arXiv:2603.16666, 2026

    Tianyuan Yuan, Zibin Dong, Yicheng Liu, and Hang Zhao. Fast-wam: Do world action models need test-time future imagination?arXiv preprint arXiv:2603.16666, 2026

  42. [42]

    Lora: Low-rank adaptation of large language models

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. InInternational Conference on Learning Representations, 2022

  43. [43]

    Tenenbaum, and Alberto Rodriguez

    Nima Fazeli, Miquel Oller, Jiajun Wu, Zheng Wu, Joshua B. Tenenbaum, and Alberto Rodriguez. See, feel, act: Hierarchical learning for complex manipulation skills with multisensory fusion.Science Robotics, 4(26):eaav3123,

  44. [44]

    Representation learning via global temporal alignment and cycle-consistency

    Isma Hadji, Konstantinos G Derpanis, and Allan D Jepson. Representation learning via global temporal alignment and cycle-consistency. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11068–11077, 2021

  45. [45]

    Temporal cycle- consistency learning

    Debidatta Dwibedi, Yusuf Aytar, Jonathan Tompson, Pierre Sermanet, and Andrew Zisserman. Temporal cycle- consistency learning. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1801–1810, 2019

  46. [46]

    Qwen3-vl technical report.arXiv preprint arXiv:2511.21631, 2025

    Shuai Bai, Yuxuan Cai, Ruizhe Chen, Keqin Chen, Xionghui Chen, Zesen Cheng, Lianghao Deng, Wei Ding, Chang Gao, Chunjiang Ge, et al. Qwen3-vl technical report.arXiv preprint arXiv:2511.21631, 2025

  47. [47]

    Wan: Open and advanced large-scale video generative models.arXiv preprint arXiv:2503.20314, 2025

    Team Wan, Ang Wang, Baole Ai, Bin Wen, Chaojie Mao, Chen-Wei Xie, Di Chen, Feiwu Yu, Haiming Zhao, Jianxiao Yang, et al. Wan: Open and advanced large-scale video generative models.arXiv preprint arXiv:2503.20314, 2025

  48. [48]

    Mixture-of-transformers: A sparse and scalable architecture for multi-modal foundation models.Transactions on Machine Learning Research, 2025

    Weixin Liang, Lili Yu, Liang Luo, Srini Iyer, Ning Dong, Chunting Zhou, Gargi Ghosh, Mike Lewis, Wen-tau Yih, Luke Zettlemoyer, and Xi Victoria Lin. Mixture-of-transformers: A sparse and scalable architecture for multi-modal foundation models.Transactions on Machine Learning Research, 2025. ISSN 2835-8856. URL https://openreview.net/forum?id=Nu6N69i8SB

  49. [49]

    Unimax: Fairer and more effective language sampling for large-scale multilingual pretraining

    Hyung Won Chung, Xavier Garcia, Adam Roberts, Yi Tay, Orhan Firat, Sharan Narang, and Noah Constant. Unimax: Fairer and more effective language sampling for large-scale multilingual pretraining. InInternational Conference on Learning Representations (ICLR), 2023

  50. [50]

    Flow matching for generative modeling

    Yaron Lipman, Ricky TQ Chen, Heli Ben-Hamu, Maximilian Nickel, and Matthew Le. Flow matching for generative modeling. InThe Eleventh International Conference on Learning Representations, 2023. 24

  51. [51]

    A survey of robot learning from demonstra- tion.Robotics and Autonomous Systems, 57(5):469–483, 2009

    Brenna D Argall, Sonia Chernova, Manuela Veloso, and Brett Browning. A survey of robot learning from demonstra- tion.Robotics and Autonomous Systems, 57(5):469–483, 2009

  52. [52]

    Recent advances in robot learning from demonstration.Annual Review of Control, Robotics, and Autonomous Systems, 3:297–330, 2020

    Harish Ravichandar, Athanasios S Polydoros, Sonia Chernova, and Aude Billard. Recent advances in robot learning from demonstration.Annual Review of Control, Robotics, and Autonomous Systems, 3:297–330, 2020

  53. [53]

    Deep imitation learning for complex manipulation tasks from virtual reality teleoperation

    Tianhao Zhang, Zoe McCarthy, Owen Jow, Dennis Lee, Xi Chen, Ken Goldberg, and Pieter Abbeel. Deep imitation learning for complex manipulation tasks from virtual reality teleoperation. In2018 IEEE International Conference on Robotics and Automation (ICRA), pages 5628–5635. IEEE, 2018

  54. [54]

    White, De Ru Tsai, Richard Jaepyeong Cha, Jeffrey Jopling, Chelsea Finn, and Axel Krieger

    Ji Woong (Brian) Kim, Juo-Tung Chen, Pascal Hansen, Lucy Xiaoyang Shi, Antony Goldenberg, Samuel Schmidgall, Paul Maria Scheikl, Anton Deguet, Brandon M. White, De Ru Tsai, Richard Jaepyeong Cha, Jeffrey Jopling, Chelsea Finn, and Axel Krieger. SRT-H: A hierarchical framework for autonomous surgery via language-conditioned imitation learning.Science Robot...

  55. [55]

    Pomerleau

    Dean A. Pomerleau. ALVINN: An autonomous land vehicle in a neural network. In David S. Touretzky, editor, Advances in Neural Information Processing Systems, volume 1. Morgan-Kaufmann, 1988

  56. [56]

    DexMV: Imitation learning for dexterous manipulation from human videos

    Yuzhe Qin, Yueh-Hua Wu, Shaowei Liu, Hanwen Jiang, Ruihan Yang, Yang Fu, and Xiaolong Wang. DexMV: Imitation learning for dexterous manipulation from human videos. InEuropean Conference on Computer Vision, pages 570–587. Springer, 2022

  57. [57]

    DexCap: Scalable and portable mocap data collection system for dexterous manipulation

    Chen Wang, Haochen Shi, Weizhuo Wang, Ruohan Zhang, Li Fei-Fei, and C Karen Liu. DexCap: Scalable and portable mocap data collection system for dexterous manipulation. InProceedings of Robotics: Science and Systems (RSS), 2024

  58. [58]

    Time- contrastive networks: Self-supervised learning from video

    Pierre Sermanet, Corey Lynch, Yevgen Chebotar, Jasmine Hsu, Eric Jang, Stefan Schaal, and Sergey Levine. Time- contrastive networks: Self-supervised learning from video. In2018 IEEE International Conference on Robotics and Automation (ICRA), pages 1134–1141. IEEE, 2018

  59. [59]

    XIRL: Cross-embodiment inverse reinforcement learning

    Kevin Zakka, Andy Zeng, Pete Florence, Jonathan Tompson, Jeannette Bohg, and Debidatta Dwibedi. XIRL: Cross-embodiment inverse reinforcement learning. InConference on Robot Learning, pages 537–546. PMLR, 2022

  60. [60]

    Learning latent plans from play

    Corey Lynch, Mohi Khansari, Ted Xiao, Vikash Kumar, Jonathan Tompson, Sergey Levine, and Pierre Sermanet. Learning latent plans from play. InConference on Robot Learning, pages 1113–1132. PMLR, 2020

  61. [61]

    WHIRL: Human-to-robot imitation in the wild

    Shikhar Bahl, Abhinav Gupta, and Deepak Pathak. WHIRL: Human-to-robot imitation in the wild. InProceedings of Robotics: Science and Systems (RSS), 2022

  62. [62]

    MimicPlay: Long-horizon imitation learning by watching human play

    Chen Wang, Linxi Fan, Jiankai Sun, Ruohan Zhang, Li Fei-Fei, Danfei Xu, Yuke Zhu, and Animashree Anandkumar. MimicPlay: Long-horizon imitation learning by watching human play. InConference on Robot Learning, volume 229, pages 201–221. PMLR, 2023

  63. [63]

    EgoScale: Scaling dexterous manipulation with diverse egocentric human data.arXiv preprint arXiv:2602.16710, 2026

    Ruijie Zheng, Dantong Niu, Yuqi Xie, Jing Wang, Mengda Xu, Yunfan Jiang, Fernando Castañeda, Fengyuan Hu, You Liang Tan, Letian Fu, Trevor Darrell, Furong Huang, Yuke Zhu, Danfei Xu, and Linxi Fan. EgoScale: Scaling dexterous manipulation with diverse egocentric human data.arXiv preprint arXiv:2602.16710, 2026

  64. [64]

    Graphmimic: Graph-to-graphs generative modeling from videos for policy learning

    Guangyan Chen, Te Cui, Meiling Wang, Chengcai Yang, Mengxiao Hu, Haoyang Lu, Yao Mu, Zicai Peng, Tianxing Zhou, Xinran Jiang, et al. Graphmimic: Graph-to-graphs generative modeling from videos for policy learning. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 1756–1768, 2025

  65. [65]

    Learning from videos through graph-to-graphs generative modeling for robotic manipulation.IEEE Transactions on Robotics, 2026

    Guangyan Chen, Meiling Wang, Te Cui, Chengcai Yang, Mengxiao Hu, Haoyang Lu, Zicai Peng, Tianxing Zhou, Xinran Jiang, Yi Yang, et al. Learning from videos through graph-to-graphs generative modeling for robotic manipulation.IEEE Transactions on Robotics, 2026

  66. [66]

    Unifying latent action and latent state pre-training for policy learning from videos

    Guangyan Chen, Meiling Wang, Te Cui, Luojie Yang, Qi Shao, Lin Zhao, Tianle Zhang, Yihang Li, Yi Yang, and Yufeng Yue. Unifying latent action and latent state pre-training for policy learning from videos. InProceedings of the SIGGRAPH Asia 2025 Conference Papers, pages 1–11, 2025

  67. [67]

    What foundation models can bring for robot learning in manipulation: A survey.The International Journal of Robotics Research, 45(7):1091–1142, 2026

    Dingzhe Li, Yixiang Jin, YuHao Sun, Yong A, Hongze Yu, Jun Shi, Xiaoshuai Hao, Peng Hao, Huaping Liu, Xiang Li, Xinde Li, Fuchun Sun, Jianwei Zhang, and Bin Fang. What foundation models can bring for robot learning in manipulation: A survey.The International Journal of Robotics Research, 45(7):1091–1142, 2026. doi: 10.1177/02783649251390579

  68. [68]

    A generalist agent.Transactions on Machine Learning Research, 2022

    Scott Reed, Konrad Zolna, Emilio Parisotto, Sergio Gómez Colmenarejo, Alexander Novikov, Gabriel Barth-Maron, Mai Giménez, Yury Sulsky, Jackie Kay, Jost Tobias Springenberg, et al. A generalist agent.Transactions on Machine Learning Research, 2022

  69. [69]

    OpenVLA: An open-source vision-language-action model.arXiv preprint arXiv:2406.09246, 2024

    Moo Jin Kim, Karl Pertsch, Siddharth Karamcheti, Ted Xiao, Ashwin Balakrishna, Suraj Nair, Rafael Rafailov, Ethan Foster, Grace Lam, Pannag Sanketi, et al. OpenVLA: An open-source vision-language-action model.arXiv preprint arXiv:2406.09246, 2024. 25

  70. [70]

    RDT-1B: A diffusion foundation model for bimanual manipulation.arXiv preprint arXiv:2410.07864, 2024

    Songming Liu, Lingxuan Wu, Bangguo Li, Hengkai Tan, Huayu Chen, Zhengyi Wang, Ke Xu, Hang Su, and Jun Zhu. RDT-1B: A diffusion foundation model for bimanual manipulation.arXiv preprint arXiv:2410.07864, 2024

  71. [71]

    GR00T N1: An open foundation model for generalist humanoid robots.arXiv preprint arXiv:2503.14734, 2025

    Johan Bjorck et al. GR00T N1: An open foundation model for generalist humanoid robots.arXiv preprint arXiv:2503.14734, 2025

  72. [72]

    DexVLA: Vision-language model with plug-in diffusion expert for general robot control

    Junjie Wen, Yichen Zhu, Jinming Li, Zhibin Tang, Chaomin Shen, and Feifei Feng. DexVLA: Vision-language model with plug-in diffusion expert for general robot control. InConference on Robot Learning, volume 305. PMLR, 2025

  73. [73]

    World models.arXiv preprint arXiv:1803.10122, 2018

    David Ha and Jürgen Schmidhuber. World models.arXiv preprint arXiv:1803.10122, 2018

  74. [74]

    Christensen, Hao Su, Jiajun Wu, and Yunzhu Li

    Bo Ai, Stephen Tian, Haochen Shi, Yixuan Wang, Tobias Pfaff, Cheston Tan, Henrik I. Christensen, Hao Su, Jiajun Wu, and Yunzhu Li. A review of learning-based dynamics models for robotic manipulation.Science Robotics, 10 (106):eadt1497, 2025. doi: 10.1126/scirobotics.adt1497

  75. [75]

    Dream to control: Learning behaviors by latent imagination

    Danijar Hafner, Timothy Lillicrap, Jimmy Ba, and Mohammad Norouzi. Dream to control: Learning behaviors by latent imagination. InInternational Conference on Learning Representations (ICLR), 2020

  76. [76]

    Mastering atari with discrete world models

    Danijar Hafner, Timothy Lillicrap, Mohammad Norouzi, and Jimmy Ba. Mastering atari with discrete world models. InInternational Conference on Learning Representations, 2021

  77. [77]

    Mastering diverse control tasks through world models.Nature, 640:647–653, 2025

    Danijar Hafner, Jurgis Pasukonis, Jimmy Ba, and Timothy Lillicrap. Mastering diverse control tasks through world models.Nature, 640:647–653, 2025. arXiv:2301.04104

  78. [78]

    Transformers are sample-efficient world models

    Vincent Micheli, Eloi Alonso, and François Fleuret. Transformers are sample-efficient world models. InInternational Conference on Learning Representations (ICLR), 2023

  79. [79]

    Learning latent dynamics for planning from pixels

    DanijarHafner, TimothyLillicrap, IanFischer, Ruben Villegas, David Ha, Honglak Lee, andJames Davidson. Learning latent dynamics for planning from pixels. InInternational Conference on Machine Learning, pages 2555–2565. PMLR, 2019

  80. [80]

    TD-MPC2: Scalable, robust world models for continuous control

    Nicklas Hansen, Hao Su, and Xiaolong Wang. TD-MPC2: Scalable, robust world models for continuous control. In International Conference on Learning Representations (ICLR), 2024

Showing first 80 references.