Pith. sign in

REVIEW 4 major objections 5 minor 71 references

Egocentric human recovery clips, routed through a gated corrective-intent bottleneck, lift real-robot recovery success from failure starts to 85%, up from 52.5% with robot-only recovery and 71.2% with direct human-recovery co-training.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

T0 review · deepseek-v4-flash

2026-08-01 11:49 UTC pith:S2ZKIIZR

load-bearing objection Genuinely new combination and a clean controlled comparison; single-seed runs and an unproven DCT target keep the mechanism claim from being airtight. the 4 major comments →

arxiv 2607.19745 v2 pith:S2ZKIIZR submitted 2026-07-22 cs.RO cs.CV

EgoRecovery: Acquiring Failure Recovery Ability Through Human Recovery Demonstration

classification cs.RO cs.CV
keywords failure recoveryimitation learningegocentric videohuman-robot data transfercorrective intentco-trainingreal-robot manipulationrecovery gate
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper is trying to establish that egocentric human recovery demonstrations—short clips of a person correcting a staged failure—are a scalable source of failure-recovery supervision for robots, and that a specific transfer mechanism makes them usable. That mechanism, EgoRecovery, shares only a compact corrective-intent signal between embodiments: the timing and magnitude of the remaining correction, compressed into a low-frequency magnitude envelope of future end-effector motion, while direction, contact, and executable actions are learned from robot data. A learned recovery gate switches this signal on only when the robot is in a recovery state. On four real tabletop tasks, this design raises average Recovery SR to 85.0% from 52.5% with robot-only recovery and 71.2% with direct human-recovery co-training, and it raises Initial SR to 80.0%. The reason to care is cost: under the paper's collection protocol, a human operator produces about 10 times more valid recovery episodes per hour than robot teleoperation, so the result points to a cheaper path to robust closed-loop behavior.

Core claim

The central claim is that failure-recovery behavior transfers from human to robot when the human signal is compressed to a corrective-intent target rather than used as action supervision. EgoRecovery co-trains robot success, robot recovery, and human recovery data: the shared intent head predicts a 4D DCT magnitude envelope of the next 16 phase points of end-effector motion on valid recovery windows; the recovery gate head predicts whether the current observation is a recovery state; and the robot action decoder is modulated by the predicted intent through a gated residual feature-wise modulation only when the gate is high. Robot recovery data supplies the same intent target together with ex

What carries the argument

The load-bearing object is corrective intent, defined in Eq. (1) as the magnitude envelope of the next L=16 phase points of end-effector motion, projected onto K=4 low-frequency DCT basis functions and scaled by embodiment scale. It captures how much correction remains and how it unfolds over time, while deliberately discarding direction and contact. The second component is the recovery gate head, which predicts the probability that the current observation requires recovery and produces a scalar that gates the residual feature-wise modulation of the robot action decoder. Together they form a bottleneck: human recovery broadens the failure states that train the intent predictor, robot recover

Load-bearing premise

The load-bearing premise is that a directionless, embodiment-scaled magnitude profile of the remaining correction is an informative signal shared by human and robot recovery; if corrective information lives mainly in direction or contact rather than magnitude, the transfer target is vacuous and the reported gains would come from elsewhere in the pipeline.

What would settle it

Look for a recovery task whose correction is predominantly directional with little translation—e.g., reorienting a grasped object in place. Compute the paper's DCT magnitude features for human and robot recovery clips: if the two distributions show no more overlap than success pairs, and EgoRecovery's Recovery SR advantage over direct human-recovery co-training disappears when the intent loss is removed, the corrective-intent target is not carrying the transfer. The paper's own cup-brush diagnostic (KSR=0.21 vs KSS=0.17) is the first place to look for this failure.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • If correct, egocentric human recovery segments are a practical supervision source: under the paper's protocol, humans produce roughly 10.5× more accepted recovery episodes per operator hour than robot teleoperation, and scaling human recovery from 200 to 800 episodes raises Recovery SR from 75.0% to 90.0% under a fixed 50+50 robot anchor.
  • The gated bottleneck is the active ingredient, not the extra data alone: with identical data, EgoRecovery reaches 85.0% Recovery SR versus 71.2% for direct human-recovery co-training; removing the intent loss drops Recovery SR to 65.0%, removing the corrective-segment mask to 55.0%, and removing gated modulation to 65.0%.
  • Robot recovery grounding remains necessary: human recovery alone (300 episodes plus robot success data) reaches only 8.8% Recovery SR, and with only 20 robot recovery episodes, adding 600 human recovery episodes reaches 75.0%, below the 85.0% achieved with 50 robot + 300 human episodes.
  • The recovery gate preserves nominal execution: forcing intent modulation on all states drops Initial SR from 80.0% to 70.0%, while the gated version keeps nominal behavior intact and still improves recovery.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: if the throughput and transfer results hold beyond the four tabletop tasks, the collection recipe 'stage a failure, record a short human clip, keep a modest robot anchor' could make recovery supervision cheap enough to gather at scale for many manipulation skills, including in non-laboratory settings.
  • Editorial inference: the magnitude-only intent target is an explicit wager that correction information lives in 'how much' rather than 'which way'; a natural extension is a directional or orientation-aware intent target, motivated by the paper's own cup-brush diagnostic in which recovery DCT features are less aligned across embodiments (KS 0.21 vs 0.17 for success).
  • Editorial inference: the recovery gate's sharp drop near the annotated recovery boundary (within ±1 frame on all tested robot episodes) hints that the same head could serve as a runtime failure detector; the paper does not claim this.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes EgoRecovery, a co-training framework for robot failure recovery that augments limited robot recovery demonstrations with egocentric human recovery segments. Human and robot data are aligned through a compact corrective-intent target: a low-frequency DCT magnitude envelope of future end-effector motion (Eq. 1). A learned recovery gate modulates robot decoder features via gated FiLM so that corrective intent affects action decoding only in recovery states. Experiments on four real-world tabletop manipulation tasks report that EgoRecovery raises average Recovery SR from 52.5% (robot-only) and 71.2% (direct human recovery mix) to 85.0%, while preserving Initial SR at 80.0%. The appendix provides confidence intervals, ablations, and mechanism diagnostics for the intent target, the learned bottleneck, and the gate.

Significance. If the result holds, the paper makes a strong practical contribution: egocentric human recovery data can be collected at roughly 10x the throughput of robot teleoperation, and a gated corrective-intent bottleneck makes that data usable for robot recovery while leaving embodiment-specific execution to robot supervision. The experimental design is a genuine strength: the key comparison (direct human recovery mix vs. EgoRecovery) holds data composition and backbone fixed, the ablation study isolates the intent loss, the mask, the modulation path, and the gate, and the appendix reports Wilson confidence intervals and count-level audits. The code release and the detailed collection/annotation protocol support reproducibility. The main weaknesses are statistical (single seed, overlapping confidence intervals for the headline contrast) and mechanistic (the paper's own KS diagnostic weakens the claim that Eq. (1)'s DCT target is the shared carrier of recovery information for cup-brush insertion).

major comments (4)
  1. [§4.2, Table 6, Appendix H, Appendix I] The headline claim that EgoRecovery improves Recovery SR over the direct human recovery mix rests on a difference of 85.0 vs. 71.2, but the Wilson 95% intervals overlap: EgoRecovery [75.6, 91.2], direct mix [60.5, 80.0]. Moreover, Appendix H states that each reported model was trained with a single seed. With only 80 recovery rollouts per condition and one seed, the central 13.8-point gain is not statistically distinguished at the aggregate level. Please report multiple seeds (at least 3) and their spread, or a significance test (e.g., a paired or permutation test over the fixed starts). Without this, the main quantitative claim is not established.
  2. [§3.2, Eq. (1), App. J, Fig. 8, Table 2] The proposed mechanism is that the DCT magnitude envelope in Eq. (1) is a shared corrective-intent signal that transfers human recovery information to the robot. The paper's own KS diagnostic in Fig. 8 shows the opposite for cup-brush insertion: the recovery-window KS distance (KSR=0.21) is larger than the success-window distance (KSS=0.17). Recovery windows are exactly where transfer should be most visible. The 'without intent loss' ablation (Table 2) shows that the intent pathway is used, but an auxiliary prediction target can regularize the trunk or gate even if y_t carries little cross-embodiment information; the 'c_t zero' intervention shows only that the predicted intent is used, not that its values encode human correction. Since the aggregate 85.0% is the primary evidence for the contribution, this leaves the mechanism under-supported for one of the four tasks. Please provide a pe
  3. [Appendix D and Limitations] The evaluation starts are drawn from the same task-level failure families used to collect human and robot recovery training data. The benchmark therefore measures within-family recovery, not generalization to unseen failure modes or to qualitatively new recovery skills. This is honestly acknowledged in the Limitations section, but the abstract and introduction state the contribution in broader terms ('acquiring failure recovery ability'). The framing should be adjusted throughout to make the within-family scope explicit, or new evidence should be added for held-out failure families.
  4. [Abstract and Table 1] The abstract claims human recovery yields 'more than 10x as much valid recovery data per hour' compared to robot teleoperation. Table 1 gives per-task ratios of 10.9x, 10.3x, 11.4x, and 9.8x; cube stacking is below 10x. The claim should be qualified as an average (10.5x) or the per-task values should be reported in the abstract. This is not load-bearing but is a factual inconsistency in a headline claim.
minor comments (5)
  1. [Fig. 10] The left panel title reads 'Cu brush insertion' (missing hyphen/space). Please fix.
  2. [§3.2 and App. G] The embodiment scale s_d is referenced in Eq. (1) and App. G but never defined precisely. State how s_d is computed for each embodiment and task.
  3. [App. J] The dimension-normalized sliced Wasserstein distance is central to the bottleneck diagnostic, but the normalization formula is only described verbally ('1/sqrt(d) dependence'). Please give the exact expression used.
  4. [App. H] The appendix says all reported models used a single seed. Even for exploratory scalability experiments, it would be useful to state this prominently in the main text, since the main result is currently a point estimate without seed variance.
  5. [Table 5] Color jitter and ImageNet normalization are listed as the only input augmentations; typical augmentation magnitudes and normalization statistics should be given for reproducibility.

Circularity Check

0 steps flagged

No significant circularity: corrective intent and gate are supervised auxiliary signals, and the headline Recovery SR is an independent closed-loop measurement.

full rationale

The paper's central comparison is empirical: EgoRecovery vs direct mix vs robot-only are closed-loop rollouts from fixed recovery starts, with the same data composition for EgoRecovery and the direct mix. Eq. (1) defines a hand-designed DCT-magnitude intent target from future end-effector motion; it is not derived from or fitted to the evaluation outcome. The intent head and gate are trained by supervised losses (Eq. 3) on targets computed from human/robot recovery segments and annotations; deployment uses only robot observations, so no fitted parameter is renamed as a prediction of the reported Recovery SR. The gain (71.2 to 85.0) is an empirical result and is not forced by construction. The paper discloses the main scope limitation in the Limitations section ('These findings are limited to related off-nominal states within the same task-level recovery family in tabletop manipulation.') and in Appendix D ('The benchmark therefore tests transfer within related recovery behavior rather than unrelated skills.'); this within-family evaluation is a validity/external-validity caveat, not a circularity. The Fig. 8 KS diagnostic (KSR=0.21 vs KSS=0.17 for cup brush) weakens the claim that the specific DCT magnitude target carries the shared recovery signal for that task, but a weak or contradicted assumption is a correctness risk, not a reduction of the result to its inputs. No load-bearing self-citations appear: architecture references (HPT [53], EgoMimic-style co-training [19]) and compact-behavior-transfer citations ([17,21,58]) are external works. The cost-matched budget uses the independently measured 10x throughput of Table 1; the conversion is an accounting device, not a fitted output. Accordingly, no circular step can be exhibited from the paper's equations or citations.

Axiom & Free-Parameter Ledger

5 free parameters · 6 axioms · 1 invented entities

The method's load-bearing elements are: (1) the corrective-intent target computed from DCT magnitudes, (2) the recovery gate labels, (3) the shared HPT backbone, and (4) the within-family evaluation protocol. The first and fourth are the most fragile; neither has independent external evidence beyond the paper's own diagnostics and ablations.

free parameters (5)
  • DCT coefficient count K = 4
    Chosen in main experiments; controls compression of the corrective-intent envelope (Eq. 1, Appendix G). A different K would change the target and likely the results.
  • Future window length L = 16 phase points (8% of episode)
    Chosen via phase resampling P=200; sets the temporal horizon of the correction-magnitude envelope.
  • Embodiment scale s_d = per-embodiment calibration, value not reported
    Normalizes human/robot end-effector motion in Eq. 1; if chosen per dataset, it is a fitted normalizer that affects transfer.
  • Loss weights lambda_c, lambda_g, lambda_n = 0.05, 0.05, 0.01
    Hand-chosen training weights (Appendix H) balancing intent, gate, and nominal regularization; no sensitivity sweep is reported.
  • Active effector selection = one task-designated active arm
    Hand-specified per task; ignores the other arm, constraining the intent signal.
axioms (6)
  • domain assumption DCT magnitude envelope of normalized future end-effector motion (Eq. 1) is an informative shared corrective-intent signal across embodiments
    The transfer mechanism assumes that a directionless magnitude profile of remaining correction carries usable recovery information; Fig. 8 shows weak alignment for cup-brush (KSR=0.21 vs KSS=0.17).
  • domain assumption Recovery boundary t_rec and phase labels s_t/m_t can be annotated reliably
    All gate and intent supervision depends on human-annotated t_rec (Appendix E); annotation errors propagate into training labels.
  • standard math HPT-style shared transformer with embodiment-specific adapters can co-train human and robot observations
    Borrowed architecture from [53]; assumed to transfer representation, not re-derived here.
  • standard math Behavior cloning on 100-step action chunks from robot teleoperation is a valid policy-learning objective
    Standard imitation-learning assumption used for the robot action head (Appendix H).
  • domain assumption Recovery evaluation from off-nominal starts in the same task-level failure families as training is a meaningful test of recovery ability
    Appendix D defines evaluation starts from the same failure families used to collect recovery training data; this is a within-distribution test, not a novel-failure generalization test.
  • domain assumption HaWoR hand-pose reconstruction from egocentric video is accurate enough for intent targets
    Human trajectories are used for the auxiliary decoder and intent target; reconstruction noise is filtered but not independently validated.
invented entities (1)
  • Corrective-intent space (DCT magnitude envelope c_t) no independent evidence
    purpose: Cross-embodiment transfer target; aligns human and robot recovery frames through a 4D low-frequency magnitude profile.
    Introduced by this paper as the shared variable; only internal evidence (Figs. 8-9, ablations) supports its validity. No external benchmark or independent predictive handle is provided.

pith-pipeline@v1.3.0-alltime-deepseek · 23097 in / 13762 out tokens · 136564 ms · 2026-08-01T11:49:20.245566+00:00 · methodology

0 comments
read the original abstract

Robust embodied robots should be able to recover from failures and retry tasks in order to operate reliably in unstructured and noisy real-world environments. Achieving this capability requires training policies on data that captures recovery behaviors. However, collecting such data through robot teleoperation is difficult to scale, as it is time-consuming to induce diverse failure states, perform corrective actions, and reset the environment. This challenge is further exacerbated by the high diversity of failure modes, which demands substantially more recovery data than success demonstrations. In this work, we show that egocentric human data capturing failure recovery processes provides a scalable alternative. By efficiently arranging task-level failure configurations and recording short recovery segments, human operators can generate more than 10x as much valid recovery data per hour compared to robot teleoperation under our protocol. To address the embodiment gap between human and robot, we propose EgoRecovery, a co-training framework for learning recovery behavior, where human recovery demonstrations are aligned to a compact corrective-intent space shared with robot data, which captures the timing and magnitude of correction. Only a small number of robot recovery demonstrations are required to connect this intent to executable robot actions. At deployment, a learned recovery gate predicts when correction is needed from robot observations and activates the corrective intent only in recovery states. Experiments on real-world recovery tasks show that EgoRecovery improves success from failure starts over robot-only recovery, direct co-training with human recovery data, and direct intent-transfer baselines.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

71 extracted references · 26 linked inside Pith

  1. [1]

    Human-to-robotimitationinthewild

    ShikharBahl,AbhinavGupta,andDeepakPathak. Human-to-robotimitationinthewild. In ProceedingsofRobotics: Science and Systems(RSS), 2022

  2. [2]

    H-RDT: Human manipulation enhanced bimanual robotic manipulation.arXivpreprintarXiv:2507.23523, 2025

    Hongzhe Bi, Lingxuan Wu, Tianwei Lin, Hengkai Tan, Zhizhong Su, Hang Su, and Jun Zhu. H-RDT: Human manipulation enhanced bimanual robotic manipulation.arXivpreprintarXiv:2507.23523, 2025

  3. [3]

    In-N-On: Scaling egocentric manipulation with in-the-wild and on-task data.arXivpreprintarXiv:2511.15704, 2025

    Xiongyi Cai, Ri-Zhao Qiu, Geng Chen, Lai Wei, Isabella Liu, Tianshu Huang, Xuxin Cheng, and Xiaolong Wang. In-N-On: Scaling egocentric manipulation with in-the-wild and on-task data.arXivpreprintarXiv:2511.15704, 2025

  4. [4]

    Diffusion policy: Visuomotor policy learning via action diffusion

    Cheng Chi, Zhenjia Xu, Siyuan Feng, Eric Cousineau, Yilun Du, Benjamin Burchfiel, Russ Tedrake, and Shuran Song. Diffusion policy: Visuomotor policy learning via action diffusion. InProceedings of Robotics: Science and Systems (RSS), 2023

  5. [5]

    Universal manipulation interface: In-the-wild robot teaching without in-the-wild robots

    Cheng Chi, Zhenjia Xu, Chuer Pan, Eric Cousineau, Benjamin Burchfiel, Siyuan Feng, Russ Tedrake, and Shuran Song. Universal manipulation interface: In-the-wild robot teaching without in-the-wild robots. InProceedings of Robotics: Science and Systems(RSS), 2024

  6. [6]

    ConLA:Contrastive latent action learning from human videos for robotic manipulation.arXiv preprintarXiv:2602.00557, 2026

    WeishengDai,KaiLan,JianyiZhou,BoZhao,XiuSu,JunwenTong,WeiliGuan,andShuoYang. ConLA:Contrastive latent action learning from human videos for robotic manipulation.arXiv preprintarXiv:2602.00557, 2026

  7. [7]

    Spatial-aware VLA pretraining through visual-physical alignment from human videos.arXivpreprintarXiv:2512.13080, 2025

    Yicheng Feng, Wanpeng Zhang, Ye Wang, Hao Luo, Haoqi Yuan, Sipeng Zheng, and Zongqing Lu. Spatial-aware VLA pretraining through visual-physical alignment from human videos.arXivpreprintarXiv:2512.13080, 2025

  8. [8]

    WARPED: Wrist-aligned rendering for robot policy learning from egocentric human demonstrations.arXivpreprintarXiv:2604.10809, 2026

    Harry Freeman, Chung Hee Kim, and George Kantor. WARPED: Wrist-aligned rendering for robot policy learning from egocentric human demonstrations.arXivpreprintarXiv:2604.10809, 2026

  9. [9]

    Shenyuan Gao, William Liang, Kaiyuan Zheng, Ayaan Malik, Seonghyeon Ye, Sihyun Yu, Wei-Cheng Tseng, Yuzhu Dong, Kaichun Mo, Chen-Hsuan Lin, Qianli Ma, Seungjun Nah, Loic Magne, Jiannan Xiang, Yuqi Xie, Ruijie Zheng, Dantong Niu, You Liang Tan, K. R. Zentner, George Kurian, Suneel Indupuru, Pooya Jannaty, Jinwei Gu, Jun Zhang, Jitendra Malik, Pieter Abbeel,...

  10. [10]

    UniLACT: Depth-aware RGB latent action learning for vision-language-action models.arXivpreprintarXiv:2602.20231, 2026

    Manish Kumar Govind, Dominick Reilly, Pu Wang, and Srijan Das. UniLACT: Depth-aware RGB latent action learning for vision-language-action models.arXivpreprintarXiv:2602.20231, 2026

  11. [11]

    Ego4D: Around the world in 3,000 hours of egocentric video

    Kristen Grauman, Andrew Westbury, Eugene Byrne, Zachary Chavis, Antonino Furnari, Rohit Girdhar, Jackson Hamburger, Hao Jiang, Miao Liu, Xingyu Liu, et al. Ego4D: Around the world in 3,000 hours of egocentric video. In ProceedingsoftheIEEE/CVFConferenceonComputerVisionandPatternRecognition(CVPR),pages18995–19012, 2022

  12. [12]

    BridgeACT: Bridging human demonstrations to robot actions via unified tool-target affordances.arXiv preprintarXiv:2604.23249, 2026

    Yifan Han, Jianxiang Liu, Haoyu Zhang, Yuqi Gu, Yunhan Guo, and Wenzhao Lian. BridgeACT: Bridging human demonstrations to robot actions via unified tool-target affordances.arXiv preprintarXiv:2604.23249, 2026

  13. [13]

    2HandedAfforder: Learning precise actionable bimanual affordances from human videos

    Marvin Heidinger, Snehal Jauhri, Vignesh Prasad, and Georgia Chalvatzaki. 2HandedAfforder: Learning precise actionable bimanual affordances from human videos. InProceedingsofthe IEEE/CVFInternational Conferenceon ComputerVision(ICCV), pages 14743–14753, 2025

  14. [14]

    ThriftyDAg- ger: Budget-aware novelty and risk gating for interactive imitation learning

    RyanHoque, AshwinBalakrishna, EllenNovoseller, AlbertWilcox, DanielS.Brown, andKenGoldberg. ThriftyDAg- ger: Budget-aware novelty and risk gating for interactive imitation learning. InProceedings of the 5th Conference on RobotLearning (CoRL), volume 164 ofProceedingsof MachineLearning Research, pages 598–608. PMLR, 2021

  15. [15]

    Yoon, Mouli Sivapurapu, and Jian Zhang

    Ryan Hoque, Peide Huang, David J. Yoon, Mouli Sivapurapu, and Jian Zhang. EgoDex: Learning dexterous manipulation from large-scale egocentric video.arXivpreprintarXiv:2505.11709, 2025

  16. [16]

    RaC: Robot learning for long-horizon tasks by scaling recovery and correction.arXivpreprintarXiv:2509.07953, 2025

    Zheyuan Hu, Robyn Wu, Naveen Enock, Jasmine Li, Riya Kadakia, Zackory Erickson, and Aviral Kumar. RaC: Robot learning for long-horizon tasks by scaling recovery and correction.arXivpreprintarXiv:2509.07953, 2025

  17. [17]

    MINT: Mimic intent, not just trajectories.arXiv preprintarXiv:2602.08602, 2026

    Renming Huang, Chendong Zeng, Wenjing Tang, Jintian Cai, Cewu Lu, and Panpan Cai. MINT: Mimic intent, not just trajectories.arXiv preprintarXiv:2602.08602, 2026

  18. [18]

    AnchorRefine: Synergy-manipulationbasedontrajectoryanchorandresidualrefinementforvision-language- action models.arXiv preprintarXiv:2604.17787, 2026

    Tingzheng Jia, Kan Guo, Lanping Qian, Yongli Hu, Daxin Tian, Guixian Qu, Chunmian Lin, Baocai Yin, and Jiapu Wang. AnchorRefine: Synergy-manipulationbasedontrajectoryanchorandresidualrefinementforvision-language- action models.arXiv preprintarXiv:2604.17787, 2026. 11

  19. [19]

    EgoMimic: Scaling imitation learning via egocentric video

    SimarKareer,DhruvPatel,RyanPunamiya,PranayMathur,ShuoCheng,ChenWang,JudyHoffman,andDanfeiXu. EgoMimic: Scaling imitation learning via egocentric video. InProceedingsofthe IEEEInternational Conferenceon RoboticsandAutomation(ICRA). IEEE, 2025

  20. [20]

    Kochenderfer

    Michael Kelly, Chelsea Sidrane, Katherine Driggs-Campbell, and Mykel J. Kochenderfer. HG-DAgger: Interactive imitation learning with human experts. InProceedings of the IEEE International Conference on Robotics and Automation(ICRA), pages 8077–8083. IEEE, 2019

  21. [21]

    UniSkill: Imitating human videos via cross-embodiment skill representations

    Hanjung Kim, Jaehyun Kang, Hyolim Kang, Meedeum Cho, Seon Joo Kim, and Youngwoon Lee. UniSkill: Imitating human videos via cross-embodiment skill representations. InProceedingsofthe 9th ConferenceonRobotLearning (CoRL), volume 305 ofProceedingsofMachineLearning Research, pages 4269–4294. PMLR, 2025

  22. [22]

    DART: Noise injection for robust imitation learning

    Michael Laskey, Jonathan Lee, Roy Fox, Anca Dragan, and Ken Goldberg. DART: Noise injection for robust imitation learning. In Proceedings of the 1st Annual Conference on Robot Learning (CoRL), volume 78 ofProceedings of MachineLearning Research, pages 143–156. PMLR, 2017

  23. [23]

    MVP-LAM:Learningaction-centriclatentactionviacross-viewpointreconstruction

    Jung Min Lee, Dohyeok Lee, Seokhun Ju, Taehyun Cho, Jin Woo Koo, Li Zhao, Sangwoo Hong, and Jungwoo Lee. MVP-LAM:Learningaction-centriclatentactionviacross-viewpointreconstruction. arXivpreprintarXiv:2602.03668, 2026

  24. [24]

    A mechanistic analysis of sim-and-real co-training in generative robot policies.arXivpreprint arXiv:2604.13645, 2026

    Yu Lei, Minghuan Liu, Abhiram Maddukuri, Zhenyu Jiang, and Yuke Zhu. A mechanistic analysis of sim-and-real co-training in generative robot policies.arXivpreprint arXiv:2604.13645, 2026

  25. [25]

    GazeVLA: Learning human intention for robotic manipulation.arXivpreprint arXiv:2604.22615, 2026

    Chengyang Li, Kaiyi Xiong, Yuan Xu, Lei Qian, Yizhou Wang, and Wentao Zhu. GazeVLA: Learning human intention for robotic manipulation.arXivpreprint arXiv:2604.22615, 2026

  26. [26]

    ManipTrans: Efficient dexterous bimanual manipulation transfer via residual learning

    Kailin Li, Puhao Li, Tengyu Liu, Yuyang Li, and Siyuan Huang. ManipTrans: Efficient dexterous bimanual manipulation transfer via residual learning. InProceedingsofthe IEEE/CVFConferenceonComputerVisionand PatternRecognition(CVPR), 2025

  27. [27]

    Scalable vision-language-action model pretraining for robotic manipulation with real-life human activity videos

    Qixiu Li, Yu Deng, Yaobo Liang, Lin Luo, Lei Zhou, Chengtang Yao, Lingqi Zeng, Zhiyuan Feng, Huizhi Liang, Sicheng Xu, Yizhong Zhang, Xi Chen, Hao Chen, Lily Sun, Dong Chen, Jiaolong Yang, and Baining Guo. Scalable vision-language-action model pretraining for robotic manipulation with real-life human activity videos. In Proceedings ofthe IEEE Internationa...

  28. [28]

    Hi-WM: Human-in-the-world-model for scalable robot post-training.arXiv preprintarXiv:2604.21741, 2026

    Yaxuan Li, Zhongyi Zhou, Yefei Chen, Yanjiang Guo, Jiaming Liu, Shanghang Zhang, Jianyu Chen, and Yichen Zhu. Hi-WM: Human-in-the-world-model for scalable robot post-training.arXiv preprintarXiv:2604.21741, 2026

  29. [29]

    EgoLive: A large-scale egocentric dataset from real-world human tasks

    Yihang Li, Xuelong Wei, Jingzhou Luo, Yingjing Xiao, Yibo Bai, Guangyuan Zhou, Teng Zou, Chenguang Gui, Jiajun Wen, He Zhang, Kangliang Chen, Xing Pan, Shuaiyan Liu, Daming Wang, Tao An, Jiayi Li, Shibo Jin, Wanwan Zhang, Tianyu Wang, Boren Wei, Zhixuan Huang, Fangsheng Liu, Ruodai Li, Hui Zhang, Anson Li, Yicheng Gong, Peng Cao, Jiaming Liang, and Liang ...

  30. [30]

    A systematic study of data modalities and strategies for co-training large behavior models for robot manipulation.arXiv preprintarXiv:2602.01067, 2026

    Fanqi Lin, Kushal Arora, Jean Mercat, Haruki Nishimura, Paarth Shah, Chen Xu, Mengchao Zhang, Mark Zolotas, Maya Angeles, Owen Pfannenstiehl, Andrew Beaulieu, and Jose Barreiros. A systematic study of data modalities and strategies for co-training large behavior models for robot manipulation.arXiv preprintarXiv:2602.01067, 2026

  31. [31]

    Model-based runtime monitoring with interactive imitationlearning

    Huihan Liu, Shivin Dass, Roberto Martín-Martín, and Yuke Zhu. Model-based runtime monitoring with interactive imitationlearning. In ProceedingsoftheIEEEInternationalConferenceonRoboticsandAutomation(ICRA).IEEE, 2024

  32. [32]

    EgoZero: Robot learning from smart glasses.arXivpreprintarXiv:2505.20290, 2025

    Vincent Liu, Ademi Adeniji, Haotian Zhan, Siddhant Haldar, Raunaq Bhirangi, Pieter Abbeel, and Lerrel Pinto. EgoZero: Robot learning from smart glasses.arXivpreprintarXiv:2505.20290, 2025

  33. [33]

    REFLECT: Summarizing robot experiences for failure explanation and correction

    Zeyi Liu, Arpit Bahety, and Shuran Song. REFLECT: Summarizing robot experiences for failure explanation and correction. InProceedingsofthe7thConferenceonRobotLearning(CoRL),volume229of ProceedingsofMachine Learning Research. PMLR, 2023

  34. [34]

    Joint-aligned latent action: Towards scalable VLA pretraining in the wild.arXivpreprintarXiv:2602.21736, 2026

    Hao Luo, Ye Wang, Wanpeng Zhang, Haoqi Yuan, Yicheng Feng, Haiweng Xu, Sipeng Zheng, and Zongqing Lu. Joint-aligned latent action: Towards scalable VLA pretraining in the wild.arXivpreprintarXiv:2602.21736, 2026. 12

  35. [35]

    Being-H0.5: Scaling human-centric robot learning for cross-embodiment generalization.arXiv preprintarXiv:2601.12993, 2026

    Hao Luo, Ye Wang, Wanpeng Zhang, Sipeng Zheng, Ziheng Xi, Chaoyi Xu, Haiweng Xu, Haoqi Yuan, Chi Zhang, Yiqing Wang, Yicheng Feng, and Zongqing Lu. Being-H0.5: Scaling human-centric robot learning for cross-embodiment generalization.arXiv preprintarXiv:2601.12993, 2026

  36. [36]

    Learning latent plans from play

    Corey Lynch, Mohi Khansari, Ted Xiao, Vikash Kumar, Jonathan Tompson, Sergey Levine, and Pierre Sermanet. Learning latent plans from play. InProceedings of the 3rd Conference on RobotLearning (CoRL), volume 100 of Proceedings ofMachineLearning Research, pages 1113–1132. PMLR, 2020

  37. [37]

    Robot learning from human videos: A survey.arXiv preprintarXiv:2604.27621, 2026

    Junyi Ma, Erhang Zhang, Haoran Yang, Ditao Li, Chenyang Xu, Guangming Wang, and Hesheng Wang. Robot learning from human videos: A survey.arXiv preprintarXiv:2604.27621, 2026

  38. [38]

    What matters in learning from offline human demonstrations for robot manipulation

    AjayMandlekar,DanfeiXu,JosiahWong,SoroushNasiriany,ChenWang,RohunKulkarni,LiFei-Fei,SilvioSavarese, Yuke Zhu, and Roberto Martín-Martín. What matters in learning from offline human demonstrations for robot manipulation. In Proceedings of the 5th Conference on Robot Learning (CoRL), volume 164 ofProceedings of MachineLearning Research, pages 1678–1690. PMLR, 2021

  39. [39]

    MimicGen: A data generation system for scalable robot learning using human demonstrations

    Ajay Mandlekar, Soroush Nasiriany, Bowen Wen, Iretiayo Akinola, Yashraj Narang, Linxi Fan, Yuke Zhu, and Dieter Fox. MimicGen: A data generation system for scalable robot learning using human demonstrations. InProceedings ofthe 7th ConferenceonRobotLearning (CoRL), volume 229 ofProceedingsofMachineLearning Research, pages 1820–1864. PMLR, 2023

  40. [40]

    R3M: A universal visual representation for robot manipulation

    Suraj Nair, Aravind Rajeswaran, Vikash Kumar, Chelsea Finn, and Abhinav Gupta. R3M: A universal visual representation for robot manipulation. InProceedings of the 6th Conference on RobotLearning (CoRL), volume 205 ofProceedingsofMachineLearning Research, pages 892–909. PMLR, 2022

  41. [41]

    Open X-Embodiment: Robotic learning datasets and RT-X models

    Abhishek Padalkar, Acorn Pooley, Ajinkya Jain, Alex Bewley, Alex Herzog, Alex Irpan, Alexander Khazatsky, Anant Rai, Anikait Singh, Anthony Brohan, et al. Open X-Embodiment: Robotic learning datasets and RT-X models. InProceedings of the IEEE International Conferenceon Roboticsand Automation(ICRA), pages 6892–6903. IEEE, 2024

  42. [42]

    SOP: A scalable online post-training system for vision-language-action models

    Mingjie Pan, Siyuan Feng, Qinglin Zhang, Xinchen Li, Jianheng Song, Chendi Qu, Yi Wang, Chuankang Li, Ziyu Xiong, Zhi Chen, Yi Liu, and Jianlan Luo. SOP: A scalable online post-training system for vision-language-action models. arXivpreprint arXiv:2601.03044, 2026

  43. [43]

    FAST: Efficient action tokenization for vision-language-action models

    Karl Pertsch, Kyle Stachowicz, Brian Ichter, Danny Driess, Suraj Nair, Quan Vuong, Oier Mees, Chelsea Finn, and Sergey Levine. FAST: Efficient action tokenization for vision-language-action models. InProceedings of Robotics: Science and Systems(RSS), 2025

  44. [44]

    Zhu, Simar Kareer, Judy Hoffman, and Danfei Xu

    Ryan Punamiya, Dhruv Patel, Patcharapong Aphiwetsa, Pranav Kuppili, Lawrence Y. Zhu, Simar Kareer, Judy Hoffman, and Danfei Xu. EgoBridge: Domain adaptation for generalizable imitation from egocentric human data. In Advancesin NeuralInformationProcessingSystems(NeurIPS), 2025

  45. [45]

    Ryan Punamiya, Simar Kareer, Zeyi Liu, Josh Citron, Ri-Zhao Qiu, Xiongyi Cai, Alexey Gavryushin, Jiaqi Chen, Davide Liconti, Lawrence Y. Zhu, Patcharapong Aphiwetsa, Baoyu Li, Aniketh Cheluva, Pranav Kuppili, Yangcen Liu,DhruvPatel,AidanGao,Hye-YoungChung,RyanCo,ReneeZbizika,JeffLiu,XiaomengXu,HaoyuXiong,Geng Chen, Sebastiano Oliani, Chenyu Yang, Xi Wang,...

  46. [46]

    Gordon, and J

    Stéphane Ross, Geoffrey J. Gordon, and J. Andrew Bagnell. A reduction of imitation learning and structured predictiontono-regretonlinelearning. In Proceedingsofthe14thInternationalConferenceonArtificialIntelligence and Statistics(AISTATS), volume 15 ofProceedingsofMachineLearning Research, pages 627–635. PMLR, 2011

  47. [47]

    CARE: Multi-task pretraining for latent continuous action representation in robot control.arXivpreprintarXiv:2601.22467, 2026

    Jiaqi Shi, Xulong Zhang, Xiaoyang Qu, and Jianzong Wang. CARE: Multi-task pretraining for latent continuous action representation in robot control.arXivpreprintarXiv:2601.22467, 2026

  48. [48]

    ZeroMimic: Distilling robotic manipulation skills from web videos

    Junyao Shi, Zhuolun Zhao, Tianyou Wang, Ian Pedroza, Amy Luo, Jie Wang, Jason Ma, and Dinesh Jayaraman. ZeroMimic: Distilling robotic manipulation skills from web videos. InProceedings of the IEEE International Conferenceon Roboticsand Automation(ICRA). IEEE, 2025

  49. [49]

    Hand-object interaction pretraining from videos

    Himanshu Gaurav Singh, Antonio Loquercio, Carmelo Sferrazza, Jane Wu, Haozhi Qi, Pieter Abbeel, and Jitendra 13 Malik. Hand-object interaction pretraining from videos. InProceedings of the IEEE International Conference on RoboticsandAutomation(ICRA). IEEE, 2025

  50. [50]

    arXivpreprintarXiv:2602.10098, 2026

    Jingwen Sun, Wenyao Zhang, Zekun Qi, Shaojie Ren, Zezhi Liu, Hanxin Zhu, Guangzhong Sun, Xin Jin, and Zhibo Chen.VLA-JEPA:Enhancingvision-language-actionmodelwithlatentworldmodel. arXivpreprintarXiv:2602.10098, 2026

  51. [51]

    MimicPlay: Long-horizon imitation learning by watching human play

    Chen Wang, Linxi Fan, Jiankai Sun, Ruohan Zhang, Li Fei-Fei, Danfei Xu, Yuke Zhu, and Anima Anandkumar. MimicPlay: Long-horizon imitation learning by watching human play. InProceedings of the 7th Conference on RobotLearning (CoRL), volume 229 ofProceedingsofMachineLearning Research. PMLR, 2023

  52. [52]

    XRZero-G0: Pushing the frontier of dexterous robotic manipulation with interfaces, quality and ratios.arXivpreprintarXiv:2604.13001, 2026

    James Wang, Primo Pu, Zephyr Fung, Alex Wang, Sam Wang, Bender Deng, Kevin Wang, Zivid Liu, Chris Pan, Panda Yang, Andy Zhai, Lucy Liang, Shalfun Li, Johnny Sun, Jacky Xu, Will Tian, Kai Yan, Kohler Ye, Scott Li, Qian Wang, Roy Gan, and Hao Wang. XRZero-G0: Pushing the frontier of dexterous robotic manipulation with interfaces, quality and ratios.arXivpre...

  53. [53]

    Scalingproprioceptive-visuallearningwithheterogeneous pre-trained transformers

    LiruiWang, XinleiChen, JialiangZhao, andKaimingHe. Scalingproprioceptive-visuallearningwithheterogeneous pre-trained transformers. InAdvancesinNeuralInformationProcessingSystems(NeurIPS), 2024

  54. [54]

    Libra-VLA: Achieving learning equilibrium via asynchronous coarse-to-fine dual-system.arXiv preprintarXiv:2604.24921, 2026

    Yifei Wei, Linqing Zhong, Yi Liu, Yuxiang Lu, Xindong He, Maoqing Yao, and Guanghui Ren. Libra-VLA: Achieving learning equilibrium via asynchronous coarse-to-fine dual-system.arXiv preprintarXiv:2604.24921, 2026

  55. [55]

    FlowCorrect: Efficient interactive correction of generative flow policies for robotic manipulation.arXivpreprintarXiv:2602.22056, 2026

    Edgar Welte, Yitian Shi, Rosa Wolf, Maximillian Gilles, and Rania Rayyes. FlowCorrect: Efficient interactive correction of generative flow policies for robotic manipulation.arXivpreprintarXiv:2602.22056, 2026

  56. [56]

    TAMEn: Tactile-aware manipulation engine for closed-loop data collection in contact-rich tasks.arXiv preprint arXiv:2604.07335, 2026

    LongyanWu, JiejiRen, Chenghang Jiang, JunxiZhou, ShijiaPeng, RanHuang, Guoying Gu, Li Chen, andHongyang Li. TAMEn: Tactile-aware manipulation engine for closed-loop data collection in contact-rich tasks.arXiv preprint arXiv:2604.07335, 2026

  57. [57]

    Learning human-intention priors from large-scale human demonstrations for robotic manipulation.arXivpreprintarXiv:2604.24681, 2026

    Yifan Xie, YuAn Wang, Guangyu Chen, Jinkun Liu, Yu Sun, and Wenbo Ding. Learning human-intention priors from large-scale human demonstrations for robotic manipulation.arXivpreprintarXiv:2604.24681, 2026

  58. [58]

    XSkill: Cross embodiment skill discovery

    Mengda Xu, Zhenjia Xu, Cheng Chi, Manuela Veloso, and Shuran Song. XSkill: Cross embodiment skill discovery. InProceedings of the 7th Conferenceon RobotLearning (CoRL), volume 229 ofProceedings of Machine Learning Research, pages 3536–3555. PMLR, 2023

  59. [59]

    LIDEA: Human-to-robot imitation learning via implicit feature distillation and explicit geometry alignment.arXivpreprintarXiv:2604.10677, 2026

    Yifu Xu, Bokai Lin, Xinyu Zhan, Hongjie Fang, Yong-Lu Li, Cewu Lu, and Lixin Yang. LIDEA: Human-to-robot imitation learning via implicit feature distillation and explicit geometry alignment.arXivpreprintarXiv:2604.10677, 2026

  60. [60]

    AoE: Always-on egocentric human video collection for embodied AI.arXiv preprintarXiv:2602.23893, 2026

    Bowen Yang, Zishuo Li, Yang Sun, Changtao Miao, Yifan Yang, Man Luo, Xiaotong Yan, Feng Jiang, Jinchuan Shi, Yankai Fu, Ning Chen, Junkai Zhao, Pengwei Wang, Guocai Yao, Shanghang Zhang, Hao Chen, Zhe Li, and Kai Zhu. AoE: Always-on egocentric human video collection for embodied AI.arXiv preprintarXiv:2602.23893, 2026

  61. [61]

    EgoVLA: Learning vision-language-action models from egocentric human videos.arXivpreprint arXiv:2507.12440, 2025

    RuihanYang,QinxiYu,YechengWu,RuiYan,BoruiLi,An-ChiehCheng,XueyanZou,YunhaoFang,XuxinCheng,Ri- Zhao Qiu, Hongxu Yin, Sifei Liu, Song Han, Yao Lu, and Xiaolong Wang. EgoVLA: Learning vision-language-action models from egocentric human videos.arXivpreprint arXiv:2507.12440, 2025

  62. [62]

    WM-DAgger: Enabling efficient data aggregation for imitation learning with world models.arXivpreprint arXiv:2604.11351, 2026

    Anlan Yu, Zaishu Chen, Peili Song, Zhiqing Hong, Haotian Wang, Desheng Zhang, Tian He, Yi Ding, and Daqing Zhang. WM-DAgger: Enabling efficient data aggregation for imitation learning with world models.arXivpreprint arXiv:2604.11351, 2026

  63. [63]

    CLAP: Contrastive latent action pretraining for learning vision-language-action models from human videos.arXivpreprint arXiv:2601.04061, 2026

    Chao Zhang, Jun Wang, Zixuan Gao, Yixuan Su, Tao Dai, Chao Zhou, Jiajun Lu, and Yansong Tang. CLAP: Contrastive latent action pretraining for learning vision-language-action models from human videos.arXivpreprint arXiv:2601.04061, 2026

  64. [64]

    UniDex: A robot foundation suite for universal dexterous hand control from egocentric human videos.arXiv preprintarXiv:2603.22264, 2026

    Gu Zhang, Qicheng Xu, Haozhe Zhang, Jianhan Ma, Long He, Yiming Bao, Zeyu Ping, Zhecheng Yuan, Chenhao Lu, Chengbo Yuan, Tianhai Liang, Xiaoyu Tian, Maanping Shao, Feihong Zhang, Mingyu Ding, Yang Gao, Hao Zhao, Hang Zhao, and Huazhe Xu. UniDex: A robot foundation suite for universal dexterous hand control from egocentric human videos.arXiv preprintarXiv:...

  65. [65]

    HaWoR: World-space hand motion reconstruction from egocentric videos

    Jinglei Zhang, Jiankang Deng, Chao Ma, and Rolandos Alexandros Potamias. HaWoR: World-space hand motion reconstruction from egocentric videos. InProceedings of the IEEE/CVF Conference on Computer Vision and PatternRecognition(CVPR), pages 1805–1815, 2025. 14

  66. [66]

    EasyMimic: A low-cost framework for robot imitation learning from human videos.arXiv preprintarXiv:2602.11464, 2026

    Tao Zhang, Song Xia, Ye Wang, and Qin Jin. EasyMimic: A low-cost framework for robot imitation learning from human videos.arXiv preprintarXiv:2602.11464, 2026

  67. [67]

    Rewind-IL:Onlinefailuredetection and state respawning for imitation learning.arXivpreprintarXiv:2604.16683, 2026

    GehanZheng,SanjaySeenivasan,MatthewJohnson-Roberson,andWeimingZhi. Rewind-IL:Onlinefailuredetection and state respawning for imitation learning.arXivpreprintarXiv:2604.16683, 2026

  68. [68]

    EgoScale: Scaling dexterous manipulation with diverse egocentric human data.arXivpreprintarXiv:2602.16710, 2026

    Ruijie Zheng, Dantong Niu, Yuqi Xie, Jing Wang, Mengda Xu, Yunfan Jiang, Fernando Castañeda, Fengyuan Hu, You Liang Tan, Letian Fu, Trevor Darrell, Furong Huang, Yuke Zhu, Danfei Xu, and Linxi Fan. EgoScale: Scaling dexterous manipulation with diverse egocentric human data.arXivpreprintarXiv:2602.16710, 2026

  69. [69]

    From noise to intent: Anchoring generative VLA policies with residual bridges.arXiv preprint arXiv:2604.21391, 2026

    Yiming Zhong, Yaoyu He, Zemin Yang, Pengfei Tian, Yifan Huang, Qingqiu Huang, Xinge Zhu, and Yuexin Ma. From noise to intent: Anchoring generative VLA policies with residual bridges.arXiv preprint arXiv:2604.21391, 2026

  70. [70]

    Zhu, Pranav Kuppili, Ryan Punamiya, Patcharapong Aphiwetsa, Dhruv Patel, Simar Kareer, Sehoon Ha, and Danfei Xu

    Lawrence Y. Zhu, Pranav Kuppili, Ryan Punamiya, Patcharapong Aphiwetsa, Dhruv Patel, Simar Kareer, Sehoon Ha, and Danfei Xu. EMMA: Scaling mobile manipulation via egocentric human data.IEEERoboticsandAutomation Letters, 2025

  71. [71]

    ActiveGlasses: Learning manipulation with active vision from ego-centric human demonstration.arXivpreprintarXiv:2604.08534, 2026

    Yifei Zou, Chenguang Shi, Wenhao Yu, Haoran Xue, Jiaqi Lv, Yuzhe Pan, Cheng Wen, and Cewu Lu. ActiveGlasses: Learning manipulation with active vision from ego-centric human demonstration.arXivpreprintarXiv:2604.08534, 2026. 15 A Data Sources and Budgets All episode counts in the experiments are per task and source specific. They indicate how many episodes...