Pith. sign in

REVIEW 4 major objections 5 minor 64 references

A robotic policy trained on human egocentric video performs active perception as act-sense-act, with 83.3% mean success on hidden-target manipulation tasks.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 04:30 UTC pith:SRFHXFSC

load-bearing objection Big margins on real tasks, but the cognitive head's information-gain story is a framing, not a measurement, and the evidence needs error bars. the 4 major comments →

arxiv 2602.04600 v2 pith:SRFHXFSC submitted 2026-02-04 cs.RO

Act, Sense, Act: Learning Active Perception from Large-Scale Egocentric Human Data

Jialiang Li , Yi Qiao , Yunhan Guo , Changwen Chen , Wenzhao Lian This is my paper
classification cs.RO
keywords active perceptionnon-Markovian decision processegocentric human videovision-language-actioninformation gaincognitive auxiliary headdual-track memorymanipulation learning
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper tries to establish that active perception — the ability of a robot to deliberately move, look, and manipulate in order to resolve uncertainty about where task-relevant objects are — can be learned from large-scale human egocentric video, not just from robot demonstrations. It formalizes active perception as a history-dependent decision process in which actions are chosen partly for the information they will reveal, and in which different perceptual outcomes branch into different subsequent actions. The proposed framework, CoMe-VLA, couples a cognitive head that detects when a sub-task's uncertainty is resolved with a dual-track memory that keeps visual and proprioceptive history, and is trained in three stages on 800k human samples and 100k robot samples per task. The central empirical claim is 83.3% mean success across five tasks in which targets start hidden or far, while the strongest comparison policy reaches at most 16.7%; ablations attribute much of this gap to the cognitive decomposition and memory. If true, the result means human 'act, sense, act' behavior transfers to robots and lowers the amount of robot-specific teleoperation data needed.

Core claim

The paper's central claim is that a single policy can learn to treat perception as an intentional act — scanning, opening drawers, uncovering bowls, and bringing objects closer are generated to resolve uncertainty, and a learned binary cognitive label decides when to switch from exploration to exploitation. This is formalized as a non-Markovian decision process with information gain and decision branching (Eqs. 1–2); the framework approximates both with a cognitive auxiliary head (a proxy for information gain) and a dual-track memory that pools visual and proprioceptive history in an egocentric action space aligning human and robot head, wrist, and gripper commands. On a wheeled humanoid acr

What carries the argument

The load-bearing mechanism is the cognitive auxiliary head: a binary classifier on a dedicated token that predicts whether the current sub-task's uncertainty is resolved, supervised by manually labeling the final 90 robot frames or 30 human frames of each sub-task as 'resolved,' and deployed with a threshold of 0.7 sustained over three consecutive frames to trigger a switch to the next sub-task instruction. Around this sits a dual-track memory: a temporal window of visual frames (current plus five past frames at 1-second spacing) and a transformer-encoded proprioceptive history, both attended by a flow-matching action decoder that outputs a 29-dimensional chunk of head, bimanual end-effector

Load-bearing premise

The cognitive head is trained on a manual rule that the last 90 robot frames (or 30 human frames) of each sub-task count as 'information resolved,' and this binary label is assumed to faithfully stand in for the mutual-information gain in the formulation, with the deployment threshold of 0.7 over three consecutive frames assumed to generalize to new episodes.

What would settle it

Concrete test: in episodes where the target's true location distribution is known (e.g., left/right randomization), compute or approximate the actual mutual information gain at each timestep and compare it with when the cognitive label fires; if the label fires before uncertainty is actually resolved, or if varying the threshold of 0.7 and the three-frame persistence rule collapses success rates, the cognitive mechanism's role is not established. A second test would train a variant with no manual sub-task boundary labels, using only action-consistency-based switching, and see whether the 83.3%

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • With 800k human samples and only 100k robot samples per task, the policy reaches 83.3% mean success; dropping human data to 400k lowers success to 72.0%, and training on robot data alone leaves the robot-trained variant at 42.7% — so human egocentric pretraining, not robot data volume, is what drives active-perception performance.
  • Removing the cognitive decomposition drops mean success from 83.3% to 22.7%, and removing cognitive pretraining (stage 1) drops it to 52.0%; the cognitive label is not a minor add-on but central to long-horizon execution.
  • Removing memory drops mean success to 40.7%, and replacing the dual-track memory with a single fused track, visual-only, or proprio-only variants all degrade success, supporting the claim that active perception is genuinely history-dependent.
  • In a 10-minute adversarial test, when the target disappears the robot resumes scanning, and when it reappears the robot re-acquires and holds gaze — evidence that the exploration–exploitation switch is continuous rather than a hard-coded script.
  • The taxonomy of information discovery via viewpoint, information discovery via manipulation, and information enrichment gives a systematic way to design and evaluate active-perception tasks across different ways actions resolve uncertainty.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Extension: the binary completion label is a manual proxy for information gain; a follow-up could train the cognitive head on estimated information gain or temporal-difference task-progress signals and test whether success is less sensitive to the threshold rules.
  • Extension: the alignment recipe — episode-relative egocentric frame, gripper width from fingertip distances, and chassis/head decomposition — is demonstrated on one wheeled humanoid; whether it transplants to other embodiments is open.
  • Extension: comparison policies in the paper train on robot data alone, so the reported gap likely bundles human-data scale with architecture choices; giving the same human pretraining to the baselines would isolate the cognitive and memory contribution.
  • Extension: the formulation defines information gain as the driving quantity, but deployment uses a binary switch; estimating that quantity online could enable continuous exploration bonuses beyond sub-task switching.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper formalizes active perception as a non-Markovian decision process driven by information gain and decision branching (Section III-A, Eqs. (1)-(2)), and proposes CoMe-VLA, a vision-language-action model with a cognitive auxiliary head for sub-task transitions and a dual-track memory system. Human egocentric data (CaptainCook4D and Ego-Exo4D) and robot teleoperation data are aligned in a body-centric egocentric action space, and the model is trained in three stages. On five real-robot active-perception tasks, CoMe-VLA reports a mean success rate of 83.3%, substantially above baselines (best baseline 16.7%), with ablations attributing large gains to the cognitive switching mechanism and to the dual-track memory.

Significance. If the reported results hold, the paper would provide a practical demonstration that large-scale human egocentric video can transfer active-perception priors to a real robot and reduce the required amount of robot-specific demonstration data. The task taxonomy (viewpoint vs. manipulation discovery, information enrichment) is useful, and the real-robot evaluation with a blind teleoperation protocol is a strength. The paper also includes valuable ablations on data composition, memory architecture, and memory length. However, the central cognitive mechanism — the learned sub-task transition label — is the main risk: it is defined as a temporal completion heuristic rather than a validated information-gain proxy, and the key ablation shows that this mechanism carries much of the reported performance. The statistical reporting is also too thin to fully support the quantitative claims. These issues are addressable with additional experiments, so the contribution is potentially significant but not yet fully established.

major comments (4)
  1. [Section V-A, Eq. (4); Supplementary III-B.1 and IV-B; Table V] The central claim that CoMe-VLA learns information-gain-driven transitions is not established. The cognitive label c_t is defined as the final 90 robot frames / 30 human frames of each manually segmented sub-task (Supplementary III-B.1), which is a temporal completion annotation, not a measure of uncertainty reduction or mutual information. Eq. (4) merely asserts an unspecified mapping f. Deployment uses a fixed threshold tau=0.7 and a 3-frame hysteresis (Supplementary IV-B), and Table V shows that removing this mechanism drops mean SR from 83.3% to 22.7%. Please provide (i) an oracle-switching upper bound, (ii) threshold and hysteresis sensitivity analysis, and (iii) evidence that predicted transitions coincide with actual perceptual disambiguation moments (e.g., target entering field of view) rather than a generic time-to-go signal.
  2. [Supplementary V-A.1; Table V] The 'w/o Cog' ablation is implemented by replacing sub-task instructions with the full task instruction and disabling the task-switching mechanism during deployment. This appears to be an inference-time change on a model trained with sub-task-conditioned prompts, not a model trained without the cognitive objective. Full-instruction inputs may be out of distribution, so the large drop may reflect distribution shift rather than the value of cognitive decomposition. Please retrain the no-cognition condition from scratch, or compare instead against oracle sub-task transitions and a retrained full-instruction policy.
  3. [Tables I, V, VI, VII] All success counts are single runs of 30 trials per configuration; no error bars, confidence intervals, repeated seeds, or significance tests are reported. For several comparisons (e.g., 18/30 vs 21/30 in Table I, or memory-length rows in Table VI), the margins are small relative to binomial sampling noise. Please report binomial confidence intervals or multiple random seeds, and provide a statistical test for the headline comparisons (e.g., CoMe-VLA vs. baselines, w/o Cog vs. Ours).
  4. [Section III-A, Eqs. (1)-(2)] The formalization of information gain is problematic as stated. With A_t = pi(h_t, o_t), if the policy is deterministic then I(o_{t+1}; A_t | h_t, o_t) is zero; more generally, this mutual information does not measure how an action reduces uncertainty about a task-relevant hidden state. The standard active-perception quantity would be something like I(theta; o_{t+1} | h_t, o_t, a_t) for a hidden state theta. Please clarify the stochastic semantics of A_t and explain how Eq. (1) connects to the cognitive loss in Eq. (5) and Eq. (7); as written, the formalization is not used by the method and may be internally inconsistent.
minor comments (5)
  1. [Supplementary II-A] Typo: 'CaptionCook4D' should be 'CaptainCook4D' in the subsection heading and first sentence.
  2. [Section V-A, 'Implicit Data-Driven Decisions'] The text says decision branching is learned 'implicitly' rather than as discrete choices, but the cognitive head explicitly switches sub-task textual instructions at inference. This terminology is confusing; please reconcile or reword.
  3. [Section IV-C] The unified body-centric frame is a reasonable design choice, but no quantitative analysis is provided for how much domain shift the episode-base transformation removes. A small visualization or metric would strengthen the claim of cross-embodiment alignment.
  4. [Supplementary IV-B] The choice of threshold tau=0.7 and the 3-consecutive-frame requirement is not justified apart from being 'empirically' chosen. If the threshold sensitivity study is added as suggested above, please also report the selection procedure.
  5. [Section VI-B and Supplementary V-B] The paper does not compare quantitatively with the recent active-perception systems ViA, ActiveUMI, and EgoMI, citing different hardware and evaluation settings. This is a reasonable scope decision, but the Discussion should state it more prominently in the main text rather than only the supplementary.

Circularity Check

1 steps flagged

Cognitive 'information-gain' label is the manual final-frame annotation relabeled; the task-success evaluation is otherwise external and independent.

specific steps
  1. fitted input called prediction [Eq. (4) in Sec. V-A; Supplementary III-B.1; deployment threshold in Supplementary IV-B]
    "Formally, the cognitive label can be interpreted as a low-dimensional proxy of information gain: c_t ≈ f(I(o_{t+1}; A_t | h_t, o_t)). ... For robot trajectories, the final 90 frames of each sub-task are designated as 'sub-task completion' and assigned a cognitive label c_t = 1. For human data, this completion window is set to the final 30 frames."

    The cognitive signal that is presented as evidence for the NMDP/information-gain formalization is defined by manual temporal annotation—the final 90 robot / 30 human frames of each segmented sub-task—not by any computation of Eq. (1). This same label supervises the cognitive head in Stages 1–3 and, through the τ=0.7 over three consecutive frames rule, triggers the sub-task switches that the w/o-Cog ablation credits with 83.3% vs 22.7% SR. The learned 'information-gain cognition' thus reduces, by construction, to a supervised completion detector trained to output the manual boundary annotation, and that annotation is then reported as evidence of information-gain-driven active perception.

full rationale

The paper's central empirical result—83.3% mean SR against 12.7–16.7% for general-purpose VLAs and imitation baselines, plus the data-composition and memory ablations—is measured on external task success and does not reduce to any fitted output. There is no load-bearing self-citation chain and no imported uniqueness theorem; the only author overlap in the references (Jialong Li in [33]) is not used to justify a central premise. The one definitional short-circuit is localized to the cognitive label: Eq. (4) calls c_t a learned proxy for Eq. (1)'s mutual information, while Supplementary III-B.1 defines c_t as the final 90/30 frames of manually segmented sub-tasks. Since the same annotation supervises the cognitive head and triggers deployment switches, the cognitive component's claim to learn information-gain-driven cognition is partly circular—it is better described as supervised sub-task-completion detection. This affects the interpretation of the w/o-Cog ablation, but does not by itself undermine the standalone success-rate comparison, so the overall circularity is partial rather than global.

Axiom & Free-Parameter Ledger

6 free parameters · 5 axioms · 0 invented entities

The paper's load-bearing assumptions are the transferability of human egocentric priors and the validity of the hand-annotated cognitive labels. The latter is an ad hoc construction: cognitive supervision is generated by labeling the last 90/30 frames of each sub-task as completion, and then a thresholded version of this learned label is used to switch sub-tasks at deployment. The framework contains no measure that verifies Eq. (4); many controller/threshold parameters are hand-tuned. No new physical entities are introduced.

free parameters (6)
  • Cognitive label threshold and hysteresis = τ=0.70; 3 consecutive timesteps
    Deployment threshold for subtask switching; hand-tuned, no sensitivity analysis. (Supplementary IV-B)
  • Subtask-completion annotation window = 90 frames (robot), 30 frames (human)
    Frames labeled c_t=1; arbitrary heuristic for 'information gain' proxy. (Supplementary III-B)
  • Memory window length = 5 history frames at 1 s intervals
    Ablated; changing it moves SR from 44.6% to 83.3%. (Section V-A / Table VI)
  • Chassis proportional controller gains and deadzones = k_v=0.2, k_ω=0.5, v_max=0.15 m/s, ω_max=0.3 rad/s, etc.
    Hand-picked, kept fixed across tasks. (Supplementary IV-D)
  • Chassis/head decomposition threshold = |ψ|>0.7 rad for chassis takeover
    Heuristic to map composite head pose to control commands. (Supplementary II-C)
  • Loss weights and focal-loss parameters = λ_vr=1.5, λ_vp=1.0, λ_er=1.0, λ_ep=1.0, λ_g=1.0, α_t=0.25, γ=2.0
    Manually chosen; balance action components and cognitive loss. (Table IV)
axioms (5)
  • standard math Non-Markovian decision process framework of Bacchus et al. and Bellman
    Used to frame active perception; accepted from prior literature, not proved here. (Section III-A)
  • domain assumption Mutual information in Eq. (1) is the correct objective for active perception
    No derivation that maximizing this information gain leads to task success. (Section III-A)
  • domain assumption Egocentric human video contains transferable eye-hand coordination and exploration priors
    Core premise of the transfer; supported only by the included ablation, not by an external theoretical argument. (Section IV-A)
  • ad hoc to paper Episode base frame / body-centric local frame removes embodiment and environment shifts
    Normalization chosen by the authors; no theoretical guarantee of structural isomorphism. (Section IV-C, Eq. 3)
  • ad hoc to paper Final N frames of a manually segmented sub-task are a valid proxy for information gain
    Feeds the cognitive head; no independent validation against actual information gain. (Supplementary III-B)

pith-pipeline@v1.3.0-alltime-deepseek · 20087 in / 11865 out tokens · 120635 ms · 2026-08-03T04:30:08.129649+00:00 · methodology

0 comments
read the original abstract

Achieving generalizable manipulation in unconstrained environments requires the robot to proactively resolve information uncertainty, i.e., the capability of active perception. However, existing methods are often confined in limited types of sensing behaviors, restricting their applicability to complex environments. In this work, we formalize active perception as a history-dependent perception-action loop driven by information-seeking action and decision branching, providing a structured categorization of visual active perception paradigms. Building on this perspective, we introduce CoMe-VLA, a cognitive and memory-aware vision-language-action (VLA) framework that leverages large-scale human egocentric data to learn versatile exploration and manipulation priors. Our framework integrates a cognitive auxiliary head for autonomous sub-task transitions and a dual-track memory system to maintain consistent self and environmental awareness by fusing proprioceptive and visual temporal contexts. By aligning human and robot hand-eye coordination behaviors in a unified egocentric action space, we train the model progressively in three stages. Extensive experiments on a wheel-based humanoid have demonstrated strong robustness and adaptability of our proposed method across diverse long-horizon tasks spanning multiple active perception scenarios.

Figures

Figures reproduced from arXiv: 2602.04600 by Changwen Chen, Jialiang Li, Wenzhao Lian, Yi Qiao, Yunhan Guo.

Figure 1
Figure 1. Figure 1: By unifying large-scale egocentric human data and robot data within a shared action space, a wheel-based humanoid develops robust [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Comparison between passive perception and active perception. [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: VR-based immersive teleoperation system. The operator performs perception-driven control based solely on egocentric video streaming from the robot. are mapped to the robot’s 2-DoF head actuation in real￾time, while hand controllers govern bimanual manipulation and chassis movement. Crucially, as shown in [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: CoMe-VLA Overview. CoMe-VLA integrates a pre-trained VLM (Qwen3-VL-2B [2]) with a transformer-based proprioceptive memory encoder to construct temporal visual-semantic and proprioceptive contexts, which are fed into a flow-matching action decoder to generate a 29-D action chunk. The VLM also outputs a cognitive latent token for the cognitive auxiliary head, which predicts a binary label for autonomous task… view at source ↗
Figure 5
Figure 5. Figure 5: Evaluated Tasks. All tasks are designed with uncertain initial conditions, where the locations of the target or task-critical objects are unknown to the model before execution, and can vary across multiple configurations. Croissant Search: Find a croissant on the table but initially out of view. Can Disposal: Throw cans into a dustbin initially out of view but in the room. Bottle Retrieval: Open the cabine… view at source ↗
Figure 6
Figure 6. Figure 6: Ablation on Cognitive Grounding. 1) Ablation on Cognition: As illustrated in [PITH_FULL_IMAGE:figures/full_fig_p007_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Ablation on Memory Architecture and Temporal Horizon Design. first allows the model to utilize high-level awareness to refine low-level motor commands. 2) Ablation on Memory: As shown in [PITH_FULL_IMAGE:figures/full_fig_p008_7.png] view at source ↗
Figure 9
Figure 9. Figure 9: Interface of the VR-based teleoperation system. [PITH_FULL_IMAGE:figures/full_fig_p012_9.png] view at source ↗
Figure 8
Figure 8. Figure 8: The Corenetic Monte02 robot platform. B. Teleoperation Based on Vuer [46], we develop a VR-based teleoperation system, as illustrated in [PITH_FULL_IMAGE:figures/full_fig_p012_8.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

64 extracted references · 13 linked inside Pith

  1. [1]

    Structured solution methods for non-markovian decision processes

    Fahiem Bacchus, Craig Boutilier, and Adam Grove. Structured solution methods for non-markovian decision processes. AAAI’97/IAAI’97, page 112–117. AAAI Press, 1997. ISBN 0262510952

    1997

  2. [2]

    Qwen3-vl technical report

    Shuai Bai, Yuxuan Cai, Ruizhe Chen, Keqin Chen, Xionghui Chen, Zesen Cheng, Lianghao Deng, Wei Ding, Chang Gao, Chunjiang Ge, Wenbin Ge, Zhifang Guo, Qidong Huang, Jie Huang, Fei Huang, Binyuan Hui, Shu- tong Jiang, Zhaohai Li, Mingsheng Li, Mei Li, Kaixin Li, Zicheng Lin, Junyang Lin, Xuejing Liu, Jiawei Liu, Chenglong Liu, Yang Liu, Dayiheng Liu, Shixua...

    Pith/arXiv arXiv 2025

  3. [3]

    A markovian decision process.Journal of Mathematics and Mechanics, pages 679–684, 1957

    Richard Bellman. A markovian decision process.Journal of Mathematics and Mechanics, pages 679–684, 1957

    1957

  4. [4]

    Towards generalizable zero-shot manipulation via translating human interaction plans

    Homanga Bharadhwaj, Abhinav Gupta, Vikash Kumar, and Shubham Tulsiani. Towards generalizable zero-shot manipulation via translating human interaction plans. In 2024 IEEE International Conference on Robotics and Automation (ICRA), pages 6904–6911. IEEE, 2024

    2024

  5. [5]

    arXiv preprint arXiv:2410.24164, 2024

    Kevin Black, Noah Brown, Danny Driess, Adnan Es- mail, Michael Equi, Chelsea Finn, Niccolo Fusai, Lachy Groom, Karol Hausman, Brian Ichter, et al.π 0: A vision- language-action flow model for general robot control. arXiv preprint arXiv:2410.24164, 2024

    Pith/arXiv arXiv 2024

  6. [6]

    In9th Annual Conference on Robot Learning, 2025

    Kevin Black, Noah Brown, James Darpinian, Karan Dhabalia, Danny Driess, Adnan Esmail, Michael Robert Equi, Chelsea Finn, Niccolo Fusai, Manuel Y Galliker, et al.π 0.5: a vision-language-action model with open- world generalization. In9th Annual Conference on Robot Learning, 2025

    2025

  7. [7]

    Gennbv: Generalizable next-best-view policy for active 3d reconstruction

    Xiao Chen, Quanyi Li, Tai Wang, Tianfan Xue, and Jiangmiao Pang. Gennbv: Generalizable next-best-view policy for active 3d reconstruction. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16436–16445, 2024

    2024

  8. [8]

    Universal manipulation interface: In-the- wild robot teaching without in-the-wild robots

    Cheng Chi, Zhenjia Xu, Chuer Pan, Eric Cousineau, Benjamin Burchfiel, Siyuan Feng, Russ Tedrake, and Shuran Song. Universal manipulation interface: In-the- wild robot teaching without in-the-wild robots. Robotics: Science and Systems, 2024

    2024

  9. [9]

    Diffusion policy: Visuomotor policy learning via action diffusion.The International Journal of Robotics Research, 44(10-11):1684–1704, 2025

    Cheng Chi, Zhenjia Xu, Siyuan Feng, Eric Cousineau, Yilun Du, Benjamin Burchfiel, Russ Tedrake, and Shuran Song. Diffusion policy: Visuomotor policy learning via action diffusion.The International Journal of Robotics Research, 44(10-11):1684–1704, 2025

    2025

  10. [10]

    Look, focus, act: Efficient and robust robot learning via human gaze and foveated vision transform- ers.arXiv preprint arXiv:2507.15833, 2025

    Ian Chuang, Jinyu Zou, Andrew Lee, Dechen Gao, and Iman Soltani. Look, focus, act: Efficient and robust robot learning via human gaze and foveated vision transform- ers.arXiv preprint arXiv:2507.15833, 2025

    Pith/arXiv arXiv 2025

  11. [11]

    Corenetic. Monte02. URL https://www.corenetic.ai/ product

  12. [12]

    Ego-exo4d: Understanding skilled human activity from first-and third-person perspectives

    Kristen Grauman, Andrew Westbury, Lorenzo Torresani, Kris Kitani, Jitendra Malik, Triantafyllos Afouras, Kumar Ashutosh, Vijay Baiyya, Siddhant Bansal, Bikram Boote, et al. Ego-exo4d: Understanding skilled human activity from first-and third-person perspectives. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 193...

    2024

  13. [13]

    Macarons: Mapping and coverage antic- ipation with rgb online self-supervision

    Antoine Gu ´edon, Tom Monnier, Pascal Monasse, and Vincent Lepetit. Macarons: Mapping and coverage antic- ipation with rgb online self-supervision. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 940–951, 2023

    2023

  14. [14]

    2handedafforder: Learning precise actionable bimanual affordances from human videos

    Marvin Heidinger, Snehal Jauhri, Vignesh Prasad, and Georgia Chalvatzaki. 2handedafforder: Learning precise actionable bimanual affordances from human videos. arXiv preprint arXiv:2503.09320, 2025

    Pith/arXiv arXiv 2025

  15. [15]

    6d rotation representation for unconstrained head pose estimation

    Thorsten Hempel, Ahmed A Abdelrahman, and Ayoub Al-Hamadi. 6d rotation representation for unconstrained head pose estimation. In2022 IEEE International Con- ference on Image Processing (ICIP), pages 2496–2500. IEEE, 2022

    2022

  16. [16]

    Egodex: Learning dexterous manipulation from large-scale egocentric video.arXiv preprint arXiv:2505.11709, 2025

    Ryan Hoque, Peide Huang, David J Yoon, Mouli Siva- purapu, and Jian Zhang. Egodex: Learning dexterous manipulation from large-scale egocentric video.arXiv preprint arXiv:2505.11709, 2025

    Pith/arXiv arXiv 2025

  17. [17]

    Egomimic: Scaling imitation learning via egocentric video

    Simar Kareer, Dhruv Patel, Ryan Punamiya, Pranay Mathur, Shuo Cheng, Chen Wang, Judy Hoffman, and Danfei Xu. Egomimic: Scaling imitation learning via egocentric video. In2025 IEEE International Confer- ence on Robotics and Automation (ICRA), pages 13226– 13233. IEEE, 2025

    2025

  18. [18]

    Multi-task real-robot data with gaze attention for dual-arm fine manipulation

    Heecheol Kim, Yoshiyuki Ohmura, and Yasuo Ku- niyoshi. Multi-task real-robot data with gaze attention for dual-arm fine manipulation. In2024 IEEE/RSJ Inter- national Conference on Intelligent Robots and Systems (IROS), pages 8516–8523. IEEE, 2024

    2024

  19. [19]

    Openvla: An open-source vision-language-action model.arXiv preprint arXiv:2406.09246, 2024

    Moo Jin Kim, Karl Pertsch, Siddharth Karamcheti, Ted Xiao, Ashwin Balakrishna, Suraj Nair, Rafael Rafailov, Ethan Foster, Grace Lam, Pannag Sanketi, et al. Openvla: An open-source vision-language-action model.arXiv preprint arXiv:2406.09246, 2024

    Pith/arXiv arXiv 2024

  20. [20]

    Fine- tuning vision-language-action models: Optimizing speed and success.arXiv preprint arXiv:2502.19645, 2025

    Moo Jin Kim, Chelsea Finn, and Percy Liang. Fine- tuning vision-language-action models: Optimizing speed and success.arXiv preprint arXiv:2502.19645, 2025

    Pith/arXiv arXiv 2025

  21. [21]

    Aria gen 2 pilot dataset.arXiv preprint arXiv:2510.16134, 2025

    Chen Kong, James Fort, Aria Kang, Jonathan Wittmer, Simon Green, Tianwei Shen, Yipu Zhao, Cheng Peng, Gustavo Solaira, Andrew Berkovich, et al. Aria gen 2 pilot dataset.arXiv preprint arXiv:2510.16134, 2025

    arXiv 2025

  22. [22]

    Scalable vision-language-action model pretraining for robotic manipulation with real-life human activity videos.arXiv preprint arXiv:2510.21571, 2025

    Qixiu Li, Yu Deng, Yaobo Liang, Lin Luo, Lei Zhou, Chengtang Yao, Lingqi Zeng, Zhiyuan Feng, Huizhi Liang, Sicheng Xu, et al. Scalable vision-language-action model pretraining for robotic manipulation with real-life human activity videos.arXiv preprint arXiv:2510.21571, 2025

    arXiv 2025

  23. [23]

    Focal loss for dense object detection

    Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Doll ´ar. Focal loss for dense object detection. InProceedings of the IEEE International Conference on Computer Vision, pages 2980–2988, 2017

    2017

  24. [24]

    Hoi4d: A 4d egocentric dataset for category- level human-object interaction

    Yunze Liu, Yun Liu, Che Jiang, Kangbo Lyu, Weikang Wan, Hao Shen, Boqiang Liang, Zhoujie Fu, He Wang, and Li Yi. Hoi4d: A 4d egocentric dataset for category- level human-object interaction. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 21013–21022, 2022

    2022

  25. [25]

    Vip: Towards universal visual reward and representation via value-implicit pre-training

    Yecheng Jason Ma, Shagun Sodhani, Dinesh Jayaraman, Osbert Bastani, Vikash Kumar, and Amy Zhang. Vip: Towards universal visual reward and representation via value-implicit pre-training. InThe Eleventh International Conference on Learning Representations, 2022

    2022

  26. [26]

    Aria glasses,

    Meta. Aria glasses, . URL https://www.projectaria.com/

  27. [27]

    Quest 3,

    Meta. Quest 3, . URL https://www.meta.com/quest/ quest-3

  28. [28]

    Hololens

    Microsoft. Hololens. URL https://learn.microsoft.com/ zh-cn/hololens/

  29. [29]

    Captaincook4d: A dataset for understanding errors in procedural activities.Advances in Neural Information Processing Systems, 37:135626–135679, 2024

    Rohith Peddi, Shivvrat Arya, Bharath Challa, Likhitha Pallapothula, Akshay Vyas, Bhavya Gouripeddi, Qifan Zhang, Jikai Wang, Vasundhara Komaragiri, Eric Ragan, et al. Captaincook4d: A dataset for understanding errors in procedural activities.Advances in Neural Information Processing Systems, 37:135626–135679, 2024

    2024

  30. [30]

    A backpack full of skills: Egocentric video understanding with diverse task per- spectives

    Simone Alberto Peirone, Francesca Pistilli, Antonio Al- liegro, and Giuseppe Averta. A backpack full of skills: Egocentric video understanding with diverse task per- spectives. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18275–18285, 2024

    2024

  31. [31]

    Hd-epic: A highly-detailed egocentric video dataset

    Toby Perrett, Ahmad Darkhalil, Saptarshi Sinha, Omar Emara, Sam Pollard, Kranti Kumar Parida, Kaiting Liu, Prajwal Gatti, Siddhant Bansal, Kevin Flanagan, et al. Hd-epic: A highly-detailed egocentric video dataset. In Proceedings of the Computer Vision and Pattern Recog- nition Conference, pages 23901–23913, 2025

    2025

  32. [32]

    Egobridge: Domain adaptation for generalizable imitation from egocentric human data

    Ryan Punamiya, Dhruv Patel, Patcharapong Aphiwetsa, Pranav Kuppili, Lawrence Y Zhu, Simar Kareer, Judy Hoffman, and Danfei Xu. Egobridge: Domain adaptation for generalizable imitation from egocentric human data. InHuman to Robot: Workshop on Sensorizing, Modeling, and Learning from Humans, 2025

    2025

  33. [33]

    Humanoid policy˜ human policy.arXiv preprint arXiv:2503.13441, 2025

    Ri-Zhao Qiu, Shiqi Yang, Xuxin Cheng, Chaitanya Chawla, Jialong Li, Tairan He, Ge Yan, David J Yoon, Ryan Hoque, Lars Paulsen, et al. Humanoid policy˜ human policy.arXiv preprint arXiv:2503.13441, 2025

    arXiv 2025

  34. [34]

    Javier Romero, Dimitrios Tzionas, and Michael J. Black. Embodied hands: Modeling and capturing hands and bodies together.ACM Transactions on Graphics, (Proc. SIGGRAPH Asia), 36(6), November 2017

    2017

  35. [35]

    Gaze-guided task decomposition for imitation learning in robotic manipulation.arXiv preprint arXiv:2501.15071, 2025

    Ryo Takizawa, Yoshiyuki Ohmura, and Yasuo Kuniyoshi. Gaze-guided task decomposition for imitation learning in robotic manipulation.arXiv preprint arXiv:2501.15071, 2025

    Pith/arXiv arXiv 2025

  36. [36]

    Agrl: Affordance-guided reinforce- ment learning from human video

    Laura Tran-Dubois. Agrl: Affordance-guided reinforce- ment learning from human video. InHuman to Robot: Workshop on Sensorizing, Modeling, and Learning from Humans, 2025

    2025

  37. [37]

    Dexcap: Scalable and portable mocap data collection system for dexterous manipulation.arXiv preprint arXiv:2403.07788, 2024

    Chen Wang, Haochen Shi, Weizhuo Wang, Ruohan Zhang, Li Fei-Fei, and C Karen Liu. Dexcap: Scalable and portable mocap data collection system for dexterous manipulation.arXiv preprint arXiv:2403.07788, 2024

    Pith/arXiv arXiv 2024

  38. [38]

    Observe then act: Asynchronous active vision-action model for robotic manipulation.IEEE Robotics and Automation Letters, 2025

    Guokang Wang, Hang Li, Shuyuan Zhang, Di Guo, Yanhong Liu, and Huaping Liu. Observe then act: Asynchronous active vision-action model for robotic manipulation.IEEE Robotics and Automation Letters, 2025

    2025

  39. [39]

    Graspview: Active perception scoring and best-view optimization for robotic grasping in cluttered environments.arXiv preprint arXiv:2511.04199, 2025

    Shenglin Wang, Mingtong Dai, Jingxuan Su, Lingbo Liu, Chunjie Chen, Xinyu Wu, and Liang Lin. Graspview: Active perception scoring and best-view optimization for robotic grasping in cluttered environments.arXiv preprint arXiv:2511.04199, 2025

    arXiv 2025

  40. [40]

    Egonav: Egocentric scene-aware human trajectory pre- diction.arXiv preprint arXiv:2403.19026, 2024

    Weizhuo Wang, C Karen Liu, and Monroe Kennedy III. Egonav: Egocentric scene-aware human trajectory pre- diction.arXiv preprint arXiv:2403.19026, 2024

    Pith/arXiv arXiv 2024

  41. [41]

    Observer actor: Active vision imitation learning with sparse view gaussian splatting.arXiv preprint arXiv:2511.18140, 2025

    Yilong Wang, Cheng Qian, Ruomeng Fan, and Edward Johns. Observer actor: Active vision imitation learning with sparse view gaussian splatting.arXiv preprint arXiv:2511.18140, 2025

    arXiv 2025

  42. [42]

    Mean squared error: Love it or leave it? a new look at signal fidelity measures.IEEE signal processing magazine, 26(1):98–117, 2009

    Zhou Wang and Alan C Bovik. Mean squared error: Love it or leave it? a new look at signal fidelity measures.IEEE signal processing magazine, 26(1):98–117, 2009

    2009

  43. [43]

    Vision in action: Learning active perception from human demonstrations

    Haoyu Xiong, Xiaomeng Xu, Jimmy Wu, Yifan Hou, Jeannette Bohg, and Shuran Song. Vision in action: Learning active perception from human demonstrations. arXiv preprint arXiv:2506.15666, 2025

    Pith/arXiv arXiv 2025

  44. [44]

    Understanding and improving layer normalization.Advances in neural information process- ing systems, 32, 2019

    Jingjing Xu, Xu Sun, Zhiyuan Zhang, Guangxiang Zhao, and Junyang Lin. Understanding and improving layer normalization.Advances in neural information process- ing systems, 32, 2019

    2019

  45. [45]

    Dexumi: Using human hand as the universal manipulation in- terface for dexterous manipulation.arXiv preprint arXiv:2505.21864, 2025

    Mengda Xu, Han Zhang, Yifan Hou, Zhenjia Xu, Linxi Fan, Manuela Veloso, and Shuran Song. Dexumi: Using human hand as the universal manipulation in- terface for dexterous manipulation.arXiv preprint arXiv:2505.21864, 2025

    arXiv 2025

  46. [46]

    VUER: An event-driven, declarative visu- alization toolkit for genai and robotics, 2025

    Ge Yang. VUER: An event-driven, declarative visu- alization toolkit for genai and robotics, 2025. URL https://github.com/vuer-ai/vuer

    2025

  47. [47]

    Egomi: Learning active vision and whole-body manipulation from egocentric human demonstrations.arXiv preprint arXiv:2511.00153, 2025

    Justin Yu, Yide Shentu, Di Wu, Pieter Abbeel, Ken Goldberg, and Philipp Wu. Egomi: Learning active vision and whole-body manipulation from egocentric human demonstrations.arXiv preprint arXiv:2511.00153, 2025

    arXiv 2025

  48. [48]

    Activeumi: Robotic manipulation with active perception from robot-free human demonstrations.arXiv preprint arXiv:2510.01607, 2025

    Qiyuan Zeng, Chengmeng Li, Jude St John, Zhongyi Zhou, Junjie Wen, Guorui Feng, Yichen Zhu, and Yi Xu. Activeumi: Robotic manipulation with active perception from robot-free human demonstrations.arXiv preprint arXiv:2510.01607, 2025

    arXiv 2025

  49. [49]

    Learning fine-grained bimanual manipulation with low-cost hardware.arXiv preprint arXiv:2304.13705, 2023

    Tony Z Zhao, Vikash Kumar, Sergey Levine, and Chelsea Finn. Learning fine-grained bimanual manipulation with low-cost hardware.arXiv preprint arXiv:2304.13705, 2023

    Pith/arXiv arXiv 2023

  50. [50]

    Emma: Scaling mobile manipulation via egocentric human data.arXiv preprint arXiv:2509.04443, 2025

    Lawrence Y Zhu, Pranav Kuppili, Ryan Punamiya, Patcharapong Aphiwetsa, Dhruv Patel, Simar Kareer, Sehoon Ha, and Danfei Xu. Emma: Scaling mobile manipulation via egocentric human data.arXiv preprint arXiv:2509.04443, 2025

    arXiv 2025

  51. [51]

    Vision-based manipulation from single human video with open-world object graphs.arXiv preprint arXiv:2405.20321, 2024

    Yifeng Zhu, Arisrei Lim, Peter Stone, and Yuke Zhu. Vision-based manipulation from single human video with open-world object graphs.arXiv preprint arXiv:2405.20321, 2024. Supplementary Material I. ROBOTPLATFORM A. Hardware We conduct all experiments on the Corenetic Monte02 wheel-based humanoid robot [11], as illustrated in Figure 8. The robot features 23...

    Pith/arXiv arXiv 2024

  52. [52]

    CaptionCook4D:CaptionCook4D is captured using a HoloLens device [28] and provides high-quality egocentric visual observations together with accurate head and hand actions and manipulation annotations. The dataset primarily consists of structured cooking procedures with relatively fixed execution flows, while still allowing variations in low-level actions ...

  53. [53]

    We target the Cooking, Bike Repair, and Covid subsets, as these scenarios inherently require active visual search in cluttered scenes

    Ego-Exo4D:In contrast to the structured environment of CaptainCook4D, EgoExo4D offers data captured via Aria glasses [26] in diverse, unconstrained real-world settings. We target the Cooking, Bike Repair, and Covid subsets, as these scenarios inherently require active visual search in cluttered scenes. However, a major challenge with Ego-Exo4D is that the...

  54. [54]

    Head and Hand Poses:All poses are expressed in a unified body-centric local frameL, defined as a right-handed coordinate system shared across human and robot embodi- ments. The base frame is initialized using the head pose in the first frame of each episode, and all subsequent observations and actions are represented relative to this reference via rigid- ...

  55. [55]

    This value is then linearly normalized to the range[0,1]to obtain a unified representation of hand opening

    Gripper Actions:Human data provides hand joint po- sitions, from which we compute a scalar gripper aperture by averaging the Euclidean distances between the thumb tip and the tips of the remaining fingers. This value is then linearly normalized to the range[0,1]to obtain a unified representation of hand opening. For robot data, the raw gripper width is si...

    2048

  56. [56]

    sub-task completion

    Data Annotation and Preprocessing:To provide explicit supervision for the cognitive component of our framework, we manually annotate sub-task boundaries in each demonstration. For robot trajectories, the final 90 frames of each sub-task are designated as “sub-task completion” and assigned a cognitive labelc t = 1. For human data, this completion window is...

  57. [57]

    Training Details:We mix collected robot data from all tasks to train a unified CoMe-VLA model. Across all training stages, following the official guidance of Qwen3-VL, we only update the vision projector (visual merger), while keeping the language backbone, vision encoder, and most transformer blocks frozen to preserve pretrained visual and semantic knowl...

  58. [58]

    To minimize rotation time, the controller is bi-directional: it selects the orientation (forward or backward) that requires the smallest angular displacement

    Heading Alignment: The robot rotates in place to align its longitudinal axis with the bearing line to the target. To minimize rotation time, the controller is bi-directional: it selects the orientation (forward or backward) that requires the smallest angular displacement. The angular velocity is computed asω=k ωeψ

  59. [59]

    Linear Approach: Once aligned, the robot translates towards the target with linear velocityv=k ved, while maintaining zero angular velocity

  60. [60]

    All velocity commands are clamped within predefined safety bounds, i.e.,v∈[v min, vmax]andω∈[ω min, ωmax]

    Final Orientation: Upon reaching the target position, the robot performs a final rotation to match the target yaw: ω=k ωeθ. All velocity commands are clamped within predefined safety bounds, i.e.,v∈[v min, vmax]andω∈[ω min, ωmax]. To suppress oscillations near the target, dead-zone thresholds ϵp andϵ θ are applied to translational and rotational errors, r...

  61. [61]

    VLA-based baselines:For VLA-based baselines, which support text conditioning, we train each baseline with a single model on all collected robot data across tasks. Training follows our sub-task decomposition setting, where each sub-task is provided as a textual prompt during training and inference, enabling the models to condition their action generation o...

  62. [62]

    These models are thus required to directly learn end-to-end execution of the entire task sequence without any sub-task switching mechanism

    Imitation learning baselines:For imitation learning baselines, which are originally designed for single-task settings without text conditioning, we train a separate model for each task using full long-horizon trajectories. These models are thus required to directly learn end-to-end execution of the entire task sequence without any sub-task switching mecha...

  63. [63]

    Ablation on Cognition:We conduct ablations on the cognition-based task decomposition strategy by replacing sub- task instructions with the full task instruction and disabling the task-switching mechanism during deployment. In addition, we ablate the cognitive state pretraining (Stage 1) by training the model using only Stage 2 and Stage 3, i.e., full-mode...

  64. [64]

    Below we describe the implementation details of the ablations on the memory architecture

    Ablation on Memory:We detail the performance of different memory length choices in Table VI. Below we describe the implementation details of the ablations on the memory architecture. •No memory. We feed only the current visual frame and the current robot state into the model during training and deployment, without incorporating any historical information....