Pith. sign in

REVIEW 4 major objections 7 minor 29 references

A kinematic intermediate representation lets robots open drawers and doors from half the usual demonstration data.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-31 13:14 UTC pith:NL4OL6XO

load-bearing objection Clean inductive-bias story with real ablations; the half-data claim holds against Seer, but several other baselines look mis-tuned and should not carry the headline. the 4 major comments →

arxiv 2607.24493 v1 pith:NL4OL6XO submitted 2026-07-27 cs.RO

KAI: A Kinematic-Aware Interface for Data-Efficient Articulated Object Manipulation

Yaping Li , Zhaxizhuoma , Qiaojun Yu , Jia Zeng , Dahua Lin , Jiangmiao Pang This is my paper
classification cs.RO
keywords articulated object manipulationdata-efficient learningkinematic priorsimitation learninghuman-video co-trainingintermediate representationrobot learning
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Articulated objects move under hard geometric constraints—hinges rotate, drawers slide—yet end-to-end robot policies usually have to rediscover those constraints from raw pixels and actions. This paper argues that making those constraints an explicit intermediate target, called KAI, gives the learner a strong inductive bias and cuts the need for costly robot demonstrations. KAI predicts a small set of keypoints on the moving part plus their short-horizon future displacements, regularized so the points stay rigid and move like a prismatic or revolute joint. Placed between perception and action in a phased architecture, it also accepts supervision from human interaction videos that have no robot actions. In simulation the approach reaches about 83% average success across six open/close tasks while matching or beating strong baselines on half the data; in the real world, policies trained in one clean scene stay effective under background and object clutter, and video co-training pushes average success above 70%.

Core claim

Embedding an explicit kinematic interface—keypoints on the moving part plus regularized future displacements—into a Perception–KAI–Action policy yields high success on articulated manipulation with substantially fewer robot demonstrations, and the same action-agnostic interface supports co-training on human videos for cluttered real scenes.

What carries the argument

KAI (Kinematic-Aware Articulation Interface): K interaction keypoints on the moving part with N-step future locations and displacements, trained under isometric structural loss plus joint-type motion losses (coherent translation for prismatic joints; fixed-axis anchors for revolute joints), then fed as intermediate tokens to an action decoder.

Load-bearing premise

The geometric losses assume the joint type is known and that a few fixed keypoints plus pure translation or single-axis rotation adequately describe how the part moves.

What would settle it

Train and evaluate the same policy on multi-DoF or unknown-joint articulated objects without supplying joint type, or ablate the joint-type motion losses: if sample-efficiency and success collapse to baseline levels, the claimed inductive bias does not hold as stated.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Policies for common single-joint household articulations can be trained with roughly half the robot demonstrations once KAI-style kinematic targets are supervised.
  • Human interaction video datasets become usable training signal for robot policies without needing action labels, via the action-agnostic KAI head.
  • Single-scene real-world collection can transfer to cluttered backgrounds and distractors more reliably when kinematics are predicted explicitly rather than left implicit in pixels-to-actions maps.
  • Phased causal attention (structure before control) is a practical recipe for mixing heterogeneous robot and human data in one model.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If joint type must be known a priori, automatic joint-type classification or mixture losses would be the natural next bottleneck before scaling beyond doors, drawers, and laptops.
  • The same keypoint-plus-displacement interface could serve as a shared latent for cross-embodiment transfer, not only human-to-robot video co-training.
  • Short-horizon KAI chunks may compose into longer sequential articulation plans if a higher-level controller sequences part-level goals.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 7 minor

Summary. The paper introduces KAI (Kinematic-Aware Articulation Interface), a structured intermediate representation for articulated-object manipulation: K interaction keypoints on the moving part with predicted N-step future locations and displacements, regularized by an isometric structural loss (Eq. 2) and joint-type-specific motion losses for prismatic (Eq. 3) and revolute (Eq. 4) joints. A phased Perception–KAI–Action transformer with unidirectional attention (KAI tokens cannot attend to action tokens) makes the representation action-agnostic, enabling co-training with human interaction videos (HOI4D) supervised from part bounding boxes. On six Isaac Sim tasks with held-out seeds, backgrounds, and lighting, the method reports 82.9% average success (84.6% with video co-training) vs. the best baseline Seer at 73.4%, and claims parity with baselines at half the demonstration data. Ablations (Tables 2–3) attribute most of the gain to KAI itself, with additional contributions from the geometric loss and causal attention masking. Real-world experiments on a Franka arm (3 tasks, 15 trials per condition) show robustness to background and object distractors, with video co-training giving +6.6 to +13.4 pp.

Significance. If the results hold, this is a useful contribution to data-efficient articulated-object manipulation: the KAI representation is simple, the phased unidirectional-attention design cleanly enables action-agnostic video co-training, and the paper ships the right supporting evidence — data-volume curves (100–800 demos), component ablations isolating KAI (44.1% → 75.0%) and Lgeo (→ 82.9%), non-trivial architectural controls (future-depth prediction under the same backbone, DINOv2-on-DP3, bidirectional attention), quantitative KAI accuracy correlated with success (Table 7), and a real-world robustness study under three distraction settings with a falsifiable mechanism (kinematic priors reduce reliance on surface visual correlations). The known-joint-type prior limits scope but is a reasonable engineering assumption for the evaluated regime. The principal risk to significance is not the idea but the strength of the comparative evidence.

major comments (4)
  1. [Table 1, Table 6 (App. A.3), App. A.2] The headline claim is comparative ('matching or surpassing baseline performance while using only half the demonstration data', Abstract and §4.3), but three of the five baselines show signatures of misconfiguration rather than genuine capability. (i) DP3's data curve in Table 6 is non-monotonic: 20.1% (100 demos) → 13.0% (200) → 24.3% (400) → 37.3% (800), and its per-task profile is bimodal (97.0% Close Door vs. 2.0% Close Drawer at 800 demos; Close Drawer drops from 11.0% at 100 demos to 2.0% at 800). Degrading performance with 8x more data indicates a training-stability or checkpoint-selection problem, not a property of the method. (ii) RISE-2 at 22.3% is far below what its architecture achieves in its source paper; the authors' own adaptation note ('modifying the output action dimension', App. A.2) suggests a possibly lossy port, and the 'DP3 w/ DINOv2' ablation (Table 3, 17.7%) furth
  2. [§3.1, Eqs. (3)–(4)] L_pris_motion (Eq. 3) and L_rev_motion (Eq. 4) presuppose that the joint type is known at training and deployment time, and the revolute loss additionally requires a designated anchor subset Ka 'located on the rotation axis' and an axis for proj(·). The manuscript never states where these come from: is joint type hand-labeled per task? How are Ka keypoints and the rotation axis obtained for real-world objects where no simulator state exists? This is load-bearing for the paper's generality claim, since the inductive bias that drives the sample-efficiency gains only applies when the correct motion regularizer is selected and the axis is correct. §7 (Limitations) discusses only task horizon and single-embodiment evaluation; it should also address the known-joint-type assumption, sensitivity to mislabeled or multi-DoF joints, and how K=4 keypoints are chosen/placed on the moving part in prac
  3. [Table 4, §5.1] Each real-world cell is 5 initial states × 3 trials = 15 trials (45 per distraction setting across 3 tasks). With n=15, the reported margins (e.g., Ours w/ video 73.3% vs. Seer 26.7% under background distractions) are directionally convincing, but per-task differences and the +6.6/+11.1/+13.4 pp video co-training gains carry binomial standard errors of roughly ±11 pp per cell and ±7 pp per setting average. No confidence intervals or trial counts per aggregate are reported. The 'over 70% average success rate' claim should either be supported with more trials or reported with uncertainty estimates; as written, small re-runs could move individual cells substantially.
  4. [§3.3, Eqs. (5)–(6)] The co-training claim depends on supervising KAI from HOI4D using 'bounding boxes of articulated parts' (§3.3), but Eqs. (5)–(6) require 3D keypoint locations and displacement sequences. How 3D keypoint trajectories in the robot's world frame are derived from 2D part bounding boxes in egocentric human video — camera calibration, depth source, coordinate alignment, handling of occlusion — is not described. Likewise, the source of ground-truth KAI targets for the real-robot demonstrations (App. A.1 mentions manual base calibration only) is unspecified. This gap makes the video co-training pipeline difficult to reproduce or assess; a short subsection or appendix detailing target extraction for both data sources is needed.
minor comments (7)
  1. [§1, paragraph 3] Typo: 'we introduceKinematic-Aware Articulation Interface' — missing space and missing article ('the').
  2. [Figure 4] Figure 4's inline data labels (e.g., '20.08 / 13.0 / 24.25 / 37.33') are cramped and hard to attribute to curves; Table 6 contains the same numbers, so consider simplifying the figure or moving labels to a legend.
  3. [App. A.2] Inconsistent checkpoint-selection protocols across baselines (ACT: best validation checkpoint; Seer: average of epochs 30/34/39; DP3/RISE-2/ArticuBot and the authors' own method: unstated). Please state the protocol for every method, including 'Ours'.
  4. [Table 5] Hyperparameters in Table 5 are unusual in scale (α3 = 0.001, λ1 = 1000.0). A brief note on how sensitive results are to λ1 (the KAI-vs-action loss balance) would strengthen the ablation section, since LKAI effectively dominates the objective.
  5. [§3.1] The KAI keypoint count K and prediction horizon N are never given values in the main text (K=4 appears only in the Fig. 3 caption). Please state K, N, and the keypoint placement procedure explicitly in §3.1.
  6. [App. A.7] The unseen-instance results (Tables 9–10) are encouraging but thin: sim uses one task (close laptop), and the real-world test is a single unseen laptop with unspecified trial count. Please report trial counts and, if available, standard errors.
  7. [Table 6] DP3's Close Laptop success actually drops from 89.5% (100 demos) to 62.0% (800 demos) in Table 6 — worth flagging explicitly when discussing baseline behavior, since readers will notice it.

Circularity Check

0 steps flagged

Empirical imitation-learning paper: success rates are held-out measurements, not algebraic restatements of KAI losses or fitted constants.

full rationale

KAI is an engineered intermediate representation (K keypoints plus N-step locations/displacements, regularized by isometric and joint-type motion losses in Eqs. 1–4) inserted into a Perception–KAI–Action policy. The load-bearing claims—82.9% average sim SR with half the demos, real-world transfer, and video co-training gains—are evaluated by task success on held-out seeds, unseen backgrounds, and real trials (Tables 1, 4, 6; Figs. 4, 6–7). LKAI supervises predicted geometry against demo- or bbox-derived targets; Laction and the SR metric are separate. Nothing in the chain equates a fitted constant to a reported prediction by construction, imports uniqueness from author-overlapping theorems, or renames a known law as a derivation. Ordinary design choices (K, N, loss weights, causal masks) are hyperparameters, not circular steps. Baseline-tuning concerns affect comparative strength, not circularity of the method’s own results.

Axiom & Free-Parameter Ledger

4 free parameters · 5 axioms · 1 invented entities

Load-bearing content is mostly standard robot-learning practice plus the paper’s kinematic modeling choices. Claims rest on: (i) articulated motion being well-modeled as prismatic or revolute with isometric part geometry; (ii) demo-derived keypoint/displacement labels being faithful supervision; (iii) several hand-chosen scales (K, N, loss weights); (iv) HOI4D part boxes transferring as KAI targets. No new physical entities; KAI is an engineered representation.

free parameters (4)
  • K (number of interaction keypoints) = 4 (illustrated)
    Cardinality of the keypoint set is chosen (illustrated with K=4); not derived from data or theory.
  • N (prediction horizon timesteps)
    Future location/displacement and action chunk length is a design choice shared with action-chunking policies.
  • Loss weights α1, α2, α3, λ1, λ2 (and λs, λm inside Lgeo) = α1=1, α2=1, α3=0.001, λ1=1000, λ2=1
    Table 5 sets α1=α2=1.0, α3=0.001, λ1=1000, λ2=1.0; geometric sub-weights are free knobs that shape the inductive bias.
  • Ka / Km partition for revolute loss
    Which keypoints are treated as axis anchors vs moving points is a modeling choice required by Eq. 4.
axioms (5)
  • domain assumption Target articulated motions are well approximated as single prismatic or revolute joints with rigid moving parts (isometric keypoint distances).
    Lstruct (Eq. 2) and the split Lpris/Lrev (Eqs. 3–4) encode this; multi-DoF or non-rigid cases are outside the loss design (§3.1).
  • ad hoc to paper Joint type (prismatic vs revolute) is known so the correct motion regularizer can be applied.
    Method switches between Eq. 3 and Eq. 4; no joint-type classifier is described in the main method.
  • domain assumption Expert demonstrations and HOI4D part bounding boxes yield reliable ground-truth keypoint trajectories for LKAI.
    §3.3 derives KAI supervision from articulated-part boxes; label noise would directly corrupt the intermediate.
  • ad hoc to paper Unidirectional attention (KAI cannot attend to action tokens) is the right information bottleneck for structural understanding and heterogeneous co-training.
    Phased-reasoning design in §3.2; ablation without causal mask drops avg SR to 78.5% (Table 3).
  • domain assumption Standard imitation-learning evaluation (success within time limit on held-out seeds/scenes) measures the intended capability.
    Used throughout §4–5; common in the field but defines the reported metric.
invented entities (1)
  • Kinematic-Aware Articulation Interface (KAI) no independent evidence
    purpose: Structured intermediate: K keypoints on the moving part plus N-step locations/displacements with geometric losses, bridging perception and action.
    Core postulated representation; not a physical particle/force but an engineered latent interface the policy is trained to predict. Falsifiable only via task metrics and MDE in this paper’s setups.

pith-pipeline@v1.2.0-grok45-kimik3 · 17985 in / 3767 out tokens · 76696 ms · 2026-07-31T13:14:29.345436+00:00 · methodology

0 comments
read the original abstract

Articulated object manipulation requires an understanding of kinematic structure that is difficult and costly to learn from robot demonstrations alone. We introduce the Kinematic-Aware Articulation Interface (KAI), a structured intermediate representation that captures the kinematic structure of articulated objects. By embedding interpretable geometric and kinematic priors into policy learning, KAI provides a strong inductive bias aligned with the underlying structure of articulated motion. This design effectively improves sample efficiency, with gains particularly pronounced in low-data regimes: across six simulation tasks, our method achieves an average success rate of 82.9%, matching or surpassing baseline performance while using only half the demonstration data. Our method also exhibits robust generalization to unseen backgrounds and visual distractors, transferring from a single clean training environment to cluttered real-world scenes. KAI's action-agnostic design further enables co-training with human interaction videos to enhance real-world robustness: under diverse visual distractions, our method with video co-training achieves over 70% average success rate.

Figures

Figures reproduced from arXiv: 2607.24493 by Dahua Lin, Jiangmiao Pang, Jia Zeng, Qiaojun Yu, Yaping Li, Zhaxizhuoma.

Figure 1
Figure 1. Figure 1: Overview. (a) End-to-end policies map observations to actions without modeling kine￾matic structure, relying on large demonstration sets. (b) KAI injects an explicit structured represen￾tation that encodes articulated object kinematics into policy learning. (c) This design yields strong sample efficiency and robust generalization under diverse distractions. Abstract: Articulated object manipulation require… view at source ↗
Figure 2
Figure 2. Figure 2: Method overview. (a) Perception: RGB and point cloud inputs are encoded and fused into visual features. (b) KAI: The KAI module predicts keypoint locations and motion displace￾ments to represent the kinematic structure of the articulated object. (c) Action: The action decoder generates robot actions conditioned on both visual features and KAI predictions. bias that captures the underlying kinematic structu… view at source ↗
Figure 3
Figure 3. Figure 3: Instantiation of KAI. The KAI rep￾resentation visualized on two manipulation tasks (laptop opening and drawer closing). KAI represents the kinematic structure of an articulated object through K interaction key￾points on the moving part. For each key￾point i ∈ {1, . . . , K}, given an observation at timestep t, KAI predicts a sequence of fu￾ture locations l i t+1, . . . , l i t+N and correspond￾ing displace… view at source ↗
Figure 4
Figure 4. Figure 4: Data efficiency comparison. Success rates of our method, Seer, and DP3 across dif￾ferent training data volumes. We evaluate our method against the 2D baseline Seer [27] and the 3D baseline DP3 [28] under varying training data sizes (100, 200, 400, and 800 demonstration episodes). As shown in Fig￾ure 4, while both our method and Seer show rapid performance improvement with increasing data before converging,… view at source ↗
Figure 6
Figure 6. Figure 6: Sequential visualizations of three real￾world tasks. Each row shows a task progressing from left to right over time. 5.1 Experimental Setups Real-world Setup. We evaluate our method on a Franka Research 3 arm with a Robotiq 2F-85 gripper across three articulated object manipulation tasks, using two RealSense D455 cameras for RGB-D visual perception, as shown in [PITH_FULL_IMAGE:figures/full_fig_p007_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Visualization of generalization tasks under different types of distractions. [PITH_FULL_IMAGE:figures/full_fig_p007_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Visualization of five initial states per task under background distraction generalization [PITH_FULL_IMAGE:figures/full_fig_p014_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Visualization of five initial states per task under object distractor generalization setup. [PITH_FULL_IMAGE:figures/full_fig_p015_9.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

29 extracted references · 10 linked inside Pith

  1. [1]

    C. Ning, R. Wu, H. Lu, K. Mo, and H. Dong. Where2explore: Few-shot affordance learning for unseen novel categories of articulated objects. Advances in Neural Information Processing Systems, 36:4585–4596, 2023

    2023

  2. [2]

    Y . Geng, B. An, H. Geng, Y . Chen, Y . Yang, and H. Dong. Rlafford: End-to-end affordance learning for robotic manipulation. In 2023 IEEE International conference on robotics and automation (ICRA), pages 5880–5886. IEEE, 2023

    2023

  3. [3]

    R. Wu, Y . Zhao, K. Mo, Z. Guo, Y . Wang, T. Wu, Q. Fan, X. Chen, L. Guibas, and H. Dong. Vat-mart: Learning visual action trajectory proposals for manipulating 3d articulated objects. arXiv preprint arXiv:2106.14440, 2021

    Pith/arXiv arXiv 2021

  4. [4]

    Y . Wang, R. Wu, K. Mo, J. Ke, Q. Fan, L. J. Guibas, and H. Dong. Adaafford: Learning to adapt manipulation affordance for 3d articulated objects via few-shot interactions. InEuropean conference on computer vision, pages 90–107. Springer, 2022

    2022

  5. [5]

    K. Mo, L. J. Guibas, M. Mukadam, A. Gupta, and S. Tulsiani. Where2act: From pixels to actions for articulated 3d objects. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 6813–6823, 2021

    2021

  6. [6]

    Eisner, H

    B. Eisner, H. Zhang, and D. Held. Flowbot3d: Learning 3d articulation flow to manipulate articulated objects. arXiv preprint arXiv:2205.04382, 2022

    Pith/arXiv arXiv 2022

  7. [7]

    Zhang, B

    H. Zhang, B. Eisner, and D. Held. Flowbot++: Learning generalized articulated objects ma- nipulation via articulation projection. arXiv preprint arXiv:2306.12893, 2023

    Pith/arXiv arXiv 2023

  8. [8]

    Z. Xu, Z. He, and S. Song. Universal manipulation policy network for articulated objects. IEEE robotics and automation letters , 7(2):2447–2454, 2022

    2022

  9. [9]

    W. Cui, C. Zhao, S. Wei, J. Zhang, H. Geng, Y . Chen, H. Li, and H. Wang. Gapartma- nip: A large-scale part-centric dataset for material-agnostic articulated object manipulation. In 2025 IEEE International Conference on Robotics and Automation (ICRA) , pages 14791– 14798. IEEE, 2025

    2025

  10. [10]

    C. P. Morlans, C. Chen, Y . Weng, M. Yi, Y . Huang, N. Heppert, L. Zhou, L. Guibas, and J. Bohg. Ao-grasp: Articulated object grasp generation. In 2024 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) , pages 13096–13103. IEEE, 2024

    2024

  11. [11]

    Q. Yu, J. Wang, W. Liu, C. Hao, L. Liu, L. Shao, W. Wang, and C. Lu. Gamma: Generalizable articulation modeling and manipulation for articulated objects. In 2024 IEEE International Conference on Robotics and Automation (ICRA) , pages 5419–5426. IEEE, 2024

    2024

  12. [12]

    A. Jain, R. Lioutikov, C. Chuck, and S. Niekum. Screwnet: Category-independent articu- lation model estimation from depth images using screw theory. In 2021 IEEE International Conference on Robotics and Automation (ICRA) , pages 13670–13677. IEEE, 2021

    2021

  13. [13]

    A. Jain, S. Giguere, R. Lioutikov, and S. Niekum. Distributional depth-based estimation of object articulation models. In Conference on Robot Learning, pages 1611–1621. PMLR, 2022

    2022

  14. [14]

    V . Zeng, T. E. Lee, J. Liang, and O. Kroemer. Visual identification of articulated object parts. In 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) , pages 2443–2450. IEEE, 2021

    2021

  15. [15]

    X. Li, H. Wang, L. Yi, L. J. Guibas, A. L. Abbott, and S. Song. Category-level articulated object pose estimation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 3706–3715, 2020. 9

    2020

  16. [16]

    H. Geng, H. Xu, C. Zhao, C. Xu, L. Yi, S. Huang, and H. Wang. Gapartnet: Cross-category domain-generalizable object perception and manipulation via generalizable and actionable parts. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogni- tion, pages 7081–7091, 2023

    2023

  17. [17]

    J. Wang, W. Liu, Q. Yu, Y . You, L. Liu, W. Wang, and C. Lu. Rpmart: Towards robust percep- tion and manipulation for articulated objects. In 2024 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) , pages 7270–7277. IEEE, 2024

    2024

  18. [18]

    Y . Wang, Z. Wang, M. Nakura, P. Bhowal, C.-L. Kuo, Y .-T. Chen, Z. Erickson, and D. Held. Articubot: Learning universal articulated object manipulation policy via large scale simulation. arXiv preprint arXiv:2503.03045, 2025

    Pith/arXiv arXiv 2025

  19. [19]

    X. Wang, T. Chen, Q. Yu, T. Xu, Z. Chen, Y . Fu, Z. He, C. Lu, Y . Mu, and P. Luo. Articulated object manipulation using online axis estimation with sam2-based tracking. arXiv preprint arXiv:2409.16287, 2024

    Pith/arXiv arXiv 2024

  20. [20]

    Q. Yu, S. Huang, X. Yuan, Z. Jiang, C. Hao, X. Li, H. Chang, J. Wang, L. Liu, H. Li, et al. Uniaff: A unified representation of affordances for tool usage and articulation with vision- language models. In 2025 IEEE International Conference on Robotics and Automation (ICRA), pages 8980–8987. IEEE, 2025

    2025

  21. [21]

    H. Fang, C. Wang, Y . Wang, J. Chen, S. Xia, J. Lv, Z. He, X. Yi, Y . Guo, X. Zhan, et al. Airexo-2: Scaling up generalizable robotic imitation learning with low-cost exoskeletons. arXiv preprint arXiv:2503.03081, 2025

    Pith/arXiv arXiv 2025

  22. [22]

    Oquab, T

    M. Oquab, T. Darcet, T. Moutakanni, H. V o, M. Szafraniec, V . Khalidov, P. Fernandez, D. Haz- iza, F. Massa, A. El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07193, 2023

    Pith/arXiv arXiv 2023

  23. [23]

    Radford, J

    A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, et al. Learning transferable visual models from natural language supervi- sion. In International conference on machine learning , pages 8748–8763. PmLR, 2021

    2021

  24. [24]

    Y . Liu, Y . Liu, C. Jiang, K. Lyu, W. Wan, H. Shen, B. Liang, Z. Fu, H. Wang, and L. Yi. Hoi4d: A 4d egocentric dataset for category-level human-object interaction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 21013–21022, 2022

    2022

  25. [25]

    Isaac Sim

    NVIDIA. Isaac Sim. URL https://github.com/isaac-sim/IsaacSim

  26. [26]

    T. Z. Zhao, V . Kumar, S. Levine, and C. Finn. Learning fine-grained bimanual manipulation with low-cost hardware. arXiv preprint arXiv:2304.13705, 2023

    Pith/arXiv arXiv 2023

  27. [27]

    Y . Tian, S. Yang, J. Zeng, P. Wang, D. Lin, H. Dong, and J. Pang. Predictive inverse dynamics models are scalable learners for robotic manipulation. arXiv preprint arXiv:2412.15109, 2024

    Pith/arXiv arXiv 2024

  28. [28]

    Y . Ze, G. Zhang, K. Zhang, C. Hu, M. Wang, and H. Xu. 3d diffusion policy: Generalizable visuomotor policy learning via simple 3d representations. arXiv preprint arXiv:2403.03954 , 2024. 10 A Appendix A.1 Implementation Details Vision. At each timestep, visual inputs are captured from two viewpoints: an eye-on-hand camera and an eye-on-base camera. For s...

    Pith/arXiv arXiv 2024

  29. [128]

    The spatial aligner module subsequently fuses the 2D semantic features and 3D geometric features, producing an output of 14 tokens with 512 chan- nels each

    All 3D coordinates are defined in the world coordinate system, with the robot base position manually calibrated for real-world experiments. The spatial aligner module subsequently fuses the 2D semantic features and 3D geometric features, producing an output of 14 tokens with 512 chan- nels each. Finally, an MLP projects this representation to 384 dimensio...