Pith. sign in

REVIEW 3 major objections 4 minor 1 cited by

Generating 6DoF Object Manipulation Trajectories from Action Description in Egocentric Vision

T0 review · 3 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Ordinary egocentric video can supply 28,497 text-labeled 6DoF manipulation trajectories, and models trained on them turn action descriptions into valid object motions.

desk verdict Useful dataset contribution, but the rotation labels are unvalidated and the flat GD numbers suggest the rotation part of the claimed 6DoF generation may be trivial. read the letter →

arxiv 2506.03605 v1 pith:AUMW64YF submitted 2025-06-04 cs.CV

Tomoya Yoshida , Shuhei Kurita , Taichi Nishimura , Shinsuke Mori This is my paper
classification cs.CV
keywords 6DoFobjectmanipulationegocentricvisiontrajectorygenerationvision-languagemodelspointcloudsactiondescriptionsdatasetconstructionsingularvaluedecomposition
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that precise 6DoF object manipulation demonstrations can be extracted automatically from ordinary egocentric video, without pre-recorded camera coordinates or depth sensors, and that this extracted data can train models to generate object trajectories from an action description. The authors build EgoTraj, a dataset of 28,497 trajectories with associated action descriptions, by combining temporal action localization, open-vocabulary segmentation, dense 3D point tracking, monocular depth estimation, and point-cloud registration. They then train vision- and point-cloud-based language models to emit discretized trajectory tokens, and show on a separate egocentric benchmark with motion-capture ground truth that the models generate valid, action-dependent trajectories. If correct, this turns the large body of already-recorded egocentric video into a scalable source of training data for text-conditioned robot manipulation.

What carries the argument

The carrying mechanism is the extraction pipeline's chain of geometry estimates. Temporal action localization fixes the action span and object name; open-vocabulary segmentation isolates the object in the first frame; a dense 3D point tracker followed by monocular depth estimation produces an object point-cloud sequence; and point-cloud registration (RANSAC feature matching refined by colored ICP) projects every frame into the first frame's camera coordinates. For rotation, the pipeline computes the covariance matrix $H$ between the initial and current object point clouds, factorizes it as $H = U\Sigma V^T$, and sets the rotation matrix to $R = V U^T$, assuming the object is rigid. On the generation side, each of the six pose coordinates is quantized into 256 bins and mapped to special tokens, so a vision-language model emits the trajectory autoregressively.

What would settle it

Run the extraction pipeline on a motion-capture egocentric dataset whose true 6DoF object poses are known, compute the per-frame geodesic distance between the extracted and ground-truth rotations, and check whether the extraction error alone is already as large as the prediction errors the paper reports; if it is, the rotation labels are too corrupted to support the claimed trajectory validity.

Watch

Extended reading notes

Core claim

The central claim is that a scalable, fully automatic pipeline can recover 6DoF object manipulation trajectories from a single moving egocentric camera, and that these noisy but large-scale trajectories are sufficient supervision for text-conditioned trajectory generation. Concretely, the paper extracts 28,497 trajectories from the Ego-Exo4D videos, covering 228 action verbs and 4,158 objects, using a four-stage framework, and then trains several language-model backbones by discretizing each pose dimension into 256 tokens and treating generation as next-token prediction. On the HOT3D benchmark, which supplies high-precision optical-marker trajectories, the resulting models produce valid trajectories, and the point-cloud-based model achieves the lowest displacement error and qualitatively different motions for different action verbs such as 'transfer' and 'stir'. The paper also reports that adding trajectory information improves action-description generation, especially verb similarity, which it takes as evidence that the extracted motions reflect the described manipulation.

Load-bearing premise

The approach depends on the assumption that the automatically tracked point clouds are clean and the objects rigid enough for the computed rotations to be close to the true rotations, and this assumption is never checked against ground truth.

Editorial extensions

If this is right

  • Automatically extracted trajectories transfer to a motion-capture benchmark, so large collections of everyday egocentric video can substitute for expensive laboratory demonstration recordings.
  • Because the pipeline needs no pre-recorded camera coordinates or depth hardware, it can be applied to most existing egocentric video datasets, not only purpose-built manipulation collections.
  • Trajectory-conditioned models generate more accurate action descriptions, with verb similarity rising by roughly ten percent, indicating the extracted motion encodes the action's semantics.
  • Sampling from the language model yields multiple plausible trajectories for one action description, matching the fact that a manipulation can be performed in several ways.
  • Scaling the automatically extracted training set improves displacement errors, implying that the data-harvesting approach has continued returns as more egocentric video becomes available.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper leaves the rotation labels unvalidated; a natural extension is to compare SVD-extracted rotations against ground truth on motion-capture clips and, if they are noisy, to train position-only models to see how much of the reported validity comes from translation alone.
  • The rigid-object filter excludes deformable objects, so the framework's generality hinges on extending tracking and pose recovery to clothes, rope, and other non-rigid items; that is a testable next milestone rather than a claim of the paper.
  • Because the models are trained on one video collection and evaluated on another, an obvious stress test is to apply the same pipeline to additional egocentric corpora and measure whether the transfer gap widens or narrows with dataset scale.
  • A direct probe of whether the action description is driving generation, rather than the initial visual state, would be to fix the image and vary the text; the paper's qualitative examples suggest the text matters, but it does not quantify this.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper proposes an automated pipeline for extracting 6DoF object manipulation trajectories from egocentric videos in Ego-Exo4D, without using pre-recorded camera extrinsic parameters. The pipeline uses GPT-4o for temporal localization and action annotation, Grounded SAM for segmentation, monocular depth estimation and SpaTracker for 3D point tracking, point-cloud registration for camera alignment, and SVD alignment of tracked point clouds for rotation extraction. This yields EgoTraj, a dataset of 28,497 trajectories with action descriptions. The authors then train several VLM backbones (BLIP-2, VILA, PointLLM, MiniGPT-3D) with discretized trajectory tokens and evaluate on HOT3D, reporting ADE, FDE, and geodesic distance. They also show that conditioning captioning on trajectories improves verb similarity. The central claim is that the extracted trajectories are valid and that the trained models generate valid 6DoF object manipulation trajectories from action descriptions.

Significance. If the rotation labels were validated, the contribution would be significant: EgoTraj would be the first large-scale, text-annotated 6DoF manipulation trajectory dataset extracted from unconstrained egocentric video, with an automated and scalable pipeline. The evaluation on HOT3D uses external ground-truth tracking, so the main generation experiment is not circular, and the release of dataset and code is a concrete strength. The comparison against USST and the trajectory-conditioned captioning experiments also provide useful baselines. However, the paper's central claim hinges on the correctness of the automatically extracted rotation supervision, and that component is never validated. The near-constant geodesic distance across models and sampling sizes is a red flag that the rotation channel may be degenerate, so the significance of the work is currently conditional on resolving this issue.

major comments (3)
  1. [Sec. 3.1 (Rotation Sequence Extraction); Limitations; App. A.1 (Failure Cases)] The rotation labels in EgoTraj are obtained by SVD alignment of SpaTracker point clouds under the assumptions of object rigidity, accurate 3D point tracking, and reliable monocular depth, but no quantitative validation of these labels is provided. The paper's own Limitations section and Appendix Fig. 11 state that object segmentation and point-cloud registration can fail, and the Data Curation Methods discard such clips, yet the impact of these failures on the resulting rotation distribution is never measured. Because the central claim of 'valid 6DoF trajectories' depends directly on the rotation channel, I request a validation experiment: run the extraction pipeline on HOT3D videos (where ground-truth 6DoF trajectories exist) and report the rotation error between extracted and ground-truth rotations, along with the fraction of clips rejected at each stage. Without such a check, the training supervision for rotations is unverified.
  2. [Table 2; Table 4; Sec. 5.2] The rotation results are suspiciously flat. In Table 2, the geodesic distance is approximately 0.54 rad for nearly every model regardless of backbone or input modality, and Table 4 shows that increasing sampling from 1 to 10 improves GD only from 0.541 to 0.540 while ADE improves from 0.271 to 0.212. This pattern is consistent with the model emitting a near-constant rotation (for example, the identity or the training-set mean), and the text itself acknowledges 'the difficulty in generating accurate rotational movements.' To support the claim that the models generate valid 6DoF trajectories, the authors should compare against trivial baselines (identity rotation, mean rotation, constant-rotation Seq2Seq) and report the distribution of predicted rotations, such as per-axis variance and the fraction of predictions at bin boundaries. If the GD of these trivial baselines matches the reported values, the rotation-generation claim should be weakened accordingly.
  3. [Sec. 3.2 and App. A.2 (Action Description Generation); Sec. 5.1] The test-time action descriptions for HOT3D are generated by GPT-4o using a prompt nearly identical to the prompt used to create training descriptions in EgoTraj, and the same model is used for temporal action localization in both. This introduces a style and vocabulary coupling between training and test text: the models may be tuned to reproduce GPT-4o's phrasing rather than to understand human-written instructions. This is not circularity in the 6DoF ground truth, but it is a confound for the claim that the model generalizes from action descriptions. I ask the authors to report whether evaluation results change when human-written or paraphrased descriptions are used, or at minimum to quantify the lexical overlap between training and test descriptions.
minor comments (4)
  1. [Introduction] The text contains a typo: 'pre-recored camera coordinates' should be 'pre-recorded camera coordinates.'
  2. [Table 1 caption] The caption contains a typo: 'repored in [5]' should be 'reported in [5].'
  3. [App. A.1] The statement that a hand-object detection model [81] is used is confusing because reference [81] is 'Understanding human hands in contact at internet scale,' which is a contact-detection dataset paper, not an obvious hand-object detection method; please clarify the actual detection model used.
  4. [Sec. 3.1] The introduction says the method works 'without pre-recorded camera coordinates or depth,' but the pipeline itself relies on monocular depth estimation. This is not an error, but the wording should be clarified to 'without pre-recorded camera coordinates or depth sensors' to avoid confusion.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the trajectory-generation claim is tested against external HOT3D ground truth, and no prediction reduces to its own inputs by construction.

full rationale

The paper's central contribution is an automatically extracted 6DoF training set (EgoTraj) plus VLM baselines that generate trajectories from action descriptions, evaluated on HOT3D. No step in this chain reduces to its own input by construction. The rotation labels in EgoTraj are computed by SVD alignment of tracked object point clouds (H=UΣV^T, R=VU^T), but these labels are training supervision, not the evaluation target; the models are tested against HOT3D's optical-marker-based ground-truth trajectories, which are external to the extraction pipeline. The predicted rotations are scored with geodesic distance against that independent ground truth, so the 'valid trajectory generation' claim is not statistically forced by the SVD extraction. The one legitimate concern is that GPT-4o is used both to localize action spans and name objects in EgoTraj (Sec. 3.1) and to synthesize action descriptions for HOT3D (Appendix A.2), so the text-conditioning evaluation only demonstrates generalization to GPT-4o-style language rather than to independently human-authored instructions. This is a shared-annotator limitation and an external-validity caveat, not a circular reduction: the trajectory targets remain independent, no fitted parameter is renamed as a prediction, and no load-bearing self-citation or imported uniqueness theorem appears. Hence the appropriate circularity finding is none.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The central results rest on the reliability of automatically extracted training labels and on the equivalence between machine-generated and human test descriptions. The pipeline introduces several hand-chosen thresholds and assumes rigidity, accurate monocular depth, and successful point cloud registration. No new physical entities are introduced.

free parameters (4)
  • Detection confidence threshold = 0.3
    Filters video clips for which a hand-object detector's confidence is below 0.3; directly controls which clips become EgoTraj samples (Appendix A.1).
  • Trajectory length cap = 20 frames
    Trajectories longer than 20 frames are cropped during training and evaluation due to token limits, changing the target distribution (Sec 5.1).
  • Discretization bin count = 256
    Each pose dimension is quantized into 256 uniform bins within dataset bounds, setting the output resolution (Sec 4).
  • Point cloud registration thresholds = RANSAC distance 0.03, ICP distance 0.008
    Hand-set thresholds for camera extrinsic estimation; affect trajectory projection accuracy (Appendix A.1).
assumptions (5)
  • domain assumption The manipulated object is rigid, so its rotation can be recovered by SVD alignment of tracked point clouds.
    The framework filters non-rigid objects and assumes remaining objects are rigid (Sec. 3.1 Rotation Sequence Extraction). If an object deforms, the extracted rotation labels are invalid.
  • domain assumption Camera motion between consecutive frames is small and can be estimated by point cloud registration.
    The pipeline assumes adjacent frames can be aligned with RANSAC and colored ICP (Sec. 3.1 Trajectory Projection). Large or fast camera motions cause registration failures acknowledged in the Limitations.
  • domain assumption Monocular depth estimates from Depth Anything are geometrically consistent across frames.
    Depth maps are converted to point clouds for tracking and registration; errors in relative depth scaling would distort trajectories (Sec. 3.1).
  • domain assumption GPT-4o temporal localization and object naming are accurate.
    Action spans and object names are obtained from GPT-4o with few-shot prompts (Sec. 3.1, Appendix A.1); mistakes propagate into the extracted trajectories.
  • domain assumption HOT3D test action descriptions generated by GPT-4o are a valid proxy for human instructions.
    HOT3D lacks temporal descriptions, so the authors generate action descriptions and time spans with GPT-4o (Appendix A.2); the evaluation measures text-conditioned generation using these synthetic labels.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Generating 6DoF Object Manipulation Trajectories from Action Description in Egocentric Vision." pith.science (2026). https://pith.science/paper/AUMW64YF

@misc{pith2026250603605,
  author       = {Pith},
  title        = {Pith review of: Generating 6DoF Object Manipulation Trajectories from Action Description in Egocentric Vision},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/AUMW64YF}},
  note         = {Machine review of arXiv:2506.03605}
}
read the original abstract

Learning to use tools or objects in common scenes, particularly handling them in various ways as instructed, is a key challenge for developing interactive robots. Training models to generate such manipulation trajectories requires a large and diverse collection of detailed manipulation demonstrations for various objects, which is nearly unfeasible to gather at scale. In this paper, we propose a framework that leverages large-scale ego- and exo-centric video datasets -- constructed globally with substantial effort -- of Exo-Ego4D to extract diverse manipulation trajectories at scale. From these extracted trajectories with the associated textual action description, we develop trajectory generation models based on visual and point cloud-based language models. In the recently proposed egocentric vision-based in-a-quality trajectory dataset of HOT3D, we confirmed that our models successfully generate valid object trajectories, establishing a training dataset and baseline models for the novel task of generating 6DoF manipulation trajectories from action descriptions in egocentric vision.

Figures

Figures reproduced from arXiv: 2506.03605 by the authors.

Figure 1
Figure 1. 6DoF object manipulation trajectory. This task aims to generate a sequence of 6DoF object poses from an action description and an initial state comprising the visual input and the object’s initial pose. Abstract Learning to use tools or objects in common scenes, par￾ticularly handling them in various ways as instructed, is a key challenge for developing interactive robots. Train￾ing models to generate such manipulat… view at source ↗
Figure 2
Figure 2. Trajectory extraction from egocentric videos. Four steps of (1) temporal action localization, (2) position sequence extraction, (3) trajectory projection, and (4) rotation sequence extraction. Our project page includes visualization of resulting trajectories with videos. each frame using a depth estimation model. For SpaTracker, we input the sequence of frames, the corresponding depth maps, and the segmentation map … view at source ↗
Figure 3
Figure 3. Overview of model architecture. Our model archi￾tecture utilizes visual and point cloud-based language models as backbones and extends them by incorporating extended vocabu￾laries for trajectory tokenization. clouds. Further details are provided in the Appendix. 4. 6DoF Object Manipulation Generation Task Setting. Our task is to generate a sequence of 6DoF object poses for object manipulation, based on an action de￾… view at source ↗
Figures from the paper (10 more)
Figure 4
Figure 4. Figure 4: Qualitative results of PointLLM [97]. Generated trajectories are illustrated using 3D bounding boxes for visualization. 3D pos. 2D pos. 3D rot. Sampling ADE FDE ADE FDE GD 1 0.271 0.458 0.208 0.327 0.541 3 0.236 0.401 0.178 0.280 0.543 10 0.212 0.363 0.158 0.252 0.540 …
Figure 5
Figure 5. Figure 5: Comparison of performance across different dataset scales for PointLLM [97]. as it achieved the highest performance in Tab. 2. The results, especially in ADE and FDE, indicate that model perfor￾mance improves as the dataset size increases, demonstrating the effectivene…
Figure 7
Figure 7. Figure 7: Qualitative results in image and 3D captioning task. The top and bottom figures depict generated captions of BLIP￾2 [50] and PointLLM [97], respectively. “w/” and “w/o” indicate whether models utilize trajectory information. cate that utilizing trajectories boosts perf…
Figure 8
Figure 8. Figure 8: System prompt to obtain manipulated objects [PITH_FULL_IMAGE:figures/full_fig_p015_8.png]
Figure 9
Figure 9. Figure 9: Image frames embedded sequential indices for GPT-4o input. cluded incorrect segmentation results using a hand-object detector [81], such as [PITH_FULL_IMAGE:figures/full_fig_p015_9.png]
Figure 10
Figure 10. Figure 10: System prompt for temporal action localization. (a) Object segmentation failure “Pick the cup in the sink with his right hand.” Correct object Incorrect object “Drop the wine bottle on the chopping board with his right hand.” (b) Point cloud registration failure [PIT…
Figure 11
Figure 11. Figure 11: Failure cases. A.3. Dataset Statistics Here, we provide detailed statistics for our dataset and the HOT3D evaluation dataset. Vocabularies [PITH_FULL_IMAGE:figures/full_fig_p016_11.png]
Figure 12
Figure 12. Figure 12: System prompt for HOT3D annotation. Word cloud of objects in our dataset Word cloud of verbs in our dataset Word cloud of objects in HOT3D Word cloud of verbs in HOT3D [PITH_FULL_IMAGE:figures/full_fig_p017_12.png]
Figure 13
Figure 13. Figure 13: Word cloud of objects and verbs in our dataset and HOT3D dataset [3]. (a) Our dataset (b) HOT3D dataset [PITH_FULL_IMAGE:figures/full_fig_p017_13.png]
Figure 15
Figure 15. Figure 15: Distribution of variations in each trajectory element for our dataset and HOT3D dataset [PITH_FULL_IMAGE:figures/full_fig_p018_15.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. MotionForesight: Re-purposing Video Models for Future 3D Scene-Flow Prediction

    cs.CV 2026-07 conditional novelty 6.0 of 10

    A 12.19M-parameter adapter on a frozen video/tracking model forecasts future 3D object-point tracks from 7 observed frames, trained on 40k human videos without language or action labels.

Reference graph

Works this paper leans on

111 extracted references · 58 canonical work pages · cited by 1 Pith paper

  1. [1]

    Gpt-4 technical report.arXiv preprint arXiv:2303.08774, 2023

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report.arXiv preprint arXiv:2303.08774, 2023. 3, 1, 2

    arXiv 2023

  2. [2]

    Affordances from human videos as a versatile representation for robotics

    Shikhar Bahl, Russell Mendonca, Lili Chen, Unnat Jain, and Deepak Pathak. Affordances from human videos as a versatile representation for robotics. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023. 2

    2023

  3. [3]

    Introducing hot3d: An egocentric dataset for 3d hand and object tracking

    Prithviraj Banerjee, Sindi Shkodrani, Pierre Moulon, Shreyas Hampali, Fan Zhang, Jade Fountain, Edward Miller, Selen Basol, Richard Newcombe, Robert Wang, Jakob Julian Engel, and Tomas Hodan. Introducing hot3d: An egocentric dataset for 3d hand and object tracking. arXiv preprint arXiv:2406.09598, 2024. 1, 2, 3, 4, 5

    arXiv 2024

  4. [4]

    METEOR: An auto- matic metric for MT evaluation with improved correlation with human judgments

    Satanjeev Banerjee and Alon Lavie. METEOR: An auto- matic metric for MT evaluation with improved correlation with human judgments. InProceedings of the ACL Work- shop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization, 2005. 8

    2005

  5. [5]

    Uncertainty-aware state space transformer for egocentric 3d hand trajectory forecasting

    Wentao Bao, Lele Chen, Libing Zeng, Zhong Li, Yi Xu, Junsong Yuan, and Yu Kong. Uncertainty-aware state space transformer for egocentric 3d hand trajectory forecasting. InProceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2023. 2, 5, 6

    2023

  6. [6]

    ARKitscenes - a diverse real-world dataset for 3d indoor scene understanding using mobile RGB-d data

    Gilad Baruch, Zhuoyuan Chen, Afshin Dehghan, Tal Dimry, Yuri Feigin, Peter Fu, Thomas Gebauer, Brandon Joffe, Daniel Kurz, Arik Schwartz, and Elad Shulman. ARKitscenes - a diverse real-world dataset for 3d indoor scene understanding using mobile RGB-d data. InThirty- fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (...

    2021

  7. [7]

    Yu, and Jianbo Shi

    Gedas Bertasius, Hyun Soo Park, Stella X. Yu, and Jianbo Shi. First-person action-object detection with egotnet. In Proceedings of Robotics: Science and Systems (RSS), 2017. 2

    2017

  8. [8]

    Kemp, and James Hays

    Samarth Brahmbhatt, Cusuh Ham, Charles C. Kemp, and James Hays. Contactdb: Analyzing and predicting grasp contact via thermal imaging. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019. 2

    2019

Show all 111 references
  1. [9]

    RT-1: Robotics Transformer for Real- World Control at Scale

    Anthony Brohan, Noah Brown, Justice Carbajal, Yev- gen Chebotar, Joseph Dabis, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alexander Herzog, Jas- mine Hsu, et al. RT-1: Robotics Transformer for Real- World Control at Scale. InProceedings of Robotics: Sci- ence and S...

    2023

  2. [10]

    Deep regression on manifolds: A 3d ro- tation case study

    Romain Br ´egier. Deep regression on manifolds: A 3d ro- tation case study. In2021 International Conference on 3D Vision (3DV), 2021. 4

    2021

  3. [11]

    Text2hoi: Text-guided 3d motion generation for hand-object interaction

    Junuk Cha, Jihyeon Kim, Jae Shin Yoon, and Seungryul Baek. Text2hoi: Text-guided 3d motion generation for hand-object interaction. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024. 2

    2024

  4. [12]

    Fleet, and Geoffrey Hinton

    Ting Chen, Saurabh Saxena, Lala Li, David J. Fleet, and Geoffrey Hinton. Pix2seq: A language modeling frame- work for object detection. In10th International Conference on Learning Representations, ICLR, 2022. 3, 5

    2022

  5. [13]

    Looking to relations for future trajectory forecast

    Chiho Choi and Behzad Dariush. Looking to relations for future trajectory forecast. InProceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2019. 6

    2019

  6. [14]

    Learning to act properly: Predicting and explain- ing affordances from images

    Ching-Yao Chuang, Jiaman Li, Antonio Torralba, and Sanja Fidler. Learning to act properly: Predicting and explain- ing affordances from images. InProceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018. 2

    2018

  7. [15]

    Open x-embodiment: Robotic learning datasets and rt-x models: Open x- embodiment collaboration

    Open X-Embodiment Collaboration. Open x-embodiment: Robotic learning datasets and rt-x models: Open x- embodiment collaboration. InIEEE International Confer- ence on Robotics and Automation (ICRA), 2024. 1, 2, 5

    2024

  8. [16]

    Ganhand: Predicting human grasp affordances in multi-object scenes

    Enric Corona, Albert Pumarola, Guillem Alenya, Francesc Moreno-Noguer, and Gregory Rogez. Ganhand: Predicting human grasp affordances in multi-object scenes. InPro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020. 2

    2020

  9. [17]

    Scaling egocentric vision: The epic- kitchens dataset

    Dima Damen, Hazel Doughty, Giovanni Maria Farinella, Sanja Fidler, Antonino Furnari, Evangelos Kazakos, Da- vide Moltisanti, Jonathan Munro, Toby Perrett, Will Price, and Michael Wray. Scaling egocentric vision: The epic- kitchens dataset. InProceedings of the European Confer-...

    2018

  10. [18]

    Rescaling egocentric vision: Collection, pipeline and challenges for epic-kitchens-100.Interna- tional Journal of Computer Vision, 130(1):33–55, 2022

    Dima Damen, Hazel Doughty, Giovanni Maria Farinella, Antonino Furnari, Evangelos Kazakos, Jian Ma, Davide Moltisanti, Jonathan Munro, Toby Perrett, Will Price, and Michael Wray. Rescaling egocentric vision: Collection, pipeline and challenges for epic-kitchens-100.Interna- tio...

    2022

  11. [19]

    Affor- dancenet: An end-to-end deep learning approach for object affordance detection

    Thanh-Toan Do, Anh Nguyen, and Ian Reid. Affor- dancenet: An end-to-end deep learning approach for object affordance detection. InIEEE International Conference on Robotics and Automation (ICRA), 2018. 2

    2018

  12. [20]

    Guoguang Du, Kai Wang, Shiguo Lian, and Kaiyong Zhao. Vision-based robotic grasping from object localization, ob- ject pose estimation to grasp estimation for parallel grip- pers: a review.Artificial Intelligence Review, 54(3):1677– 1734, 2021. 3

    2021

  13. [21]

    Project aria: A new tool for egocentric multi-modal ai research

    Jakob Engel, Kiran Somasundaram, Michael Goesele, Al- bert Sun, Alexander Gamino, Andrew Turner, Arjang Talat- tof, Arnie Yuan, Bilal Souti, Brighid Meredith, et al. Project aria: A new tool for egocentric multi-modal ai research. arXiv preprint arXiv:2308.13561, 2023. 1, 4

    2023 arXiv

  14. [22]

    Egopat3dv2: Predicting 3d action target from 2d egocentric vision for human-robot interac- tion

    Irving Fang, Yuzhong Chen, Yifan Wang, Jianghan Zhang, Qiushi Zhang, Jiali Xu, Xibo He, Weibo Gao, Hao Su, Yim- ing Li, and Chen Feng. Egopat3dv2: Predicting 3d action target from 2d egocentric vision for human-robot interac- tion. InIEEE International Conference on Robotics a...

    2024

  15. [23]

    Fischler and Robert C

    Martin A. Fischler and Robert C. Bolles. Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography.Communica- tions of the ACM, 24(6), 1981. 4

    1981

  16. [24]

    Zhao, and Chelsea Finn

    Zipeng Fu, Tony Z. Zhao, and Chelsea Finn. Mobile ALOHA: Learning bimanual mobile manipulation using low-cost whole-body teleoperation. In8th Annual Confer- ence on Robot Learning, 2024. 1

    2024

  17. [25]

    What would you expect? anticipating egocentric actions with rolling-unrolling lstms and modality attention

    Antonino Furnari and Giovanni Maria Farinella. What would you expect? anticipating egocentric actions with rolling-unrolling lstms and modality attention. InProceed- ings of the IEEE/CVF International Conference on Com- puter Vision (ICCV), 2019. 2

    2019

  18. [26]

    Next-active-object predic- tion from egocentric videos.Journal of Visual Communi- cation and Image Representation, 49:401–411, 2017

    Antonino Furnari, Sebastiano Battiato, Kristen Grauman, and Giovanni Maria Farinella. Next-active-object predic- tion from egocentric videos.Journal of Visual Communi- cation and Image Representation, 49:401–411, 2017. 2

    2017

  19. [27]

    So predictable! continuous 3d hand trajectory prediction in virtual reality

    Nisal Menuka Gamage, Deepana Ishtaweera, Martin Weigel, and Anusha Withana. So predictable! continuous 3d hand trajectory prediction in virtual reality. InThe 34th Annual ACM Symposium on User Interface Software and Technology, 2021. 2

    2021

  20. [28]

    Transformer networks for trajectory forecasting

    Francesco Giuliari, Irtiza Hasan, Marco Cristani, and Fabio Galasso. Transformer networks for trajectory forecasting. In25th International Conference on Pattern Recognition (ICPR), 2021. 5

    2021

  21. [29]

    Ego4d: Around the world in 3,000 hours of egocentric video

    Kristen Grauman, Andrew Westbury, Eugene Byrne, Zachary Chavis, Antonino Furnari, Rohit Girdhar, Jackson Hamburger, Hao Jiang, Miao Liu, Xingyu Liu, et al. Ego4d: Around the world in 3,000 hours of egocentric video. In Proceedings of the IEEE/CVF Conference on Computer Vi- sio...

    2022

  22. [30]

    Ego-exo4d: Understanding skilled human activity from first-and third-person perspectives.arXiv preprint arXiv:2311.18259, 2023

    Kristen Grauman, Andrew Westbury, Lorenzo Torresani, Kris Kitani, Jitendra Malik, Triantafyllos Afouras, Kumar Ashutosh, Vijay Baiyya, Siddhant Bansal, Bikram Boote, et al. Ego-exo4d: Understanding skilled human activity from first-and third-person perspectives.arXiv preprint ...

    2023 arXiv

  23. [31]

    Handal: A dataset of real-world manipulable object cate- gories with pose annotations, affordances, and reconstruc- tions

    Andrew Guo, Bowen Wen, Jianhe Yuan, Jonathan Trem- blay, Stephen Tyree, Jeffrey Smith, and Stan Birchfield. Handal: A dataset of real-world manipulable object cate- gories with pose annotations, affordances, and reconstruc- tions. In2023 IEEE/RSJ International Conference on In...

    2023

  24. [32]

    Honnotate: A method for 3d annotation of hand and object poses

    Shreyas Hampali, Mahdi Rad, Markus Oberweger, and Vin- cent Lepetit. Honnotate: A method for 3d annotation of hand and object poses. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020. 3

    2020

  25. [33]

    Ego3dt: Tracking every 3d object in ego-centric videos

    Shengyu Hao, Wenhao Chai, Zhonghan Zhao, Meiqi Sun, Wendi Hu, Jieyang Zhou, Yixian Zhao, Qi Li, Yizhou Wang, Xi Li, and Gaoang Wang. Ego3dt: Tracking every 3d object in ego-centric videos. InProceedings of the 32nd ACM International Conference on Multimedia, 2024. 3

    2024

  26. [34]

    Onepose++: Keypoint-free one- shot object pose estimation without cad models

    Xingyi He, Jiaming Sun, Yuang Wang, Di Huang, Hujun Bao, and Xiaowei Zhou. Onepose++: Keypoint-free one- shot object pose estimation without cad models. InAd- vances in Neural Information Processing Systems, 2022. 3

    2022

  27. [35]

    Pvn3d: A deep point-wise 3d keypoints voting network for 6dof pose estimation

    Yisheng He, Wei Sun, Haibin Huang, Jianran Liu, Hao- qiang Fan, and Jian Sun. Pvn3d: A deep point-wise 3d keypoints voting network for 6dof pose estimation. InPro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020. 3

    2020

  28. [36]

    Fs6d: Few-shot 6d pose estimation of novel objects

    Yisheng He, Yao Wang, Haoqiang Fan, Jian Sun, and Qifeng Chen. Fs6d: Few-shot 6d pose estimation of novel objects. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022. 3

    2022

  29. [37]

    The curious case of neural text degeneration

    Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. The curious case of neural text degeneration. In 8th International Conference on Learning Representations, ICLR, 2020. 7

    2020

  30. [38]

    3d-llm: Inject- ing the 3d world into large language models

    Yining Hong, Haoyu Zhen, Peihao Chen, Shuhong Zheng, Yilun Du, Zhenfang Chen, and Chuang Gan. 3d-llm: Inject- ing the 3d world into large language models. InAdvances in Neural Information Processing Systems, 2023. 3

    2023

  31. [39]

    Lita: Language instructed temporal-localization assistant

    De-An Huang, Shijia Liao, Subhashree Radhakrishnan, Hongxu Yin, Pavlo Molchanov, Zhiding Yu, and Jan Kautz. Lita: Language instructed temporal-localization assistant. arXiv preprint arXiv:2403.19046, 2024. 3, 5

    2024 arXiv

  32. [40]

    V oxposer: Composable 3d value maps for robotic manipulation with language models

    Wenlong Huang, Chen Wang, Ruohan Zhang, Yunzhu Li, Jiajun Wu, and Li Fei-Fei. V oxposer: Composable 3d value maps for robotic manipulation with language models. In 7th Annual Conference on Robot Learning, 2023. 2

    2023

  33. [41]

    Technical report for ego4d long term action antici- pation challenge 2023.arXiv preprint arXiv:2307.01467,

    Tatsuya Ishibashi, Kosuke Ono, Noriyuki Kugo, and Yuji Sato. Technical report for ego4d long term action antici- pation challenge 2023.arXiv preprint arXiv:2307.01467,

    2023 arXiv

  34. [42]

    Jacobs, Michael I

    Robert A. Jacobs, Michael I. Jordan, Steven J. Nowlan, and Geoffrey E. Hinton. Adaptive mixtures of local experts. Neural Computation, 3(1):79–87, 1991. 5

    1991

  35. [43]

    Jordan and R.A

    M.I. Jordan and R.A. Jacobs. Hierarchical mixtures of ex- perts and the em algorithm. InProceedings of 1993 Inter- national Conference on Neural Networks (IJCNN), 1993. 5

    1993

  36. [44]

    Open- vla: An open-source vision-language-action model.arXiv preprint arXiv:2406.09246, 2024

    Moo Jin Kim, Karl Pertsch, Siddharth Karamcheti, Ted Xiao, Ashwin Balakrishna, Suraj Nair, Rafael Rafailov, Ethan Foster, Grace Lam, Pannag Sanketi, et al. Open- vla: An open-source vision-language-action model.arXiv preprint arXiv:2406.09246, 2024. 2, 3, 5

    2024 arXiv

  37. [45]

    Koppula and Ashutosh Saxena

    Hema S. Koppula and Ashutosh Saxena. Anticipating hu- man activities using object affordances for reactive robotic response.IEEE Transactions on Pattern Analysis and Ma- chine Intelligence, 38(1):14–29, 2016. 2

    2016

  38. [46]

    H2o: Two hands manipulating objects for first person interaction recognition

    Taein Kwon, Bugra Tekin, Jan St ¨uhmer, Federica Bogo, and Marc Pollefeys. H2o: Two hands manipulating objects for first person interaction recognition. InProceedings of the IEEE/CVF International Conference on Computer Vi- sion (ICCV), 2021. 2, 3, 6

    2021

  39. [47]

    Choy, Philip H

    Namhoon Lee, Wongun Choi, Paul Vernaza, Christopher B. Choy, Philip H. S. Torr, and Manmohan Chandraker. De- sire: Distant future prediction in dynamic scenes with inter- acting agents. InProceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017. 6

    2017

  40. [48]

    Locate: Localize and transfer object parts for weakly supervised affordance grounding

    Gen Li, Varun Jampani, Deqing Sun, and Laura Sevilla- Lara. Locate: Localize and transfer object parts for weakly supervised affordance grounding. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023. 2

    2023

  41. [49]

    Learning precise affordances from ego- centric videos for robotic manipulation.arxiv preprint arXiv:2408.10123, 2024

    Gen Li, Nikolaos Tsagkas, Jifei Song, Ruaridh Mon- Williams, Sethu Vijayakumar, Kun Shao, and Laura Sevilla-Lara. Learning precise affordances from ego- centric videos for robotic manipulation.arxiv preprint arXiv:2408.10123, 2024. 2

    2024

  42. [50]

    BLIP-2: Bootstrapping language-image pre-training with frozen image encoders and large language models

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. BLIP-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. InPro- ceedings of the 40th International Conference on Machine Learning, 2023. 3, 5, 6, 8

    2023

  43. [51]

    Yin Li, Miao Liu, and James M. Rehg. In the eye of be- holder: Joint learning of gaze and actions in first person video. InProceedings of the European Conference on Com- puter Vision (ECCV), 2018. 3

    2018

  44. [52]

    Deepim: Deep iterative matching for 6d pose estimation

    Yi Li, Gu Wang, Xiangyang Ji, Yu Xiang, and Dieter Fox. Deepim: Deep iterative matching for 6d pose estimation. InProceedings of the European Conference on Computer Vision (ECCV), 2018. 3

    2018

  45. [53]

    Egocentric predic- tion of action target in 3d

    Yiming Li, Ziang Cao, Andrew Liang, Benjamin Liang, Lu- oyao Chen, Hang Zhao, and Chen Feng. Egocentric predic- tion of action target in 3d. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022. 2, 6

    2022

  46. [54]

    Vila: On pre-training for vi- sual language models

    Ji Lin, Hongxu Yin, Wei Ping, Pavlo Molchanov, Moham- mad Shoeybi, and Song Han. Vila: On pre-training for vi- sual language models. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024. 5, 6

    2024

  47. [55]

    Visual instruction tuning

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. InAdvances in Neural In- formation Processing Systems, 2023. 3

    2023

  48. [56]

    Improved baselines with visual instruction tuning

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. InPro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024. 3

    2024

  49. [57]

    Miao Liu, Siyu Tang, Yin Li, and James M. Rehg. Fore- casting human-object interaction: Joint prediction of motor attention and actions in first person video. InProceedings of the European Conference on Computer Vision (ECCV),

  50. [58]

    Joint hand motion and interaction hotspots prediction from egocentric videos

    Shaowei Liu, Subarna Tripathi, Somdeb Majumdar, and Xi- aolong Wang. Joint hand motion and interaction hotspots prediction from egocentric videos. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022. 2, 3

    2022

  51. [59]

    Grounding dino: Marrying dino with grounded pre-training for open-set object detection.arXiv preprint arXiv:2303.05499, 2023

    Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Chunyuan Li, Jianwei Yang, Hang Su, Jun Zhu, et al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection.arXiv preprint arXiv:2303.05499, 2023. 1

    2023 arXiv

  52. [60]

    Hoi4d: A 4d egocentric dataset for category-level human-object interaction

    Yunze Liu, Yun Liu, Che Jiang, Kangbo Lyu, Weikang Wan, Hao Shen, Boqiang Liang, Zhoujie Fu, He Wang, and Li Yi. Hoi4d: A 4d egocentric dataset for category-level human-object interaction. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVP...

    2022

  53. [61]

    Decoupled weight decay regularization

    Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In7th International Conference on Learning Representations, ICLR, 2019. 3

    2019

  54. [62]

    Phrase-based affordance detection via cyclic bi- lateral interaction.IEEE Transactions on Artificial Intelli- gence, 4(5):1186–1198, 2023

    Liangsheng Lu, Wei Zhai, Hongchen Luo, Yu Kang, and Yang Cao. Phrase-based affordance detection via cyclic bi- lateral interaction.IEEE Transactions on Artificial Intelli- gence, 4(5):1186–1198, 2023. 2

    2023

  55. [63]

    Madiff: Motion-aware mamba diffusion models for hand trajectory prediction on egocentric videos

    Junyi Ma, Xieyuanli Chen, Wentao Bao, Jingyi Xu, and Hesheng Wang. Madiff: Motion-aware mamba diffusion models for hand trajectory prediction on egocentric videos. arXiv preprint arXiv:2409.02638, 2024. 2

    2024

  56. [64]

    Diff-ip2d: Diffusion-based hand-object interac- tion prediction on egocentric videos.arXiv preprint arXiv:2405.04370, 2024

    Junyi Ma, Jingyi Xu, Xieyuanli Chen, and Hesheng Wang. Diff-ip2d: Diffusion-based hand-object interac- tion prediction on egocentric videos.arXiv preprint arXiv:2405.04370, 2024. 2

    2024

  57. [65]

    Quest 3, 2023

    Meta. Quest 3, 2023. 1, 4

    2023

  58. [66]

    Leveraging the present to anticipate the future in videos

    Antoine Miech, Ivan Laptev, Josef Sivic, Heng Wang, Lorenzo Torresani, and Du Tran. Leveraging the present to anticipate the future in videos. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, 2019. 2

    2019

  59. [67]

    Rt-affordance: Affordances are versatile intermedi- ate representations for robot manipulation.arXiv preprint arXiv:2411.02704, 2024

    Soroush Nasiriany, Sean Kirmani, Tianli Ding, Laura Smith, Yuke Zhu, Danny Driess, Dorsa Sadigh, and Ted Xiao. Rt-affordance: Affordances are versatile intermedi- ate representations for robot manipulation.arXiv preprint arXiv:2411.02704, 2024. 2

    2024 arXiv

  60. [68]

    Open-vocabulary affordance detection in 3d point clouds

    Toan Nguyen, Minh Nhat Vu, An Vuong, Dzung Nguyen, Thieu V o, Ngan Le, and Anh Nguyen. Open-vocabulary affordance detection in 3d point clouds. In2023 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2023. 2

    2023

  61. [69]

    Bleu: a method for automatic evaluation of machine translation

    Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation of machine translation. InProceedings of the 40th Annual Meeting of the Association for Computational Linguistics, 2002. 8

    2002

  62. [70]

    Colored point cloud registration revisited

    Jaesik Park, Qian-Yi Zhou, and Vladlen Koltun. Colored point cloud registration revisited. In2017 IEEE Interna- tional Conference on Computer Vision (ICCV), 2017. 4

    2017

  63. [71]

    Pix2pose: Pixel-wise coordinate regression of objects for 6d pose es- timation

    Kiru Park, Timothy Patten, and Markus Vincze. Pix2pose: Pixel-wise coordinate regression of objects for 6d pose es- timation. InProceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2019. 3

    2019

  64. [72]

    GloVe: Global vectors for word representation

    Jeffrey Pennington, Richard Socher, and Christopher Man- ning. GloVe: Global vectors for word representation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2014. 8

    2014

  65. [73]

    Detecting activities of daily living in first-person camera views

    Hamed Pirsiavash and Deva Ramanan. Detecting activities of daily living in first-person camera views. InProceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2012. 2

    2012

  66. [74]

    Spatial cognition from egocentric video: Out of sight, not out of mind.arXiv preprint arXiv:2404.05072, 2024

    Chiara Plizzari, Shubham Goel, Toby Perrett, Jacob Chalk, Angjoo Kanazawa, and Dima Damen. Spatial cognition from egocentric video: Out of sight, not out of mind.arXiv preprint arXiv:2404.05072, 2024. 3

    2024 arXiv

  67. [75]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. InProceedings of the...

  68. [76]

    Grounded sam: Assembling open-world models for diverse visual tasks.arXiv preprint arXiv:2401.14159,

    Tianhe Ren, Shilong Liu, Ailing Zeng, Jing Lin, Kunchang Li, He Cao, Jiayu Chen, Xinyu Huang, Yukang Chen, Feng Yan, et al. Grounded sam: Assembling open-world models for diverse visual tasks.arXiv preprint arXiv:2401.14159,

    arXiv

  69. [77]

    Action scene graphs for long-form understanding of egocentric videos

    Ivan Rodin, Antonino Furnari, Kyle Min, Subarna Tripathi, and Giovanni Maria Farinella. Action scene graphs for long-form understanding of egocentric videos. InProceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024. 3

    2024

  70. [78]

    Fast point feature histograms (fpfh) for 3d registration

    Radu Bogdan Rusu, Nico Blodow, and Michael Beetz. Fast point feature histograms (fpfh) for 3d registration. In IEEE International Conference on Robotics and Automa- tion (ICRA), 2009. 4

    2009

  71. [79]

    What object should i use? - task driven object detection

    Johann Sawatzky, Yaser Souri, Christian Grund, and Jur- gen Gall. What object should i use? - task driven object detection. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019. 2

    2019

  72. [80]

    As- sembly101: A large-scale multi-view video dataset for un- derstanding procedural activities

    Fadime Sener, Dibyadip Chatterjee, Daniel Shelepov, Kun He, Dipika Singhania, Robert Wang, and Angela Yao. As- sembly101: A large-scale multi-view video dataset for un- derstanding procedural activities. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern R...

    2022

  73. [81]

    Dandan Shan, Jiaqi Geng, Michelle Shu, and David F. Fouhey. Understanding human hands in contact at inter- net scale. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020. 1, 2

    2020

  74. [82]

    Ego4d goal-step: Toward hierarchical understanding of procedural activities

    Yale Song, Eugene Byrne, Tushar Nagarajan, Huiyu Wang, Miguel Martin, and Lorenzo Torresani. Ego4d goal-step: Toward hierarchical understanding of procedural activities. InAdvances in Neural Information Processing Systems,

  75. [83]

    Onepose: One-shot object pose estimation without cad models

    Jiaming Sun, Zihao Wang, Siyu Zhang, Xingyi He, Hongcheng Zhao, Guofeng Zhang, and Xiaowei Zhou. Onepose: One-shot object pose estimation without cad models. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022. 3

    2022

  76. [84]

    Yoshiyuki Tanaka. Robot-aided rehabilitation methodol- ogy for enhancing movement smoothness by using a hu- man hand trajectory generation model with task-related constraints.Journal of Human-Robot Interaction, 4(3): 101–119, 2015. 2

    2015

  77. [85]

    Minigpt-3d: Efficiently aligning 3d point clouds with large language models using 2d priors

    Yuan Tang, Xu Han, Xianzhi Li, Qiao Yu, Yixue Hao, Long Hu, and Min Chen. Minigpt-3d: Efficiently aligning 3d point clouds with large language models using 2d priors. arXiv preprint arXiv:2405.01413, 2024. 3, 5, 6

    2024 arXiv

  78. [86]

    Leveraging next-active ob- jects for context-aware anticipation in egocentric videos

    Sanket Thakur, Cigdem Beyan, Pietro Morerio, Vittorio Murino, and Alessio Del Bue. Leveraging next-active ob- jects for context-aware anticipation in egocentric videos. In Proceedings of the IEEE/CVF Winter Conference on Appli- cations of Computer Vision (WACV), 2024. 2

    2024

  79. [87]

    Llama: Open and efficient foundation language mod- els.arXiv preprint arXiv:2302.13971, 2023

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timoth ´ee Lacroix, Bap- tiste Rozi `ere, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language mod- els.arXiv preprint arXiv:2302.13971, 2023. 5

    2023 arXiv

  80. [88]

    Epic fields: Marrying 3d geometry and video un- derstanding

    Vadim Tschernezki, Ahmad Darkhalil, Zhifan Zhu, David Fouhey, Iro Laina, Diane Larlus, Dima Damen, and Andrea Vedaldi. Epic fields: Marrying 3d geometry and video un- derstanding. InAdvances in Neural Information Processing Systems, 2023. 3

    2023

  81. [89]

    Attention is all you need

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. Attention is all you need. InAdvances in Neural Information Processing Systems, 2017. 5, 6, 3

    2017

  82. [90]

    Omnivid: A gener- ative framework for universal video understanding

    Junke Wang, Dongdong Chen, Chong Luo, Bo He, Lu Yuan, Zuxuan Wu, and Yu-Gang Jiang. Omnivid: A gener- ative framework for universal video understanding. InPro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024. 3, 5

    2024

  83. [91]

    OFA: Unifying architectures, tasks, and modalities through a simple sequence-to-sequence learning framework

    Peng Wang, An Yang, Rui Men, Junyang Lin, Shuai Bai, Zhikang Li, Jianxin Ma, Chang Zhou, Jingren Zhou, and Hongxia Yang. OFA: Unifying architectures, tasks, and modalities through a simple sequence-to-sequence learning framework. InProceedings of the 39th International Con- fe...

    2022

  84. [92]

    Dust3r: Geometric 3d vision made easy

    Shuzhe Wang, Vincent Leroy, Yohann Cabon, Boris Chidlovskii, and Jerome Revaud. Dust3r: Geometric 3d vision made easy. InProceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition (CVPR),

  85. [93]

    Holoassist: an egocentric human interaction dataset for interactive ai assistants in the real world

    Xin Wang, Taein Kwon, Mahdi Rad, Bowen Pan, Ishani Chakraborty, Sean Andrist, Dan Bohus, Ashley Feniello, Bugra Tekin, Felipe Vieira Frujeri, Neel Joshi, and Marc Pollefeys. Holoassist: an egocentric human interaction dataset for interactive ai assistants in the real world. In...

    2023

  86. [94]

    Foundationpose: Unified 6d pose estimation and tracking of novel objects

    Bowen Wen, Wei Yang, Jan Kautz, and Stan Birchfield. Foundationpose: Unified 6d pose estimation and tracking of novel objects. InProceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition (CVPR),

  87. [95]

    Learning descriptors for object recognition and 3d pose estimation

    Paul Wohlhart and Vincent Lepetit. Learning descriptors for object recognition and 3d pose estimation. InProceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2015. 6

    2015

  88. [96]

    Spatialtracker: Tracking any 2d pixels in 3d space

    Yuxi Xiao, Qianqian Wang, Shangzhan Zhang, Nan Xue, Sida Peng, Yujun Shen, and Xiaowei Zhou. Spatialtracker: Tracking any 2d pixels in 3d space. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024. 2, 3

    2024

  89. [97]

    Pointllm: Empowering large language models to understand point clouds.arXiv preprint arXiv:2308.16911, 2023

    Runsen Xu, Xiaolong Wang, Tai Wang, Yilun Chen, Jiang- miao Pang, and Dahua Lin. Pointllm: Empowering large language models to understand point clouds.arXiv preprint arXiv:2308.16911, 2023. 3, 5, 6, 7, 8

    2023 arXiv

  90. [98]

    Ulip- 2: Towards scalable multimodal pre-training for 3d under- standing

    Le Xue, Ning Yu, Shu Zhang, Artemis Panagopoulou, Jun- nan Li, Roberto Mart´ın-Mart´ın, Jiajun Wu, Caiming Xiong, Ran Xu, Juan Carlos Niebles, and Silvio Savarese. Ulip- 2: Towards scalable multimodal pre-training for 3d under- standing. InProceedings of the IEEE/CVF Conferenc...

    2024

  91. [99]

    Active object detection with knowledge aggregation and distillation from large models

    Dejie Yang and Yang Liu. Active object detection with knowledge aggregation and distillation from large models. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024. 2

    2024

  92. [100]

    Depth anything: Un- leashing the power of large-scale unlabeled data.arXiv preprint arXiv:2401.10891, 2024

    Lihe Yang, Bingyi Kang, Zilong Huang, Xiaogang Xu, Ji- ashi Feng, and Hengshuang Zhao. Depth anything: Un- leashing the power of large-scale unlabeled data.arXiv preprint arXiv:2401.10891, 2024. 3, 4

    2024 arXiv

  93. [101]

    What’s in your hands? 3d reconstruction of generic objects in hands

    Yufei Ye, Abhinav Gupta, and Shubham Tulsiani. What’s in your hands? 3d reconstruction of generic objects in hands. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022. 2

    2022

  94. [102]

    Point-bert: Pre-training 3d point cloud transformers with masked point modeling

    Xumin Yu, Lulu Tang, Yongming Rao, Tiejun Huang, Jie Zhou, and Jiwen Lu. Point-bert: Pre-training 3d point cloud transformers with masked point modeling. InProceedings of the IEEE/CVF Conference on Computer Vision and Pat- tern Recognition (CVPR), 2022. 5

    2022

  95. [103]

    Video-LLaMA: An instruction-tuned audio-visual language model for video understanding

    Hang Zhang, Xin Li, and Lidong Bing. Video-LLaMA: An instruction-tuned audio-visual language model for video understanding. InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing: Sys- tem Demonstrations, 2023. 3

    2023

  96. [104]

    Hoidiffusion: Generating real- istic 3d hand-object interaction data

    Mengqi Zhang, Yang Fu, Zheng Ding, Sifei Liu, Zhuowen Tu, and Xiaolong Wang. Hoidiffusion: Generating real- istic 3d hand-object interaction data. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024. 2

    2024

  97. [105]

    Opt: Open pre-trained transformer language models.arXiv preprint arXiv:2205.01068, 2022

    Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. Opt: Open pre-trained transformer language models.arXiv preprint arXiv:2205.01068, 2022. 5

    2022 arXiv

  98. [106]

    Pear: Phrase-based hand-object interaction anticipa- tion.arXiv preprint arXiv:2407.21510, 2024

    Zichen Zhang, Hongchen Luo, Wei Zhai, Yang Cao, and Yu Kang. Pear: Phrase-based hand-object interaction anticipa- tion.arXiv preprint arXiv:2407.21510, 2024. 2

    2024 arXiv

  99. [107]

    Zhao, Vikash Kumar, Sergey Levine, and Chelsea Finn

    Tony Z. Zhao, Vikash Kumar, Sergey Levine, and Chelsea Finn. Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware. InProceedings of Robotics: Science and Systems, 2023. 1

    2023

  100. [108]

    Judging llm-as-a-judge with mt-bench and chatbot arena

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, Hao Zhang, Joseph E Gonzalez, and Ion Stoica. Judging llm-as-a-judge with mt-bench and chatbot arena. InAdvances in Neural Information Process- ing...

    2023

  101. [109]

    On the continuity of rotation representations in neural networks

    Yi Zhou, Connelly Barnes, Jingwan Lu, Jimei Yang, and Hao Li. On the continuity of rotation representations in neural networks. InProceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition (CVPR),

  102. [110]

    3d-vista: Pre-trained transformer for 3d vision and text alignment

    Ziyu Zhu, Xiaojian Ma, Yixin Chen, Zhidong Deng, Siyuan Huang, and Qing Li. 3d-vista: Pre-trained transformer for 3d vision and text alignment. InProceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2023. 3

    2023

  103. [111]

    grab a cup

    Brianna Zitkovich, Tianhe Yu, Sichun Xu, Peng Xu, Ted Xiao, Fei Xia, Jialin Wu, Paul Wohlhart, Stefan Welker, Wahid, et al. Rt-2: Vision-language-action models transfer web knowledge to robotic control. InProceedings of The 7th Conference on Robot Learning, 2023. 2, 3, 5 Gener...

    2023

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.