pith. sign in

arxiv: 2605.18214 · v1 · pith:U67BHLJ2new · submitted 2026-05-18 · 💻 cs.CV

EgoInteract: Synthetic Egocentric Videos Generation for Interaction Understanding and Anticipation

Rosario Leonardi , Francesco Ragusa , Daniele Materia , Alessandro Passanisi , James Fort , Jakob Engel , Giovanni Maria Farinella This is my paper

Pith reviewed 2026-05-20 10:57 UTC · model grok-4.3

classification 💻 cs.CV
keywords egocentric videosynthetic datahuman-object interactionaction segmentationinteraction anticipationvideo simulatortransfer learningdense annotation
0
0 comments X

The pith

A controllable simulator generates synthetic egocentric videos with dense annotations that improve models for interaction understanding and anticipation on real benchmarks.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper introduces EgoInteract, a simulator that produces synthetic egocentric videos by allowing precise control over camera motion, human body and hand movements, object manipulations, and scene setups in varied environments. This framework creates a dataset equipped with automatic dense spatial and temporal labels for four key tasks: temporal action segmentation, next-active object detection, interaction anticipation, and hand-object interaction detection. Models trained on the synthetic data, alone or combined with real footage, are then evaluated on multiple existing real-world egocentric video collections that differ in settings, objects, and interaction styles. The results indicate steady gains over established baselines, pointing to simulation as a practical route around the high cost, privacy limits, and coverage gaps of real data gathering. A sympathetic reader would care because this offers a scalable path to stronger first-person perception systems for robotics, augmented reality, and assistive technologies.

Core claim

EgoInteract is a controllable simulator for egocentric video generation designed to model fine-grained human-object interactions and their temporal dynamics. It enables exact control over camera, human body and hand motion, object manipulation, and scene composition across diverse environments. The authors generate a synthetic egocentric video dataset with dense spatial and temporal annotations for temporal action segmentation, next-active object detection, interaction anticipation, and hand-object interaction detection. Models trained with this simulated data are evaluated on multiple real-world egocentric benchmarks spanning diverse environments, object categories, and interaction patterns

What carries the argument

The EgoInteract simulator, which supplies controllable parameters for camera, body, hands, objects, and environments to produce temporally coherent egocentric interactions together with automatic dense annotations.

If this is right

  • Training for egocentric tasks can proceed with far less reliance on slow and expensive real video collection.
  • The generated annotations supply error-free ground truth for spatial and temporal labels across all four tasks.
  • The same simulation pipeline transfers effectively across varied real environments and object sets.
  • Interaction anticipation and hand-object detection models become easier to improve through repeated synthetic data generation.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The simulator could be extended to generate rare or safety-critical interaction sequences that real recordings rarely capture.
  • Mixed training that blends synthetic and real clips might further close remaining domain differences.
  • Similar controllable simulation could support downstream applications such as robot learning from first-person demonstrations.

Load-bearing premise

The synthetic videos and annotations match the statistical distributions, motion patterns, and interaction variations of real egocentric data closely enough that models trained on them generalize to actual benchmarks without large domain gaps.

What would settle it

Run the same models on a fresh real-world egocentric dataset recorded in a previously unseen environment with novel objects and interaction sequences; the performance gains over real-data-only baselines would disappear or reverse if the central claim fails.

Figures

Figures reproduced from arXiv: 2605.18214 by Alessandro Passanisi, Daniele Materia, Francesco Ragusa, Giovanni Maria Farinella, Jakob Engel, James Fort, Rosario Leonardi.

Figure 1
Figure 1. Figure 1: EgoInteract generates temporally coherent videos of humans interacting with diverse objects, enabling the study of egocentric interaction understanding at multiple levels. generation of hand-object interactions [24], or on diffusion-based egocentric video synthesis methods that emphasize visual realism and world modeling rather than task-oriented supervision [37, 17]. However, restricting simulation to sta… view at source ↗
Figure 2
Figure 2. Figure 2: The EgoInteract simulator. The generated egocentric videos are automatically labeled with several spatial and temporal annotations. 3.1 Overview and Episode Definition In EgoInteract, interactions are organized as episodes, where each episode corresponds to a complete first-person interaction sequence centered on a single target object ( [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Examples of HM3D environments used in EgoInteract. Blue regions denote the navigable surfaces used for agent placement and motion, while purple regions indicate support surfaces where objects can be placed. A Technical appendices This supplementary material provides additional technical details, implementation specifics, ablation results, visual examples, and qualitative results of our study that complemen… view at source ↗
Figure 4
Figure 4. Figure 4: Examples of procedurally generated tabletop scenes in [PITH_FULL_IMAGE:figures/full_fig_p016_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Examples of avatar appearance randomization in [PITH_FULL_IMAGE:figures/full_fig_p017_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Examples of full-body inverse kinematics in [PITH_FULL_IMAGE:figures/full_fig_p017_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Examples of the collision hand proxy used for grasp generation in [PITH_FULL_IMAGE:figures/full_fig_p018_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Sequence of full-body poses generated during interaction execution in [PITH_FULL_IMAGE:figures/full_fig_p018_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Examples of interaction episodes generated by [PITH_FULL_IMAGE:figures/full_fig_p019_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Example of interaction episodes generated by [PITH_FULL_IMAGE:figures/full_fig_p019_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: Qualitative TAS predictions on representative test videos from [PITH_FULL_IMAGE:figures/full_fig_p020_11.png] view at source ↗
Figure 12
Figure 12. Figure 12: Qualitative comparison between models trained with real data only ( [PITH_FULL_IMAGE:figures/full_fig_p021_12.png] view at source ↗
Figure 13
Figure 13. Figure 13: Qualitative comparison between models trained with real data only ( [PITH_FULL_IMAGE:figures/full_fig_p022_13.png] view at source ↗
Figure 14
Figure 14. Figure 14: Qualitative examples of synthetic finetuning improving interaction anticipation. For each [PITH_FULL_IMAGE:figures/full_fig_p023_14.png] view at source ↗
read the original abstract

Collecting large-scale egocentric video datasets with dense spatial and temporal annotations is costly, slow, and often constrained by environmental biases, privacy constraints, and limited coverage of interaction patterns. While synthetic data has shown strong potential in several vision domains, its use for egocentric perception remains relatively underexplored, especially for tasks requiring temporally coherent human-object interactions. In this work, we introduce EgoInteract, a controllable simulator for egocentric video generation designed to model fine-grained egocentric interactions and their temporal dynamics. The simulator enables precise control over camera, human body and hand motion, object manipulation, and scene composition across diverse environments. Building on this framework, we generate a synthetic egocentric video dataset with dense spatial and temporal annotations for temporal action segmentation, next-active object detection, interaction anticipation, and hand-object interaction detection. We evaluate models trained with simulated data on multiple real-world egocentric benchmarks spanning diverse environments, object categories, and interaction patterns. Results show consistent improvements over strong baselines across tasks and datasets, demonstrating the effectiveness and transferability of our simulation-based approach.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 2 minor

Summary. The paper introduces EgoInteract, a controllable simulator for generating synthetic egocentric videos that model fine-grained human-object interactions and temporal dynamics. It produces a large annotated dataset for temporal action segmentation, next-active object detection, interaction anticipation, and hand-object interaction detection. Models trained exclusively on this synthetic data are evaluated on multiple real-world egocentric benchmarks and reported to yield consistent improvements over strong baselines, supporting the claim of effective simulation-to-real transfer.

Significance. If the reported transfer holds after rigorous validation of domain gaps and experimental controls, the work could meaningfully address data scarcity, annotation costs, and environmental biases in egocentric vision. Controllable synthetic generation for temporally coherent interactions is a promising direction that could accelerate progress on anticipation and interaction tasks where real data collection is particularly constrained.

major comments (2)
  1. [§4] §4 (Experiments): The central transfer claim rests on consistent gains across tasks and datasets, yet the manuscript provides insufficient detail on whether synthetic data is used exclusively or mixed with real data, the exact training protocols, and statistical significance testing of the improvements. This information is load-bearing for distinguishing simulator effectiveness from other factors.
  2. [§3] §3 (Simulator Design): The description of how the simulator ensures statistical match in motion patterns, interaction variations, and scene composition to real egocentric data lacks quantitative comparisons (e.g., distribution distances or motion statistics). Without this, the weakest assumption—that synthetic videos sufficiently approximate real distributions—remains unverified and directly affects generalizability claims.
minor comments (2)
  1. The abstract and introduction would benefit from explicit dataset statistics (number of videos, total frames, diversity of environments and objects) to allow readers to assess scale and coverage.
  2. Figure captions and method diagrams should more clearly label the controllable parameters (camera pose, hand articulation, object affordances) to improve reproducibility.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the detailed and constructive review of our manuscript. We address each major comment below and have revised the paper to provide greater clarity and supporting analyses where feasible.

read point-by-point responses

  1. Referee: [§4] §4 (Experiments): The central transfer claim rests on consistent gains across tasks and datasets, yet the manuscript provides insufficient detail on whether synthetic data is used exclusively or mixed with real data, the exact training protocols, and statistical significance testing of the improvements. This information is load-bearing for distinguishing simulator effectiveness from other factors.

    Authors: We thank the referee for highlighting this. The original manuscript states that models were trained exclusively on synthetic data (see abstract and Section 4). To address the request for additional detail, the revised version expands Section 4 with a full description of the training protocols, including optimizer settings, learning rate schedules, batch sizes, data augmentation, and network initializations. We have also added results from multiple independent runs with statistical significance testing via paired t-tests, confirming that the observed improvements are significant at p < 0.05 for the primary metrics across tasks. revision: yes

  2. Referee: [§3] §3 (Simulator Design): The description of how the simulator ensures statistical match in motion patterns, interaction variations, and scene composition to real egocentric data lacks quantitative comparisons (e.g., distribution distances or motion statistics). Without this, the weakest assumption—that synthetic videos sufficiently approximate real distributions—remains unverified and directly affects generalizability claims.

    Authors: We agree that explicit quantitative validation would further support the claims. In the revised Section 3 we now include direct comparisons of motion statistics (e.g., distributions of hand velocities, grasp durations, and camera motion magnitudes) and scene composition metrics (object category frequencies and interaction type histograms) between EgoInteract and real datasets such as EPIC-KITCHENS. Full high-dimensional distribution distances such as Fréchet Video Distance were not computed in the original work because of computational cost and because downstream task performance on multiple real benchmarks already serves as the primary evidence of transfer; we view the added statistics as a useful but partial strengthening of the argument. revision: partial

Circularity Check

0 steps flagged

No significant circularity detected

full rationale

The paper presents a simulator for generating synthetic egocentric videos with dense annotations, followed by training models on this data and evaluating performance on multiple external real-world benchmarks. The central claim rests on empirical transfer gains rather than any internal derivation, equation, or self-referential fit. No load-bearing steps reduce predictions to quantities defined or fitted within the paper itself; the evaluation uses independent real datasets, making the chain self-contained against external benchmarks.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 0 invented entities

Review based on abstract only; no explicit free parameters, axioms, or invented entities are detailed beyond the high-level claim that the simulator enables effective transfer.

axioms (1)
  • domain assumption Synthetic data with precise motion and scene control can approximate real egocentric interaction distributions sufficiently for model transfer.
    This premise underpins the claim that training on generated data yields improvements on real benchmarks.

pith-pipeline@v0.9.0 · 5733 in / 1333 out tokens · 45923 ms · 2026-05-20T10:57:24.090631+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Lean theorems connected to this paper

Citations machine-checked in the Pith Canon. Every link opens the source theorem in the public Lean library.

  • IndisputableMonolith/Foundation/RealityFromDistinction.lean reality_from_one_distinction unclear
    ?
    unclear

    Relation between the paper passage and the cited Recognition theorem.

    We introduce EgoInteract, a Unity-based simulator for the generation of egocentric interaction data. EgoInteract enables the generation of first-person hand-object interaction episodes within diverse 3D environments, providing fine-grained control over agents, objects, camera behavior, and interaction parameters.

  • IndisputableMonolith/Cost/FunctionalEquation.lean washburn_uniqueness_aczel unclear
    ?
    unclear

    Relation between the paper passage and the cited Recognition theorem.

    The simulator automatically produces dense spatial annotations, including bounding boxes and semantic segmentation masks for both objects and hands as well as temporal annotations by assigning action labels with explicit start and end times.

What do these tags mean?
matches
The paper's claim is directly supported by a theorem in the formal canon.
supports
The theorem supports part of the paper's argument, but the paper may add assumptions or extra steps.
extends
The paper goes beyond the formal theorem; the theorem is a base layer rather than the whole result.
uses
The paper appears to rely on the theorem as machinery.
contradicts
The paper's claim conflicts with a theorem or certificate in the canon.
unclear
Pith found a possible connection, but the passage is too broad, indirect, or ambiguous to say the theorem truly supports the claim.

Reference graph

Works this paper leans on

55 extracted references · 55 canonical work pages · 4 internal anchors

  1. [1]

    LLaVA-OneVision-1.5: Fully Open Framework for Democratized Multimodal Training

    Xiang An, Yin Xie, Kaicheng Yang, Wenkang Zhang, Xiuwei Zhao, Zheng Cheng, Yirui Wang, Songcen Xu, Changrui Chen, Didi Zhu, Chunsheng Wu, Huajie Tan, Chunyuan Li, Jing Yang, Jie Yu, Xiyao Wang, Bin Qin, Yumeng Wang, Zizhen Yan, Ziyong Feng, Ziwei Liu, Bo Li, and Jiankang Deng. Llava-onevision-1.5: Fully open framework for democratized multimodal training,...

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  2. [2]

    Scenescript: Reconstructing scenes with an autoregressive structured language model.arXiv preprint arXiv:2403.13064, 2024

    Armen Avetisyan, Christopher Xie, Henry Howard-Jenkins, Tsun-Yi Yang, Samir Aroudj, Suvam Patra, Fuyang Zhang, Duncan Frost, Luke Holland, Campbell Orme, et al. Scenescript: Reconstructing scenes with an autoregressive structured language model.arXiv preprint arXiv:2403.13064, 2024

    work page arXiv 2024

  3. [3]

    Towards a richer 2d understanding of hands at scale

    Tianyi Cheng, Dandan Shan, Ayda Sultan Hassen, Richard Ely Locke Higgins, and David Fouhey. Towards a richer 2d understanding of hands at scale. InThirty-seventh Conference on Neural Information Processing Systems, 2023

    work page 2023

  4. [4]

    Scaling egocentric vision: The epic-kitchens dataset

    Dima Damen, Hazel Doughty, Giovanni Maria Farinella, Sanja Fidler, Antonino Furnari, Evangelos Kazakos, Davide Moltisanti, Jonathan Munro, Toby Perrett, Will Price, et al. Scaling egocentric vision: The epic-kitchens dataset. InECCV, pages 720–736, 2018

    work page 2018

  5. [5]

    Rescaling egocentric vision: Collection, pipeline and challenges for epic-kitchens-100.IJCV, pages 1–23, 2021

    Dima Damen, Hazel Doughty, Giovanni Maria Farinella, , Antonino Furnari, Jian Ma, Evangelos Kazakos, Davide Moltisanti, Jonathan Munro, Toby Perrett, Will Price, and Michael Wray. Rescaling egocentric vision: Collection, pipeline and challenges for epic-kitchens-100.IJCV, pages 1–23, 2021

    work page 2021

  6. [6]

    Epic-kitchens visor benchmark: Video segmentations and object relations

    Ahmad Darkhalil, Dandan Shan, Bin Zhu, Jian Ma, Amlan Kar, Richard Higgins, Sanja Fidler, David Fouhey, and Dima Damen. Epic-kitchens visor benchmark: Video segmentations and object relations. InNeurIPS, pages 13745–13758, 2022

    work page 2022

  7. [7]

    Objaverse-XL: A Universe of 10M+ 3D Objects

    Matt Deitke, Ruoshi Liu, Matthew Wallingford, Huong Ngo, Oscar Michel, Aditya Kusupati, Alan Fan, Christian Laforte, Vikram V oleti, Samir Yitzhak Gadre, Eli VanderBilt, Aniruddha Kembhavi, Carl V ondrick, Georgia Gkioxari, Kiana Ehsani, Ludwig Schmidt, and Ali Farhadi. Objaverse-xl: A universe of 10m+ 3d objects.arXiv preprint arXiv:2307.05663, 2023

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  8. [8]

    CARLA: An open urban driving simulator

    Alexey Dosovitskiy, German Ros, Felipe Codevilla, Antonio Lopez, and Vladlen Koltun. CARLA: An open urban driving simulator. InProceedings of the 1st Annual Conference on Robot Learning, pages 1–16, 2017

    work page 2017

  9. [9]

    Everingham, L

    M. Everingham, L. Van Gool, C. K. I. Williams, J. Winn, and A. Zisserman. The PASCAL Visual Object Classes Challenge 2010 (VOC2010) Results.International Journal of Computer Vision, 88(2):303–338, 2010. URL https://www.robots.ox.ac.uk/~vgg/projects/pascal/ VOC/pubs/everingham10.html. 10

    work page 2010

  10. [10]

    Motsynth: How can synthetic data help pedestrian detection and tracking? InICCV, 2021

    Matteo Fabbri, Guillem Brasó, Gianluca Maugeri, Aljoša Ošep, Riccardo Gasparini, Orcun Cetintas, Simone Calderara, Laura Leal-Taixé, and Rita Cucchiara. Motsynth: How can synthetic data help pedestrian detection and tracking? InICCV, 2021

    work page 2021

  11. [11]

    Next- active-object prediction from egocentric videos.Journal of Visual Communication and Image Representation, 49:401–411, November 2017

    Antonino Furnari, Sebastiano Battiato, Kristen Grauman, and Giovanni Maria Farinella. Next- active-object prediction from egocentric videos.Journal of Visual Communication and Image Representation, 49:401–411, November 2017. ISSN 1047-3203

    work page 2017

  12. [12]

    Personal-location-based temporal segmentation of egocentric video for lifelogging applications.Journal of Visual Communication and Image Representation, 52:1–12, 2018

    Antonino Furnari, Sebastiano Battiato, and Giovanni Maria Farinella. Personal-location-based temporal segmentation of egocentric video for lifelogging applications.Journal of Visual Communication and Image Representation, 52:1–12, 2018. ISSN 1047-3203. doi: https: //doi.org/10.1016/j.jvcir.2018.01.019

    work page doi:10.1016/j.jvcir.2018.01.019 2018

  13. [13]

    Domain-adversarial training of neural networks

    Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario March, and Victor Lempitsky. Domain-adversarial training of neural networks. Journal of Machine Learning Research, 17(59):1–35, 2016. URLhttp://jmlr.org/papers/ v17/15-239.html

    work page 2016

  14. [14]

    Chavis, Antonino Furnari, Rohit Girdhar, Jackson Hamburger, Hao Jiang, Miao Liu, Xingyu Liu, Miguel Martin, Tushar Nagarajan, Ilija Radosavovic, Santhosh K

    Kristen Grauman, Andrew Westbury, Eugene Byrne, Zachary Q. Chavis, Antonino Furnari, Rohit Girdhar, Jackson Hamburger, Hao Jiang, Miao Liu, Xingyu Liu, Miguel Martin, Tushar Nagarajan, Ilija Radosavovic, Santhosh K. Ramakrishnan, Fiona Ryan, Jayant Sharma, Michael Wray, Mengmeng Xu, Eric Z. Xu, Chen Zhao, Siddhant Bansal, Dhruv Batra, Vincent Cartillier, ...

    work page 2021

  15. [15]

    Kristen Grauman, Andrew Westbury, Lorenzo Torresani, Kris Kitani, Jitendra Malik, Tri- antafyllos Afouras, Kumar Ashutosh, Vijay Baiyya, Siddhant Bansal, Bikram Boote, Eugene Byrne, Zach Chavis, Joya Chen, Feng Cheng, Fu-Jen Chu, Sean Crane, Avijit Dasgupta, Jing Dong, Maria Escobar, Cristhian Forigua, Abrham Gebreselasie, Sanjay Haresh, Jing Huang, Md Mo...

    work page doi:10.1007/s11263-025-02557-6 2025

  16. [16]

    Maisi: Medical ai for synthetic imaging

    Pengfei Guo, Can Zhao, Dong Yang, Ziyue Xu, Vishwesh Nath, Yucheng Tang, Benjamin Simon, Mason Belue, Stephanie Harmon, Baris Turkbey, and Daguang Xu. Maisi: Medical ai for synthetic imaging. InProceedings of the Winter Conference on Applications of Computer Vision (WACV), pages 4430–4441, February 2025. 11

    work page 2025

  17. [17]

    Egosim: Egocentric world simulator for embodied interaction generation.arXiv preprint arXiv:2604.01001, 2026

    Jinkun Hao, Mingda Jia, Ruiyan Wang, Xihui Liu, Ran Yi, Lizhuang Ma, Jiangmiao Pang, and Xudong Xu. Egosim: Egocentric world simulator for embodied interaction generation.arXiv preprint arXiv:2604.01001, 2026

    work page arXiv 2026

  18. [18]

    LoRA: Low-Rank Adaptation of Large Language Models

    Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models, 2021. URL https://arxiv.org/abs/2106.09685. Accessed 24 April 2026

    work page internal anchor Pith review Pith/arXiv arXiv 2021

  19. [19]

    Pointrend: Image segmentation as rendering

    Alexander Kirillov, Yuxin Wu, Kaiming He, and Ross Girshick. Pointrend: Image segmentation as rendering. InCVPR, pages 9799–9808, 2020

    work page 2020

  20. [20]

    AI2-THOR: An Interactive 3D Environment for Visual AI.arXiv, 2017

    Eric Kolve, Roozbeh Mottaghi, Winson Han, Eli VanderBilt, Luca Weihs, Alvaro Herrasti, Daniel Gordon, Yuke Zhu, Abhinav Gupta, and Ali Farhadi. AI2-THOR: An Interactive 3D Environment for Visual AI.arXiv, 2017

    work page 2017

  21. [21]

    Temporal convolutional networks for action segmentation and detection

    Colin Lea, Michael D Flynn, Rene Vidal, Austin Reiter, and Gregory D Hager. Temporal convolutional networks for action segmentation and detection. Inproceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 156–165, 2017

    work page 2017

  22. [22]

    Egocen- tric human-object interaction detection exploiting synthetic data

    Rosario Leonardi, Francesco Ragusa, Antonino Furnari, and Giovanni Maria Farinella. Egocen- tric human-object interaction detection exploiting synthetic data. InInternational Conference on Image Analysis and Processing, pages 237–248. Springer, 2022

    work page 2022

  23. [23]

    Exploit- ing multimodal synthetic data for egocentric human-object interaction detection in an industrial scenario.Computer Vision and Image Understanding, 242:103984, 2024

    Rosario Leonardi, Francesco Ragusa, Antonino Furnari, and Giovanni Maria Farinella. Exploit- ing multimodal synthetic data for egocentric human-object interaction detection in an industrial scenario.Computer Vision and Image Understanding, 242:103984, 2024

    work page 2024

  24. [24]

    Are synthetic data useful for egocentric hand-object interaction detection? InEuropean Conference on Computer Vision, pages 36–54

    Rosario Leonardi, Antonino Furnari, Francesco Ragusa, and Giovanni Maria Farinella. Are synthetic data useful for egocentric hand-object interaction detection? InEuropean Conference on Computer Vision, pages 36–54. Springer, 2025

    work page 2025

  25. [25]

    Llava-onevision: Easy visual task transfer,

    Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Ziwei Liu, and Chunyuan Li. Llava-onevision: Easy visual task transfer,

    work page

  26. [26]

    LLaVA-OneVision: Easy Visual Task Transfer

    URLhttps://arxiv.org/abs/2408.03326. Accessed: 24 April 2026

    work page internal anchor Pith review Pith/arXiv arXiv 2026

  27. [27]

    Egogen: An egocentric synthetic data generator

    Gen Li, Kaifeng Zhao, Siwei Zhang, Xiaozhong Lyu, Mihai Dusmanu, Yan Zhang, Marc Pollefeys, and Siyu Tang. Egogen: An egocentric synthetic data generator. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14497–14509, 2024

    work page 2024

  28. [28]

    Ms-tcn++: Multi- stage temporal convolutional network for action segmentation.IEEE transactions on pattern analysis and machine intelligence, 2020

    Shi-Jie Li, Yazan AbuFarha, Yun Liu, Ming-Ming Cheng, and Juergen Gall. Ms-tcn++: Multi- stage temporal convolutional network for action segmentation.IEEE transactions on pattern analysis and machine intelligence, 2020

    work page 2020

  29. [29]

    Cross-domain adaptive teacher for object detection

    Yu-Jhe Li, Xiaoliang Dai, Chih-Yao Ma, Yen-Cheng Liu, Kan Chen, Bichen Wu, Zijian He, Kris Kitani, and Peter Vajda. Cross-domain adaptive teacher for object detection. InCVPR, pages 7581–7590, 2022

    work page 2022

  30. [30]

    Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C

    Tsung-Yi Lin, Michael Maire, Serge J. Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft coco: Common objects in context. InECCV, pages 740–755, 2014

    work page 2014

  31. [31]

    Forecasting human-object interaction: Joint prediction of motor attention and actions in first person video

    Miao Liu, Siyu Tang, Yin Li, et al. Forecasting human-object interaction: Joint prediction of motor attention and actions in first person video. InECCV, pages 704–721, 2020

    work page 2020

  32. [32]

    Fact: Frame-action cross-attention temporal modeling for efficient action segmentation

    Zijia Lu and Ehsan Elhamifar. Fact: Frame-action cross-attention temporal modeling for efficient action segmentation. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18175–18185, 2024

    work page 2024

  33. [33]

    V olumetric hierarchical approximate convex decom- position.Game engine gems, 3:141–158, 2016

    Khaled Mamou, E Lengyel, and A Peters. V olumetric hierarchical approximate convex decom- position.Game engine gems, 3:141–158, 2016

    work page 2016

  34. [34]

    Habitat: A Platform for Embodied AI Research

    Manolis Savva*, Abhishek Kadian*, Oleksandr Maksymets*, Yili Zhao, Erik Wijmans, Bhavana Jain, Julian Straub, Jia Liu, Vladlen Koltun, Jitendra Malik, Devi Parikh, and Dhruv Batra. Habitat: A Platform for Embodied AI Research. InICCV, 2019. 12

    work page 2019

  35. [35]

    Leveraging gaze and set- of-mark in vllms for human-object interaction anticipation from egocentric videos

    Daniele Materia, Francesco Ragusa, and Giovanni Maria Farinella. Leveraging gaze and set- of-mark in vllms for human-object interaction anticipation from egocentric videos. InICPR, 2026

    work page 2026

  36. [36]

    Grounded human-object interaction hotspots from video

    Tushar Nagarajan, Christoph Feichtenhofer, and Kristen Grauman. Grounded human-object interaction hotspots from video. InICCV, pages 8687–8696, 2019

    work page 2019

  37. [37]

    Nvidia isaac sim, 2021.https://developer.nvidia.com/isaac-sim

    NVIDIA. Nvidia isaac sim, 2021.https://developer.nvidia.com/isaac-sim

    work page 2021

  38. [38]

    Egocontrol: Controllable egocentric video generation via 3d full-body poses

    Enrico Pallotta, Sina Mokhtarzadeh Azar, Lars Doorenbos, Serdar Ozsoy, Umar Iqbal, and Juergen Gall. Egocontrol: Controllable egocentric video generation via 3d full-body poses. arXiv preprint arXiv:2511.18173, 2025

    work page arXiv 2025

  39. [39]

    Georgios Pavlakos, Vasileios Choutas, Nima Ghorbani, Timo Bolkart, Ahmed A. A. Osman, Dimitrios Tzionas, and Michael J. Black. Expressive body capture: 3D hands, face, and body from a single image. InProceedings IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), pages 10975–10985, 2019

    work page 2019

  40. [40]

    Hd-epic: A highly-detailed egocentric video dataset

    Toby Perrett, Ahmad Darkhalil, Saptarshi Sinha, Omar Emara, Sam Pollard, Kranti Parida, Kaiting Liu, Prajwal Gatti, Siddhant Bansal, Kevin Flanagan, Jacob Chalk, Zhifan Zhu, Rhodri Guerrier, Fahd Abdelazim, Bin Zhu, Davide Moltisanti, Michael Wray, Hazel Doughty, and Dima Damen. Hd-epic: A highly-detailed egocentric video dataset. InCVPR, pages 23901– 239...

    work page 2025

  41. [41]

    The meccano dataset: Understanding human-object interactions from egocentric videos in an industrial-like domain

    Francesco Ragusa, Antonino Furnari, Salvatore Livatino, and Giovanni Maria Farinella. The meccano dataset: Understanding human-object interactions from egocentric videos in an industrial-like domain. InWinter Conference on Applications of Computer Vision, pages 1569–1578, 2021

    work page 2021

  42. [42]

    Meccano: A multimodal egocentric dataset for humans behavior understanding in the industrial-like domain.Comput

    Francesco Ragusa, Antonino Furnari, and Giovanni Maria Farinella. Meccano: A multimodal egocentric dataset for humans behavior understanding in the industrial-like domain.Comput. Vis. Image Underst., 235(C), October 2023. ISSN 1077-3142. doi: 10.1016/j.cviu.2023.103764. URLhttps://doi.org/10.1016/j.cviu.2023.103764

    work page doi:10.1016/j.cviu.2023.103764 2023

  43. [43]

    Enigma-51: Towards a fine-grained under- standing of human behavior in industrial scenarios

    Francesco Ragusa, Rosario Leonardi, Michele Mazzamuto, Claudia Bonanno, Rosario Scavo, Antonino Furnari, and Giovanni Maria Farinella. Enigma-51: Towards a fine-grained under- standing of human behavior in industrial scenarios. InProceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 4549–4559, 2024

    work page 2024

  44. [44]

    Enigma-360: An ego-exo dataset for human behavior understanding in industrial scenarios

    Francesco Ragusa, Rosario Leonardi, Michele Mazzamuto, Daniele Di Mauro, Camillo Quat- trocchi, Alessandro Passanisi, Irene D’Ambra, Antonino Furnari, and Giovanni Maria Farinella. Enigma-360: An ego-exo dataset for human behavior understanding in industrial scenarios. arXiv preprint arXiv:2603.09741, 2026

    work page arXiv 2026

  45. [45]

    Rockstar advanced game engine (rage)

    Rockstar Games. Rockstar advanced game engine (rage). https://www.rockstargames. com/, Accessed 2024. Proprietary game engine developed by Rockstar Games

    work page 2024

  46. [46]

    Assembly101: A large-scale multi-view video dataset for understanding procedural activities

    Fadime Sener, Dibyadip Chatterjee, Daniel Shelepov, Kun He, Dipika Singhania, Robert Wang, and Angela Yao. Assembly101: A large-scale multi-view video dataset for understanding procedural activities. InCVPR, pages 21096–21106, 2022

    work page 2022

  47. [47]

    Dandan Shan, Jiaqi Geng, Michelle Shu, and David F. Fouhey. Understanding human hands in contact at internet scale. InCVPR, pages 9869–9878, 2020

    work page 2020

  48. [48]

    Coarse to fine multi-resolution temporal convolutional network.arXiv preprint arXiv:2105.10859, 2021

    Dipika Singhania, Rahul Rahaman, and Angela Yao. Coarse to fine multi-resolution temporal convolutional network.arXiv preprint arXiv:2105.10859, 2021

    work page arXiv 2021

  49. [49]

    Habitat 2.0: Training home assistants to rearrange their habitat.Advances in Neural Information Processing Systems, 34:251–266, 2021

    Andrew Szot, Alexander Clegg, Eric Undersander, Erik Wijmans, Yili Zhao, John Turner, Noah Maestre, Mustafa Mukadam, Devendra Singh Chaplot, Oleksandr Maksymets, et al. Habitat 2.0: Training home assistants to rearrange their habitat.Advances in Neural Information Processing Systems, 34:251–266, 2021. 13

    work page 2021

  50. [50]

    Action recognition in rgb-d egocentric videos

    Yansong Tang, Yi Tian, Jiwen Lu, Jianjiang Feng, and Jie Zhou. Action recognition in rgb-d egocentric videos. In2017 IEEE International Conference on Image Processing (ICIP), pages 3410–3414. IEEE, 2017

    work page 2017

  51. [51]

    Zamir, Zhi-Yang He, Alexander Sax, Jitendra Malik, and Silvio Savarese

    Fei Xia, Amir R. Zamir, Zhi-Yang He, Alexander Sax, Jitendra Malik, and Silvio Savarese. Gibson env: real-world perception for embodied agents. InCVPR, 2018

    work page 2018

  52. [52]

    Asformer: Transformer for action segmentation

    Fangqiu Yi, Hongyu Wen, and Tingting Jiang. Asformer: Transformer for action segmentation. InThe British Machine Vision Conference (BMVC), 2021

    work page 2021

  53. [53]

    Fine-grained egocentric hand- object segmentation: Dataset, model, and applications

    Lingzhi Zhang, Shenghao Zhou, Simon Stent, and Jianbo Shi. Fine-grained egocentric hand- object segmentation: Dataset, model, and applications. InECCV, pages 127–145, 2022

    work page 2022

  54. [54]

    Deep future gaze: Gaze anticipation on egocentric videos using adversarial networks

    Mengmi Zhang, Keng Teck Ma, et al. Deep future gaze: Gaze anticipation on egocentric videos using adversarial networks. InCVPR, pages 3539–3548, 2017. 14 Figure 3: Examples of HM3D environments used inEgoInteract. Blue regions denote the navigable surfaces used for agent placement and motion, while purple regions indicate support surfaces where objects ca...

    work page 2017

  55. [55]

    19 Figure 11: Qualitative TAS predictions on representative test videos fromEPIC-KITCHENS(top) andEgo-Exo4D(bottom)

    "The Agent.“ VQA Figure 10: Example of interaction episodes generated byEgoInteractwith the set of temporal and spatial annotations obtained automatically. 19 Figure 11: Qualitative TAS predictions on representative test videos fromEPIC-KITCHENS(top) andEgo-Exo4D(bottom). Red segments indicateTakeactions, while blue segments indicateRelease actions. A.2.1...

    work page