Pith. sign in

REVIEW 3 major objections 6 minor 121 references

CoMind: Understanding Collaborative Human Activity from Multiple Minds and Views

T0 review · 3 major / 6 minor · reviewed 2026-07-10 · grok-4.5

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read CoMind is a multimodal cooking-collaboration dataset and three Theory-of-Mind vision tasks that show current vision-language models lack social grounding, while fine-tuning on the data substantially closes the gap.

desk verdict Strong multi-agent ego-exo cooking resource with social-cue labels and three hard prediction tasks; the ToM framing is marketing, not a load-bearing flaw. read the letter →

arxiv 2607.06691 v1 pith:Q2VH56CY submitted 2026-07-07 cs.CV

Alexey Gavryushin , Dingxi Zhang , Zhao Huang , Alexandros Delitzas , Jiaqi Chen , Ben Ellis , Cedric Zöllner , Manthan Patel
show 3 more authors
Manuel Kaufmann Marc Pollefeys Xi Wang
This is my paper · ORCID
classification cs.CV
keywords egocentricvisiontheoryofmindhuman-humancollaborationjointattentionactionanticipationobjecthandovermultimodaldatasetsocialcues
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that Theory of Mind—the ability to infer others’ intentions from social cues—is essential for real collaboration yet almost unmeasurable in natural settings because prior datasets and benchmarks stay textual, single-agent, or short-horizon. CoMind fills that gap with 41 hours of synchronized egocentric and exocentric video, audio, gaze, hand tracking, and 3D scans of people cooking together, plus dense annotations of shared attention, social cues, and object interactions. Three escalating tasks are defined: estimating joint attention on an object, anticipating a helper’s next socially conditioned action, and predicting a handover before any reach begins. Zero-shot vision-language models score near zero on spatial localization and only modestly on cue and category recognition; fine-tuning open-weight models on CoMind training splits produces large gains, especially on action verbs and initiator identity. The claim is that this resource is now the practical foundation for training and evaluating AI that can read social signals and assist proactively in shared physical work.

What carries the argument

Three interdependent vision tasks that operationalize Theory of Mind for physical collaboration: Joint Attention Estimation (shared object, dual-view boxes, cue type), Socially Conditioned Object Interaction Anticipation (helper’s next verb-noun-box given leader cues), and Collaborative Handover Prediction (time-to-handover, delivery flow, initiator, cue, object box before any reach).

What would settle it

Train models to high accuracy on the three CoMind tasks, then test whether the same models correctly predict a held-out partner’s next need or successful assistance in a new, unscripted kitchen session whose social-cue distribution differs from the training kitchens; failure of transfer would falsify the claim that the tasks capture general collaborative ToM.

Watch

Extended reading notes

Core claim

Current state-of-the-art vision-language models exhibit severe deficiencies on three new social-reasoning tasks grounded in real dual-person cooking collaboration—particularly near-zero accuracy on bounding-box localization of jointly attended or soon-to-be-handed objects—while fine-tuning the same open-weight models on CoMind’s training split yields large, consistent lifts (for example action-verb accuracy rising from roughly 0.09–0.14 to 0.64–0.65), establishing the dataset as a usable training and evaluation foundation for socially aware AI.

Load-bearing premise

That success on these three hand-crafted vision tasks is a valid stand-in for Theory of Mind in real collaboration, without independent cognitive or behavioral validation that the tasks measure mental-state inference rather than pattern matching of surface cues.

Editorial extensions

If this is right

  • Multimodal perception systems can be trained to detect joint attention and social cues from synchronized first- and third-person video plus gaze and speech.
  • Proactive assistive agents can be scored on whether they correctly anticipate a partner’s next object interaction or handover before physical motion begins.
  • Collaborative planning models gain temporally aligned, 3D-grounded training data for long-horizon kitchen tasks that include verbal and gestural intent.
  • Open-weight vision-language models can be domain-adapted for social grounding, turning near-zero spatial scores into competitive ones after fine-tuning on CoMind.
  • Future 3D extensions become feasible by lifting the existing 2D boxes into the provided scene and object scans for embodied spatial reasoning.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the tasks fix the helper as the agent that must react to the leader, the same protocol can be reused to train robot helpers that must act without explicit commands.
  • The large zero-shot gap on bounding boxes versus moderate category accuracy suggests current VLMs already parse linguistic intent but lack the cross-view geometric binding needed for embodied collaboration.
  • Gaze-following and body-pose pseudo-labels already reconstructible from the dual views and Aria MPS data could bootstrap denser social-cue supervision without new manual annotation.
  • If the three tasks truly track ToM, performance curves on CoMind should correlate with independent ToM battery scores of the same models on classic false-belief or intention-inference tests.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. CoMind presents a multimodal ego–exocentric dataset of unscripted two-person cooking collaboration (80 sessions, ~41 h dual-view / ~81 h single-view, 125 participants) with synchronized Aria egocentric video, dual GoPro exocentric views, gaze and hand tracking, audio/transcripts, camera trajectories, dense kitchen scans, and scanned object meshes. The authors define three vision benchmarks intended to operationalize Theory of Mind in physical collaboration—Joint Attention Estimation, Socially Conditioned Object Interaction Anticipation, and Collaborative Handover Prediction—with manual annotations for shared objects, social cue types, verbs/nouns, delivery flow, initiator, and time-to-handover. They evaluate multiple closed- and open-weight VLMs under a shared prompting protocol with participant-disjoint train/test splits, report near-zero spatial grounding for most proprietary models, and show large gains after LoRA fine-tuning of Qwen3-VL (e.g., action-verb accuracy rising from ~0.09–0.14 to ~0.64–0.65).

Significance. If the resource and benchmarks hold as described, CoMind fills a clear gap relative to prior ego/exo and multi-agent datasets (Table 1): long-horizon goal-directed physical collaboration with gaze, verbal/gestural cue labels, and 3D scene/object grounding. The participant-disjoint evaluation and the documented fine-tuning gains provide concrete evidence that the training split is useful for socially conditioned perception, not only as a zero-shot stress test. The three tasks are well-specified for proactive assistance research even if one disputes the strongest ToM rhetoric. Public release of data and benchmarks is a genuine contribution to multimodal and embodied social AI.

major comments (3)
  1. Abstract, Introduction, and §3.1 frame the three vision tasks as a formalization of Theory of Mind for physical collaboration, yet success is defined solely by matching annotated boxes, verbs/nouns, cue types, flow, initiator, and TTH. No human study, correlation with established ToM instruments (those cited in Related Work), or ablation that isolates mental-state inference from surface multimodal pattern matching is provided. The empirical VLM numbers remain valid as a social-perception benchmark; the stronger claim that CoMind advances ToM-capable AI should be tempered to “social cue–conditioned collaborative perception,” or supported by an explicit validation argument.
  2. §3.4 describes multi-stage human annotation with QA and author review, but the manuscript reports no inter-annotator agreement (e.g., IoU agreement on boxes, Cohen/Fleiss κ on cue type, initiator, delivery flow, or verb/noun labels). For a dataset paper whose central value is the annotations (Tables 2–4, Figs. 13–15, Supp. annotation guides), IAA (or at least double-annotation on a held-out subset) is load-bearing for trusting the reported metrics and the fine-tuning gains.
  3. §4 evaluates only general VLMs (plus random/most-frequent priors). For Joint Attention Estimation and handover timing/localization, the literature already has specialized gaze, mutual-attention, and action-anticipation models (cited in §2). Without at least one non-VLM or modular baseline (e.g., gaze-intersection + object detector, or a standard anticipation transformer), it is hard to separate “VLMs fail at social grounding” from “the tasks are hard for any current method.” Adding such baselines would strengthen the claim of a significant performance deficiency.
minor comments (6)
  1. Abstract vs. body: the abstract lists “Action Anticipation” while §3.1 and Tables use “Socially Conditioned Object Interaction Anticipation”; align naming throughout.
  2. §4: only 5 uniformly sampled frames from the 10 s context are fed to VLMs. An ablation on frame count / video input (beyond the partial Supp. Table S6) would clarify whether spatial failures are partly an input bottleneck.
  3. Table 4 TTH metric uses a tight ±0.25 s window; report also mean absolute error or a coarser bin so temporal performance is easier to interpret.
  4. Object hierarchy (L1–L3) and synonym handling for Cat. (L1) are important for reproducibility; point more explicitly from the main text to Supp. §S5 / Fig. S9.
  5. Fig. 1 / Table 1: “81” hours ego vs. “40h 43m” single-view wording can confuse; state dual-view vs. single-view totals once in a consistent way.
  6. Minor polish: arXiv id and some model version strings (Claude Opus 4.5/4.6, GPT 5.x) will age quickly—cite system cards with access dates as already partly done in references.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: empirical dataset and VLM benchmarks with held-out evaluation; no fitted parameters re-labeled as predictions and no self-referential derivation.

full rationale

CoMind is a resource paper that collects multimodal cooking collaboration data, defines three annotation-driven vision tasks (Joint Attention Estimation, Socially Conditioned Object Interaction Anticipation, Collaborative Handover Prediction), and reports zero-shot and fine-tuned VLM numbers on a participant-disjoint test split. Performance metrics (IoU@0.5, cue-type accuracy, verb/noun matches, TTH, etc.) are obtained by comparing model outputs against human annotations; fine-tuning gains are measured on the same held-out set. There are no equations that define a quantity in terms of itself, no parameters fitted to data and then re-presented as independent predictions, and no uniqueness theorems or ansatzes imported via self-citation that force the central claims. The framing that the three tasks operationalize Theory of Mind is an interpretive claim about task design, not a circular derivation. The empirical results are therefore self-contained against external models and held-out data.

Assumptions & free parameters 3 free parameters · 3 assumptions · 2 invented entities

As a dataset paper the central claims rest on the fidelity of capture, the correctness of human annotations, and the claim that the three tasks measure socially conditioned reasoning. No free parameters are fitted to produce a scientific constant; the free choices are design decisions of the benchmark.

free parameters (3)
  • context window length (10 s)
    Fixed by authors for all three tasks; not derived from data but chosen for VLM context limits and annotation practicality.
  • IoU threshold 0.5 and TTH tolerance 0.25 s
    Standard but arbitrary evaluation thresholds that directly affect reported scores.
  • LoRA rank r=64, alpha=128, lr=2e-4, epochs=3–5
    Hyper-parameters of the fine-tuning experiments that produce the largest reported gains.
assumptions (3)
  • ad hoc to paper Success on the three vision tasks is a valid proxy for Theory-of-Mind ability in physical collaboration.
    Stated in Introduction and §3.1; no independent cognitive or behavioral validation is supplied.
  • domain assumption Cooking in real kitchens with unscripted pairs yields representative collaborative social cues.
    Underpins the claim of ecological validity; participants are young adults (18–38) in 55 kitchens.
  • domain assumption Human annotations of joint attention, cue type and handover initiator are sufficiently reliable after QA review.
    No quantitative inter-annotator agreement is reported; reliability is asserted via multi-stage review.
invented entities (2)
  • Socially Conditioned Object Interaction Anticipation task
    purpose: Operationalize intent inference from social cues into a structured prediction problem (verb, noun, box, cue type).
    New task definition introduced by the paper; no prior benchmark uses exactly this formulation.
  • Collaborative Handover Prediction task (pre-reach TTH + initiator + flow)
    purpose: Measure proactive physical coordination before any reaching motion.
    Distinct from prior kinematic handover datasets that annotate the transfer itself rather than its cognitive precursors.

how reviews work

0 comments
Cite this review

Pith. "Pith review of CoMind: Understanding Collaborative Human Activity from Multiple Minds and Views." pith.science (2026). https://pith.science/paper/Q2VH56CY

@misc{pith2026260706691,
  author       = {Pith},
  title        = {Pith review of: CoMind: Understanding Collaborative Human Activity from Multiple Minds and Views},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/Q2VH56CY}},
  note         = {Machine review of arXiv:2607.06691}
}
read the original abstract

Human-human collaboration is a fundamental aspect of everyday life, essential to success in a wide range of goal-directed activities from household tasks to professional teamwork. While much research has focused on modeling coordination and task execution, the cognitive processes that support such collaboration, particularly Theory of Mind (the ability to infer the mental states of others), remain difficult to study in natural settings. To address this gap, we introduce a novel egocentric and exocentric video dataset capturing real-world collaboration in cooking scenarios. The dataset integrates multi-perspective video, high-quality audio, gaze tracking, and 3D scene and object scans, with annotations for shared attention to objects, social cues and interactions between agents, as well as agent-object interactions. We establish benchmarks for Joint Attention Estimation, Socially Conditioned Object Interaction Anticipation, and Collaborative Handover Prediction, enabling research on multimodal perception, proactive assistance, and collaborative planning. By providing temporally aligned, richly annotated multimodal data, CoMind facilitates the development and evaluation of AI systems capable of modeling complex social interactions and reasoning about human behaviors in collaborative environments. Our dataset and benchmarks are made available at https://comind.ethz.ch/.

Figures

Figures reproduced from arXiv: 2607.06691 by the authors.

Figure 1
Figure 1. The CoMind dataset captures 41 hours of human-human collaboration across 80 sessions, combining egocentric/exocentric videos, audio, gaze, and high-quality 3D scene and object scans, as well as rich annotations for three novel social reasoning tasks. Abstract. Human-human collaboration is a fundamental aspect of ev￾eryday life, essential to success in a wide range of goal-directed activities from household tasks to … view at source ↗
Figure 2
Figure 2. Joint Attention Estimation Task. From the audio or transcribed speech in a context win￾dow, together with a target frame (green), the model predicts the jointly attended object, its bounding box in the left and right views, and the social cue type (blue). "Can you please wash the garlic and onion and peel them?" "Yeah, just cut it like that" [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 5
Figure 5. Example anno￾tations. The prediction frame shows the ground￾truth action verb, noun, ob￾ject bounding box, and cue type(s). of the cue eliciting the joint attention on the object. This task measures the capacity to recognize a collaborative agent’s shared mental state. Task 2: Socially Conditioned Object Interaction Anticipation. We define a socially conditioned action of participant i ∈ {ℓ, h} as any action perform… view at source ↗
Figures from the paper (10 more)
Figure 6
Figure 6. Figure 6: Given a prediction frame f before any reaching or grasping motion towards a handed-over object, together with the two synchronized egocentric [PITH_FULL_IMAGE:figures/full_fig_p006_6.png]
Figure 6
Figure 6. Figure 6: Collaborative Handover Prediction Task. From context frames (or video) and audio or transcribed speech, together with a prediction frame (green), the model predicts the delivery flow (who hands to whom), the object category and bounding box in the view of the handing p…
Figure 9
Figure 9. Figure 9: 3D scans of ob￾jects used in the record￾ings. Participants work with objects that have been 3D-scanned. BLK2GO scanner prior to the session. Lastly, we reuse a set of common cooking utensils (pans and pots) for each session, which are scanned using an Artec 3D Leo [6] …
Figure 10
Figure 10. Figure 10: Scan Alignment. To transform the 3D scan of the environment into Aria’s world frame, we follow a two-stage procedure. In the first stage, we localize the BLK2GO images within Aria’s semi-dense point cloud using Hierarchical Localization [72, 73]. Each successfully loc…
Figure 10
Figure 10. Figure 10: Modalities in the CoMind dataset. The dataset provides synchronized multimodal recordings of collaborative sessions, including egocentric video with eye gaze and hand tracking, audio and IMU signals, exocentric camera views, camera trajectories and spatial mapping, de…
Figure 11
Figure 11. Figure 11: Histogram of recording lengths. Our lengthy recordings allow for the study of human-human collaborative work in long-context settings. In total, we record 41 hours of two-person resp. 81 hours of single-view data. 51.2% 48.1% 0.6% Male Female Undisclosed Gender 54.4% …
Figure 13
Figure 13. Figure 13: Action verb and action noun distribution of the Socially Conditioned [PITH_FULL_IMAGE:figures/full_fig_p010_13.png]
Figure 14
Figure 14. Figure 14: Annotation statistics for the Joint Attention Estimation task. [PITH_FULL_IMAGE:figures/full_fig_p011_14.png]
Figure 15
Figure 15. Figure 15: Annotation statistics for the Collaborative Handover Prediction task. [PITH_FULL_IMAGE:figures/full_fig_p011_15.png]
Figure 13
Figure 13. Figure 13: We further provide statistics regarding the object categories for the [PITH_FULL_IMAGE:figures/full_fig_p011_13.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

121 extracted references · 121 canonical work pages

  1. [1]

    In: Proceedings of the IEEE conference on computer vision and pattern recognition

    Abu Farha, Y., Richard, A., Gall, J.: When will you do what?-anticipating temporal occurrences of activities. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 5343–5352 (2018) 16 A. Gavryushin et al

    work page 2018

  2. [2]

    Current directions in psy- chological science26(3), 243–248 (2017)

    Adams Jr, R.B., Albohn, D.N., Kveraga, K.: Social vision: Applying a social- functional approach to face and expression perception. Current directions in psy- chological science26(3), 243–248 (2017)

    work page 2017

  3. [3]

    System card (Nov 2025),https://www

    Anthropic: Claude opus 4.5 system card. System card (Nov 2025),https://www. anthropic.com/claude-opus-4-5-system-card, accessed 2026-03-04

    work page 2025

  4. [4]

    Anthropic News (Feb 2026),https://www

    Anthropic: Introducing claude opus 4.6. Anthropic News (Feb 2026),https://www. anthropic.com/news/claude-opus-4-6, accessed 2026-03-04

    work page 2026

  5. [5]

    theory of mind

    Apperly, I.: Mindreaders: the cognitive basis of" theory of mind". Psychology Press (2010)

    work page 2010

  6. [6]

    com/portable-3d-scanners/artec-leo, accessed: 2026-03-04

    Artec 3D: Artec leo: Wireless professional 3d scanner (2024),https://www.artec3d. com/portable-3d-scanners/artec-leo, accessed: 2026-03-04

    work page 2024

  7. [7]

    Qwen3-VL Technical Report

    Bai, S., Cai, Y., Chen, R., Chen, K., Chen, X., Cheng, Z., Deng, L., Ding, W., Gao, C., Ge, C., et al.: Qwen3-vl technical report. arXiv preprint arXiv:2511.21631 (2025)

    work page Pith review arXiv 2025

  8. [8]

    Bain, M., Huh, J., Han, T., Zisserman, A.: Whisperx: Time-accurate speech tran- scription of long-form audio (2023),https://arxiv.org/abs/2303.00747

    work page Pith review arXiv 2023

Show all 121 references
  1. [9]

    arXiv preprint arXiv:2411.19167 (2024)

    Banerjee, P., Shkodrani, S., Moulon, P., Hampali, S., Han, S., Zhang, F., Zhang, L., Fountain, J., Miller, E., Basol, S., et al.: Hot3d: Hand and object tracking in 3d from egocentric multi-view videos. arXiv preprint arXiv:2411.19167 (2024)

    2024 arXiv

  2. [10]

    In: NeuRIPS Workshop on Gaze Meets ML (2023)

    de Belen, R.A., Mohammadi, G., Sowmya, A.: Temporal understanding of gaze communication with gazetransformer. In: NeuRIPS Workshop on Gaze Meets ML (2023)

    2023

  3. [11]

    arXiv preprint arXiv:2506.01608 (2025)

    Bonnetto, A., Qi, H., Leong, F., Tashkovska, M., Rad, M., Shokur, S., Hummel, F., Micera, S., Pollefeys, M., Mathis, A.: Epfl-smart-kitchen-30: Densely annotated cooking dataset with 3d kinematics to challenge video and language models. arXiv preprint arXiv:2506.01608 (2025)

    2025 arXiv

  4. [12]

    In: 2011 IEEE/RSJ International Conference on Intelligent Robots and Systems

    Cakmak, M., Srinivasa, S.S., Lee, M.K., Forlizzi, J., Kiesler, S.: Human prefer- ences for robot-human hand-over configurations. In: 2011 IEEE/RSJ International Conference on Intelligent Robots and Systems. pp. 1986–1993. IEEE (2011)

    2011

  5. [13]

    arXiv preprint arXiv:2411.00081 (2024)

    Chang, M., Chhablani, G., Clegg, A., Cote, M.D., Desai, R., Hlavac, M., Karashchuk, V., Krantz, J., Mottaghi, R., Parashar, P., et al.: Partnr: A benchmark for planning and reasoning in embodied multi-agent tasks. arXiv preprint arXiv:2411.00081 (2024)

    2024 arXiv

  6. [14]

    In: Advances in Neural Information Processing Systems (NeurIPS) Datasets and Benchmarks Track (2025)

    Chavan, V., Imgrund, Y., Dao, T., Bai, S., Wang, B., Lu, Z., Heimann, O., Krüger, J.: Indego: A dataset of industrial scenarios and collaborative work for egocentric assistants. In: Advances in Neural Information Processing Systems (NeurIPS) Datasets and Benchmarks Track (2025)

    2025

  7. [15]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Chen, J., Mittal, G., Yu, Y., Kong, Y., Chen, M.: Gatehub: Gated history unit with background suppression for online action detection. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 19925– 19934 (2022)

    2022

  8. [16]

    In: Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)

    Chen, Z., Wu, J., Zhou, J., Wen, B., Bi, G., Jiang, G., Cao, Y., Hu, M., Lai, Y., Xiong, Z., et al.: Tombench: Benchmarking theory of mind in large language models. In: Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pape...

    2024

  9. [17]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

    Chong, E., Wang, Y., Ruiz, N., Rehg, J.M.: Detecting attended visual targets in video. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 5396–5406 (2020)

    2020

  10. [18]

    Trends in Cognitive Sciences13(4), 148–153 (2009) CoMind: Understanding Collaborative Human Activity 17

    Csibra, G., Gergely, G.: Natural pedagogy. Trends in Cognitive Sciences13(4), 148–153 (2009) CoMind: Understanding Collaborative Human Activity 17

    2009

  11. [19]

    In: Proceedings of the European conference on computer vision (ECCV)

    Damen, D., Doughty, H., Farinella, G.M., Fidler, S., Furnari, A., Kazakos, E., Moltisanti, D., Munro, J., Perrett, T., Price, W., et al.: Scaling egocentric vision: The epic-kitchens dataset. In: Proceedings of the European conference on computer vision (ECCV). pp. 720–736 (2018)

    2018

  12. [20]

    International Journal of Computer Vision (IJCV)130, 33–55 (2022),https://doi.org/10.1007/s11263- 021-01531-2

    Damen, D., Doughty, H., Farinella, G.M., Furnari, A., Ma, J., Kazakos, E., Molti- santi, D., Munro, J., Perrett, T., Price, W., Wray, M.: Rescaling egocentric vision: Collection, pipeline and challenges for epic-kitchens-100. International Journal of Computer Vision (IJCV)130,...

    2022 doi

  13. [21]

    https://deepmind.google/models/gemma/gemma- 4/ (2026), accessed: July 2026

    DeepMind, G.: Gemma 4. https://deepmind.google/models/gemma/gemma- 4/ (2026), accessed: July 2026

    2026

  14. [22]

    https://blog.google/products-and-platforms/products/gemini/gemini- 3-flash/(Dec 2025), google Blog

    Doshi, T., the Gemini Team: Gemini 3 flash: Frontier intelligence built for speed. https://blog.google/products-and-platforms/products/gemini/gemini- 3-flash/(Dec 2025), google Blog

    2025

  15. [23]

    In: Proceedings of the 27th ACM International Conference on Multimedia

    Dutta, A., Zisserman, A.: The VIA annotation software for images, audio and video. In: Proceedings of the 27th ACM International Conference on Multimedia. MM ’19, ACM, New York, NY, USA (2019)

    2019

  16. [24]

    arXiv preprint arXiv:2308.13561 (2023)

    Engel, J., Somasundaram, K., Goesele, M., Sun, A., Gamino, A., Turner, A., Talattof, A., Yuan, A., Souti, B., Meredith, B., et al.: Project aria: A new tool for egocentric multi-modal ai research. arXiv preprint arXiv:2308.13561 (2023)

    2023 arXiv

  17. [25]

    In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)

    Fan, L., Wang, W., Huang, S., Tang, X., Zhu, S.C.: Understanding human gaze com- munication by spatio-temporal graph reasoning. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV). pp. 5724–5733 (2019)

    2019

  18. [26]

    In: Proceedings of the European Conference on Computer Vision (ECCV) Workshops

    Fathi, A., Li, Y., Rehg, J.M.: Learning to recognize daily actions using gaze. In: Proceedings of the European Conference on Computer Vision (ECCV) Workshops. pp. 314–327. Springer (2012)

    2012

  19. [27]

    In: Proceedings of the Computer Vision and Pattern Recognition Conference

    Fu, R., Zhang, D., Jiang, A., Fu, W., Funk, A., Ritchie, D., Sridhar, S.: Gigahands: A massive annotated dataset of bimanual hand activities. In: Proceedings of the Computer Vision and Pattern Recognition Conference. pp. 17461–17474 (2025)

    2025

  20. [28]

    IEEE transactions on pattern analysis and machine intelligence 43(11), 4021–4036 (2020)

    Furnari, A., Farinella, G.M.: Rolling-unrolling lstms for action anticipation from first-person video. IEEE transactions on pattern analysis and machine intelligence 43(11), 4021–4036 (2020)

    2020

  21. [29]

    In: 2022 26th International Conference on Pattern Recognition (ICPR)

    Furnari, A., Farinella, G.M.: Towards streaming egocentric action anticipation. In: 2022 26th International Conference on Pattern Recognition (ICPR). pp. 1250–1257. IEEE (2022)

    2022

  22. [30]

    arXiv preprint arXiv:1707.04818 (2017)

    Gao, J., Yang, Z., Nevatia, R.: Red: Reinforced encoder-decoder networks for action anticipation. arXiv preprint arXiv:1707.04818 (2017)

    2017 arXiv

  23. [31]

    In: Proceedings of the IEEE/CVF international conference on computer vision

    Girdhar, R., Grauman, K.: Anticipative video transformer. In: Proceedings of the IEEE/CVF international conference on computer vision. pp. 13505–13515 (2021)

    2021

  24. [32]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Grauman, K., Westbury, A., Byrne, E., Chavis, Z., Furnari, A., Girdhar, R., Hamburger, J., Jiang, H., Liu, M., Liu, X., et al.: Ego4d: Around the world in 3,000 hours of egocentric video. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp...

    2022

  25. [33]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Grauman, K., Westbury, A., Torresani, L., Kitani, K., Malik, J., Afouras, T., Ashutosh, K., Baiyya, V., Bansal, S., Boote, B., et al.: Ego-exo4d: Understanding skilled human activity from first-and third-person perspectives. In: Proceedings of the IEEE/CVF Conference on Comput...

    2024

  26. [34]

    Neural Computing and Applications35(2), 2007–2024 (2023) 18 A

    Guo, Z., Hou, Y., Wang, P., Gao, Z., Xu, M., Li, W.: Ft-hid: a large-scale rgb-d dataset for first-and third-person human interaction analysis. Neural Computing and Applications35(2), 2007–2024 (2023) 18 A. Gavryushin et al

    2007

  27. [35]

    In: ICLR (2022)

    Hu, E.J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., Chen, W.: LoRA: Low-rank adaptation of large language models. In: ICLR (2022)

    2022

  28. [36]

    arXiv preprint arXiv:2306.16545 (2023)

    Huang, D., Hilliges, O., Van Gool, L., Wang, X.: Palm: Predicting actions through language models@ ego4d long-term action anticipation challenge 2023. arXiv preprint arXiv:2306.16545 (2023)

    2023 arXiv

  29. [37]

    IEEE Transactions on Human-Machine Systems50(4), 306–316 (2020)

    Huang, Y., Cai, M., Sato, Y.: An ego-vision system for discovering human joint attention. IEEE Transactions on Human-Machine Systems50(4), 306–316 (2020)

    2020

  30. [38]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (2024)

    Huang, Y., Chen, G., Xu, J., Zhang, M., Yang, L., Pei, B., Zhang, H., Lu, D., Wang, Y., Wang, L., Qiao, Y.: Egoexolearn: A dataset for bridging asynchronous ego- and exo-centric view of procedural activities in real world. In: Proceedings of the IEEE/CVF Conference on Computer...

    2024

  31. [39]

    In: Proceedings of the IEEE/CVF International Conference on Computer Vision Workshops

    Jang, Y., Sullivan, B., Ludwig, C., Gilchrist, I., Damen, D., Mayol-Cuevas, W.: Epic-tent: An egocentric video dataset for camping tent assembly. In: Proceedings of the IEEE/CVF International Conference on Computer Vision Workshops. pp. 0–0 (2019)

    2019

  32. [40]

    In: European Conference on Computer Vision

    Jia, B., Chen, Y., Huang, S., Zhu, Y., Zhu, S.C.: Lemma: A multi-view dataset for le arning m ulti-agent m ulti-task a ctivities. In: European Conference on Computer Vision. pp. 767–786. Springer (2020)

    2020

  33. [41]

    arXiv preprint arXiv:2305.16487 (2023)

    Khirodkar, R., Bansal, A., Ma, L., Newcombe, R., Vo, M., Kitani, K.: Egohumans: An egocentric 3d multi-human benchmark. arXiv preprint arXiv:2305.16487 (2023)

    2023 arXiv

  34. [42]

    arXiv preprint arXiv:2503.22152 (2025)

    Li, Y., Veerabadran, V., Iuzzolino, M.L., Roads, B.D., Celikyilmaz, A., Ridgeway, K.: Egotom: Benchmarking theory of mind reasoning from egocentric videos. arXiv preprint arXiv:2503.22152 (2025)

    2025 arXiv

  35. [43]

    In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition

    Liu, S., Tripathi, S., Majumdar, S., Wang, X.: Joint hand motion and interaction hotspots prediction from egocentric videos. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 3282–3292 (2022)

    2022

  36. [44]

    arXiv preprint arXiv:2406.19353 (2024)

    Liu, Y., Zhang, C., Xing, R., Tang, B., Yang, B., Yi, L.: Core4d: A 4d human- object-human interaction dataset for collaborative object rearrangement. arXiv preprint arXiv:2406.19353 (2024)

    2024 arXiv

  37. [45]

    International Journal of Computer Vision (IJCV) 106, 282–296 (2014)

    Marin-Jimenez, M.J., Zisserman, A., Eichner, M., Ferrari, V.: Detecting people looking at each other in videos. International Journal of Computer Vision (IJCV) 106, 282–296 (2014)

    2014

  38. [46]

    In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision

    Mascaró, E.V., Ahn, H., Lee, D.: Intention-conditioned long-term human egocentric action anticipation. In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. pp. 6048–6057 (2023)

    2023

  39. [47]

    Technical report, arXiv (2025), https://arxiv.org/pdf/2510.16258, arXiv preprint

    McLean, C., Meendering, M., Swartz, T., Gabbay, O., Olsen, A., Jacobs, R., Rosen, N., de Bree, P., Garcia, T., Merrill, G., Sandakly, J., Buffalini, J., Jain, N., Krenn, S., Kumar, M., Markovic, D., Ng, E., Prada, F., Saba, A., Zhang, S., Agrawal, V., Godisart, T., Richard, A....

    2025

  40. [48]

    IEEE transactions on pattern analysis and machine intelligence (TPAMI)10(2021)

    Medina, M.J.M.K.V.: Suárez p zisserman a laeo-net++: revisiting people looking at each other in videos. IEEE transactions on pattern analysis and machine intelligence (TPAMI)10(2021)

    2021

  41. [49]

    https://facebookresearch.github.io/projectaria_tools/docs/ARK/mps (2024), accessed: 2026-03-04

    Meta Reality Labs Research: Project aria machine perception services (mps). https://facebookresearch.github.io/projectaria_tools/docs/ARK/mps (2024), accessed: 2026-03-04

    2024

  42. [50]

    Advances in Neural Information Processing Systems35, 23765–23779 (2022) CoMind: Understanding Collaborative Human Activity 19

    Mittal, H., Morgado, P., Jain, U., Gupta, A.: Learning state-aware visual repre- sentations from audible interactions. Advances in Neural Information Processing Systems35, 23765–23779 (2022) CoMind: Understanding Collaborative Human Activity 19

    2022

  43. [51]

    Current directions in psychological science16, 269–274 (11 2007)

    Mundy, P., Newell, L.: Attention, joint attention, and social cognition. Current directions in psychological science16, 269–274 (11 2007)

    2007

  44. [52]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Ngo, T.D., Hua, B.S., Nguyen, K.: Isbnet: a 3d point cloud instance segmenta- tion network with instance-aware sampling and box-aware dynamic convolution. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 13550–13559 (2023)

    2023

  45. [53]

    System card (Aug 2024),https://openai.com/index/ gpt-4o-system-card/, accessed 2026-03-04

    OpenAI: Gpt-4o system card. System card (Aug 2024),https://openai.com/index/ gpt-4o-system-card/, accessed 2026-03-04

    2024

  46. [54]

    System card addendum (Nov 2025),https://openai.com/index/gpt-5-system-card-addendum- gpt-5-1/, accessed 2026-03-04

    OpenAI: Gpt-5.1 instant and gpt-5.1 thinking system card addendum. System card addendum (Nov 2025),https://openai.com/index/gpt-5-system-card-addendum- gpt-5-1/, accessed 2026-03-04

    2025

  47. [55]

    OpenAI Index (Dec 2025),https://openai.com/ index/introducing-gpt-5-2/, accessed 2026-03-04

    OpenAI: Introducing gpt-5.2. OpenAI Index (Dec 2025),https://openai.com/ index/introducing-gpt-5-2/, accessed 2026-03-04

    2025

  48. [56]

    IEEE Transactions on Robotics37(6), 1855–1873 (2021)

    Ortenzi, V., Cosgun, A., Pardi, T., Chan, W.P., Croft, E., Kulić, D.: Object handovers: a review for robotics. IEEE Transactions on Robotics37(6), 1855–1873 (2021)

    2021

  49. [57]

    In: Proceedings of the IEEE/CVF International Conference on Computer Vision

    Osman, N., Camporese, G., Coscia, P., Ballan, L.: Slowfast rolling-unrolling lstms for action anticipation in egocentric videos. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 3437–3445 (2021)

    2021

  50. [58]

    In: Conference on Computer Vision and Pattern Recognition 2024 (2024),https://eth-ait.github

    Pasca, R., Gavryushin, A., Hamza, M., Kuo, Y.L., Mo, K., Van Gool, L., Hilliges, O., Wang, X.: Summarize the past to predict the future: Natural language descriptions of context boost multimodal object interaction anticipation. In: Conference on Computer Vision and Pattern Rec...

    2024

  51. [59]

    In: Proceedings of the Computer Vision and Pattern Recognition Conference

    Perrett, T., Darkhalil, A., Sinha, S., Emara, O., Pollard, S., Parida, K.K., Liu, K., Gatti, P., Bansal, S., Flanagan, K., et al.: Hd-epic: A highly-detailed egocentric video dataset. In: Proceedings of the Computer Vision and Pattern Recognition Conference. pp. 23901–23913 (2025)

    2025

  52. [60]

    Behavioral and Brain Sciences1, 515 – 526 (12 1978)

    Premack, D., Woodruff, G.: Does a chimpanzee have a theory of mind. Behavioral and Brain Sciences1, 515 – 526 (12 1978)

    1978

  53. [61]

    Puig, X., Ra, K., Boben, M., Li, J., Wang, T., Fidler, S., Torralba, A.: Virtualhome: Simulating household activities via programs (2018)

    2018

  54. [62]

    arXiv preprint arXiv:2010.09890 (2020)

    Puig, X., Shu, T., Li, S., Wang, Z., Liao, Y.H., Tenenbaum, J.B., Fidler, S., Torralba, A.: Watch-and-help: A challenge for social perception and human-ai collaboration. arXiv preprint arXiv:2010.09890 (2020)

    2010 arXiv

  55. [63]

    IEEE Transactions on Pattern Analysis and Machine Intelligence (2021)

    Qi, Z., Wang, S., Su, C., Su, L., Huang, Q., Tian, Q.: Self-regulated learning for egocentric video activity anticipation. IEEE Transactions on Pattern Analysis and Machine Intelligence (2021)

    2021

  56. [64]

    arXiv preprint arXiv:2501.19061 (2025)

    Qiu, H., Shi, Z., Wang, L., Xiong, H., Li, X., Li, H.: Egome: A new dataset and challenge for following me via egocentric view in real world. arXiv preprint arXiv:2501.19061 (2025)

    2025 arXiv

  57. [65]

    In: International conference on machine learning

    Rabinowitz, N., Perbet, F., Song, F., Zhang, C., Eslami, S.A., Botvinick, M.: Machine theory of mind. In: International conference on machine learning. pp. 4218–4227. PMLR (2018)

    2018

  58. [66]

    In: International conference on machine learning

    Radford, A., Kim, J.W., Xu, T., Brockman, G., McLeavey, C., Sutskever, I.: Robust speech recognition via large-scale weak supervision. In: International conference on machine learning. pp. 28492–28518. PMLR (2023)

    2023

  59. [67]

    Recasens, A., Khosla, A., Vondrick, C., Torralba, A.: Where are they looking? In: NeurIPS (2015)

    2015

  60. [68]

    Joint attention: Communication and other minds: Issues in philosophy and psychology p

    Reddy, V.: Understanding attention to self. Joint attention: Communication and other minds: Issues in philosophy and psychology p. 85 (2005) 20 A. Gavryushin et al

    2005

  61. [69]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Rizve, M.N., Mittal, G., Yu, Y., Hall, M., Sajeev, S., Shah, M., Chen, M.: Piv- otal: Prior-driven supervision for weakly-supervised temporal action localization. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 22992–23002 (2023)

    2023

  62. [70]

    In: International Conference on Image Analysis and Processing

    Rodin, I., Furnari, A., Mavroeidis, D., Farinella, G.M.: Untrimmed action anticipa- tion. In: International Conference on Image Analysis and Processing. pp. 337–348. Springer (2022)

    2022

  63. [71]

    In: Proceedings third international conference on 3-D digital imaging and modeling

    Rusinkiewicz, S., Levoy, M.: Efficient variants of the icp algorithm. In: Proceedings third international conference on 3-D digital imaging and modeling. pp. 145–152. IEEE (2001)

    2001

  64. [72]

    In: CVPR (2019)

    Sarlin, P.E., Cadena, C., Siegwart, R., Dymczyk, M.: From coarse to fine: Robust hierarchical localization at large scale. In: CVPR (2019)

    2019

  65. [73]

    In: CVPR (2020)

    Sarlin, P.E., DeTone, D., Malisiewicz, T., Rabinovich, A.: SuperGlue: Learning feature matching with graph neural networks. In: CVPR (2020)

    2020

  66. [74]

    Behavioral and Brain Sciences 36(4), 393–414 (2013)

    Schilbach, L., Timmermans, B., Reddy, V., Costall, A., Bente, G., Schlicht, T., Vogeley, K.: Toward a second-person neuroscience. Behavioral and Brain Sciences 36(4), 393–414 (2013)

    2013

  67. [75]

    Trends in Cognitive Sciences10(2), 70–76 (2006)

    Sebanz, N., Bekkering, H., Knoblich, G.: Joint action: bodies and minds moving together. Trends in Cognitive Sciences10(2), 70–76 (2006)

    2006

  68. [76]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Sener, F., Chatterjee, D., Shelepov, D., He, K., Singhania, D., Wang, R., Yao, A.: Assembly101: A large-scale multi-view video dataset for understanding procedural activities. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 21096–21106 (2022)

    2022

  69. [77]

    In: Proceedings of the AAAI Conference on Artificial Intelligence (2025)

    Shi, H., Ye, S., Fang, X., Jin, C., Isik, L., Kuo, Y.L., Shu, T.: Muma-tom: Multi- modal multi-agent theory of mind. In: Proceedings of the AAAI Conference on Artificial Intelligence (2025)

    2025

  70. [78]

    In: Proceedings of the European Conference on Computer Vision (ECCV)

    Shi, Y., Fernando, B., Hartley, R.: Action anticipation with rbf kernelized feature mapping rnn. In: Proceedings of the European Conference on Computer Vision (ECCV). pp. 301–317 (2018)

    2018

  71. [79]

    In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition

    Shridhar, M., Thomason, J., Gordon, D., Bisk, Y., Han, W., Mottaghi, R., Zettle- moyer, L., Fox, D.: Alfred: A benchmark for interpreting grounded instructions for everyday tasks. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 10740–...

    2020

  72. [80]

    Journal of Multimodal User Interfaces9(3), 223–229 (2015).https: //doi.org/10.1007/s12193-015-0196-1

    Six, J., Leman, M.: Synchronizing Multimodal Recordings Using Audio-To-Audio Alignment. Journal of Multimodal User Interfaces9(3), 223–229 (2015).https: //doi.org/10.1007/s12193-015-0196-1

    2015 doi

  73. [81]

    Journal of Human-Robot Interaction2(1), 112–132 (2013)

    Strabala, K.W., Lee, M.K., Dragan, A.D., Forlizzi, J.L., Srinivasa, S., Cakmak, M., Micelli, V.: Towards seamless human-robot handovers. Journal of Human-Robot Interaction2(1), 112–132 (2013)

    2013

  74. [82]

    In: European conference on computer vision

    Taheri, O., Ghorbani, N., Black, M.J., Tzionas, D.: Grab: A dataset of whole- body human grasping of objects. In: European conference on computer vision. pp. 581–600. Springer (2020)

    2020

  75. [83]

    Behavioral and Brain Sciences 28, 675–735 (11 2005)

    Tomasello, M., Carpenter, M., Call, J., Behne, T., Moll, H.: Understanding and sharing intentions: The origins of cultural cognition. Behavioral and Brain Sciences 28, 675–735 (11 2005)

    2005

  76. [84]

    arXiv preprint arXiv:2507.04415 (2025)

    Villa-Cueva, E., Ahmed, S., Chevi, R., Cruz, J.C.B., Elzeky, K., Cristobal, F., Aji, A.F., Wang, S., Mihalcea, R., Solorio, T.: Moments: A comprehensive multimodal benchmark for theory of mind. arXiv preprint arXiv:2507.04415 (2025)

    2025

  77. [85]

    In: Proceedings of the IEEE conference on computer vision and pattern recognition

    Vondrick, C., Pirsiavash, H., Torralba, A.: Anticipating visual representations from unlabeled video. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 98–106 (2016) CoMind: Understanding Collaborative Human Activity 21

    2016

  78. [86]

    In: Proceedings of the IEEE/CVF International Conference on Computer Vision

    Wang, X., Kwon, T., Rad, M., Pan, B., Chakraborty, I., Andrist, S., Bohus, D., Feniello, A., Tekin, B., Frujeri, F.V., et al.: Holoassist: an egocentric human interaction dataset for interactive ai assistants in the real world. In: Proceedings of the IEEE/CVF International Con...

    2023

  79. [87]

    arXiv preprint arXiv:2411.01455 (2024)

    Wang, Z., Zhao, X., Stepputtis, S., Kim, W., Wu, T., Sycara, K., Xie, Y.: Himem- former: Hierarchical memory-aware transformer for multi-agent action anticipation. arXiv preprint arXiv:2411.01455 (2024)

    2024 arXiv

  80. [88]

    Advances in Neural Information Processing Systems36, 68736–68748 (2023)

    Wiederhold, N., Megyeri, A., Paris, D., Banerjee, S., Banerjee, N.: Hoh: Marker- less multimodal human-object-human handover dataset with large object count. Advances in Neural Information Processing Systems36, 68736–68748 (2023)

    2023

  81. [89]

    In: Proceedings of the Computer Vision and Pattern Recognition Conference

    Yang, J., Liu, S., Guo, H., Dong, Y., Zhang, X., Zhang, S., Wang, P., Zhou, Z., Xie, B., Wang, Z., et al.: Egolife: Towards egocentric life assistant. In: Proceedings of the Computer Vision and Pattern Recognition Conference. pp. 28885–28900 (2025)

    2025

  82. [90]

    arXiv preprint arXiv:2602.15989 (2026)

    Yang, X., Kukreja, D., Pinkus, D., Sagar, A., Fan, T., Park, J., Shin, S., Cao, J., Liu, J., Ugrinovic, N., Feiszli, M., Malik, J., Dollar, P., Kitani, K.: Sam 3d body: Robust full-body human mesh recovery. arXiv preprint arXiv:2602.15989 (2026)

    2026

  83. [91]

    In: ICCV (2025)

    Yang, Y.H., Piccinelli, L., Segu, M., Li, S., Huang, R., Fu, Y., Pollefeys, M., Blum, H., Bauer, Z.: 3d-mood: Lifting 2d to 3d for monocular open-set object detection. In: ICCV (2025)

    2025

  84. [92]

    In: IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG)

    Ye, Z., Li, Y., Liu, Y., Bridges, C., Rozga, A., Rehg, J.M.: Detecting bids for eye contact using a wearable camera. In: IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG). vol. 1, pp. 1–8. IEEE (2015)

    2015

  85. [93]

    In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)

    Zhang, C., Fu, C., Wang, S., Agarwal, N., Lee, K., Choi, C., Sun, C.: Object- centric video representation for long-term action anticipation. In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV). pp. 6751–6761 (January 2024)

    2024

  86. [94]

    arXiv preprint arXiv:2307.02485 (2023)

    Zhang, H., Du, W., Shan, J., Zhou, Q., Du, Y., Tenenbaum, J.B., Shu, T., Gan, C.: Building cooperative embodied agents modularly with large language models. arXiv preprint arXiv:2307.02485 (2023)

    2023 arXiv

  87. [95]

    In: European conference on computer vision

    Zhang, S., Ma, Q., Zhang, Y., Qian, Z., Kwon, T., Pollefeys, M., Bogo, F., Tang, S.: Egobody: Human body shape and motion of interacting people from head-mounted devices. In: European conference on computer vision. pp. 180–200. Springer (2022)

    2022

  88. [96]

    arXiv preprint arXiv:2505.18943 (2025)

    Zhang, X., Chen, Y., Yeh, S., Li, S.: Metamind: Modeling human social thoughts with metacognitive multi-agent systems. arXiv preprint arXiv:2505.18943 (2025)

    2025

  89. [97]

    {transcript}

    Zhao, Q., Wang, S., Zhang, C., Fu, C., Do, M.Q., Agarwal, N., Lee, K., Sun, C.: AntGPT: Can large language models help long-term action anticipation from videos? In: The Twelfth International Conference on Learning Representations (2024) 22 A. Gavryushin et al. Supplementary M...

    2024

  90. [98]

    Cue type: Analyze the frame and classify the underlying cue(s) that reveal the joint attention. Choose from one or more of these four categories (1 to 4 choices): - <gaze>: The eye gaze or head pose of both individuals clearly converges onto the same target object or spatial a...

  91. [99]

    cue type

    Object grounding: Look at the provided image frame. Based on your answers to the previous questions, locate and identify the precise object of joint attention in BOTH views SEPARATELY: - Object category: Provide a short, precise category name of the object being mutually atten...

  92. [104]

    handover happen offset

    Object Grounding: Look at the PROVIDED SINGLE REFERENCE IMAGE FRAME. Based on the context of the video clip and your answers to the previous questions, locate the precise object that is most likely going to be handed over in the future: - Provide a short, precise object catego...

  93. [105]

    Handover Timing: Approximately how many seconds after the reference image frame will the object handover between two persons occur? (Provide a decimal time offset)

  94. [106]

    Delivering Flow: Whether the object will be handed over from left person to right person or from right person to left person? (Choose STRICTLY: <left to right> or <right to left>)

  95. [107]

    (Choose STRICTLY: <left> or <right>)

    Initiator: Who initiates the handover? ’Initiation’ refers to the first cue that prompts the handover to occur. (Choose STRICTLY: <left> or <right>)

  96. [108]

    - <gestural>: The cue is PURELY a physical gesture (e.g., reaching out, pointing) with NO speech

    Initiation Type: Classify the initiator’s cue STRICTLY into one of these four categories: - <verbal>: The cue is PURELY spoken (e.g., a verbal request/command). - <gestural>: The cue is PURELY a physical gesture (e.g., reaching out, pointing) with NO speech. - <verbal and gest...

  97. [109]

    handover happen offset

    Object Grounding: Look at the PROVIDED SINGLE REFERENCE IMAGE FRAME. Based on the context of the video clip and your answers to the previous questions, locate the precise object that is most likely going to be handed over in the future: - Provide a short, precise object catego...

  98. [110]

    Maintain that focus for a short but continuous duration (typically ≥ 0.5 seconds ) 3

    Visually attend to the same object or region in the scene, 2. Maintain that focus for a short but continuous duration (typically ≥ 0.5 seconds ) 3. Show behavioral coordination or awareness of the shared focus (e.g., verbal reference, handover, shared gaze, or gesture). Eviden...

  99. [111]

    Note: Gaze can be noisy

    Gaze Convergence (Gaze): The gaze overlays (dots) from both participants land on the same object or within the same small work area. Note: Gaze can be noisy. If the dot is slightly away from an object but the person is clearly reaching for or looking at, consider it "on" the object

  100. [112]

    If one person is holding an object while the other person cuts it, pours into it, or reaches for it, they are in Joint Attention

    Manual Interaction (Action): The strongest cue. If one person is holding an object while the other person cuts it, pours into it, or reaches for it, they are in Joint Attention

  101. [113]

    presents

    Deictic Gestures (Gestures): One person points at, holds up, or "presents" an object to the other. JA begins as soon as the other person looks at that object

  102. [114]

    Pass me the salt

    Verbal/Social Context (Communication): Listen for instructions ("Pass me the salt") or descriptions ("Look at this color"). Use the audio to find the start of the mental connection. If cues conflict, follow this order of importance:

  103. [115]

    Look at this

    Action (Physical touch/interaction is the most reliable). 2. Verbal/Gestures (Explicit intent). 3. Gaze Overlays (Confirmation of visual focus) Confidence level Select the confidence level that best describes the alignment of evidence. Use the Evidence Cues (Gaze, Action, Gest...

  104. [116]

    Mark the time interval (start and end time). 2. Draw 2 bounding boxes around the object or area both participants are focused on each view

  105. [117]

    start_time

    Assign an object or region label (e.g., cutting board , pan , chicken ). Example output JSON: { "start_time": 12.345, // seconds "end_time": 12.400, // seconds "bbox_left": [x, y, w, h], // pixel coordinates for helper view on the left "bbox_right": [x, y, w, h], // pixel coor...

  106. [118]

    Would this action still happen if the Leader did nothing?

    Play video and watch for Social Cues from Leader 2. When a cue is identified, monitor the Helper 3. Identify the FIRST valid socially cued action 4. Pause the video 5. Create a new timeline with label: <verb> <noun> 6. Select the correct tag from the dropdown associated with t...

  107. [119]

    space” to pause at the handover moment 2. Press “o

    Press “space” to pause at the handover moment 2. Press “o” to annotate the timestamp (press “enter” when textbox appears, you don’t need to input anything)

  108. [120]

    Input “t_h” in the textbox above the dropdown box (see the example below) for each annotation line

  109. [121]

    Skip”. 5. click “Add

    select anything for dropdown boxes, just as blank filler to add the next annotation line (you don’t need to worry about it, just select any options for each dropdown box). Do not click “Skip”. 5. click “Add” to add a new annotation timeline (one timeline for one handover actio...

  110. [122]

    Play the video normally and observe both participants’ hands and grasped objects

  111. [123]

    Identify handover moments where the giver passes the object to the receiver, specifically the transfer point moment where both the giver and receiver are touching the object

  112. [124]

    left to right

    Pause the video and press o to annotate the transfer point timestamp T_h ① 5. Select “left to right” or “right to left” from Delivering Flow ② dropdown box 6. Roll backward to the moment before giver starts reaching for the object 7. Then Slowly Roll backward frame by frame fr...

  113. [125]

    verbal”, “gestural

    Identify initiation type and Initiator by go through the frames before T_h ○ select “verbal”, “gestural” or “implicit” from Initiation Type ⑥ dropdown box ○ select “left” or “right” from Initiator ⑦ dropdown box Save annotations (File → Export annotations) in VIA JSON format. ...

Pith tools

Reviewed July 10, 2026 · model on record in the stance chip above.