pith. sign in

arxiv: 2511.20162 · v2 · submitted 2025-11-25 · 💻 cs.CV · cs.AI· q-bio.NC

Action Without Interaction: Probing the Physical Foundations of Video LMMs via Contact-Release Detection

Daniel Harari , Michael Sidorov , Chen Shterental , Liel David , Abrham Kahsay Gebreselasie , Muhammad Haris Khan This is my paper

Pith reviewed 2026-05-17 04:43 UTC · model grok-4.3

classification 💻 cs.CV cs.AIq-bio.NC
keywords video LMMsphysical groundingcontact-release detectionshortcut learningaction understandingSomething-Something-V2multimodal modelsinteraction localization
0
0 comments X

The pith

Video LMMs name objects and actions in interactions but fail to identify the frames where contact or release begins or ends.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper tests how well video large multimodal models ground their understanding in actual physical events rather than surface patterns. The authors built a dataset of over 20,000 annotated hand-object interactions drawn from Something-Something-V2 videos, with labels marking the exact timing and location of contact and release moments. When state-of-the-art models including GPT, Gemini, and Qwen were asked to locate these events in short clips, they performed well at naming the objects and actions yet consistently missed the correct frames and spatial positions. The results point to a reliance on semantic shortcuts that does not extend to reasoning about basic physical primitives.

Core claim

State-of-the-art video LMMs exhibit shortcut learning in which they reliably name target objects and identify actions in hand-object videos, yet they consistently fail to identify the frame where the interaction begins or ends and poorly localize the physical event within the scene. This pattern was measured using a new dataset of more than 20K human-annotated contact and release events. The findings indicate that the models excel at System 1 intuitive pattern recognition but lack the System 2 foundations required to reason about physical primitives such as contact and release.

What carries the argument

The contact-release detection task on a new annotated video dataset, which requires models to report the precise frame and spatial location of attachment or detachment events in short interaction clips.

If this is right

  • Semantic success at describing actions does not imply accurate detection of when physical interactions start or stop.
  • Models that pass high-level action recognition benchmarks can still lack basic physical event localization.
  • Current video LMMs may require new training signals focused on contact and release to achieve grounded dynamic scene understanding.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Applications such as robotic planning or video-based simulation could be limited until models improve at physical timing.
  • The same evaluation approach could be extended to other physical primitives like support or collision in future datasets.
  • Explicit training objectives that reward correct contact-release localization might close the observed gap.

Load-bearing premise

The failures on timing and localization are caused by missing physical grounding in the models rather than by prompt design, video compression, or noise in the annotations.

What would settle it

Re-testing the same models on the identical videos after adding clear visual markers at contact points or increasing frame rate to check whether frame-identification accuracy rises above current levels.

read the original abstract

Large multi-modal models (LMMs) show increasing performance in realistic visual tasks for images and, more recently, for videos. For example, given a video sequence, such models are able to describe in detail objects, the surroundings and dynamic actions. In this study, we explored the extent to which these models ground their semantic understanding in the actual visual input. Specifically, given sequences of hands interacting with objects, we asked models when and where the interaction begins or ends. For this purpose, we introduce a first of its kind, large-scale dataset with more than 20K annotated interactions on videos from the Something-Something-V2 dataset. 250 AMTurk human annotators labeled core interaction events, particularly when and where objects and agents become attached (`contact') or detached (`release'). We asked SoTA LMMs, including GPT, Gemini and Qwen to locate these events in short videos, each with a single event. The results show that while models reliably name target objects and identify actions, they exhibit a form of `shortcut learning' where semantic success masks a failure in physical grounding. Specifically, they consistently fail to identify the frame where the interaction begins or ends and poorly localize the physical event within the scene. This disconnect suggests that while LMMs excel at System 1 intuitive pattern recognition (naming the action and objects), they lack the System 2 cognitive foundations required to reason about physical primitives like `contact' and `release', hence truly ground dynamic scenes in physical reality.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

3 major / 2 minor

Summary. The paper introduces a new dataset of over 20K human-annotated contact and release events drawn from Something-Something-V2 videos. It evaluates several state-of-the-art video LMMs (GPT, Gemini, Qwen) on their ability to name objects/actions versus their ability to report the precise onset/offset frames and spatial localization of those physical events. The central claim is that semantic success on naming masks a fundamental failure of physical grounding, which the authors interpret as evidence that current models lack System-2-style reasoning about contact and release primitives.

Significance. If the quantitative results hold after proper controls and metrics, the work would supply a concrete, falsifiable probe for physical grounding in video LMMs and a reusable benchmark focused on contact/release timing. This could usefully steer research away from purely semantic pattern matching toward models that explicitly track physical state changes.

major comments (3)
  1. [Abstract / Results] Abstract and Results sections: the central claim of 'consistent failure' and 'shortcut learning' is supported only by qualitative statements; no accuracy percentages, frame-error distributions, temporal IoU scores, or per-model success rates are reported, nor are error bars or statistical tests provided to quantify the magnitude or reliability of the observed deficits.
  2. [Experimental Setup] Experimental Setup: the evaluation uses a single prompting regime on compressed Something-Something-V2 clips without ablations on prompt phrasing, frame sampling rate, or input resolution; therefore the reported failures cannot yet be attributed specifically to absence of physical grounding rather than to tokenization or prompt artifacts.
  3. [Dataset] Dataset section: although 250 AMTurk annotators produced >20K labels, no inter-annotator agreement statistics, annotation protocol details, or quality-control procedures are described, leaving the reliability of the ground-truth timing and localization labels unverified.
minor comments (2)
  1. [Abstract] The abstract invokes 'System 1' and 'System 2' without a brief citation or clarification; adding one sentence linking the terminology to the specific temporal-localization task would improve accessibility.
  2. [Figures / Results] Figure captions and result tables (if present) should explicitly state the exact prompt template and video preprocessing steps used for each model.

Simulated Author's Rebuttal

3 responses · 0 unresolved

We thank the referee for the constructive and detailed feedback. We address each major comment below and describe the revisions that will be incorporated to strengthen the manuscript's rigor and clarity.

read point-by-point responses

  1. Referee: [Abstract / Results] Abstract and Results sections: the central claim of 'consistent failure' and 'shortcut learning' is supported only by qualitative statements; no accuracy percentages, frame-error distributions, temporal IoU scores, or per-model success rates are reported, nor are error bars or statistical tests provided to quantify the magnitude or reliability of the observed deficits.

    Authors: We agree that the abstract and high-level Results summary rely primarily on qualitative language. Although quantitative metrics appear in the Experiments section, we will revise the abstract to report explicit per-model accuracy percentages for naming versus contact/release detection, frame-error distributions, temporal IoU scores, and success rates. Error bars and basic statistical tests (e.g., paired t-tests across models) will also be added to the Results section to quantify the deficits more precisely. revision: yes

  2. Referee: [Experimental Setup] Experimental Setup: the evaluation uses a single prompting regime on compressed Something-Something-V2 clips without ablations on prompt phrasing, frame sampling rate, or input resolution; therefore the reported failures cannot yet be attributed specifically to absence of physical grounding rather than to tokenization or prompt artifacts.

    Authors: We acknowledge that a single prompting regime limits attribution. To isolate physical grounding from prompt or tokenization effects, we will add ablations varying prompt phrasing, frame sampling rates, and input resolutions. These new experiments and results will be reported in the Experimental Setup and Results sections to show that the core failures persist across conditions. revision: yes

  3. Referee: [Dataset] Dataset section: although 250 AMTurk annotators produced >20K labels, no inter-annotator agreement statistics, annotation protocol details, or quality-control procedures are described, leaving the reliability of the ground-truth timing and localization labels unverified.

    Authors: We agree that annotation reliability must be documented. The revised manuscript will include inter-annotator agreement statistics (e.g., percentage agreement and Cohen's kappa for frame timing and spatial localization), a full description of the annotation protocol and interface, and quality-control steps such as gold-standard validation and filtering of inconsistent annotations. revision: yes

Circularity Check

0 steps flagged

No circularity: direct empirical test on new human-annotated dataset

full rationale

The paper introduces a new dataset of over 20K contact-release annotations on Something-Something-V2 clips and performs direct empirical evaluation of existing LMMs (GPT, Gemini, Qwen) on timing and localization tasks. No equations, fitted parameters, predictions derived from prior fits, or self-citation chains are present; the reported failures follow from comparison against fresh human labels rather than any reduction to the paper's own inputs or prior results by construction. The analysis is therefore self-contained.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 0 invented entities

The claim rests on the assumption that human annotations constitute accurate ground truth for physical contact and release events and that standard LMM prompting elicits the models' true capabilities on this task.

axioms (1)
  • domain assumption Human annotations on AMTurk provide reliable ground truth for the timing and location of contact and release events.
    The study uses 250 annotators to label core interaction events as the benchmark against which model performance is measured.

pith-pipeline@v0.9.0 · 5604 in / 1281 out tokens · 55042 ms · 2026-05-17T04:43:48.404254+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

77 extracted references · 77 canonical work pages · 2 internal anchors

  1. [1]

    Qwen2.5-VL Technical Report

    Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhao- hai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Hang Zhang, Zhibo Yang, Haiyang Xu, and Jun- yang Lin. Qwen2.5-vl technical repor...

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  2. [2]

    Physical reasoning in infancy.The cog- nitive neurosciences, pages 181–204, 1995

    Renee Baillargeon. Physical reasoning in infancy.The cog- nitive neurosciences, pages 181–204, 1995. 1

    work page 1995

  3. [3]

    Visrl: Intention-driven visual perception via reinforced reasoning

    Zhangquan Chen, Xufang Luo, and Dongsheng Li. Visrl: Intention-driven visual perception via reinforced reasoning. arXiv preprint arXiv:2503.07523, 2025. 2

    work page arXiv 2025

  4. [4]

    Smith, and Joshua B

    Anoop Cherian, Kuan-Chuan Peng, Suhas Lohit, Kevin A. Smith, and Joshua B. Tenenbaum. Are deep neural networks smarter than second graders? In2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 10834–10844, 2023. 1

    work page 2023

  5. [5]

    Unifying video self-supervised learning across families of tasks: A survey.Preprints, 2024

    Ishan Dave, Malitha Gunawardhana, Limalka Sadith, Honglu Zhou, Liel David, Daniel Harari, Mubarak Shah, and Muhammad Khan. Unifying video self-supervised learning across families of tasks: A survey.Preprints, 2024. 1

    work page 2024

  6. [6]

    A survey on in- context learning

    Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Jingyuan Ma, Rui Li, Heming Xia, Jingjing Xu, Zhiyong Wu, Baobao Chang, Xu Sun, Lei Li, and Zhifang Sui. A survey on in- context learning. InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 1107–1128, Miami, Florida, USA, 2024. Association for Computational Linguistics. 4, 6

    work page 2024

  7. [7]

    A survey of video action recog- nition based on deep learning.Knowledge-Based Systems, 320:113594, 2025

    Ping Gong and Xudong Luo. A survey of video action recog- nition based on deep learning.Knowledge-Based Systems, 320:113594, 2025. 1

    work page 2025

  8. [8]

    The” something something” video database for learning and evaluating visual common sense

    Raghav Goyal, Samira Ebrahimi Kahou, Vincent Michal- ski, Joanna Materzynska, Susanne Westphal, Heuna Kim, Valentin Haenel, Ingo Fruend, Peter Yianilos, Moritz Mueller-Freitag, et al. The” something something” video database for learning and evaluating visual common sense. InProceedings of the IEEE international conference on com- puter vision, pages 5842...

    work page 2017

  9. [9]

    Gunawardhana, L

    M. Gunawardhana, L. Sadith, L. David, D. Harari, and M. H. Khan. How effective are self-supervised models for contact identification in videos. InProceedings of the International Joint Conference on Artificial Intelligence, pages 117–131, Singapore, 2024. Springer Nature Singapore. 8

    work page 2024

  10. [10]

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Peiyi Wang, Qihao Zhu, Runxin Xu, Ruoyu Zhang, Shirong Ma, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing Xue, Bingxuan Wang, Bochao Wu, Bei Feng, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chong Ruan, Damai Dai, Deli Chen, Dongjie Ji, ...

    work page 2025

  11. [11]

    Do vision-language models really understand visual lan- guage? InForty-second International Conference on Ma- chine Learning, 2025

    Yifan Hou, Buse Giledereli, Yilei Tu, and Mrinmaya Sachan. Do vision-language models really understand visual lan- guage? InForty-second International Conference on Ma- chine Learning, 2025. 1

    work page 2025

  12. [12]

    The perception of causality in infants.Per- ception, 11(2):173–186, 1982

    Alan M Leslie. The perception of causality in infants.Per- ception, 11(2):173–186, 1982. 1

    work page 1982

  13. [13]

    Jingping Liu, Ziyan Liu, Zhedong Cen, Yan Zhou, Yinan Zou, Weiyan Zhang, Haiyun Jiang, and Tong Ruan. Can multimodal large language models understand spatial rela- tions? InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 620–632, Vienna, Austria, 2025. Association for Computational Lin...

    work page 2025

  14. [14]

    Deep video understanding with video- language model

    Runze Liu, Yaqun Fang, Fan Yu, Ruiqi Tian, Tongwei Ren, and Gangshan Wu. Deep video understanding with video- language model. InProceedings of the 31st ACM Inter- national Conference on Multimedia, page 9551–9555, New York, NY , USA, 2023. Association for Computing Machin- ery. 1

    work page 2023

  15. [15]

    Video-chatgpt: Towards detailed video un- derstanding via large vision and language models

    Muhammad Maaz, Hanoona Rasheed, Salman Khan, and Fahad Khan. Video-chatgpt: Towards detailed video un- derstanding via large vision and language models. InPro- ceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 12585–12602, 2024. 2

    work page 2024

  16. [16]

    Basic Books,

    Albert Michotte.The perception of causality. Basic Books,

    work page

  17. [17]

    Human action recognition: A taxonomy- based survey, updates, and opportunities.Sensors, 23(4),

    Md Golam Morshed, Tangina Sultana, Aftab Alam, and Young-Koo Lee. Human action recognition: A taxonomy- based survey, updates, and opportunities.Sensors, 23(4),

    work page

  18. [18]

    GPT-4o System Card

    OpenAI. Gpt-4o system card.arXiv preprint arxiv:2410.21276, 2024. 2, 6

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  19. [19]

    What is the limitation of multimodal llms? a deeper look into multimodal llms through prompt probing.Information Processing and Management, 60(6): 103510, 2023

    Shuhan Qi, Zhengying Cao, Jun Rao, Lei Wang, Jing Xiao, and Xuan Wang. What is the limitation of multimodal llms? a deeper look into multimodal llms through prompt probing.Information Processing and Management, 60(6): 103510, 2023. 2

    work page 2023

  20. [20]

    Oxford University Press, 2014

    Gabriel A Radvansky and Jeffrey M Zacks.Event cognition. Oxford University Press, 2014. 1

    work page 2014

  21. [21]

    The perception of causality in infancy.Acta psychologica, 123(1-2):144–165, 2006

    Rebecca Saxe and Susan Carey. The perception of causality in infancy.Acta psychologica, 123(1-2):144–165, 2006. 1

    work page 2006

  22. [22]

    Schulze Buschoff, Elif Akata, Matthias Bethge, and Eric Schulz

    Luca M. Schulze Buschoff, Elif Akata, Matthias Bethge, and Eric Schulz. Visual cognition in multimodal large language models.Nature Machine Intelligence, 7(1):96, 2025. 1, 8

    work page 2025

  23. [23]

    Visual cot: advancing multi-modal language models with a comprehen- sive dataset and benchmark for chain-of-thought reasoning

    Hao Shao, Shengju Qian, Han Xiao, Guanglu Song, Zhuofan Zong, Letian Wang, Yu Liu, and Hongsheng Li. Visual cot: advancing multi-modal language models with a comprehen- sive dataset and benchmark for chain-of-thought reasoning. InProceedings of the 38th International Conference on Neu- ral Information Processing Systems, Red Hook, NY , USA,

    work page

  24. [24]

    Curran Associates Inc. 2, 8

    work page

  25. [25]

    Video understand- ing with large language models: A survey.IEEE Transac- tions on Circuits and Systems for Video Technology, 2025

    Yunlong Tang, Jing Bi, Siting Xu, Luchuan Song, Susan Liang, Teng Wang, Daoan Zhang, Jie An, Jingyang Lin, Rongyi Zhu, Ali V osoughi, Chao Huang, Zeliang Zhang, Pinxin Liu, Mingqian Feng, Feng Zheng, Jianguo Zhang, Ping Luo, Jiebo Luo, and Chenliang Xu. Video understand- ing with large language models: A survey.IEEE Transac- tions on Circuits and Systems ...

    work page 2025

  26. [26]

    Llm-enhanced action-aware multi-modal prompt tuning for image-text matching.arXiv preprint arXiv:2506.23502, 2025

    Mengxiao Tian, Xinxiao Wu, and Shuo Yang. Llm-enhanced action-aware multi-modal prompt tuning for image-text matching.arXiv preprint arXiv:2506.23502, 2025. 2

    work page arXiv 2025

  27. [27]

    VideoMAE: Masked autoencoders are data-efficient learners for self-supervised video pre-training

    Zhan Tong, Yibing Song, Jue Wang, and Limin Wang. VideoMAE: Masked autoencoders are data-efficient learners for self-supervised video pre-training. InAdvances in Neural Information Processing Systems, 2022. 1, 8

    work page 2022

  28. [28]

    Chi, Quoc V

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoc V . Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning in large language models. InProceedings of the 36th Inter- national Conference on Neural Information Processing Sys- tems, Red Hook, NY , USA, 2022. Curran Associates Inc. 2, 5, 6, 8, 1

    work page 2022

  29. [29]

    F-lmm: Grounding frozen large multimodal models

    Size Wu, Sheng Jin, Wenwei Zhang, Lumin Xu, Wentao Liu, Wei Li, and Chen Change Loy. F-lmm: Grounding frozen large multimodal models. InProceedings of the Computer Vision and Pattern Recognition Conference, pages 24710– 24721, 2025. 2

    work page 2025

  30. [30]

    Griffiths, Yuan Cao, and Karthik Narasimhan

    Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L. Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: deliberate problem solving with large lan- guage models. InProceedings of the 37th International Conference on Neural Information Processing Systems, Red Hook, NY , USA, 2023. Curran Associates Inc. 6, 1

    work page 2023

  31. [31]

    Visual question decomposition on multimodal large language mod- els

    Haowei Zhang, Jianzhe Liu, Zhen Han, Shuo Chen, Bailan He, V olker Tresp, Zhiqiang Xu, and Jindong Gu. Visual question decomposition on multimodal large language mod- els. InFindings of the Association for Computational Lin- guistics: EMNLP 2024, pages 1926–1949, Miami, Florida, USA, 2024. Association for Computational Linguistics. 8

    work page 2024

  32. [32]

    Gundavarapu, Liangzhe Yuan, Hao Zhou, Shen Yan, Jennifer J

    Long Zhao, Nitesh B. Gundavarapu, Liangzhe Yuan, Hao Zhou, Shen Yan, Jennifer J. Sun, Luke Friedman, Rui Qian, Tobias Weyand, Yue Zhao, Rachel Hornung, Florian Schroff, Ming-Hsuan Yang, David A. Ross, Huisheng Wang, Hartwig Adam, Mikhail Sirotenko, Ting Liu, and Boqing Gong. Videoprism: a foundational visual encoder for video under- standing. InProceeding...

    work page 2024

  33. [33]

    template_id

    Ablation Experiments 8.1. Action and object recognition This section provides extended descriptions of the experi- mental setup of the evaluation on the tasks related to action and object recognition, mentioned in Section 4 of the main manuscript. All experiments were conducted using the ex- tracted video frames of the full length videos from the orig- in...

    work page

  34. [34]

    Prediction

    Your output should ALWAYS be the word "Prediction" ,→followed by the frame number

    work page

  35. [35]

    You will be provided a score with absolute values in ,→the range [0, 1], representing the correctness ,→of your prediction

    work page

  36. [36]

    Score of 0 means that your prediction is incorrect, ,→while an absolute value of 1 means you have ,→found the correct frame

    work page

  37. [37]

    The absolute values in the range of [0,1] reflect the ,→proximity of the predicted frame to the true ,→target frame, where the higher value is better

    work page

  38. [38]

    If the sign is positive, ,→your next prediction should be in the same ,→direction as the previous prediction

    The sign of the score signifies the direction for ,→your next prediction. If the sign is positive, ,→your next prediction should be in the same ,→direction as the previous prediction. A negative ,→sign means you should change the direction of ,→your next prediction. For example, if your last ,→prediction was frame 5, your current prediction ,→is frame 6 a...

    work page

  39. [39]

    You should never predict the same frame twice

    work page

  40. [40]

    ,→Putting number of something onto something

    Stop your predictions only when the score is above ,→0.98. Now we will perform an iterative session, during which ,→you will need to find the frame in the provided ,→image sequence. Follow the instructions above ,→when prompted with the feedback score. Listing 8. Instruction prompt for action recognition. System: You are an expert video-interaction classi...

    work page

  41. [41]

    GPT-5 obtained a Top-1 accuracy of 63.6%and a perfect Top-5 accuracy of100%, outperform- ing both GPT-4o and Qwen-2.5VL on this subset

    Detailed results Action and object recognition.For completeness, we also evaluated GPT-5 on the action-recognition task un- der the same zero-shot protocol described in Section 4 of the main manuscript. GPT-5 obtained a Top-1 accuracy of 63.6%and a perfect Top-5 accuracy of100%, outperform- ing both GPT-4o and Qwen-2.5VL on this subset. Event bounding-box...

    work page

  42. [42]

    For each video, we cropped short 10-frame video clips around three tempo- rally separated core interaction events

    Experimental Dataset S-Table 8 lists all 33 videos included in the experimental dataset, which was used in our experiments. For each video, we cropped short 10-frame video clips around three tempo- rally separated core interaction events. The table includes the video ID, action template and object placeholders from the original SSv2 dataset. In addition, ...

    work page

  43. [43]

    **Frame 1**: The hand is positioned above the white circular object, indicating an intention to interact with it

    work page

  44. [44]

    **Frame 2**: The hand moves closer to the white circular object

    work page

  45. [45]

    **Frame 3**: The hand makes contact with the white circular object

    work page

  46. [46]

    **Frame 4**: The hand lifts the white circular object

    work page

  47. [47]

    **Frame 5**: The hand places the white circular object back on the surface

    work page

  48. [48]

    **Frame 6**: The hand moves away from the white circular object

    work page

  49. [49]

    **Frame 7**: The hand returns to the white circular object

    work page

  50. [50]

    **Frame 8**: The hand lifts the white circular object again

    work page

  51. [51]

    **Frame 9**: The hand places the white circular object back on the surface

    work page

  52. [52]

    ### Chain of Thought - The event of interest here is the interaction between the hand and the white circular object

    **Frame 10**: The hand moves away from the white circular object. ### Chain of Thought - The event of interest here is the interaction between the hand and the white circular object. - The interaction begins when the hand first makes contact with the white circular object. - This contact occurs in Frame 3. ### Prediction Prediction 3 Frame 3 Chain of Thou...

    work page

  53. [53]

    The hand is positioned as if it is about to place the black object down

    **Frame 1**: A hand is holding a small black object near a white object on a blue circular surface. The hand is positioned as if it is about to place the black object down

    work page

  54. [54]

    **Frame 2**: The hand is still holding the black object, and it appears to be in the process of placing it on the blue surface

    work page

  55. [55]

    **Frame 3**: The hand is still holding the black object, and it appears to be in the process of placing it on the blue surface

    work page

  56. [56]

    **Frame 4**: The hand is still holding the black object, and it appears to be in the process of placing it on the blue surface

    work page

  57. [57]

    The white object remains in place

    **Frame 5**: The hand is no longer visible, and the black object is now resting on the blue surface. The white object remains in place

    work page

  58. [58]

    **Frame 6**: The hand reappears, holding the black object again, suggesting a new interaction

    work page

  59. [59]

    The white object remains in place

    **Frame 7**: The hand is no longer visible, and the black object is still resting on the blue surface. The white object remains in place

    work page

  60. [60]

    The white object remains in place

    **Frame 8**: The hand is no longer visible, and the black object is still resting on the blue surface. The white object remains in place

    work page

  61. [61]

    The white object remains in place

    **Frame 9**: The hand is no longer visible, and the black object is still resting on the blue surface. The white object remains in place

    work page

  62. [62]

    From the analysis, the start of an event occurs when the hand first appears holding the black object and moves it towards the blue surface

    **Frame 10**: The hand reappears, holding the black object again, suggesting a new interaction. From the analysis, the start of an event occurs when the hand first appears holding the black object and moves it towards the blue surface. This happens in Frame 1, Frame 6, and Frame

    work page

  63. [63]

    ### Prediction: Prediction 1 Frame 1 Chain of ThoughtFrame 5 Frame 6 Frame 7 False Prediction Event type: Release ### Chain of Thought:

    Since we need to find the earliest start of an event, we focus on Frame 1. ### Prediction: Prediction 1 Frame 1 Chain of ThoughtFrame 5 Frame 6 Frame 7 False Prediction Event type: Release ### Chain of Thought:

    work page

  64. [64]

    Specifically, the hand is pointing at or touching the container

    **Identify the Event**: - The event in question is the hand interacting with the container. Specifically, the hand is pointing at or touching the container

    work page

  65. [65]

    - From Frame 2 onwards, the hand continues to interact with the container in a similar manner

    **Temporal Analysis**: - In Frame 1, the hand is already in contact with the container. - From Frame 2 onwards, the hand continues to interact with the container in a similar manner

    work page

  66. [66]

    - There is no clear indication of a prior state where the hand was not interacting with the container

    **Determine the Start of the Event**: - Since the hand is already in contact with the container in Frame 1, this indicates that the interaction has already started. - There is no clear indication of a prior state where the hand was not interacting with the container

    work page

  67. [67]

    **Conclusion**: - The start of the event, which is the hand beginning to interact with the container, is in Frame 1. ### Prediction: Prediction 1 Frame 1 Chain of ThoughtFrame 8 Frame 9 Frame 10 False Prediction Event type: Contact S-Figure 5.Example false predictions of the model Qwen-2.5VL-72B.The model provides the presented chain-of-thought under the ...

    work page

  68. [68]

    - There is no significant change in the position or action of the hand and scissors

    **Initial State (Frames 1-4)**: - The hand is consistently touching the scissors' blade. - There is no significant change in the position or action of the hand and scissors

    work page

  69. [69]

    - This is the first noticeable change in the sequence, indicating the start of a new event

    **Transition (Frame 5)**: - The hand moves away from the scissors' blade. - This is the first noticeable change in the sequence, indicating the start of a new event

    work page

  70. [70]

    - The hand moves away again in Frame 7

    **Subsequent Frames (Frames 6-10)**: - The hand returns to touch the scissors' blade in Frame 6. - The hand moves away again in Frame 7. - The hand does not interact with the scissors in Frames 8 and 9. - The hand returns to touch the scissors' blade in Frame 10. ### Conclusion The start of the event is marked by the hand moving away from the scissors' bl...

    work page

  71. [71]

    **Frame 1**: The hand is approaching the pen, but there is no clear contact yet

    work page

  72. [72]

    **Frame 2**: The hand is closer to the pen, but still no contact

    work page

  73. [73]

    **Frame 3**: The hand is making contact with the pen, indicating the start of the interaction

    work page

  74. [74]

    Given this analysis, the interaction starts in Frame 3 when the hand makes contact with the pen

    **Frames 4-10**: The hand continues to hold and manipulate the pen, showing ongoing interaction. Given this analysis, the interaction starts in Frame 3 when the hand makes contact with the pen. ### Prediction Prediction 3 Chain of ThoughtFrame 1 Frame 2 Frame 3 True Frame False Prediction Event type: Contact To identify the frame number that represents th...

    work page

  75. [75]

    There is no significant change in the position or state of the objects

    **Frames 1 to 7**: The hand is consistently interacting with the fork, holding it in place. There is no significant change in the position or state of the objects

    work page

  76. [76]

    **Frame 8**: The hand begins to move away from the fork, indicating the start of a new event where the hand is no longer interacting with the fork

    work page

  77. [77]

    Based on this analysis, the start of the event where the hand begins to move away from the fork is in **Frame 8**

    **Frames 9 and 10**: The hand continues to move away, confirming the continuation of the event started in Frame 8. Based on this analysis, the start of the event where the hand begins to move away from the fork is in **Frame 8**. ### Prediction Prediction 8 Chain of ThoughtFrame 7 Frame 8 Frame 9 Correct Prediction Event type: Release S-Figure 6.Additiona...

    work page 1979