REVIEW 4 major objections 6 minor 3 cited by
Embodied VideoAgent: Persistent Memory from Egocentric Videos and Embodied Sensors Enables Dynamic Scene Understanding
T0 review · 4 major / 6 minor · reviewed 2026-08-10 · deepseek-v4-flash
Pith's one-line read A memory agent that fuses egocentric video with depth and pose into a persistently updated object memory beats video-only models on dynamic 3D scene understanding (gains up to 11.7%).
desk verdict Solid extension of VideoAgent with a fair VQ3D result, but the OpenEQA headline gain is undermined by a subset-vs-full comparison and missing ablations. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanism is the persistent object memory: a table of object entries, each holding an object id and category, a state description, relations to nearby objects computed from 3D bounding boxes, a 3D bounding box obtained by 2D-3D lifting using depth and camera pose, and visual features of both the object crop and its surrounding frame. Construction pipelines an open-vocabulary detector into 2D detections, lifts them to 3D boxes, and merges duplicates through a two-stage re-identification procedure that separates static from dynamic objects, using intersection and volume-similarity scores for the former and a visual-similarity threshold for the latter. The companion mechanism is the vision-language-model-based memory update: when the agent perceives an action, it renders each candidate object's 3D box onto the current frame and prompts a vision-language model to decide which memory entry is the action's target, then changes that entry's state and logs the action in a history buffer. This combined memory-plus-update structure is what the paper credits for precise temporal and spatial localization and for answering event, order, and state questions in dynamic scenes.
What would settle it
Running Embodied VideoAgent and the exact same baselines on the full OpenEQA validation set, or ablating the VLM-based memory update and checking whether EnvQA Event and Order accuracies drop, would settle whether the reported gains come from the persistent memory and update mechanism.
Extended reading notes
Core claim
Embodied VideoAgent's central discovery is that fusing egocentric RGB with embodied sensor readings (depth and camera pose) into a persistent per-object memory, and refreshing that memory when actions occur, enables a tool-calling agent to track objects in dynamic scenes more accurately than either end-to-end multimodal LLMs or the video-only memory baseline. Each memory entry holds an object id and category, a state label ("normal", "open", "close", "in hand"), related objects derived from 3D bounding boxes, a 3D bounding box lifted from 2D detections via depth and pose, and visual features of the object crop and its surrounding context. Re-identification merges new detections into existing entries using visual and spatial similarity scores, with separate handling for static and dynamic objects. When an action annotation indicates an activity over an object, the agent visually prompts the vision-language model with rendered 3D boxes to identify the target entry and programmatically updates its state, logging the action in a history buffer. In evaluation, this design yields the reported gains: 85.37% success on Ego4D-VQ3D, 47.0% on the OpenEQA subset, and 25.91, 68.00, and 35.50 on EnvQA event, order, and state categories.
Load-bearing premise
The headline OpenEQA gain assumes that the randomly selected one-fifth subset of the benchmark behaves like the full set, so that baseline scores borrowed from the full benchmark remain valid for comparison.
Editorial extensions
If this is right
- Dynamic 3D scene understanding can be improved by explicit persistent memory rather than by scaling end-to-end video models, because the agent outperforms both on all three benchmarks.
- Egocentric video alone is not sufficient: depth and camera poses contribute the geometry needed for reliable object tracking and 3D localization.
- Associating perceived actions with target objects through vision-language-model visual prompting is a key ingredient for event and order understanding, where the paper reports the largest relative gains.
- The same memory supports embodied behavior beyond question answering: persistent entries let a robot retrieve an occluded object, such as an apple hidden behind a box, and complete a pick-and-place task.
- The agent architecture can generate synthetic user-assistant interaction episodes by having one LLM propose tasks and the memory-augmented agent execute them in a simulator.
Reading between the lines
- Editorial extension: the claimed 5.8% OpenEQA gain would be on firmer ground if the full benchmark were scored with identical baselines; the paper's random one-fifth subset makes the comparison with full-set borrowed baselines an assumption about representativeness.
- Editorial extension: if the memory update is the true cause of the EnvQA gains, then disabling the VLM-based update while keeping all other components should substantially reduce Event and Order accuracy; this ablation is not reported in the paper.
- Editorial extension: the method's robustness to noisy poses suggests a natural testable extension of replacing ground-truth sensor inputs with monocular depth and pose estimators end-to-end, then measuring the same three benchmarks.
- Editorial extension: the persistent memory design could be carried into long-horizon robot deployments where the same object is revisited over hours, a use case the paper illustrates only in short manipulation scenes.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes Embodied VideoAgent, an LLM-based multimodal agent that extends the prior VideoAgent system by constructing a persistent object memory from egocentric video, depth maps, and camera poses, and by updating that memory with a VLM when actions or activities are perceived. The agent also maintains action and visible-object history buffers and uses four perception tools plus seven embodied action primitives. It is evaluated on Ego4D-VQ3D object localization, OpenEQA embodied question answering, and EnvQA video question answering, with reported gains of 4.9%, 5.8%, and 11.7% respectively, and is further demonstrated on synthetic embodied interaction generation and a robot manipulation scenario. The central claim is that the persistent memory architecture, rather than the underlying LLM/VLM backbone, is responsible for the improvements.
Significance. If the empirical claims hold, the paper would demonstrate a practical and cost-effective way to handle dynamic 3D scene understanding by augmenting an LLM agent with structured persistent memory and VLM-based memory updates. The manuscript has several strengths: the VQ3D evaluation shares precomputed camera poses and depth maps across methods, making that comparison clean; the appendix provides detailed algorithms (object re-ID, memory update, tool prompts) and honest failure cases; and the application demonstrations, while qualitative, are suggestive of downstream utility. However, the headline OpenEQA number is not established because it compares a one-fifth subset score against full-benchmark baselines, and no ablation isolates the memory contribution from the stronger GPT-4o/InternVL2-8B backbone. The EnvQA and VQ3D results also lack statistical grounding. These issues are load-bearing for the paper's central claims, though they appear fixable with additional experiments.
major comments (4)
- [Section 3.2.1, Table 2] The abstract's claim of a 5.8% gain on OpenEQA is not supported by the reported experiments. Embodied VideoAgent is evaluated on a randomly selected one-fifth subset of OpenEQA, while the GPT-4 w/LLaVA-1.5 (43.6 ALL) and GPT-4 w/CG (36.5 ALL) rows are borrowed from full-benchmark evaluations. The paper argues the subset is harder based on Video-LLaVA and LLaMA-VID drops, but this does not make a subset score comparable to a full-set score for different baselines. If the intended comparison is 47.0 (subset) against 43.6 (full set), the claimed 5.8% gain is not established; if the 5.8% refers to the gap between the GPT-4o and InternVL2-8B variants (47.0 vs 41.2), then it is not a gain over an external baseline. Please evaluate all baselines on the exact same subset, or run Embodied VideoAgent on the full OpenEQA set, and report per-split counts and standard errors.
- [Section 3.2.2, Table 2] No ablation isolates the contribution of the persistent object memory, VLM-based memory update, and history buffers from the choice of backbone models. Embodied VideoAgent (GPT-4o) differs from VideoAgent not only in the memory design but also in using GPT-4o as the LLM and InternVL2-8B for VQA, so the observed gains over VideoAgent could be due in part to the stronger backbone. Please provide an ablation that either runs VideoAgent with the same GPT-4o/InternVL2-8B tools or removes the persistent memory and update mechanism from Embodied VideoAgent while keeping all other components fixed. This is necessary to support the paper's attribution that 'the strong performance of Embodied VideoAgent mainly attributed to the precise frame localization using consistent object memory and history buffer.'
- [Section 3.2.3, Table 3] The EnvQA results are reported as per-category accuracies on 200 questions per type with no variance or significance testing, and the abstract's 11.7% gain is not derivable from the table. The largest gain is in Events (25.91 vs 10.19 for Video-LLaVA), but the Orders gain over VideoAgent is only 2.5 points (68.0 vs 65.5), which is within likely noise for 200 questions. Please state the aggregation rule used to compute the 11.7% (e.g., macro-average over the three categories) and provide confidence intervals, a paired test, or a bootstrap estimate. Without this, the 'significant performance gain' is not quantitatively supported.
- [Section 3.1, Table 1] The VQ3D headline is a 4.88-point absolute gain in Succ% (85.37 vs 80.49) on a single validation set. Given that the validation set is small (QwP% values imply roughly 80-90 queries), this difference could correspond to only a few queries. Please report the number of queries and provide a bootstrap confidence interval or a McNemar test to establish that the gain is not due to chance. As it stands, the '5%' improvement is not statistically grounded.
minor comments (6)
- [Section 3.1.2] The text says Embodied VideoAgent (image) surpasses EgoLoc by 5%, but Table 1 shows 85.37 vs 80.49, which is 4.88%; please use the exact value or say 'approximately 5%'.
- [Section 2.2, Note on camera poses] The note states that camera poses for Ego4D-VQ3D are estimated using COLMAP, but Section 3.1.1 says all VQ3D methods use the same precomputed poses and depth images provided by EgoLoc. Please clarify whether EgoLoc's poses are the COLMAP poses, or whether the COLMAP remark applies only to a different experiment.
- [Section 3.2.2] The sentence 'validated by the better performance of Embodied VideoAgent (+16.4% over Video-LLaVA on subset) over GPT-4 w/CG (+4.4% over Video-LLaVA on full set)' appears to reference GPT-4 w/CG, but the +4.4% value corresponds to GPT-4 w/LLaVA-1.5; GPT-4 w/CG actually scores below Video-LLaVA. Please correct this attribution.
- [Figure 12 caption] The caption says 'placing the lamb on a table' in the context of a lamp object; this should be 'lamp'.
- [Throughout] The paper uses both 'Embodied VideoAgent' and 'E-VideoAgent' interchangeably. Define the abbreviation at first use and use it consistently.
- [Abstract and Tables 1-3] The abstract's numerical gains (4.9%, 5.8%, 11.7%) are not all directly traceable to the tables; please provide the exact formulas or comparisons used for each number.
Circularity Check
No significant circularity: the paper's claims are empirical benchmark results, not derivations; no fitted input is renamed as a prediction and no load-bearing argument reduces to a self-citation.
full rationale
Embodied VideoAgent is a system paper. Its central claims are benchmark scores on Ego4D-VQ3D, OpenEQA, and EnvQA, and the construction is an engineering pipeline rather than a derivation chain. The object memory is built from video, depth, and poses with off-the-shelf detectors (YOLO-world, SAM-2, CLIP/DINOv2) plus hand-set thresholds (0.2, 0.45, 0.7), but none of these components is fitted to or defined in terms of the benchmark answers. The re-ID visual similarity coefficients (0.15/0.85) and the VideoAgent design are inherited from the authors' prior work [7], yet this is a transparent reuse of a component, not a self-citation that supplies the paper's conclusion: the new results are measured against EgoLoc, VideoAgent, Video-LLaVA, LLaMA-VID, and GPT-4 baselines on held-out benchmarks. The VLM-based memory update and action buffer use action annotations and visual prompting to update STATE fields; answering EnvQA questions from that buffer is the system's intended operation, not a tautology. The one methodological weakness--comparing a one-fifth OpenEQA subset against full-benchmark borrowed baselines--undermines the statistical force of the 5.8% OpenEQA gain, but that is an evaluation-validity concern, not circularity: the subset scores are not constructed from the baseline numbers. No equation in the paper reduces a target quantity to its own input, and no self-citation is load-bearing in place of evidence.
Assumptions & free parameters
free parameters (5)
- Visual similarity weights =
0.15 and 0.85
- Static re-ID threshold =
0.2
- Dynamic re-ID thresholds =
0.7 and 0.45
- Dynamic-object detection visual threshold =
0.45
- Moving average window sizes =
10 static, 2 dynamic
assumptions (5)
- domain assumption Pretrained perception models (YOLO-world, SAM-2, CLIP, DINOv2, InternVL2, GPT-4o, LaViLa) provide sufficiently accurate detections, masks, features, and annotations for the memory pipeline.
- domain assumption 2D-3D lifting from depth and camera poses yields 3D bounding boxes accurate enough for re-ID and relation extraction.
- domain assumption Camera poses and depth maps are available or estimable.
- ad hoc to paper The one-fifth OpenEQA subset is representative of the full benchmark.
- domain assumption LaViLa action annotations every two seconds and VLM association correctly identify action target objects.
invented entities (2)
-
Persistent object memory entry
-
Action and visible object history buffers
Cite this review
Pith. "Pith review of Embodied VideoAgent: Persistent Memory from Egocentric Videos and Embodied Sensors Enables Dynamic Scene Understanding." pith.science (2026). https://pith.science/paper/CS5HOXXP
@misc{pith2026250100358,
author = {Pith},
title = {Pith review of: Embodied VideoAgent: Persistent Memory from Egocentric Videos and Embodied Sensors Enables Dynamic Scene Understanding},
year = {2026},
howpublished = {\url{https://pith.science/paper/CS5HOXXP}},
note = {Machine review of arXiv:2501.00358}
}
read the original abstract
This paper investigates the problem of understanding dynamic 3D scenes from egocentric observations, a key challenge in robotics and embodied AI. Unlike prior studies that explored this as long-form video understanding and utilized egocentric video only, we instead propose an LLM-based agent, Embodied VideoAgent, which constructs scene memory from both egocentric video and embodied sensory inputs (e.g. depth and pose sensing). We further introduce a VLM-based approach to automatically update the memory when actions or activities over objects are perceived. Embodied VideoAgent attains significant advantages over counterparts in challenging reasoning and planning tasks in 3D scenes, achieving gains of 4.9% on Ego4D-VQ3D, 5.8% on OpenEQA, and 11.7% on EnvQA. We have also demonstrated its potential in various embodied AI tasks including generating embodied interactions and perception for robot manipulation. The code and demo will be made public.
Figures
Figures from the paper (12 more)
Forward citations
Cited by 3 Pith papers
-
StateScribe: Towards Accessible Change Awareness Across Real-World Revisits
StateScribe uses a dual-layer memory architecture for episodic scenes and object-centric changes to deliver live and historical descriptions, achieving 83.1% F1 accuracy across revisits in evaluations and user studies...
-
Move to Understand a 3D Scene: Bridging Visual Grounding and Exploration for Efficient and Versatile Embodied Navigation
MTU3D unifies visual grounding and frontier-based exploration in a single transformer, achieving state-of-the-art success rates on HM3D-OVON, GOAT-Bench, SG3D, and A-EQA after large-scale vision-language-exploration p...
-
Dense360: Dense Understanding from Omnidirectional Panoramas
Introduces a 160K-panorama auto-annotated dataset, a dense captioning and grounding benchmark, and ERP-RoPE; fine-tuning Qwen2.5VL on the data lifts benchmark scores.
Reference graph
Works this paper leans on
-
[1]
Rocket-1: Master open-world interaction with visual-temporal context prompting
Shaofei Cai, Zihao Wang, Kewei Lian, Zhancun Mu, Xiaojian Ma, Anji Liu, and Yitao Liang. Rocket-1: Master open-world interaction with visual-temporal context prompting. arXiv preprint arXiv:2410.17856, 2024. 4
arXiv 2024
-
[2]
How far are we to gpt-4v? closing the gap to commercial multimodal models with open-source suites
Zhe Chen, Weiyun Wang, Hao Tian, Shenglong Ye, Zhangwei Gao, Erfei Cui, Wenwen Tong, Kongzhi Hu, Jiapeng Luo, Zheng Ma, et al. How far are we to gpt-4v? closing the gap to commercial multimodal models with open-source suites. arXiv preprint arXiv:2404.16821, 2024. 6
arXiv 2024
-
[3]
Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks
Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 24185–24198, 2024. 6
2024
-
[4]
Yolo-world: Real-time open- vocabulary object detection
Tianheng Cheng, Lin Song, Yixiao Ge, Wenyu Liu, Xing- gang Wang, and Ying Shan. Yolo-world: Real-time open- vocabulary object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 16901–16911, 2024. 3, 5, 12
2024
-
[5]
Rescaling egocentric vision: Collection, pipeline and challenges for epic-kitchens-100
Dima Damen, Hazel Doughty, Giovanni Maria Farinella, An- tonino Furnari, Jian Ma, Evangelos Kazakos, Davide Molti- santi, Jonathan Munro, Toby Perrett, Will Price, and Michael Wray. Rescaling egocentric vision: Collection, pipeline and challenges for epic-kitchens-100. International Journal of Computer Vision (IJCV), 130:33–55, 2022. 1, 8
2022
-
[6]
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Ab- hishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783,
-
[7]
Videoagent: A memory-augmented multi- modal agent for video understanding
Yue Fan, Xiaojian Ma, Rujie Wu, Yuntao Du, Jiaqi Li, Zhi Gao, and Qing Li. Videoagent: A memory-augmented multi- modal agent for video understanding. In European Confer- ence on Computer Vision, pages 75–92. Springer, 2025. 1, 2, 3, 6, 8, 12
2025
-
[8]
Env-qa: A video question answering benchmark for comprehensive understanding of dynamic environments
Difei Gao, Ruiping Wang, Ziyi Bai, and Xilin Chen. Env-qa: A video question answering benchmark for comprehensive understanding of dynamic environments. In Proceedings of the IEEE/CVF international conference on computer vision, pages 1675–1685, 2021. 2, 6
2021
Show all 87 references
-
[9]
Clova: A closed-loop visual assistant with tool usage and update
Zhi Gao, Yuntao Du, Xintong Zhang, Xiaojian Ma, Wenjuan Han, Song-Chun Zhu, and Qing Li. Clova: A closed-loop visual assistant with tool usage and update. CVPR, 2023. 2, 8
2023
-
[10]
Mindagent: Emergent gaming interaction
Ran Gong, Qiuyuan Huang, Xiaojian Ma, Hoi V o, Zane Du- rante, Yusuke Noda, Zilong Zheng, Song-Chun Zhu, Demetri Terzopoulos, Li Fei-Fei, et al. Mindagent: Emergent gaming interaction. arXiv preprint arXiv:2309.09971, 2023. 2, 5
2023 arXiv
-
[11]
Ego4d: Around the world in 3,000 hours of egocentric video
Kristen Grauman, Andrew Westbury, Eugene Byrne, Zachary Chavis, Antonino Furnari, Rohit Girdhar, Jackson Hamburger, Hao Jiang, Miao Liu, Xingyu Liu, et al. Ego4d: Around the world in 3,000 hours of egocentric video. In Proceedings of the IEEE/CVF Conference on Computer Vision ...
2022
-
[12]
Concept- graphs: Open-vocabulary 3d scene graphs for perception and planning
Qiao Gu, Ali Kuwajerwala, Sacha Morin, Krishna Murthy Jatavallabhula, Bipasha Sen, Aditya Agarwal, Corban Rivera, William Paul, Kirsty Ellis, Rama Chellappa, et al. Concept- graphs: Open-vocabulary 3d scene graphs for perception and planning. In 2024 IEEE International Confere...
2024
-
[13]
Semantic gaussians: Open-vocabulary scene understanding with 3d gaussian splatting
Jun Guo, Xiaojian Ma, Yue Fan, Huaping Liu, and Qing Li. Semantic gaussians: Open-vocabulary scene understanding with 3d gaussian splatting. arXiv preprint arXiv:2403.15624,
-
[14]
Visual program- ming: Compositional visual reasoning without training
Tanmay Gupta and Aniruddha Kembhavi. Visual program- ming: Compositional visual reasoning without training. In CVPR, 2023. 2, 8
2023
-
[15]
Temporal alignment networks for long-term video
Tengda Han, Weidi Xie, and Andrew Zisserman. Temporal alignment networks for long-term video. In CVPR, 2022. 1
2022
-
[16]
Chat-scene: Bridging 3d scene and large language models with object identifiers
Haifeng Huang, Yilun Chen, Zehan Wang, Rongjie Huang, Runsen Xu, Tai Wang, Luping Liu, Xize Cheng, Yang Zhao, Jiangmiao Pang, et al. Chat-scene: Bridging 3d scene and large language models with object identifiers. In The Thirty- eighth Annual Conference on Neural Information P...
2024
-
[17]
An embodied generalist agent in 3d world
Jiangyong Huang, Silong Yong, Xiaojian Ma, Xiongkun Linghu, Puhao Li, Yan Wang, Qing Li, Song-Chun Zhu, Baox- iong Jia, and Siyuan Huang. An embodied generalist agent in 3d world. arXiv preprint arXiv:2311.12871, 2023. 1, 3
2023 arXiv
-
[18]
Con- ceptfusion: Open-set multimodal 3d mapping
Krishna Murthy Jatavallabhula, Alihusein Kuwajerwala, Qiao Gu, Mohd Omama, Tao Chen, Alaa Maalouf, Shuang Li, Ganesh Iyer, Soroush Saryazdi, Nikhil Keetha, et al. Con- ceptfusion: Open-set multimodal 3d mapping. arXiv preprint arXiv:2302.07241, 2023. 1, 8
2023 arXiv
-
[19]
Lemma: A multi-view dataset for le arning m ulti-agent m ulti-task a ctivities
Baoxiong Jia, Yixin Chen, Siyuan Huang, Yixin Zhu, and Song-chun Zhu. Lemma: A multi-view dataset for le arning m ulti-agent m ulti-task a ctivities. In ECCV, 2020. 1
2020
-
[20]
Egotaskqa: Understanding human tasks in egocentric videos
Baoxiong Jia, Ting Lei, Song-Chun Zhu, and Siyuan Huang. Egotaskqa: Understanding human tasks in egocentric videos. NeurIPS, 2022. 1
2022
-
[21]
Sceneverse: Scaling 3d vision-language learning for grounded scene un- derstanding
Baoxiong Jia, Yixin Chen, Huangyue Yu, Yan Wang, Xuesong Niu, Tengyu Liu, Qing Li, and Siyuan Huang. Sceneverse: Scaling 3d vision-language learning for grounded scene un- derstanding. In European Conference on Computer Vision, pages 289–310. Springer, 2025. 3
-
[22]
Habitat synthetic scenes dataset (hssd-200): An analysis of 3d scene 9 scale and realism tradeoffs for objectgoal navigation
Mukul Khanna, Yongsen Mao, Hanxiao Jiang, Sanjay Haresh, Brennan Shacklett, Dhruv Batra, Alexander Clegg, Eric Un- dersander, Angel X Chang, and Manolis Savva. Habitat synthetic scenes dataset (hssd-200): An analysis of 3d scene 9 scale and realism tradeoffs for objectgoal nav...
2024
-
[23]
Text-conditioned resam- pler for long form video understanding
Bruno Korbar, Yongqin Xian, Alessio Tonioni, Andrew Zis- serman, and Federico Tombari. Text-conditioned resam- pler for long form video understanding. arXiv preprint arXiv:2312.11897, 2023. 1
2023 arXiv
-
[24]
Llama-vid: An im- age is worth 2 tokens in large language models
Yanwei Li, Chengyao Wang, and Jiaya Jia. Llama-vid: An im- age is worth 2 tokens in large language models. In European Conference on Computer Vision, pages 323–340. Springer,
-
[25]
Video-llava: Learning united visual rep- resentation by alignment before projection
Bin Lin, Yang Ye, Bin Zhu, Jiaxi Cui, Munan Ning, Peng Jin, and Li Yuan. Video-llava: Learning united visual rep- resentation by alignment before projection. arXiv preprint arXiv:2311.10122, 2023. 1, 6
2023 arXiv
-
[26]
Egocentric video-language pretraining
Kevin Qinghong Lin, Jinpeng Wang, Mattia Soldan, Michael Wray, Rui Yan, Eric Z Xu, Difei Gao, Rong-Cheng Tu, Wen- zhe Zhao, Weijie Kong, et al. Egocentric video-language pretraining. Advances in Neural Information Processing Sys- tems, 35:7575–7586, 2022. 1, 8
2022
-
[27]
Visual instruction tuning
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. NeurIPS, 2024. 1
2024
-
[28]
Best practices and lessons learned on synthetic data
Ruibo Liu, Jerry Wei, Fangyu Liu, Chenglei Si, Yanzhe Zhang, Jinmeng Rao, Steven Zheng, Daiyi Peng, Diyi Yang, Denny Zhou, et al. Best practices and lessons learned on synthetic data. In First Conference on Language Modeling,
-
[29]
Sqa3d: Situated question answering in 3d scenes
Xiaojian Ma, Silong Yong, Zilong Zheng, Qing Li, Yitao Liang, Song-Chun Zhu, and Siyuan Huang. Sqa3d: Situated question answering in 3d scenes. In ICLR, 2023. 8
2023
-
[30]
Video-chatgpt: Towards detailed video understanding via large vision and language models
Muhammad Maaz, Hanoona Rasheed, Salman Khan, and Fahad Shahbaz Khan. Video-chatgpt: Towards detailed video understanding via large vision and language models. arXiv preprint arXiv:2306.05424, 2023. 8
2023 arXiv
-
[31]
Egoloc: Revisiting 3d object localization from egocentric videos with visual queries
Jinjie Mai, Abdullah Hamdi, Silvio Giancola, Chen Zhao, and Bernard Ghanem. Egoloc: Revisiting 3d object localization from egocentric videos with visual queries. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 45–57, 2023. 2, 5
2023
-
[32]
Openeqa: Embodied question answering in the era of foundation models
Arjun Majumdar, Anurag Ajay, Xiaohan Zhang, Pranav Putta, Sriram Yenamandra, Mikael Henaff, Sneha Silwal, Paul Mc- vay, Oleksandr Maksymets, Sergio Arnaud, et al. Openeqa: Embodied question answering in the era of foundation models. In Proceedings of the IEEE/CVF Conference on...
2024
-
[33]
Egoschema: A diagnostic benchmark for very long- form video language understanding
Karttikeya Mangalam, Raiymbek Akshulakov, and Jitendra Malik. Egoschema: A diagnostic benchmark for very long- form video language understanding. NeurIPS, 2024. 1, 8
2024
-
[34]
Howto100m: Learning a text-video embedding by watching hundred mil- lion narrated video clips
Antoine Miech, Dimitri Zhukov, Jean-Baptiste Alayrac, Makarand Tapaswi, Ivan Laptev, and Josef Sivic. Howto100m: Learning a text-video embedding by watching hundred mil- lion narrated video clips. In ICCV, 2019. 1
2019
-
[35]
Gpt-4 technical report
OpenAI. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023. 8
2023 arXiv
-
[36]
Dinov2: Learning robust visual features without supervision
Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy V o, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07193, 2023. 12
2023 arXiv
-
[37]
Egovlpv2: Egocentric video-language pre-training with fusion in the backbone
Shraman Pramanick, Yale Song, Sayan Nag, Kevin Qinghong Lin, Hardik Shah, Mike Zheng Shou, Rama Chellappa, and Pengchuan Zhang. Egovlpv2: Egocentric video-language pre-training with fusion in the backbone. In Proceedings of the IEEE/CVF International Conference on Computer Vis...
2023
-
[38]
Learning transferable visual models from natural language supervision
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision
-
[39]
Sam 2: Segment any- thing in images and videos
Nikhila Ravi, Valentin Gabeur, Yuan-Ting Hu, Ronghang Hu, Chaitanya Ryali, Tengyu Ma, Haitham Khedr, Roman Rädle, Chloe Rolland, Laura Gustafson, et al. Sam 2: Segment any- thing in images and videos. arXiv preprint arXiv:2408.00714,
-
[40]
Concepta- gent: Llm-driven precondition grounding and tree search for robust task planning and execution
Corban Rivera, Grayson Byrd, William Paul, Tyler Feldman, Meghan Booker, Emma Holmes, David Handelman, Bethany Kemp, Andrew Badger, Aurora Schmidt, et al. Concepta- gent: Llm-driven precondition grounding and tree search for robust task planning and execution. arXiv preprint a...
-
[41]
Scene representation transformer: Geometry-free novel view synthe- sis through set-latent scene representations
Mehdi SM Sajjadi, Henning Meyer, Etienne Pot, Urs Bergmann, Klaus Greff, Noha Radwan, Suhani V ora, Mario Luˇci´c, Daniel Duckworth, Alexey Dosovitskiy, et al. Scene representation transformer: Geometry-free novel view synthe- sis through set-latent scene representations. In P...
2022
-
[42]
Habitat: A platform for embodied ai research
Manolis Savva, Abhishek Kadian, Oleksandr Maksymets, Yili Zhao, Erik Wijmans, Bhavana Jain, Julian Straub, Jia Liu, Vladlen Koltun, Jitendra Malik, et al. Habitat: A platform for embodied ai research. In ICCV, 2019. 8, 21
2019
-
[43]
Structure-from-motion revisited
Johannes Lutz Schönberger and Jan-Michael Frahm. Structure-from-motion revisited. In Conference on Computer Vision and Pattern Recognition (CVPR), 2016. 5
2016
-
[44]
Scene representation networks: Continuous 3d-structure- aware neural scene representations
Vincent Sitzmann, Michael Zollhöfer, and Gordon Wetzstein. Scene representation networks: Continuous 3d-structure- aware neural scene representations. Advances in Neural In- formation Processing Systems, 32, 2019. 1
2019
-
[45]
Vipergpt: Visual inference via python execution for reasoning
Dídac Surís, Sachit Menon, and Carl V ondrick. Vipergpt: Visual inference via python execution for reasoning. In ICCV,
-
[46]
Movieqa: Understanding stories in movies through question-answering
Makarand Tapaswi, Yukun Zhu, Rainer Stiefelhagen, Anto- nio Torralba, Raquel Urtasun, and Sanja Fidler. Movieqa: Understanding stories in movies through question-answering. In CVPR, 2016. 1
2016
-
[47]
Gemini: a family of highly capable multimodal models
Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805, 2023. 1, 8
2023 arXiv
-
[48]
Gemini 1.5: Unlocking mul- timodal understanding across millions of tokens of context
Gemini Team, Petko Georgiev, Ving Ian Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, 10 Zhufeng Pan, Shibo Wang, et al. Gemini 1.5: Unlocking mul- timodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.05530, 2024. 1, 8
2024 arXiv
-
[49]
Dust3r: Geometric 3d vision made easy
Shuzhe Wang, Vincent Leroy, Yohann Cabon, Boris Chidlovskii, and Jerome Revaud. Dust3r: Geometric 3d vision made easy. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pages 20697–20709, 2024. 5, 15
2024
-
[50]
Lstp: Language-guided spatial-temporal prompt learning for long-form video-text understanding
Yuxuan Wang, Yueqian Wang, Pengfei Wu, Jianxin Liang, Dongyan Zhao, and Zilong Zheng. Lstp: Language-guided spatial-temporal prompt learning for long-form video-text understanding. arXiv preprint arXiv:2402.16050, 2024. 8
2024 arXiv
-
[51]
Jarvis-1: Open-world multi-task agents with memory-augmented multimodal language models
Zihao Wang, Shaofei Cai, Anji Liu, Yonggang Jin, Jinbing Hou, Bowei Zhang, Haowei Lin, Zhaofeng He, Zilong Zheng, Yaodong Yang, et al. Jarvis-1: Open-world multi-task agents with memory-augmented multimodal language models. arXiv preprint arXiv:2311.05997, 2023. 1
2023 arXiv
-
[52]
Describe, explain, plan and select: Interactive planning with large language models enables open-world multi-task agents
Zihao Wang, Shaofei Cai, Anji Liu, Xiaojian Ma, and Yitao Liang. Describe, explain, plan and select: Interactive planning with large language models enables open-world multi-task agents. NeurIPS, 2023
2023
-
[53]
Omnijarvis: Unified vision-language-action tokenization enables open-world instruction following agents
Zihao Wang, Shaofei Cai, Zhancun Mu, Haowei Lin, Ceyao Zhang, Xuejie Liu, Qing Li, Anji Liu, Xiaojian Ma, and Yitao Liang. Omnijarvis: Unified vision-language-action tokenization enables open-world instruction following agents. arXiv preprint arXiv:2407.00114, 2024. 1
2024 arXiv
-
[54]
Compressed vision for efficient video understanding
Olivia Wiles, Joao Carreira, Iain Barr, Andrew Zisserman, and Mateusz Malinowski. Compressed vision for efficient video understanding. In ACCV, 2022. 1
2022
-
[55]
Visual chatgpt: Talking, draw- ing and editing with visual foundation models
Chenfei Wu, Shengming Yin, Weizhen Qi, Xiaodong Wang, Zecheng Tang, and Nan Duan. Visual chatgpt: Talking, draw- ing and editing with visual foundation models. arXiv preprint arXiv:2303.04671, 2023. 2
2023 arXiv
-
[56]
Towards long-form video understanding
Chao-Yuan Wu and Philipp Krahenbuhl. Towards long-form video understanding. In CVPR, 2021. 1
2021
-
[57]
Next-qa: Next phase of question-answering to explaining temporal actions
Junbin Xiao, Xindi Shang, Angela Yao, and Tat-Seng Chua. Next-qa: Next phase of question-answering to explaining temporal actions. In CVPR, 2021. 8
2021
-
[58]
Set-of-mark prompting unleashes extraordinary visual grounding in gpt-4v
Jianwei Yang, Hao Zhang, Feng Li, Xueyan Zou, Chunyuan Li, and Jianfeng Gao. Set-of-mark prompting unleashes extraordinary visual grounding in gpt-4v. arXiv preprint arXiv:2310.11441, 2023. 4
2023 arXiv
-
[59]
Doraemongpt: Toward understanding dynamic scenes with large language models (exemplified as a video agent)
Zongxin Yang, Guikun Chen, Xiaodi Li, Wenguan Wang, and Yi Yang. Doraemongpt: Toward understanding dynamic scenes with large language models (exemplified as a video agent). 2024. 8
2024
-
[60]
Video-llama: An instruction-tuned audio-visual language model for video un- derstanding
Hang Zhang, Xin Li, and Lidong Bing. Video-llama: An instruction-tuned audio-visual language model for video un- derstanding. arXiv preprint arXiv:2306.02858, 2023. 8
2023 arXiv
-
[61]
Worldqa: Multimodal world knowledge in videos through long-chain reasoning
Yuanhan Zhang, Kaichen Zhang, Bo Li, Fanyi Pu, Christo- pher Arif Setiadharma, Jingkang Yang, and Ziwei Liu. Worldqa: Multimodal world knowledge in videos through long-chain reasoning. arXiv preprint arXiv:2405.03272, 2024. 8
2024 arXiv
-
[62]
Mmicl: Empowering vision-language model with multi-modal in-context learning
Haozhe Zhao, Zefan Cai, Shuzheng Si, Xiaojian Ma, Kaikai An, Liang Chen, Zixuan Liu, Sheng Wang, Wenjuan Han, and Baobao Chang. Mmicl: Empowering vision-language model with multi-modal in-context learning. arXiv preprint arXiv:2309.07915, 2023. 1
2023 arXiv
-
[63]
Learning video representations from large language models
Yue Zhao, Ishan Misra, Philipp Krähenbühl, and Rohit Gird- har. Learning video representations from large language models. In CVPR, 2023. 14
2023
-
[64]
3d-vista: Pre-trained transformer for 3d vision and text alignment
Ziyu Zhu, Xiaojian Ma, Yixin Chen, Zhidong Deng, Siyuan Huang, and Qing Li. 3d-vista: Pre-trained transformer for 3d vision and text alignment. In ICCV, 2023. 1, 3
2023
-
[65]
open", "close
Ziyu Zhu, Zhuofan Zhang, Xiaojian Ma, Xuesong Niu, Yixin Chen, Baoxiong Jia, Zhidong Deng, Siyuan Huang, and Qing Li. Unifying 3d vision-language understanding via prompt- able queries. In European Conference on Computer Vision, pages 188–206. Springer, 2025. 1 11 A. Fields of...
2025
-
[66]
What is the color of the object in the image?
Use double quotes for all string arguments when using tools, avoiding any single quotes. If the tool has two arguments, format the output within brack- ets, like this: ("What is the color of the object in the image?", 16)
-
[67]
Prioritize using retrieve_objects_by_appearance, retrieve_objects_by_environment, and frame_localization for object and frame local- ization, rather than query_database
-
[68]
Once the relevant frames are located, use frame_VQA to ask questions about these frames
-
[69]
Use frame_VQA to provide these answers
For "where" questions, interpret them as spatial location queries, not as requests to identify specific frames. Use frame_VQA to provide these answers
-
[70]
What color are the blinds?
If the information is insufficient for a precise response, generate a response based on the available 15 data. Use the following format: Question: the input question you must answer Thought: you should always think about what to do Action: the action to take, should be one of ...
-
[71]
The task should involve no more than 2 objects from the room
-
[72]
The robot should complete the task using ba- sic actions like GOTO, OPEN, CLOSE, PICK, and PLACE
-
[73]
If the robot asks for the location of an object, prompt it to search rather than giving explicit details
-
[74]
a dish sponge
Use general object categories instead of specific IDs (e.g., say "a dish sponge" instead of "dish sponge 1")
-
[75]
Once the task is completed, express satisfaction and thank the robot
Adjust the task if the robot encounters difficulties. Once the task is completed, express satisfaction and thank the robot. Start by initiating a casual conversation and assign- ing a simple task! You are acting as a robot in an apartment. The available receptacles are: {recep...
-
[76]
Use the CHAT tool frequently to communicate in a casual manner, keeping the user informed of your progress
-
[77]
Example: GOTO(’glass’), then PICK(’glass’); GOTO(’fridge’), then OPEN(’fridge’)
For every action involving an object or re- ceptacle, first GOTO the target and then perform actions like PICK, PLACE, OPEN, or CLOSE. Example: GOTO(’glass’), then PICK(’glass’); GOTO(’fridge’), then OPEN(’fridge’)
-
[78]
Ensure your inventory is empty before picking up a new object
-
[79]
Use GOTO, OPEN, and CLOSE to check inside these recepta- cles
The SEARCH tool can find objects by navigating the room, but it cannot check inside articulated re- ceptacles (like fridges or microwaves). Use GOTO, OPEN, and CLOSE to check inside these recepta- cles
-
[80]
Where can I find something to drink
Before completing the task, use CHAT to confirm the user’s satisfaction. Use the following format: 21 Task: the initial task assigned by the user Thought: you should always think about what to do Action: the action to take, should be one of [{tool_names}] Action Input: the inp...
-
[81]
The stone floor is speckled with darker and lighter tones, creating a mosaic-like pattern
**Natural Stone Walls & Floor:** The walls and floor are made of natural stone, which adds to the rustic aesthetic. The stone floor is speckled with darker and lighter tones, creating a mosaic-like pattern
-
[82]
**Wooden Beams:** There are exposed wooden beams supporting the ceiling, giving a sturdy and traditional feel
-
[83]
The door on the left is ajar, revealing a room with a wooden table and a lamp stand
**Wooden Doors:** Two wooden doors are present. The door on the left is ajar, revealing a room with a wooden table and a lamp stand. The door on the right stands tall against the stone wall
-
[84]
**Lighting Fixtures:** A charming wall-mounted lamp with a cream shade is visible, providing warm lighting
-
[85]
**Interior Details:** There is also a wooden chest with a unique metal clasp, suggesting it’s either a storage piece or perhaps a decorative item
-
[86]
**Wooden Cabinets:** The small alcove on the left features wooden cabinets, which implies functional storage space within the room
-
[87]
wine glass
**Entrance/Exit:** There is an open gate leading into the room on the left, furthering the visual appeal of the space, enhancing the sense of depth and accessibility within the interior. The combination of natural materials, simple yet elegant furniture, combined with function...
Reviewed August 10, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.