Pith. sign in

REVIEW 5 major objections 8 minor 51 references

ECBench: Can Multi-modal Foundation Models Understand the Egocentric World? A Holistic Embodied Cognition Benchmark

T0 review · 5 major / 8 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read ECBench is a 4,324-question benchmark showing that top vision-language models score only about half as well as humans on embodied cognition from egocentric video.

desk verdict Useful new embodied-cognition benchmark with genuinely new question types, but the evaluation layer needs a validated judge and toned-down claims before I'd trust the rankings. read the letter →

arxiv 2501.05031 v2 pith:7VEBINRN submitted 2025-01-09 cs.CV cs.LGcs.RO

Ronghao Dang , Yuqian Yuan , Wenqi Zhang , Yifei Xin , Boqiang Zhang , Long Li , Liuyi Wang , Qinyang Zeng
show 2 more authors
Xin Li Lidong Bing
This is my paper · ORCID
classification cs.CVcs.LGcs.RO
keywords embodiedcognitionegocentricvideounderstandingvision-languagemodelsdynamicsceneperceptionhallucinationevaluationrobot-centricquestionansweringRGB-Dbenchmarkmulti-modal
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

ECBench is built on the idea that before a vision-language model can reliably drive a robot, it must be tested on the embodied skills a robot actually needs—not just object recognition in static third-person scenes. The paper constructs a benchmark of 4,324 human-annotated question-answer pairs over 386 egocentric RGB-D videos, organized into 30 cognitive dimensions spanning static scenes, dynamic scenes, and hallucination. With it, the authors report that every model they test scores far below human level: the best, GPT-4o with 32 frames, reaches 50.35% overall while humans reach 94.96%, and a blind GPT-4o given no video still scores 24.09%. The central claim is that current models lack reliable first-person self-awareness, dynamic scene perception, and the ability to reject bad user instructions, and that a systematic benchmark is what exposes these gaps.

What carries the argument

The load-bearing machinery is the 30-dimensional capability taxonomy itself, which turns 'embodied cognition' into measurable question categories: 19 static-scene abilities split into scene-based and robot-centric cognition, four dynamic-scene categories (information, quantity, spatial, and state dynamics), and seven hallucination dimensions covering over-confidence in common sense and in user input. The taxonomy is enforced by class-independent human annotation—annotators are assigned quotas per ability rather than per video, which balances rare and common skills—and by a GPT-4o blind filtering loop in which the model answers every question without video six times, and annotators rework questions it can answer correctly, repeated three rounds, to keep the benchmark visually dependent. Scores come from ECEval, which uses binary 0/1 scoring for closed-ended questions and a 0-to-1 multilevel rubric anchored by a human-written 0.5-point reference answer for open-ended questions; this design is what lets the paper claim both precision and fairness in the final rankings.

What would settle it

Have independent human annotators re-score a random sample of 300 model responses with the ECEval rubric and compare their scores with GPT-4o's; if agreement (e.g., Cohen's kappa) is low, or if replacing GPT-4o with another judge changes the overall model ordering by more than a few points, then the benchmark's claim that models lack dynamic perception would not be established.

Watch

Extended reading notes

Core claim

On the paper's own terms, ECBench is the first benchmark to break embodied cognition for large vision-language models into a fixed taxonomy of 30 abilities, and to evaluate them with mixed open and closed questions that cannot be answered from language priors alone. The empirical discovery is that capability is not uniform: robot-centric questions that require the model to reason about its own position, trajectory, or future actions are harder than scene-based third-person questions; dynamic scene questions, especially quantity dynamics, are nearly unsolved; and embodied hallucination splits into two failure modes—over-confidence in common sense and over-confidence in user input—with the latter barely addressed by any model. The strongest system, GPT-4o at 32 frames, still falls 44.61 points behind the human average, and the paper reads this as evidence that current LVLMs have third-person static-scene cognition but not yet first-person understanding in dynamic scenes.

Load-bearing premise

The headline conclusion depends on the assumption that GPT-4o, used both to filter out questions answerable without video and to score every model answer, gives scores that match human judgment and do not favor any model family; the paper reports no human-agreement check for the judge.

Editorial extensions

If this is right

  • If ECBench is measuring what it says, then scaling input frames from 8 to 20 is not a sufficient cure for dynamic-scene failures; frame count helps some models, but the dynamic sub-scores stay near or below 25%.
  • The robot-centric results imply that first-person self-awareness (trajectory review, distance and azimuth awareness, movement imagery) is a distinct bottleneck: GPT-4o scores 49.04 there versus 59.74 on scene-based questions.
  • Hallucination results imply that current models cannot reliably correct user input: on missing, erroneous, and ambiguous references, most models score near zero, which matters for safe human-robot interaction.
  • The blind GPT-4o score of 24.09, compared with the 33.5 the paper reports for OpenEQA, implies ECBench is substantially less answerable from common sense alone, so score differences on it are more likely to reflect visual understanding.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Our inference: a fair next experiment is to fine-tune one open model on dynamic egocentric video and run ECBench before and after; if scores do not move, the benchmark may be testing something other than the visual dynamics it names.
  • Our inference: because ECEval uses GPT-4o as judge, rankings may be sensitive to judge choice; re-scoring a fixed set of responses with another capable model or with human raters would tell whether the reported ordering is robust.
  • Our inference: the benchmark's videos are under five minutes, so extending the same taxonomy to hours-long egocentric streams is a natural next step and would show whether self-localization and memory degrade further over longer horizons.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 8 minor

Summary. ECBench is a new embodied-cognition benchmark for evaluating LVLMs from egocentric RGB-D indoor videos, comprising 386 videos and 4,324 QA pairs across static-scene, dynamic-scene, and hallucination subsets organized into a 30-dimension taxonomy. The authors propose ECEval, a hybrid scoring scheme that uses GPT-4o for binary scoring of close-ended questions and multi-level scoring, anchored on manually annotated partial-credit answers, for open-ended questions. They report evaluations of twelve LVLM variants plus a blind LLM baseline and a human reference, and they conclude that mainstream LVLMs are weak in dynamic scenes and robot-centric questions, that embodied hallucination is widespread, and that GPT-4o-[32f] is the best model, scoring 50.35 overall versus 94.96 for humans.

Significance. If the evaluation layer is valid, ECBench fills a genuine gap: existing embodied QA benchmarks largely lack robot-centric, dynamic-scene, and hallucination dimensions. The construction is careful in several measurable respects: category-independent manual annotation, a blind GPT-4o filtering step to remove text-only-answerable questions, transparent and reproduced prompts for benchmarking and scoring, a human ceiling score, and public release of data and code. The blind-LLM baseline of 24.09 suggests the filtering did reduce common-sense leakage relative to OpenEQA's reported 33.5. However, every headline claim about model competence is only as trustworthy as ECEval, and ECEval's validity is currently not demonstrated; the empirical conclusions are therefore conditional on additional validation of the scoring function.

major comments (5)
  1. [Sec. 4.5; Appendix 9 (Scoring); Sec. 3.1] The central empirical claims, that models perform poorly in dynamic scenes, that robot-centric questions are harder, and that GPT-4o-[32f] is the best model, all depend on scores produced by GPT-4o under the ECEval protocol, yet no evidence is reported that the judge agrees with human judgments or is unbiased across model families. GPT-4o appears in three roles: blind filter in Sec. 3.1, judge in Sec. 4.5 and Appendix 9, and top-scoring test model in Table 2. The filter role is the least problematic, since the blind baseline is only 24.09 and the filter targets common-sense-answerable items; but the judge role is load-bearing. For open-ended answers, the 0.5-point anchors are manually annotated with no inter-annotator statistics reported; for close-ended answers, 'binary' scoring still requires a GPT-4o semantic-equivalence judgment rather than exact matching, so the 90.31% close-ended share does not remove the judge from the loop. The paper itself acknowledges in Sec. 4.5 that GPT-4o-based multi-level scoring is subject to GPT-4o biases. Without a human-correlation or agreement check, a systematic judge tendency, for example toward the concise phrasing encouraged by the Appendix 9 benchmarking prompt, could partly explain the reported 50.35 versus 24.09 gap and the model ordering in Table 2. The authors should report judge-human agreement (for instance Cohen's kappa on a sample), a bias analysis by model family and answer length, and ideally a second judge before the rankings are accepted.
  2. [Sec. 4.3; Table 2; Fig. 3c] The conclusion of a 'complete inability' of LVLMs to perceive dynamic elements is drawn from a subset of only 248 QA pairs (Fig. 3c), split into four categories of roughly 60 items each; the 0.00 score of InternVL2-40B-[20f] on Quantity Dynamics and the low QD scores in general may be small-sample artifacts rather than evidence of a categorical inability. The per-category sample sizes are not reported in Table 2, and no confidence intervals are given. The authors should either enlarge the dynamic subset, which Appendix 10 acknowledges is currently limited, or reword the claims to refer to low performance on the available items, and zero-score observations should be accompanied by interval estimates.
  3. [Sec. 4.1; Tables 2-5] No confidence intervals, standard errors, or significance tests are reported anywhere in the evaluation. Several load-bearing comparisons in Sec. 4.1 involve score differences of a few points, such as the 3.05-point gain attributed to increasing the frame count for Qwen2VL-72B (41.57 to 44.62), and the per-sub-ability cells in Tables 3-5 are even smaller. With QA-level variance and judge stochasticity, differences of this size may not be reliable; bootstrap confidence intervals over QA pairs, or a paired significance test, should be added before claims such as 'Reasoning problems impose higher demands on model capabilities compared to Perception questions' in Sec. 4.2 are made.
  4. [Appendix 6, Fig. 12] The supplementary examples contain an internally inconsistent QA pair: in the second Missing Reference example under the ScanNet hallucination group, the question asks about a yellow bookshelf with stuffed toys in a third compartment, but the labeled answer states that there is no football under the desk or next to the sofa, which is the answer text of the preceding example. This looks like a copy-paste error, and if such mismatches exist in the released data, they contradict the 'stringent cross-validation' guarantee in Sec. 3.1 and could corrupt hallucination-subset scores. The authors should audit all QA pairs for answer-question mismatches and correct the example before publication.
  5. [Table 2; Sec. 4.1] The human reference score of 94.96 in Table 2 is central to the abstract's framing, but its measurement is not described anywhere: the number of participants, whether they were the same annotators who wrote the QA pairs, whether they saw the same frames and prompts as the models, and the agreement among raters are all unspecified. If the annotators who designed the questions also produced the human ceiling, the human score is likely inflated relative to an independent rater, which would exaggerate the model-human gap in the paper's headline comparison. A short protocol description and inter-rater agreement statistic should be added.
minor comments (8)
  1. [Sec. 4.1 vs Table 2] The text states that Qwen2VL achieved 44.16, but no entry in Table 2 equals 44.16; the closest value is Qwen2VL-72B-[20f] at 44.62, so the cited number should be corrected or its provenance explained.
  2. [Sec. 3.1; Appendix 9] The filtering description says GPT-4o answers all questions six times and then 'this process is iterated thrice'; the intended protocol, three iterations of six repetitions or six runs per iteration, should be stated precisely.
  3. [Sec. 3.1; Appendix 9] The scoring prompt in Appendix 9 uses a 0-5 scale and defines anchors as the '5-score answer' and '3-score answer', while Sec. 3.1 describes a 0-1 scale in 0.2 increments; the mapping between the two conventions should be made explicit.
  4. [Fig. 4] The axes and the combination of bar and overlaid distribution plots in Fig. 4 are not clearly labeled, and the units for question length and vocabulary size are ambiguous; axis titles and a legend should be added.
  5. [Table 2; Fig. 3c] The per-category sample sizes, especially the 248 dynamic-scene pairs and the small per-dimension counts in Fig. 3c, should be restated in the table captions or a footnote so that readers can assess the reliability of each cell.
  6. [References] Reference [31] is formatted as 'R OpenAI' without author names; it should be replaced with the standard GPT-4 technical report citation.
  7. [Abstract] The sentence 'All data and code is available' should read 'All data and code are available'.
  8. [Sec. 4.1] The comparison between the ECBench blind score (24.09) and the OpenEQA blind score (33.5) is presented as a 28% reduction in evaluation precision, but the two benchmarks differ in question-type mix, answer format, and difficulty; this comparison should be framed as suggestive rather than as a controlled measure of visual dependency.

Circularity Check

1 steps flagged · score 6.0 of 10

Visual-dependency claim reduces to the GPT-4o blind-filtering objective; the GPT-4o-as-judge loop is a validity gap but not by-construction circularity.

  1. fitted input called prediction [Section 3.1 (Construction of Question-Answer Pairs) and Section 4.1 (Main Results)]
    "Specifically, we employ GPT-4o to answer all the questions six times without visual inputs, followed by manual revisions of questions consistently answered correctly to enhance their reliance on visual input. This process is iterated thrice. ... Despite this, the blind evaluation score for ECBench reaches only 24.09, which is a 28% reduction compared to the entirely open-ended OpenEQA (33.5). This indicates that ECBench significantly outperforms OpenEQA in terms of evaluation precision and visual dependency."

    The surviving question set is the output of a filter whose explicit objective is to make blind, no-video GPT-4o answering fail: questions consistently answered correctly without visual input are revised or removed, iterated three times. The reported blind score of 24.09 is therefore the minimized objective of that filter, not an independent measurement of visual dependency. Using this same number to conclude that ECBench outperforms OpenEQA in visual dependency restates the screening criterion as an evaluation result. For GPT-4o, the low blind score is guaranteed by construction, so this specific evidence cannot independently establish that ECBench questions require visual input; human visual dependence may hold, but it is not demonstrated by the cited comparison.

full rationale

ECBench is a benchmark-construction and evaluation paper, not a derivation, so most classical circularity patterns (uniqueness theorems, ansatz smuggling, renaming known results) are absent. The human annotation pipeline, the 30-dimension embodied-cognition taxonomy, and the per-model evaluation numbers are independent content. The one concrete by-construction reduction is the visual-dependency validation: the question set is explicitly screened by GPT-4o blind answering, with questions consistently answered correctly without vision revised or removed over three iterations, and the paper then reports the resulting low blind score as evidence that ECBench significantly outperforms OpenEQA in visual dependency. That number is the objective function of the filter, so the supporting quality claim is forced by the screening procedure rather than independently measured. The ECEval use of GPT-4o as judge while GPT-4o is also the top-ranked evaluated model is a genuine validity gap—no human correlation, inter-annotator agreement, or bias analysis is reported—but it is not formal circularity: a judge's output is not identical by construction to the evaluated model's score, so the concern is an unvalidated measurement rather than a derivation. The only author self-citation (Video-LLaMA, ref. [40]) is illustrative and not load-bearing. Because the circular step concerns one of the benchmark's advertised quality properties while the rest of the benchmark stands independently, the score is 6, reflecting partial circularity.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The benchmark introduces no fitted constants and no new physical entities. The hand-chosen thresholds are the blind filtering consistency criterion (6/6 GPT-4o runs) and the 0.2 scoring increment for open-ended answers. The main load-bearing assumptions are the validity of GPT-4o as a proxy for common-sense answerability, the completeness of the 30-dimension taxonomy, the correctness of manual annotations, and the agreement of GPT-4o judge scores with human judgment.

free parameters (2)
  • Blind filtering consistency threshold = 6 out of 6 GPT-4o runs
    Questions answered correctly by GPT-4o in all six blind runs without visual input are considered common-sense answerable and are manually revised. This threshold is chosen by the authors and directly affects the visual dependency of the benchmark.
  • Open-ended scoring increments = 0.2 steps on a 0 to 1 scale
    ECEval scores open-ended answers in increments of 0.2 anchored to a manual 0.5-point reference. The step size is a modeling choice that affects the granularity and reported precision of model scores.
assumptions (4)
  • domain assumption GPT-4o blind answering approximates human common-sense answerability, so removing questions it answers correctly without video eliminates text-only questions.
    Used in Section 3.1 to filter common-sense questions. If GPT-4o's no-vision performance does not match human priors, some questions may remain text-answerable or be wrongly removed.
  • domain assumption The 30-dimension taxonomy covers the embodied cognition requirements of robots comprehensively.
    The paper motivates the taxonomy by analogy to general VideoQA benchmarks, but does not validate the ontology against task taxonomies in robotics or against independent expert judgment.
  • domain assumption Manual annotation and multi-round screening produce correct ground-truth answers.
    No inter-annotator agreement statistics or independent verification are reported for the QA pairs, despite the claim of meticulous human annotation.
  • domain assumption GPT-4o judge scores correlate with human correctness judgments.
    ECEval relies on GPT-4o for binary and partial-credit scoring of all model outputs, but the paper provides no human evaluation of the judge's outputs.

how reviews work

0 comments
Cite this review

Pith. "Pith review of ECBench: Can Multi-modal Foundation Models Understand the Egocentric World? A Holistic Embodied Cognition Benchmark." pith.science (2026). https://pith.science/paper/7VEBINRN

@misc{pith2026250105031,
  author       = {Pith},
  title        = {Pith review of: ECBench: Can Multi-modal Foundation Models Understand the Egocentric World? A Holistic Embodied Cognition Benchmark},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/7VEBINRN}},
  note         = {Machine review of arXiv:2501.05031}
}
read the original abstract

The enhancement of generalization in robots by large vision-language models (LVLMs) is increasingly evident. Therefore, the embodied cognitive abilities of LVLMs based on egocentric videos are of great interest. However, current datasets for embodied video question answering lack comprehensive and systematic evaluation frameworks. Critical embodied cognitive issues, such as robotic self-cognition, dynamic scene perception, and hallucination, are rarely addressed. To tackle these challenges, we propose ECBench, a high-quality benchmark designed to systematically evaluate the embodied cognitive abilities of LVLMs. ECBench features a diverse range of scene video sources, open and varied question formats, and 30 dimensions of embodied cognition. To ensure quality, balance, and high visual dependence, ECBench uses class-independent meticulous human annotation and multi-round question screening strategies. Additionally, we introduce ECEval, a comprehensive evaluation system that ensures the fairness and rationality of the indicators. Utilizing ECBench, we conduct extensive evaluations of proprietary, open-source, and task-specific LVLMs. ECBench is pivotal in advancing the embodied cognitive capabilities of LVLMs, laying a solid foundation for developing reliable core models for embodied agents. All data and code are available at https://github.com/Rh-Dang/ECBench.

Figures

Figures reproduced from arXiv: 2501.05031 by the authors.

Figure 1
Figure 1. Illustration of question answering (QA) format and representative cognitive dimensions from ECBench. There are 386 RGB-D videos, 4,324 QA pairs, and 30 distinct embodied cognitive abilities, spanning across various aspects such as perception, reasoning, self-awareness, dynamic capturing, and hallucination. ECEval employs distinct evaluation methods for different types of answers. nomenon has been extensively analyze… view at source ↗
Figure 2
Figure 2. Overview of embodied cognition dimensions in ECBench. ECBench includes three subsets: static scenes, dynamic scenes, and hallucination, evaluating a total of 30 embodied cognitive abilities. ied cognitive abilities ( [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Data analysis of ECBench reflects a rich diversity of scenario categories, video sources, and evaluation dimensions. Number of Words Amount Max Length Average Length Cognitive Category QA Pair Number Vocabulary Size 4 8 12 16 20 30 24 18 12 6 5000 4000 3000 2000 1000 5000 4000 3000 2000 1000 120 96 72 48 24 [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (10 more)
Figure 4
Figure 4. Figure 4: Comparison with OpenEQA [27] on textual data, including the distribution of question lengths, average question length, maximum question length, vocabulary size, number of questions, and number of capabilities. sual input. This process is iterated thrice. Ultimately, we…
Figure 5
Figure 5. Figure 5: Comparison of results between ECEval, Binary Scor￾ing, and Multilevel Scoring, for open-ended and closed-ended questions. Notably, only open-ended questions are annotated with 0.5-point answers . to these two typical evaluation methods. As shown in the open-ended examp…
Figure 6
Figure 6. Figure 6: Visualization of question answering examples in the static scene test set. The main focus is on the scene-based category. Part 1 out of 2 [PITH_FULL_IMAGE:figures/full_fig_p013_6.png]
Figure 7
Figure 7. Figure 7: Visualization of question answering examples in the static scene test set. The main focus is on the scene-based category. Part 2 out of 2 [PITH_FULL_IMAGE:figures/full_fig_p014_7.png]
Figure 8
Figure 8. Figure 8: Visualization of question answering examples in the static scene test set. The main focus is on the robot-centric category. Part 1 out of 2 [PITH_FULL_IMAGE:figures/full_fig_p015_8.png]
Figure 9
Figure 9. Figure 9: Visualization of question answering examples in the static scene test set. The main focus is on the robot-centric category. Part 2 out of 2 [PITH_FULL_IMAGE:figures/full_fig_p016_9.png]
Figure 10
Figure 10. Figure 10: Visualization of question answering examples in the dynamic scene test set [PITH_FULL_IMAGE:figures/full_fig_p017_10.png]
Figure 11
Figure 11. Figure 11: Visualization of question answering examples in the hallucination test set. The main focus is on the commensense over￾confidence category [PITH_FULL_IMAGE:figures/full_fig_p018_11.png]
Figure 12
Figure 12. Figure 12: Visualization of question answering examples in the hallucination test set. The main focus is on the user-input over￾confidence category [PITH_FULL_IMAGE:figures/full_fig_p019_12.png]
Figure 13
Figure 13. Figure 13: A more detailed analysis of the textual content of ECBench. [PITH_FULL_IMAGE:figures/full_fig_p021_13.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

51 extracted references · 32 canonical work pages

  1. [1]

    Scanqa: 3d question answering for spatial scene understanding

    Daichi Azuma, Taiki Miyanishi, Shuhei Kurita, and Motoaki Kawanabe. Scanqa: 3d question answering for spatial scene understanding. In CVPR, pages 19129–19139, 2022. 1, 3, 10

    work page 2022

  2. [2]

    Pca-bench: Evaluating multimodal large language models in perception-cognition-action chain

    Liang Chen, Yichi Zhang, Shuhuai Ren, Haozhe Zhao, Ze- fan Cai, Yuchi Wang, Peiyi Wang, Xiangdi Meng, Tianyu Liu, and Baobao Chang. Pca-bench: Evaluating multimodal large language models in perception-cognition-action chain. In ACL, pages 1086–1104. Association for Computational Linguistics, 2024. 3

    work page 2024

  3. [3]

    Grounded multi- hop videoqa in long-form egocentric videos

    Qirui Chen, Shangzhe Di, and Weidi Xie. Grounded multi- hop videoqa in long-form egocentric videos. arXiv preprint arXiv:2408.14469, 2024. 2, 6, 7

    arXiv 2024

  4. [4]

    Egoplan-bench: Benchmarking egocentric embodied plan- ning with multimodal large language models

    Yi Chen, Yuying Ge, Yixiao Ge, Mingyu Ding, Bohao Li, Rui Wang, Ruifeng Xu, Ying Shan, and Xihui Liu. Egoplan-bench: Benchmarking egocentric embodied plan- ning with multimodal large language models. arXiv preprint arXiv:2312.06722, 2023. 3

    arXiv 2023

  5. [5]

    How far are we to gpt-4v? closing the gap to commercial multimodal models with open-source suites

    Zhe Chen, Weiyun Wang, Hao Tian, Shenglong Ye, Zhang- wei Gao, Erfei Cui, Wenwen Tong, Kongzhi Hu, Jiapeng Luo, Zheng Ma, et al. How far are we to gpt-4v? closing the gap to commercial multimodal models with open-source suites. arXiv preprint arXiv:2404.16821, 2024. 6

    arXiv 2024

  6. [6]

    Videollama 2: Advancing spatial- temporal modeling and audio understanding in video-llms

    Zesen Cheng, Sicong Leng, Hang Zhang, Yifei Xin, Xin Li, Guanzheng Chen, Yongxin Zhu, Wenqi Zhang, Ziyang Luo, Deli Zhao, et al. Videollama 2: Advancing spatial- temporal modeling and audio understanding in video-llms. arXiv preprint arXiv:2406.07476, 2024. 6

    arXiv 2024

  7. [7]

    Chang, Manolis Savva, Maciej Hal- ber, Thomas A

    Angela Dai, Angel X. Chang, Manolis Savva, Maciej Hal- ber, Thomas A. Funkhouser, and Matthias Nießner. Scan- net: Richly-annotated 3d reconstructions of indoor scenes. In CVPR, pages 2432–2443. IEEE Computer Society, 2017. 3, 5

    work page 2017

  8. [8]

    Procthor: Large-scale embodied AI using procedural generation

    Matt Deitke, Eli VanderBilt, Alvaro Herrasti, Luca Weihs, Kiana Ehsani, Jordi Salvador, Winson Han, Eric Kolve, Aniruddha Kembhavi, and Roozbeh Mottaghi. Procthor: Large-scale embodied AI using procedural generation. In NeurIPS, 2022. 3

    work page 2022

Show all 51 references
  1. [9]

    Mmbench-video: A long-form multi-shot benchmark for holistic video under- standing

    Xinyu Fang, Kangrui Mao, Haodong Duan, Xiangyu Zhao, Yining Li, Dahua Lin, and Kai Chen. Mmbench-video: A long-form multi-shot benchmark for holistic video under- standing. arXiv preprint arXiv:2406.14515, 2024. 3

    2024 arXiv

  2. [10]

    Env-qa: A video question answering benchmark for comprehensive understanding of dynamic environments

    Difei Gao, Ruiping Wang, Ziyi Bai, and Xilin Chen. Env-qa: A video question answering benchmark for comprehensive understanding of dynamic environments. In ICCV, pages 1655–1665. IEEE, 2021. 3, 4

    2021

  3. [11]

    Knowit VQA: answering knowledge-based questions about videos

    Noa Garcia, Mayu Otani, Chenhui Chu, and Yuta Nakashima. Knowit VQA: answering knowledge-based questions about videos. In AAAI, pages 10826–10834. AAAI Press, 2020. 3

    2020

  4. [12]

    Hallusionbench: an advanced diagnostic suite for entangled language hallucination and visual illusion in large vision-language models

    Tianrui Guan, Fuxiao Liu, Xiyang Wu, Ruiqi Xian, Zongxia Li, Xiaoyu Liu, Xijun Wang, Lichang Chen, Furong Huang, Yaser Yacoob, et al. Hallusionbench: an advanced diagnostic suite for entangled language hallucination and visual illusion in large vision-language models. In CVPR,...

    2024

  5. [13]

    Gpt-4o system card

    Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perel- man, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Weli- hinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card. arXiv preprint arXiv:2410.21276, 2024. 2, 6

    2024 arXiv

  6. [14]

    Throne: An object-based hallucination benchmark for the free-form generations of large vision-language models

    Prannay Kaul, Zhizhong Li, Hao Yang, Yonatan Duk- ler, Ashwin Swaminathan, CJ Taylor, and Stefano Soatto. Throne: An object-based hallucination benchmark for the free-form generations of large vision-language models. In CVPR, pages 27228–27238, 2024. 4

    2024

  7. [15]

    Ai2-thor: An interactive 3d environment for visual ai

    Eric Kolve, Roozbeh Mottaghi, Winson Han, Eli VanderBilt, Luca Weihs, Alvaro Herrasti, Matt Deitke, Kiana Ehsani, Daniel Gordon, Yuke Zhu, et al. Ai2-thor: An interactive 3d environment for visual ai. arXiv preprint arXiv:1712.05474,

    arXiv

  8. [16]

    Openfmnav: Towards open-set zero-shot object navigation via vision- language foundation models

    Yuxuan Kuang, Hai Lin, and Meng Jiang. Openfmnav: Towards open-set zero-shot object navigation via vision- language foundation models. In Findings of the Association for Computational Linguistics: NAACL 2024, Mexico City, Mexico, June 16-21, 2024, pages 338–351. ACL, 2024. 3, 5

    2024

  9. [17]

    Building and better understanding vision- language models: insights and future directions

    Hugo Laurenc ¸on, Andr´es Marafioti, Victor Sanh, and L ´eo Tronchon. Building and better understanding vision- language models: insights and future directions. arXiv preprint arXiv:2408.12637, 2024. 6

    2024 arXiv

  10. [18]

    Jie Lei, Licheng Yu, Mohit Bansal, and Tamara L. Berg. TVQA: localized, compositional video question answering. In EMNLP, pages 1369–1379. Association for Computa- tional Linguistics, 2018. 3

    2018

  11. [19]

    Seed-bench: Bench- marking multimodal large language models

    Bohao Li, Yuying Ge, Yixiao Ge, Guangzhi Wang, Rui Wang, Ruimao Zhang, and Ying Shan. Seed-bench: Bench- marking multimodal large language models. InCVPR, pages 13299–13308. IEEE, 2024. 3

    2024

  12. [20]

    Mvbench: A comprehensive multi- modal video understanding benchmark

    Kunchang Li, Yali Wang, Yinan He, Yizhuo Li, Yi Wang, Yi Liu, Zun Wang, Jilan Xu, Guo Chen, Ping Lou, Limin Wang, and Yu Qiao. Mvbench: A comprehensive multi- modal video understanding benchmark. In CVPR, pages 22195–22206. IEEE, 2024. 3

    2024

  13. [21]

    Tem- poral reasoning transfer from text to video

    Lei Li, Yuanxin Liu, Linli Yao, Peiyuan Zhang, Chenxin An, Lean Wang, Xu Sun, Lingpeng Kong, and Qi Liu. Tem- poral reasoning transfer from text to video. arXiv preprint arXiv:2410.06166, 2024. 6

    2024 arXiv

  14. [22]

    Evaluating object hallucina- tion in large vision-language models

    Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. Evaluating object hallucina- tion in large vision-language models. arXiv preprint arXiv:2305.10355, 2023. 8

    2023 arXiv

  15. [23]

    Video-llava: Learning united visual repre- sentation by alignment before projection, 2024

    Bin Lin, Yang Ye, Bin Zhu, Jiaxi Cui, Munan Ning, Peng Jin, and Li Yuan. Video-llava: Learning united visual repre- sentation by alignment before projection, 2024. 6

    2024

  16. [24]

    Phd: A prompted visual hallucination evaluation dataset

    Jiazhen Liu, Yuhan Fu, Ruobing Xie, Runquan Xie, Xingwu Sun, Fengzong Lian, Zhanhui Kang, and Xirong Li. Phd: A prompted visual hallucination evaluation dataset. arXiv preprint arXiv:2403.11116, 2024. 4

    2024 arXiv

  17. [25]

    Kangaroo: A powerful video-language model supporting long-context video input

    Jiajun Liu, Yibing Wang, Hanghang Ma, Xiaoping Wu, Xi- aoqi Ma, Xiaoming Wei, Jianbin Jiao, Enhua Wu, and Jie Hu. Kangaroo: A powerful video-language model supporting long-context video input. arXiv preprint arXiv:2408.15542,

    arXiv

  18. [26]

    SQA3D: situ- ated question answering in 3d scenes

    Xiaojian Ma, Silong Yong, Zilong Zheng, Qing Li, Yitao Liang, Song-Chun Zhu, and Siyuan Huang. SQA3D: situ- ated question answering in 3d scenes. In ICLR, 2023. 1, 3

    2023

  19. [27]

    Openeqa: Embodied question answering in the era of foundation models

    Arjun Majumdar, Anurag Ajay, Xiaohan Zhang, Pranav Putta, Sriram Yenamandra, Mikael Henaff, Sneha Sil- wal, Paul McVay, Oleksandr Maksymets, Sergio Ar- naud, Karmesh Yadav, Qiyang Li, Ben Newman, Mo- hit Sharma, Vincent-Pierre Berges, Shiqi Zhang, Pulkit Agrawal, Yonatan Bisk,...

    2024

  20. [28]

    Egoschema: A diagnostic benchmark for very long- form video language understanding

    Karttikeya Mangalam, Raiymbek Akshulakov, and Jitendra Malik. Egoschema: A diagnostic benchmark for very long- form video language understanding. In NeurIPS, 2023. 3

    2023

  21. [29]

    Chang, and Manolis Savva

    Yongsen Mao, Yiming Zhang, Hanxiao Jiang, Angel X. Chang, and Manolis Savva. Multiscan: Scalable RGBD scanning for 3d environments with articulated objects. In NeurIPS, 2022. 3, 5

    2022

  22. [30]

    Marioqa: Answering questions by watching gameplay videos

    Jonghwan Mun, Paul Hongsuck Seo, Ilchae Jung, and Bo- hyung Han. Marioqa: Answering questions by watching gameplay videos. In ICCV, pages 2886–2894. IEEE Com- puter Society, 2017. 3

    2017

  23. [31]

    Gpt-4 technical report

    R OpenAI. Gpt-4 technical report. arxiv 2303.08774. View in Article, 2(5), 2023. 6

    2023 arXiv

  24. [32]

    Turner, Eric Undersander, Wojciech Galuba, Andrew West- bury, Angel X

    Santhosh Kumar Ramakrishnan, Aaron Gokaslan, Erik Wi- jmans, Oleksandr Maksymets, Alexander Clegg, John M. Turner, Eric Undersander, Wojciech Galuba, Andrew West- bury, Angel X. Chang, Manolis Savva, Yili Zhao, and Dhruv Batra. Habitat-matterport 3d dataset (HM3D): 1000 large-...

    2021

  25. [33]

    Explore until confi- dent: Efficient exploration for embodied question answering

    Allen Z Ren, Jaden Clark, Anushri Dixit, Masha Itkina, Anirudha Majumdar, and Dorsa Sadigh. Explore until confi- dent: Efficient exploration for embodied question answering. arXiv preprint arXiv:2403.15941, 2024. 3, 5

    2024 arXiv

  26. [34]

    Joshi, Pete Florence, Wei Han, Robert Baruch, Yao Lu, Suvir Mir- chandani, Peng Xu, Pannag Sanketi, Karol Hausman, Izhak Shafran, Brian Ichter, and Yuan Cao

    Pierre Sermanet, Tianli Ding, Jeffrey Zhao, Fei Xia, De- bidatta Dwibedi, Keerthana Gopalakrishnan, Christine Chan, Gabriel Dulac-Arnold, Sharath Maddineni, Nikhil J. Joshi, Pete Florence, Wei Han, Robert Baruch, Yao Lu, Suvir Mir- chandani, Peng Xu, Pannag Sanketi, Karol Haus...

  27. [35]

    Alanavlm: A multimodal embodied ai foun- dation model for egocentric video understanding

    Alessandro Suglia, Claudio Greco, Katie Baker, Jose L Part, Ioannis Papaionnou, Arash Eshghi, Ioannis Konstas, and Oliver Lemon. Alanavlm: A multimodal embodied ai foun- dation model for egocentric video understanding. arXiv preprint arXiv:2406.13807, 2024. 2, 6, 7

    2024 arXiv

  28. [36]

    Movieqa: Understanding stories in movies through question- answering

    Makarand Tapaswi, Yukun Zhu, Rainer Stiefelhagen, Antonio Torralba, Raquel Urtasun, and Sanja Fidler. Movieqa: Understanding stories in movies through question- answering. In CVPR, pages 4631–4640. IEEE Computer So- ciety, 2016. 3

    2016

  29. [37]

    Mitigating fine-grained hallucination by fine- tuning large vision-language models with caption rewrites

    Lei Wang, Jiabang He, Shenshen Li, Ning Liu, and Ee- Peng Lim. Mitigating fine-grained hallucination by fine- tuning large vision-language models with caption rewrites. In ACMMM, pages 32–45. Springer, 2024. 8

    2024

  30. [38]

    Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution

    Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution. arXiv preprint arXiv:2409.12191, 2024. 2, 6

    2024 arXiv

  31. [39]

    Evaluating and an- alyzing relationship hallucinations in lvlms

    Mingrui Wu, Jiayi Ji, Oucheng Huang, Jiale Li, Yuhang Wu, Xiaoshuai Sun, and Rongrong Ji. Evaluating and an- alyzing relationship hallucinations in lvlms. arXiv preprint arXiv:2406.16449, 2024. 4

    2024 arXiv

  32. [40]

    Video-llama: An instruction-tuned audio-visual language model for video un- derstanding

    Hang Zhang, Xin Li, and Lidong Bing. Video-llama: An instruction-tuned audio-visual language model for video un- derstanding. In EMNLP, pages 543–553. Association for Computational Linguistics, 2023. 1

    2023

  33. [41]

    Q-bench-video: Benchmarking the video quality understanding of lmms

    Zicheng Zhang, Ziheng Jia, Haoning Wu, Chunyi Li, Zi- jian Chen, Yingjie Zhou, Wei Sun, Xiaohong Liu, Xiongkuo Min, Weisi Lin, et al. Q-bench-video: Benchmarking the video quality understanding of lmms. arXiv preprint arXiv:2409.20063, 2024. 3

    2024 arXiv

  34. [42]

    Minigpt-4: Enhancing vision-language understanding with advanced large language models

    Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mo- hamed Elhoseiny. Minigpt-4: Enhancing vision-language understanding with advanced large language models. In ICLR, 2024. 1 ECBench: Can Multi-modal Foundation Models Understand the Egocentric World? A Holistic Embodied Cogni...

    2024

  35. [43]

    3.1 of the main paper and in Fig

    Capability Taxonomy Details In Sec. 3.1 of the main paper and in Fig. 2, we present the process of constructing the embodied cognition dimensions for ECBench. In this section, we will elaborate on the de- sign principles and specific details underlying each capabil- ity dimens...

  36. [44]

    3.1, we design ECEval, a fine-grain evaluation framework that considers different types of tasks

    Construction of Evaluation Framework As mentioned in Sec. 3.1, we design ECEval, a fine-grain evaluation framework that considers different types of tasks. That is, we also annotate a partially correct answer for each question in addition to a completely correct answer. Specif...

  37. [45]

    3.2 of the main text, we conduct a comprehensive quantitative analysis of ECBench

    Additional Dataset Analysis In Sec. 3.2 of the main text, we conduct a comprehensive quantitative analysis of ECBench. This section delves into a more detailed examination of the question-answer text within ECBench. In Fig. 13a, we present a word cloud de- picting all the ques...

  38. [46]

    For the GPT family models, we request the endpoint of GPT- 4o-0513 and GPT-4o-mini-0513 through the Azure 1

    More Details of Experiment For all experiments, we set the temperature to 0.2. For the GPT family models, we request the endpoint of GPT- 4o-0513 and GPT-4o-mini-0513 through the Azure 1. For Qwen2VL-72B/7B and InternVL2, we use vllm 2 for em- ployment. To score each model’s p...

  39. [47]

    ECBench rep- resents the first effort to systematically analyze embodied cognitive issues and to establish a comprehensive evalua- tion framework

    Limitations and Broader Impacts Limitations In this study, we introduce ECBench, a novel open-world embodied cognition benchmark. ECBench rep- resents the first effort to systematically analyze embodied cognitive issues and to establish a comprehensive evalua- tion framework. ...

  40. [48]

    Due to budget constraints, we are unable to test all pro- prietary models, such as Claude-3.5, among others

  41. [49]

    Theoretically, robots may possess a longer video mem- ory during the execution of specific tasks; however, the video inputs for ECBench are limited to under five min- utes, which may not adequately evaluate the cognitive abilities of LVLMs regarding prolonged visual memory

  42. [50]

    Although we make ef- forts to collect some real-world dynamic scene videos, these still fall short in terms of scene richness and object variety

    Cognition of dynamic scenarios in open-world environ- ments is a previously unexplored area, leading to a lack of diverse dynamic scene videos. Although we make ef- forts to collect some real-world dynamic scene videos, these still fall short in terms of scene richness and obj...

  43. [51]

    There- fore, the evolution of ECBench towards a more practical and flexible question-and-answer format will be a key direction for our future research

    The natural interaction between robots and humans should not be a singular question-and-answer exchange, but rather a streaming and interwoven dialogue. There- fore, the evolution of ECBench towards a more practical and flexible question-and-answer format will be a key directi...

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.