REVIEW 3 major objections 4 minor 17 references
EASG-Bench: Video Q&A Benchmark with Egocentric Action Scene Graphs
T0 review · 3 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read The paper introduces EASG-Bench, a video question-answering benchmark built from egocentric action scene graphs, and reports that video-LLMs trail language-only models on temporal ordering questions.
desk verdict A useful grounded egocentric QA benchmark with a solid construction pipeline, but the headline language-only vs. video-LLM gap is confounded: the language-only models are effectively handed the answer via timestamped captions. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is the Egocentric Action Scene Graph (EASG), a dynamic graph in which the camera wearer and objects appear as nodes and actions and relations appear as edges, each anchored to video frames and timestamps. The machinery is a three-stage QA generation pipeline: an LLM is conditioned on the graph sequence to propose questions, a majority-voted Llama-3.1-8B filter keeps only questions that can be answered unambiguously from direct observation, and a second filter removes generic-sounding pairs. Evaluation uses an LLM-as-a-judge that rates candidate answers on a 1-10 scale with context, question, ground truth, and grading rubric, and the scores are rescaled relative to the score the judge gives to the ground truth, producing the reported relative score per question type. For the ordering analysis, the paper adds a two-stage chain-of-thought prompt that first asks the model to locate the action and then asks what happened before or after it.
What would settle it
Take a random sample of, say, 100 EASG-Bench questions and have independent human annotators watch the corresponding clips and list every valid answer to each question; if a substantial fraction of questions admit more than one defensible answer or require information the video does not show, the benchmark's core claim of unambiguous, video-grounded QA fails.
Extended reading notes
Core claim
The paper's central claim is that EASG-Bench measures video understanding through questions that carry both spatial and temporal grounding, because each QA pair is derived from a sequence of Egocentric Action Scene Graphs instead of captions. Using Llama-3.1-8B-Instruct to generate questions from the graph sequence and to filter them by five-iteration majority voting on unambiguous answerability, the authors built 1,807 QA pairs in five categories: purpose, direct object, indirect object, before, and after. Benchmarked with open-ended answers scored by an LLM judge with relative rescaling against ground-truth scores, the best video-LLM, Qwen2.5-VL, outperformed language-only models on direct and indirect manipulation questions but fell behind them on ordering questions. The paper further shows that a two-stage chain-of-thought prompt improves Qwen2.5-VL's ordering scores, but the model still trails text-only LLMs on 'before' questions.
Load-bearing premise
The load-bearing premise is that the EASG annotations are accurate and complete enough that the questions generated from them, after Llama majority-vote filtering, are genuinely answerable in exactly one way from the video; the paper reports no human verification of this uniqueness.
Editorial extensions
If this is right
- EASG-Bench offers an evaluation where questions and answers are tied to concrete bounding boxes and timestamps, so model mistakes can be traced to specific spatio-temporal locations rather than to paraphrase mismatch.
- The measured gap identifies temporal ordering, not object recognition, as the main weakness of current video-LLMs on long egocentric video.
- The chain-of-thought result suggests that explicitly separating action localization from relational questioning is a workable way to improve video-LLM performance on ordering tasks.
- Because objects are referred to by grounding rather than by category names, the benchmark can support language-neutral evaluation across different languages.
- All five question categories are open-ended, so the benchmark can be used to probe reasoning quality rather than just answer selection.
Reading between the lines
- A natural extension would be to test the same models on a human-verified subset of EASG-Bench; if the temporal gap persists under stricter scoring, the ordering weakness is a property of the models rather than of the LLM judge.
- The benchmark's grounding in bounding boxes rather than category names could be adapted to measure compositional generalization, for example by asking about objects that were not seen during training.
- The asymmetric improvement, where chain-of-thought helps 'after' more than 'before', hints that the remaining bottleneck is look-back memory; an extension could probe whether longer context windows or explicit memory modules close the 'before' gap.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces EASG-Bench, a video question-answering benchmark for egocentric videos built from Egocentric Action Scene Graphs (EASGs) introduced in prior work [14]. Question-answer pairs are generated with Llama-3.1-8B-Instruct from graph sequences, filtered through a majority-vote uniqueness check and a genericness filter, yielding 1,807 Q&A pairs over 221 Ego4D clips in five categories (purpose, direct object, indirect object, before, after). The authors evaluate three language-only LLMs and four video-LLMs using an LLM-as-a-judge protocol with relative scores, report that language-only models outperform video-LLMs on before/after ordering questions, and show that a two-stage chain-of-thought prompt improves Qwen2.5-VL on those questions. The benchmark and code are released publicly.
Significance. If the evaluation were controlled, EASG-Bench would be a useful addition to egocentric video QA: it is grounded in structured spatio-temporal scene graphs rather than narrations, covers long untrimmed videos, and includes both spatial and temporal grounding. The public release of benchmark and code supports reproducibility, and the systematic three-stage generation pipeline is clearly described. However, the central empirical claim—that video-LLMs underperform language-only models on temporal ordering—currently rests on an asymmetric evaluation protocol and an unvalidated annotation premise, so the significance is conditional on addressing the confounds identified below.
major comments (3)
- [Section 4 (Evaluation) and Table 2] The paper does not specify what input the language-only models receive, but the evaluation section defines the context C as 'video captions with timestamps.' If language-only models are given C, then before/after questions are partially answered by reading the ordered action sequence in text, whereas video-LLMs must perceive and segment events from raw video. The large gap in Table 2 (e.g., Before: Qwen3-8B 98.92 vs. Qwen2.5-VL-7B 82.76; After: 70.81 vs. 65.97) may therefore reflect information asymmetry rather than weaker temporal reasoning in video-LLMs, which undermines the abstract's central claim. Please add a matched-control condition—for example, language-only models without C, or video-LLMs given C alongside video—and report results under both settings.
- [Section 3 (Benchmark creation)] The uniqueness filtering is performed by Llama-3.1-8B majority voting, but no human validation is reported for the generated QA pairs or for the underlying EASG annotations. If the graphs omit an action or label an action incorrectly, the generated ground-truth answer and its 'unique answerability' determination inherit that error. Please add a human study on a random sample (e.g., 150–200 pairs) measuring factual correctness, answerability, and ambiguity, and report inter-annotator agreement. Without such validation, the benchmark's grounding premise is not established.
- [Section 4, Eq. (1) and Table 2] All scores are single-run LLM-judge outputs without error bars or significance tests. The differences that drive the central claim (the before/after gap) could be within judge noise. Additionally, the judge is Llama-3.1-8B, the same model family used for QA generation and filtering, which may introduce systematic bias favoring certain answer styles. Please report multiple judge runs with variance, and correlate judge scores with human judgments on a subset to validate the evaluation protocol.
minor comments (4)
- [Section 3 and Figure 3] The 'language-neutral' property is stated as a feature of the benchmark, but the released questions and examples use object category names (e.g., 'cable', 'pipe'). Clarify whether the benchmark includes questions posed with bounding boxes and whether the language-neutral variant is actually implemented in the released dataset.
- [Section 4, Table 3] The chain-of-thought experiment is conducted on a single model (Qwen2.5-VL-7B) with one prompt template and no variance estimate; report results across models or explicitly frame this as a preliminary study.
- [Table 1] The 'Untrimmed' column is ambiguous for EASG-Bench; specify whether the 221 clips are used in their entirety or whether shorter segments are extracted from longer Ego4D videos.
- [Section 4, Eq. (1)] The normalization denominator sums over all questions of the same type, so the relative score for each question depends on the composition of the benchmark; state whether the normalization is computed on the full benchmark or per split and discuss whether scores are comparable across question types.
Circularity Check
No significant circularity: the benchmark construction and evaluation are independent measurements; the EASG dependency is a data source, not a self-referential derivation.
full rationale
EASG-Bench constructs QA pairs from EASG annotations [14], then measures model performance. The only overlapping-author dependency is the EASG dataset, which is prior published work (Rodin et al., CVPR 2024) used as ground-truth input, not as a conclusion derived within this paper. The QA generation and uniqueness filtering are performed by Llama-3.1-8B, and the same model family is also used as judge and as one evaluated baseline; however, this is a shared tool, not a step where an output is defined as its input. The relative-score formula (Eq. 1) rescales judge scores by the judge's score on the ground-truth answer, but this is a normalization for calibration, not a fitted parameter renamed as a prediction. The reported language-only vs. video-LLM gap is a measured result; the paper does not fit the gap from the benchmark's construction. The language-only models receive timestamped captions that encode order, which may confound the comparison, but that is an evaluation-validity concern rather than circular reasoning. No equation or definition in the paper makes the central claim equal to an input by construction.
Assumptions & free parameters
assumptions (4)
- domain assumption EASG annotations from Rodin et al. [14] accurately capture actors, actions, and object relations in the selected Ego4D clips.
- domain assumption Llama-3.1-8B-Instruct's majority-vote uniqueness filter selects only questions that have a unique, video-observable answer.
- domain assumption Llama3-8B-Instruct as an LLM judge produces scores that reflect answer quality, and relative rescaling removes judge bias.
- domain assumption The timestamped video captions given to the judge and to language-only models are a faithful textual proxy for the video content.
Cite this review
Pith. "Pith review of EASG-Bench: Video Q&A Benchmark with Egocentric Action Scene Graphs." pith.science (2026). https://pith.science/paper/PLVTTP34
@misc{pith2026250605787,
author = {Pith},
title = {Pith review of: EASG-Bench: Video Q&A Benchmark with Egocentric Action Scene Graphs},
year = {2026},
howpublished = {\url{https://pith.science/paper/PLVTTP34}},
note = {Machine review of arXiv:2506.05787}
}
read the original abstract
We introduce EASG-Bench, a question-answering benchmark for egocentric videos where the question-answering pairs are created from spatio-temporally grounded dynamic scene graphs capturing intricate relationships among actors, actions, and objects. We propose a systematic evaluation framework and evaluate several language-only and video large language models (video-LLMs) on this benchmark. We observe a performance gap in language-only and video-LLMs, especially on questions focusing on temporal ordering, thus identifying a research gap in the area of long-context video understanding. To promote the reproducibility of our findings and facilitate further research, the benchmark and accompanying code are available at the following GitHub page: https://github.com/fpv-iplab/EASG-bench.
Figures
Reference graph
Works this paper leans on
-
[14]
Action scene graphs for long- form understanding of egocentric videos
Ivan Rodin, Antonino Furnari, Kyle Min, Subarna Tripathi, and Giovanni Maria Farinella. Action scene graphs for long- form understanding of egocentric videos. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18622–18632, 2024. 1, 2
work page 2024
-
[1]
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhao- hai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Hang Zhang, Zhibo Yang, Haiyang Xu, and Jun- yang Lin. Qwen2.5-vl technical repor...
arXiv 2025
-
[2]
Grounded multi- hop videoqa in long-form egocentric videos
Qirui Chen, Shangzhe Di, and Weidi Xie. Grounded multi- hop videoqa in long-form egocentric videos. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 2159–2167, 2025. 1
work page 2025
-
[3]
Egoplan- bench: Benchmarking egocentric embodied planning with multimodal large language models
Yi Chen, Yuying Ge, Yixiao Ge, Mingyu Ding, Bohao Li, Rui Wang, Ruifeng Xu, Ying Shan, and Xihui Liu. Egoplan- bench: Benchmarking egocentric embodied planning with multimodal large language models. CoRR, 2023. 1
work page 2023
-
[4]
Egothink: Evalu- ating first-person perspective thinking capability of vision- language models
Sijie Cheng, Zhicheng Guo, Jingwen Wu, Kechen Fang, Peng Li, Huaping Liu, and Yang Liu. Egothink: Evalu- ating first-person perspective thinking capability of vision- language models. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pages 14291–14302, 2024. 1, 2
work page 2024
-
[5]
Amego: Active memory from long egocentric videos
Gabriele Goletto, Tushar Nagarajan, Giuseppe Averta, and Dima Damen. Amego: Active memory from long egocentric videos. In European Conference on Computer Vision, pages 92–110. Springer, 2024. 2
work page 2024
-
[6]
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Ab- hinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024. 3, 5
arXiv 2024
-
[7]
Lita: Language instructed temporal-localization assistant
De-An Huang, Shijia Liao, Subhashree Radhakrishnan, Hongxu Yin, Pavlo Molchanov, Zhiding Yu, and Jan Kautz. Lita: Language instructed temporal-localization assistant. In ECCV, 2024. 1, 2, 3, 5
work page 2024
Show all 17 references
-
[8]
Llava-onevision: Easy visual task transfer
Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Zi- wei Liu, et al. Llava-onevision: Easy visual task transfer. arXiv preprint arXiv:2408.03326, 2024. 2
2024 arXiv
-
[9]
Egoschema: A diagnostic benchmark for very long- form video language understanding
Karttikeya Mangalam, Raiymbek Akshulakov, and Jitendra Malik. Egoschema: A diagnostic benchmark for very long- form video language understanding. Advances in Neural In- formation Processing Systems , 36:46212–46244, 2023. 1, 2
2023
-
[10]
Advancing egocentric video question answering with multimodal large language models
Alkesh Patel, Vibhav Chitalia, and Yinfei Yang. Advancing egocentric video question answering with multimodal large language models. arXiv preprint arXiv:2504.04550 , 2025. 1, 2
2025 arXiv
-
[11]
Omnia de egotempo: Benchmarking temporal understanding of multi-modal llms in egocentric videos
Chiara Plizzari, Alessio Tonioni, Yongqin Xian, Achin Kul- shrestha, and Federico Tombari. Omnia de egotempo: Benchmarking temporal understanding of multi-modal llms in egocentric videos. arXiv preprint arXiv:2503.13646 ,
-
[12]
Sam 2: Segment anything in images and videos
Nikhila Ravi, Valentin Gabeur, Yuan-Ting Hu, Ronghang Hu, Chaitanya Ryali, Tengyu Ma, Haitham Khedr, Roman R¨adle, Chloe Rolland, Laura Gustafson, et al. Sam 2: Segment anything in images and videos. arXiv preprint arXiv:2408.00714, 2024. 2
2024 arXiv
-
[13]
Timechat: A time-sensitive multimodal large language model for long video understanding
Shuhuai Ren, Linli Yao, Shicheng Li, Xu Sun, and Lu Hou. Timechat: A time-sensitive multimodal large language model for long video understanding. ArXiv, abs/2312.02051,
-
[15]
Gemma 2: Improving open language models at a practical size
Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupati- raju, L´eonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ram´e, et al. Gemma 2: Improving open language models at a practical size. arXiv preprint arXiv:2408.00118,
-
[16]
Qwen3 technical report
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint arXiv:2505.09388, 2025. 5
2025 arXiv
-
[17]
Sa2va: Marrying sam2 with llava for dense grounded understanding of images and videos
Haobo Yuan, Xiangtai Li, Tao Zhang, Zilong Huang, Shilin Xu, Shunping Ji, Yunhai Tong, Lu Qi, Jiashi Feng, and Ming-Hsuan Yang. Sa2va: Marrying sam2 with llava for dense grounded understanding of images and videos. arXiv preprint arXiv:2501.04001, 2025. 2, 5 6
2025 arXiv
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.