Pith. sign in

REVIEW 3 major objections 4 minor 17 references

EASG-Bench: Video Q&A Benchmark with Egocentric Action Scene Graphs

T0 review · 3 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Pith's one-line read The paper introduces EASG-Bench, a video question-answering benchmark built from egocentric action scene graphs, and reports that video-LLMs trail language-only models on temporal ordering questions.

desk verdict A useful grounded egocentric QA benchmark with a solid construction pipeline, but the headline language-only vs. video-LLM gap is confounded: the language-only models are effectively handed the answer via timestamped captions. read the letter →

arxiv 2506.05787 v2 pith:PLVTTP34 submitted 2025-06-06 cs.CV

classification cs.CV
keywords egocentricvideoquestionansweringactionscenegraphstemporalreasoninglargelanguagemodelsopen-endedEgo4Dchain-of-thoughtprompting
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

EASG-Bench is a video question-answering benchmark whose 1,807 open-ended questions are generated from Egocentric Action Scene Graphs rather than from narrations, grounded in the actors, actions, and objects of 221 Ego4D clips. The paper argues that grounding question generation in these spatio-temporally localized graphs makes each question uniquely answerable from the video and keeps evaluation language-neutral, since objects are referenced by bounding boxes rather than category names. The central finding is a performance gap: strong language-only models answer 'before/after' ordering questions better than video-LLMs, even though video-LLMs win on direct and indirect object manipulation questions. A two-stage chain-of-thought prompt that first localizes the action and then asks about its predecessor or successor narrows the gap, especially for 'after' questions. If these results hold, the benchmark provides a grounded diagnostic for a specific weakness in current long-context video understanding.

What carries the argument

The central object is the Egocentric Action Scene Graph (EASG), a dynamic graph in which the camera wearer and objects appear as nodes and actions and relations appear as edges, each anchored to video frames and timestamps. The machinery is a three-stage QA generation pipeline: an LLM is conditioned on the graph sequence to propose questions, a majority-voted Llama-3.1-8B filter keeps only questions that can be answered unambiguously from direct observation, and a second filter removes generic-sounding pairs. Evaluation uses an LLM-as-a-judge that rates candidate answers on a 1-10 scale with context, question, ground truth, and grading rubric, and the scores are rescaled relative to the score the judge gives to the ground truth, producing the reported relative score per question type. For the ordering analysis, the paper adds a two-stage chain-of-thought prompt that first asks the model to locate the action and then asks what happened before or after it.

What would settle it

Take a random sample of, say, 100 EASG-Bench questions and have independent human annotators watch the corresponding clips and list every valid answer to each question; if a substantial fraction of questions admit more than one defensible answer or require information the video does not show, the benchmark's core claim of unambiguous, video-grounded QA fails.

Watch

Extended reading notes

Core claim

The paper's central claim is that EASG-Bench measures video understanding through questions that carry both spatial and temporal grounding, because each QA pair is derived from a sequence of Egocentric Action Scene Graphs instead of captions. Using Llama-3.1-8B-Instruct to generate questions from the graph sequence and to filter them by five-iteration majority voting on unambiguous answerability, the authors built 1,807 QA pairs in five categories: purpose, direct object, indirect object, before, and after. Benchmarked with open-ended answers scored by an LLM judge with relative rescaling against ground-truth scores, the best video-LLM, Qwen2.5-VL, outperformed language-only models on direct and indirect manipulation questions but fell behind them on ordering questions. The paper further shows that a two-stage chain-of-thought prompt improves Qwen2.5-VL's ordering scores, but the model still trails text-only LLMs on 'before' questions.

Load-bearing premise

The load-bearing premise is that the EASG annotations are accurate and complete enough that the questions generated from them, after Llama majority-vote filtering, are genuinely answerable in exactly one way from the video; the paper reports no human verification of this uniqueness.

Editorial extensions

If this is right

  • EASG-Bench offers an evaluation where questions and answers are tied to concrete bounding boxes and timestamps, so model mistakes can be traced to specific spatio-temporal locations rather than to paraphrase mismatch.
  • The measured gap identifies temporal ordering, not object recognition, as the main weakness of current video-LLMs on long egocentric video.
  • The chain-of-thought result suggests that explicitly separating action localization from relational questioning is a workable way to improve video-LLM performance on ordering tasks.
  • Because objects are referred to by grounding rather than by category names, the benchmark can support language-neutral evaluation across different languages.
  • All five question categories are open-ended, so the benchmark can be used to probe reasoning quality rather than just answer selection.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural extension would be to test the same models on a human-verified subset of EASG-Bench; if the temporal gap persists under stricter scoring, the ordering weakness is a property of the models rather than of the LLM judge.
  • The benchmark's grounding in bounding boxes rather than category names could be adapted to measure compositional generalization, for example by asking about objects that were not seen during training.
  • The asymmetric improvement, where chain-of-thought helps 'after' more than 'before', hints that the remaining bottleneck is look-back memory; an extension could probe whether longer context windows or explicit memory modules close the 'before' gap.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper introduces EASG-Bench, a video question-answering benchmark for egocentric videos built from Egocentric Action Scene Graphs (EASGs) introduced in prior work [14]. Question-answer pairs are generated with Llama-3.1-8B-Instruct from graph sequences, filtered through a majority-vote uniqueness check and a genericness filter, yielding 1,807 Q&A pairs over 221 Ego4D clips in five categories (purpose, direct object, indirect object, before, after). The authors evaluate three language-only LLMs and four video-LLMs using an LLM-as-a-judge protocol with relative scores, report that language-only models outperform video-LLMs on before/after ordering questions, and show that a two-stage chain-of-thought prompt improves Qwen2.5-VL on those questions. The benchmark and code are released publicly.

Significance. If the evaluation were controlled, EASG-Bench would be a useful addition to egocentric video QA: it is grounded in structured spatio-temporal scene graphs rather than narrations, covers long untrimmed videos, and includes both spatial and temporal grounding. The public release of benchmark and code supports reproducibility, and the systematic three-stage generation pipeline is clearly described. However, the central empirical claim—that video-LLMs underperform language-only models on temporal ordering—currently rests on an asymmetric evaluation protocol and an unvalidated annotation premise, so the significance is conditional on addressing the confounds identified below.

major comments (3)
  1. [Section 4 (Evaluation) and Table 2] The paper does not specify what input the language-only models receive, but the evaluation section defines the context C as 'video captions with timestamps.' If language-only models are given C, then before/after questions are partially answered by reading the ordered action sequence in text, whereas video-LLMs must perceive and segment events from raw video. The large gap in Table 2 (e.g., Before: Qwen3-8B 98.92 vs. Qwen2.5-VL-7B 82.76; After: 70.81 vs. 65.97) may therefore reflect information asymmetry rather than weaker temporal reasoning in video-LLMs, which undermines the abstract's central claim. Please add a matched-control condition—for example, language-only models without C, or video-LLMs given C alongside video—and report results under both settings.
  2. [Section 3 (Benchmark creation)] The uniqueness filtering is performed by Llama-3.1-8B majority voting, but no human validation is reported for the generated QA pairs or for the underlying EASG annotations. If the graphs omit an action or label an action incorrectly, the generated ground-truth answer and its 'unique answerability' determination inherit that error. Please add a human study on a random sample (e.g., 150–200 pairs) measuring factual correctness, answerability, and ambiguity, and report inter-annotator agreement. Without such validation, the benchmark's grounding premise is not established.
  3. [Section 4, Eq. (1) and Table 2] All scores are single-run LLM-judge outputs without error bars or significance tests. The differences that drive the central claim (the before/after gap) could be within judge noise. Additionally, the judge is Llama-3.1-8B, the same model family used for QA generation and filtering, which may introduce systematic bias favoring certain answer styles. Please report multiple judge runs with variance, and correlate judge scores with human judgments on a subset to validate the evaluation protocol.
minor comments (4)
  1. [Section 3 and Figure 3] The 'language-neutral' property is stated as a feature of the benchmark, but the released questions and examples use object category names (e.g., 'cable', 'pipe'). Clarify whether the benchmark includes questions posed with bounding boxes and whether the language-neutral variant is actually implemented in the released dataset.
  2. [Section 4, Table 3] The chain-of-thought experiment is conducted on a single model (Qwen2.5-VL-7B) with one prompt template and no variance estimate; report results across models or explicitly frame this as a preliminary study.
  3. [Table 1] The 'Untrimmed' column is ambiguous for EASG-Bench; specify whether the 221 clips are used in their entirety or whether shorter segments are extracted from longer Ego4D videos.
  4. [Section 4, Eq. (1)] The normalization denominator sums over all questions of the same type, so the relative score for each question depends on the composition of the benchmark; state whether the normalization is computed on the full benchmark or per split and discuss whether scores are comparable across question types.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the benchmark construction and evaluation are independent measurements; the EASG dependency is a data source, not a self-referential derivation.

full rationale

EASG-Bench constructs QA pairs from EASG annotations [14], then measures model performance. The only overlapping-author dependency is the EASG dataset, which is prior published work (Rodin et al., CVPR 2024) used as ground-truth input, not as a conclusion derived within this paper. The QA generation and uniqueness filtering are performed by Llama-3.1-8B, and the same model family is also used as judge and as one evaluated baseline; however, this is a shared tool, not a step where an output is defined as its input. The relative-score formula (Eq. 1) rescales judge scores by the judge's score on the ground-truth answer, but this is a normalization for calibration, not a fitted parameter renamed as a prediction. The reported language-only vs. video-LLM gap is a measured result; the paper does not fit the gap from the benchmark's construction. The language-only models receive timestamped captions that encode order, which may confound the comparison, but that is an evaluation-validity concern rather than circular reasoning. No equation or definition in the paper makes the central claim equal to an input by construction.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The central claims rest on four domain assumptions about the EASG source data, the LLM-based uniqueness filter, the LLM judge, and the textual proxy used for language-only baselines. There are no fitted free parameters or invented physical entities. The assumptions are reasonable for a benchmark paper but should be validated with human studies and error analysis.

assumptions (4)
  • domain assumption EASG annotations from Rodin et al. [14] accurately capture actors, actions, and object relations in the selected Ego4D clips.
    All QA generation in Section 3 starts from these graphs; errors or omissions in the graphs would propagate into questions and ground-truth answers.
  • domain assumption Llama-3.1-8B-Instruct's majority-vote uniqueness filter selects only questions that have a unique, video-observable answer.
    Section 3 and Figure 2 rely on this filter with no human agreement check; the filter uses the same model family that generated the questions, so it may share the model's blind spots.
  • domain assumption Llama3-8B-Instruct as an LLM judge produces scores that reflect answer quality, and relative rescaling removes judge bias.
    Section 4 defines evaluation entirely through this judge; no human-judge correlation or inter-judge agreement is reported.
  • domain assumption The timestamped video captions given to the judge and to language-only models are a faithful textual proxy for the video content.
    Language-only models receive text captions while video-LLMs see raw video; the performance comparison therefore measures text-grounded versus video-grounded inference.

how reviews work

0 comments
Cite this review

Pith. "Pith review of EASG-Bench: Video Q&A Benchmark with Egocentric Action Scene Graphs." pith.science (2026). https://pith.science/paper/PLVTTP34

@misc{pith2026250605787,
  author       = {Pith},
  title        = {Pith review of: EASG-Bench: Video Q&A Benchmark with Egocentric Action Scene Graphs},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/PLVTTP34}},
  note         = {Machine review of arXiv:2506.05787}
}
read the original abstract

We introduce EASG-Bench, a question-answering benchmark for egocentric videos where the question-answering pairs are created from spatio-temporally grounded dynamic scene graphs capturing intricate relationships among actors, actions, and objects. We propose a systematic evaluation framework and evaluate several language-only and video large language models (video-LLMs) on this benchmark. We observe a performance gap in language-only and video-LLMs, especially on questions focusing on temporal ordering, thus identifying a research gap in the area of long-context video understanding. To promote the reproducibility of our findings and facilitate further research, the benchmark and accompanying code are available at the following GitHub page: https://github.com/fpv-iplab/EASG-bench.

Figures

Figures reproduced from arXiv: 2506.05787 by the authors.

Figure 1
Figure 1. Dataset statistics. to generate a categorically structured set of questions that probe various aspects of the visual content. Specifically, we systematically generate questions belonging to the follow￾ing categories: • Purpose Questions: Inquiries that explore the under￾lying intent of an object or an action, e.g., What is the purpose of object X in a video? • Direct Object Questions: Questions that focus on the pri… view at source ↗
Figure 2
Figure 2. The 3-stage process of QA pairs generation, example for the ”direct” and ”after” question types. First, we prompt LLM to [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Samples of questions, ground truth answers, and the answer generated by Qwen2.5-VL for each type of question in [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Examples of evaluation. The LLM judge is tasked with providing a rating to the given answer and an explanation for the rating. [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

17 extracted references · 7 canonical work pages

  1. [14]

    Action scene graphs for long- form understanding of egocentric videos

    Ivan Rodin, Antonino Furnari, Kyle Min, Subarna Tripathi, and Giovanni Maria Farinella. Action scene graphs for long- form understanding of egocentric videos. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18622–18632, 2024. 1, 2

  2. [1]

    Qwen2.5-vl technical report

    Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhao- hai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Hang Zhang, Zhibo Yang, Haiyang Xu, and Jun- yang Lin. Qwen2.5-vl technical repor...

  3. [2]

    Grounded multi- hop videoqa in long-form egocentric videos

    Qirui Chen, Shangzhe Di, and Weidi Xie. Grounded multi- hop videoqa in long-form egocentric videos. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 2159–2167, 2025. 1

  4. [3]

    Egoplan- bench: Benchmarking egocentric embodied planning with multimodal large language models

    Yi Chen, Yuying Ge, Yixiao Ge, Mingyu Ding, Bohao Li, Rui Wang, Ruifeng Xu, Ying Shan, and Xihui Liu. Egoplan- bench: Benchmarking egocentric embodied planning with multimodal large language models. CoRR, 2023. 1

  5. [4]

    Egothink: Evalu- ating first-person perspective thinking capability of vision- language models

    Sijie Cheng, Zhicheng Guo, Jingwen Wu, Kechen Fang, Peng Li, Huaping Liu, and Yang Liu. Egothink: Evalu- ating first-person perspective thinking capability of vision- language models. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pages 14291–14302, 2024. 1, 2

  6. [5]

    Amego: Active memory from long egocentric videos

    Gabriele Goletto, Tushar Nagarajan, Giuseppe Averta, and Dima Damen. Amego: Active memory from long egocentric videos. In European Conference on Computer Vision, pages 92–110. Springer, 2024. 2

  7. [6]

    The llama 3 herd of models

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Ab- hinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024. 3, 5

  8. [7]

    Lita: Language instructed temporal-localization assistant

    De-An Huang, Shijia Liao, Subhashree Radhakrishnan, Hongxu Yin, Pavlo Molchanov, Zhiding Yu, and Jan Kautz. Lita: Language instructed temporal-localization assistant. In ECCV, 2024. 1, 2, 3, 5

Show all 17 references
  1. [8]

    Llava-onevision: Easy visual task transfer

    Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Zi- wei Liu, et al. Llava-onevision: Easy visual task transfer. arXiv preprint arXiv:2408.03326, 2024. 2

  2. [9]

    Egoschema: A diagnostic benchmark for very long- form video language understanding

    Karttikeya Mangalam, Raiymbek Akshulakov, and Jitendra Malik. Egoschema: A diagnostic benchmark for very long- form video language understanding. Advances in Neural In- formation Processing Systems , 36:46212–46244, 2023. 1, 2

  3. [10]

    Advancing egocentric video question answering with multimodal large language models

    Alkesh Patel, Vibhav Chitalia, and Yinfei Yang. Advancing egocentric video question answering with multimodal large language models. arXiv preprint arXiv:2504.04550 , 2025. 1, 2

  4. [11]

    Omnia de egotempo: Benchmarking temporal understanding of multi-modal llms in egocentric videos

    Chiara Plizzari, Alessio Tonioni, Yongqin Xian, Achin Kul- shrestha, and Federico Tombari. Omnia de egotempo: Benchmarking temporal understanding of multi-modal llms in egocentric videos. arXiv preprint arXiv:2503.13646 ,

  5. [12]

    Sam 2: Segment anything in images and videos

    Nikhila Ravi, Valentin Gabeur, Yuan-Ting Hu, Ronghang Hu, Chaitanya Ryali, Tengyu Ma, Haitham Khedr, Roman R¨adle, Chloe Rolland, Laura Gustafson, et al. Sam 2: Segment anything in images and videos. arXiv preprint arXiv:2408.00714, 2024. 2

  6. [13]

    Timechat: A time-sensitive multimodal large language model for long video understanding

    Shuhuai Ren, Linli Yao, Shicheng Li, Xu Sun, and Lu Hou. Timechat: A time-sensitive multimodal large language model for long video understanding. ArXiv, abs/2312.02051,

  7. [15]

    Gemma 2: Improving open language models at a practical size

    Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupati- raju, L´eonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ram´e, et al. Gemma 2: Improving open language models at a practical size. arXiv preprint arXiv:2408.00118,

  8. [16]

    Qwen3 technical report

    An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint arXiv:2505.09388, 2025. 5

  9. [17]

    Sa2va: Marrying sam2 with llava for dense grounded understanding of images and videos

    Haobo Yuan, Xiangtai Li, Tao Zhang, Zilong Huang, Shilin Xu, Shunping Ji, Yunhai Tong, Lu Qi, Jiashi Feng, and Ming-Hsuan Yang. Sa2va: Marrying sam2 with llava for dense grounded understanding of images and videos. arXiv preprint arXiv:2501.04001, 2025. 2, 5 6

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.