Pith. sign in

REVIEW 3 major objections 7 minor 75 references

Long-video counting fails mainly because models cannot list instances and ground evidence, not because they cannot add.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-13 15:28 UTC pith:Y34LWIMX

load-bearing objection Solid diagnostic benchmark: long-video counting fails mainly at instance ID and temporal grounding, not arithmetic, and the numbers make that hard to ignore. the 3 major comments →

arxiv 2603.29943 v2 pith:Y34LWIMX submitted 2026-03-31 cs.CV

Diagnosing Long-Video Quantitative Reasoning in Multimodal LLMs via Enumeration and Counting

Fumihiko Tsuchiya , Taiki Miyanishi , Shunsuke Yasuki , Mahiro Ukai , Nakamasa Inoue , Shuhei Kurita , Yusuke Iwasawa , Yutaka Matsuo This is my paper
classification cs.CV
keywords long-video understandingmultimodal large language modelsvideo countingenumerationtemporal groundingEC-Benchquantitative reasoningevidence spans
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Final-answer video QA only checks whether a model outputs the right number; it does not reveal which instances were counted, when the evidence occurred, or why the answer went wrong. This paper claims that reliable counting in videos longer than half an hour requires three coupled skills: enumerating every query-relevant instance without omissions or duplicates, temporally grounding the supporting evidence, and then aggregating those instances into a count. To make that diagnosis possible, the authors build EC-Bench—152 untrimmed videos over 30 minutes, 1,699 open-ended queries in six reasoning categories, and human-verified evidence spans—and evaluate 22 open-source and proprietary multimodal models on timestamped frames plus transcripts. The best systems reach only about 30% enumeration quality and 24% counting accuracy, against human scores near 80%. Analyses show enumeration quality tracks counting accuracy, better temporal localization predicts lower counting error, and accuracy falls as evidence becomes more spread out, recasting long-video counting as evidence retrieval, grounding, deduplication, and aggregation rather than simple numerical prediction.

Core claim

Counting errors on long videos are rarely isolated arithmetic mistakes. Across 22 multimodal models on EC-Bench, enumeration quality is strongly associated with counting accuracy, temporal grounding quality is associated with lower counting error, and counting accuracy drops as supporting evidence becomes more distributed; the best model reaches only 29.98% enumeration and 23.74% counting versus human 78.57% and 82.97%. Long-video counting is therefore evidence retrieval, temporal grounding, deduplication, and aggregation across the video, not pure number prediction.

What carries the argument

EC-Bench: an evidence-annotated suite that jointly evaluates enumeration (list all relevant instances), counting (produce the integer), and temporal evidence grounding (predict supporting spans) on untrimmed videos longer than 30 minutes, with six reasoning categories and human-verified multi-span annotations.

Load-bearing premise

That scoring models on a fixed budget of uniformly sampled timestamped frames plus speech transcripts, and collapsing multi-span evidence into one merged interval, fairly measures their real long-video quantitative reasoning.

What would settle it

A model that still fails enumeration and has low temporal overlap with evidence spans yet reaches near-human counting accuracy on EC-Bench, or denser adaptive sampling and native audio that erase the enumeration–counting correlation, would undercut the claim that counting errors are mainly evidence-retrieval failures.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Final-answer counting accuracy alone is an incomplete diagnostic; enumeration quality and evidence-span overlap should be reported together.
  • Enumerate-then-count prompting can raise counting accuracy without retraining the model.
  • Simply increasing uniform frame count yields limited or saturating gains; instance identification and temporal grounding are the binding constraints.
  • Performance falls as the number of evidence clues grows, so multi-hop and multi-span queries remain especially hard.
  • Closing the large human–model gap requires systems that track, deduplicate, and ground instances over tens of minutes, not architectures tuned only for short clips.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Training that rewards only the final number may encourage shortcutting; multi-stage supervision on instance lists and evidence spans is a natural training target.
  • Query-conditioned adaptive sampling or long-horizon memory that stores instance identities over tens of minutes is a direct architectural follow-on.
  • The same enumeration-plus-grounding diagnosis likely extends to other sparse long-horizon tasks such as multi-event tracking and procedural verification.
  • Collapsing multi-span evidence into a single merged interval may understate localization failures when evidence is fragmented across the timeline.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 7 minor

Summary. The paper argues that final-answer counting on long videos conceals whether models identify the right instances, when evidence occurs, and why they fail. It formalizes long-video quantitative reasoning as three coupled abilities—enumeration of query-relevant instances, temporal grounding of supporting spans, and aggregation into counts—and introduces EC-Bench: 152 untrimmed videos longer than 30 minutes, 1,699 open-ended queries in six reasoning categories, and human-verified evidence spans. Evaluating 22 open-source and proprietary MLLMs under a standardized timestamped-frame plus Whisper-transcript protocol, the best systems reach only 29.98% Enumeration and 23.74% Counting versus human 78.57%/82.97%. Analyses (Spearman ρ=0.692 between Enumeration and Counting, tIoU associations, clue-count degradation, enumeration-first prompting gains, modality and frame-density ablations) support recasting long-video counting as evidence retrieval, grounding, deduplication, and aggregation rather than isolated arithmetic.

Significance. If the diagnosis holds, EC-Bench is a useful, falsifiable diagnostic suite for a capability that short-clip counting and multiple-choice long-video QA do not isolate. Strengths include open-ended answers, explicit evidence spans, a balanced six-category design, evaluation of 22 models, a human baseline, modality and frame-density ablations, enumeration-first prompting gains, and public data/code. The LLM-as-Judge for enumeration is validated against humans (Spearman ρ=0.8233). The work is significant as a benchmark-and-diagnosis paper rather than as a new architecture: it clarifies where current MLLMs fail on ultra-long quantitative reasoning and gives the community a structured evaluation target.

major comments (3)
  1. Human Enumeration protocol vs model metric (§A Human Performance Evaluation; §3.4; Abstract/Table 2): Human Enumeration is scored as exact full-list correctness (“correct only if all required entities… were correctly listed”), while models are scored with LLM-as-Judge precision/recall/F1. The headline gap (human 78.57% vs best model 29.98%) therefore mixes incompatible metrics. Please either re-score humans with the same F1 judge (or re-score models with exact-set accuracy) and report both, or clearly state that the human number is not directly comparable to model F1 and adjust the abstract/tables accordingly. This is load-bearing for the claimed human–model gap.
  2. Merged multi-span tIoU (§3.4 Evidence spans; Fig. 7c; Fig. 10): Predicted and ground-truth multi-span evidence are collapsed into single intervals before tIoU. The paper’s central diagnosis emphasizes distributed evidence and multi-clue aggregation (Fig. 10 shows sharp Counting drops as clue count grows), yet the grounding metric cannot distinguish correct multi-span localization from a coarse envelope over the video. Please add a multi-span metric (e.g., average pairwise tIoU, F1 over matched spans, or coverage of each GT span) and re-check the association between grounding quality and Counting error under that metric; if the association weakens, qualify the temporal-grounding claim.
  3. Table 2 labeling vs §3.4 metrics: Table 2 reports “Accuracy (%)” for Enumeration categories and averages, while §3.4 defines Enumeration via precision/recall/F1 and the abstract cites “29.98% Enumeration F1.” Confirm whether Table 2 cells are F1 or exact accuracy, relabel consistently, and report precision/recall (as in Fig. 7c) alongside the main table so readers can see omission vs hallucination trade-offs that drive the diagnosis.
minor comments (7)
  1. Clarify contamination risk from using Gemini-2.5-pro for draft QA generation (§3.2) while evaluating Gemini 2.0/2.5 models (Table 2). State what was regenerated by humans (60.9% corrected) and whether any evaluated Gemini checkpoint could have seen the draft queries.
  2. Fig. 7a is referenced as “Table 7a” in the text; keep figure/table numbering consistent.
  3. §4.1: specify exact frame-sampling rule per model when context limits force fewer than 128 frames (e.g., LLaVA-Next-Video at 64, MiMo-VL at 80) so comparisons are reproducible.
  4. Abstract says “Enumeration F1” while some body text says “Enumeration accuracy”; unify terminology throughout.
  5. Fig. 5 caption compares EC-Bench to CG-AV-Counting but the body text cites CG-AV-Counting [31] and CG-Bench [6] in nearby places; ensure the length-distribution comparison cites the intended dataset.
  6. Appendix Table 3 genre breakdown is useful; consider promoting a short genre summary into the main paper near Fig. 6 for readers who skip the appendix.
  7. Minor typos/wording: e.g., “accuratelong-videocountingfundamentallyrequires” spacing artifacts in the introduction PDF text; “Speech & Audio” vs “Speech/Audio” category naming consistency.

Circularity Check

0 steps flagged

No significant circularity: EC-Bench is an external empirical evaluation of MLLMs against human-verified annotations, not a derivation that restates fitted inputs as predictions.

full rationale

The paper constructs a long-video enumeration/counting benchmark (152 videos, 1,699 queries, human-verified evidence spans), evaluates 22 independent MLLMs under a fixed protocol, and reports empirical associations (Enumeration–Counting Spearman ρ=0.692; tIoU vs. counting error; degradation with more distributed clues; enumeration-first prompting gains). These are observational results against held-out human labels and external model APIs/checkpoints, not algebraic identities or parameters fitted to the target metric and then re-reported as predictions. Draft QA generation with Gemini-2.5-pro is followed by multi-annotator human revision (60.9% of pairs corrected) and double-checking, so the evaluation set is not self-defined by the evaluated models. LLM-as-Judge for Enumeration is validated against a human study (Spearman ρ=0.8233). No uniqueness theorem, ansatz, or self-citation chain forces the central claim. The derivation chain is: build annotated suite → measure models → correlate abilities; none of these steps reduces by construction to its inputs. Score 0 is appropriate.

Axiom & Free-Parameter Ledger

3 free parameters · 4 axioms · 3 invented entities

As an empirical benchmark paper the load-bearing content is measurement design rather than free physical constants. The ledger records the operational choices that define what “correct enumeration/counting/grounding” means and the background assumptions needed to treat the reported scores as diagnostic of long-video quantitative reasoning.

free parameters (3)
  • default frame budget / sampling density = typically 128 frames
    Most models evaluated at 64–128 uniformly sampled frames (up to 2048 in ablation); absolute scores depend on this hand-chosen coverage.
  • 5-second reappearance merge rule = 5 seconds
    Annotation and answer rules count reappearances within 5 s as one occurrence; this threshold is a design choice that changes ground-truth counts.
  • merged multi-span tIoU definition = union-then-IoU
    All predicted and gold evidence intervals are collapsed to single unions before IoU; the merge policy is a free evaluation design choice.
axioms (4)
  • ad hoc to paper Accurate long-video counting requires three coupled abilities: complete instance enumeration without omission/duplication, temporal localization of supporting spans, and aggregation into a count.
    Stated as the paper’s organizing premise in the introduction and Section 3; it defines the joint evaluation design rather than being derived from prior theory.
  • domain assumption Uniformly sampled timestamped RGB frames plus Whisper transcripts constitute a fair standardized input for comparing current video MLLMs on long-form quantitative tasks.
    Section 4.1 input configuration; standard in the literature but not proven optimal for every architecture.
  • domain assumption LLM-as-Judge semantic matching of open-ended enumeration lists is a valid surrogate for human item matching when validated by correlation.
    Section 3.4; supported by a 100-query human study (ρ=0.8233) but still an external judge model.
  • domain assumption Human-revised answers and evidence spans after Gemini draft generation are sufficiently accurate ground truth.
    Section 3.2 and appendix annotation protocol (14 annotators, double-check, 60.9% of drafts corrected).
invented entities (3)
  • EC-Bench independent evidence
    purpose: Evidence-annotated evaluation suite jointly scoring enumeration, counting, and temporal grounding on ultra-long videos.
    New dataset and metric suite introduced by the paper; independent existence is the public release itself.
  • Six quantitative reasoning categories (Parallel, Causal, Speech/Audio, Appearance, Spatial, Conditional) no independent evidence
    purpose: Taxonomy used to balance and diagnose different long-video counting patterns.
    Paper-defined partition of queries; useful but not independently standardized outside this work.
  • Enumeration ability (as formalized for long video) no independent evidence
    purpose: Explicit requirement to list all query-relevant instances without omission or duplication before aggregation.
    Conceptual primitive the authors introduce to diagnose counting; operationalized via their F1 protocol.

pith-pipeline@v1.1.0-grok45 · 26130 in / 3243 out tokens · 40003 ms · 2026-07-13T15:28:40.343644+00:00 · methodology

0 comments
read the original abstract

Final-answer video QA can show whether a model predicts the right number, but not which instances it counted, when the supporting evidence occurs, or why it failed. We diagnose long-video quantitative reasoning in multimodal large language models (MLLMs) through three coupled abilities: enumerating query-relevant instances, temporally grounding supporting evidence, and aggregating the evidence into counts. To support this analysis, we build EC-Bench, an evidence-annotated evaluation suite with 152 untrimmed videos longer than 30 minutes, 1,699 open-ended queries across six reasoning categories, and human-verified evidence spans. We evaluate 22 open-source and proprietary MLLMs using timestamped visual frames and transcripts. The best average scores reach only 29.98% Enumeration F1 and 23.74% Counting accuracy, compared with human performance of 78.57% and 82.97%, respectively. Our analyses show that counting errors are rarely isolated arithmetic mistakes: Enumeration F1 is strongly associated with Counting accuracy, temporal grounding quality is associated with lower counting error, and Counting accuracy drops as supporting evidence becomes more distributed. These findings recast long-video counting as evidence retrieval, temporal grounding, deduplication, and aggregation across the video, rather than simple numerical prediction.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

75 extracted references · 14 linked inside Pith

  1. [1]

    dvd-counting (2025),https://huggingface.co/datasets/Video- R1/ DVD-counting5

    Video-r1. dvd-counting (2025),https://huggingface.co/datasets/Video- R1/ DVD-counting5

    2025

  2. [2]

    An, X., Xie, Y., Yang, K., Zhang, W., Zhao, X., Cheng, Z., Wang, Y., Xu, S., Chen, C., Wu, C., Tan, H., Li, C., Yang, J., Yu, J., Wang, X., Qin, B., Wang, Y., Yan, Z., Feng, Z., Liu, Z., Li, B., Deng, J.: Llava-onevision-1.5: Fully open framework for democratized multimodal training (2025) 9, 10, 13

    2025

  3. [3]

    Bai, J., Bai, S., Yang, S., Wang, S., Tan, S., Wang, P., Lin, J., Zhou, C., Zhou, J.: Qwen-vl: A versatile vision-language model for understanding, localization, text reading, and beyond (2023) 4

    2023

  4. [4]

    arXiv preprint arXiv:2501.01234 (2025) 9, 10, 13

    Bai, X., et al.: Qwen2.5-vl: A large multimodal language model for video under- standing. arXiv preprint arXiv:2501.01234 (2025) 9, 10, 13

    Pith/arXiv arXiv 2025

  5. [5]

    In: Proc

    Chandrasegaran, K., Gupta, A., Hadzic, L.M., Kota, T., He, J., Eyzaguirre, C., Durante, Z., Li, M., Wu, J., Li, F.F.: Hourvideo: 1-hour video-language under- standing. In: Proc. Annual Conference on Neural Information Processing Systems (NeurIPS). vol. 37 (2024) 1, 4, 5

    2024

  6. [6]

    Chen, G., Liu, Y., Huang, Y., He, Y., Pei, B., Xu, J., Wang, Y., Lu, T., Wang, L.: Cg-bench: Clue-grounded question answering benchmark for long video under- standing (2024) 4, 5, 8

    2024

  7. [7]

    In: Proceedings of the AAAI Conference on Artificial Intelligence

    Chen, Q., Di, S., Xie, W.: Grounded multi-hop videoqa in long-form egocentric videos. In: Proceedings of the AAAI Conference on Artificial Intelligence. vol. 39, pp. 2159–2167 (2025) 4

    2025

  8. [8]

    In: Proc

    Chen, Y., Xue, F., Li, D., Hu, Q., Zhu, L., Li, X., Fang, Y., Tang, H., Yang, S., Liu, Z., He, E., Yin, H., Molchanov, P., Kautz, J., Fan, L., Zhu, Y., Lu, Y., Han, S.: Longvila: Scaling long-context visual language models for long videos. In: Proc. International Conference on Learning Representations (ICLR) (2025) 4, 9, 10, 13

    2025

  9. [9]

    In: Proc

    Chen, Z., Wu, J., Wang, W., Su, W., Chen, G., Xing, S., Zhong, M., Zhang, Q., Zhu, X., Lu, L., et al.: Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. In: Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 24185–24198 (2024) 4, 9, 10, 13

    2024

  10. [10]

    Dai, W., Li, J., Li, D., Tiong, A.M.H., Zhao, J., Wang, W., Li, B., Fung, P., Hoi, S.: Instructblip: Towards general-purpose vision-language models with instruction tuning (2023) 4

    2023

  11. [11]

    arXiv preprint arXiv:2006.15418 (2020) 3, 4

    Dwibedi, D., Aytar, Y., Tompson, J., Sermanet, P., Zisserman, A.: Counting out time: Class agnostic video repetition counting in the wild. arXiv preprint arXiv:2006.15418 (2020) 3, 4

    Pith/arXiv arXiv 2006

  12. [12]

    arXiv preprint arXiv:2407.17085 (2024) 3, 4

    Dwibedi, D., Aytar, Y., Tompson, J., Zisserman, A.: Ovr: A dataset for open vo- cabulary temporal repetition counting in videos. arXiv preprint arXiv:2407.17085 (2024) 3, 4

    Pith/arXiv arXiv 2024

  13. [13]

    Fu, C., Dai, Y., Luo, Y., Li, L., Ren, S., Zhang, R., Wang, Z., Zhou, C., Shen, Y., Zhang, M., et al.: Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal llms in video analysis (2024) 4

    2024

  14. [14]

    Gemini Team, G.:Gemini 1.5:Unlockingmultimodal understanding across millions of tokens of context (2024) 4

    2024

  15. [15]

    Google Gemini Team: Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities (2025) 9, 10, 13, 2

    2025

  16. [16]

    Goyal, Y., Khot, T., Summers-Stay, D., Batra, D., Parikh, D.: Making the v in vqa matter: Elevating the role of image understanding in visual question answering. 16 F. Tsuchiya et al. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 6904–6913 (2018) 4

    2018

  17. [17]

    Hong, W., Wang, W., Ding, M., Yu, W., Lv, Q., Wang, Y., Cheng, Y., Huang, S., Ji, J., Xue, Z., Zhao, L., Yang, Z., Gu, X., Zhang, X., Feng, G., Yin, D., Wang, Z., Qi, J., Song, X., Zhang, P., Liu, D., Xu, B., Li, J., Dong, Y., Tang, J.: Cogvlm2: Visual language models for image and video understanding (2024) 4

    2024

  18. [18]

    arXiv preprint arXiv:2312.08914 (2024) 4

    Hong, W., Wang, W., Lv, Q., Xu, J., Yu, W., Ji, J., Wang, Y., Wang, Z., Zhang, Y., Li, J., Xu, B., Dong, Y., Ding, M., Tang, J.: Cogagent: A visual language model for gui agents. arXiv preprint arXiv:2312.08914 (2024) 4

    Pith/arXiv arXiv 2024

  19. [19]

    arXiv preprint arXiv:2204.01018 (2022) 4

    Hu, H., Dong, S., Zhao, Y., Lian, D., Li, Z., Gao, S.: Transrac: Encoding multi- scale temporal correlation with transformers for repetitive action counting. arXiv preprint arXiv:2204.01018 (2022) 4

    Pith/arXiv arXiv 2022

  20. [20]

    In: Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition (CVPR)

    Hudson, D.A., Manning, C.D.: Gqa: A new dataset for real-world visual reasoning and compositional question answering. In: Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition (CVPR). pp. 6700–6709 (2019) 4

    2019

  21. [21]

    In: Proc

    Jang, Y., Song, Y., Yu, Y., Kim, Y., Kim, G.: Tgif-qa: Toward spatio-temporal rea- soning in visual question answering. In: Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) 2, 4

  22. [22]

    In: Proc

    Lei, J., Yu, L., Bansal, M., Berg, T.L.: Tvqa: Localized, compositional video ques- tion answering. In: Proc. Conference on Empirical Methods in Natural Language Processing (EMNLP) (2018) 4

    2018

  23. [23]

    arXiv preprint arXiv:2307.16125 (2023) 2

    Li, B., Wang, R., Wang, G., Ge, Y., Ge, Y., Shan, Y.: Seed-bench: Benchmarking multimodal llms with generative comprehension. arXiv preprint arXiv:2307.16125 (2023) 2

    Pith/arXiv arXiv 2023

  24. [24]

    arXiv preprint arXiv:2311.17005 (2024) 1, 2, 4

    Li, K., Wang, Y., He, Y., Li, Y., Wang, Y., Liu, Y., Wang, Z., Xu, J., Chen, G., Luo, P., Wang, L., Qiao, Y.: Mvbench: A comprehensive multi-modal video understanding benchmark. arXiv preprint arXiv:2311.17005 (2024) 1, 2, 4

    Pith/arXiv arXiv 2024

  25. [25]

    International Journal of Computer Vision (IJCV) (2024) 5

    Li,X.,Wang,Z.,Zhang,J.,Chen,Y.,Yu,J.,Fang,Y.,Wang,Z.,Liu,H.,Zhang,Y., Liu, Z., Wang, C., Zhou, X.: Lvbench: A benchmark for long video understanding. International Journal of Computer Vision (IJCV) (2024) 5

    2024

  26. [26]

    arXiv preprint arXiv:2501.00574 (2024) 9, 10, 13

    Li, X., Wang, Y., Yu, J., Zeng, X., Zhu, Y., Huang, H., Gao, J., Li, K., He, Y., Wang, C., Qiao, Y., Wang, Y., Wang, L.: Videochat-flash: Hierarchical compression for long-context video modeling. arXiv preprint arXiv:2501.00574 (2024) 9, 10, 13

    Pith/arXiv arXiv 2024

  27. [27]

    Li, Y., Wang, C., Jia, J.: Llama-vid: An image is worth 2 tokens in large language models (2023) 4

    2023

  28. [28]

    Liu, H., Li, C., Wu, Q., Lee, Y.J.: Visual instruction tuning (2023) 4

    2023

  29. [29]

    Liu, Y., Duan, H., Zhang, Y., Li, B., Zhang, S., Zhao, W., Yuan, Y., Wang, J., He, C., Liu, Z., Chen, K., Lin, D.: Mmbench: Is your multi-modal model an all-around player? arXiv preprint arXiv:2307.06281 (2024) 2

    Pith/arXiv arXiv 2024

  30. [30]

    Liu, Z., Dong, Y., Liu, Z., Hu, W., Lu, J., Rao, Y.: Oryx mllm: On-demand spatial- temporal understanding at arbitrary resolution (2025) 4

    2025

  31. [31]

    Lu, L., Chen, G., Li, Z., Liu, Y., Lu, T.: Av-reasoner: Improving and benchmarking clue-grounded audio-visual counting for mllms (2025) 3, 4, 5

    2025

  32. [32]

    Mangalam, K., Akshulakov, R., Malik, J.: Egoschema: A diagnostic benchmark for very long-form video language understanding (2023) 3, 4

    2023

  33. [33]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

    Marino, K., Rastegari, M., Farhadi, A., Mottaghi, R.: Ok-vqa: A visual ques- tion answering benchmark requiring external knowledge. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 3195–3204 (2019) 4 EC-Bench 17

    2019

  34. [34]

    OpenAI: Gpt-4o technical report (2024),https://openai.com/research/gpt-4o 9, 10, 13, 2

    2024

  35. [35]

    OpenAI: Gpt-4.1 model overview (2025),https://platform.openai.com/docs/ models/gpt-4-19, 10, 13, 2

    2025

  36. [36]

    OpenAI, Achiam, J., Adler, S., et al.: Gpt-4 technical report (2024) 4

    2024

  37. [37]

    In: CVPR

    Perrett, T., Darkhalil, A., Sinha, S., Emara, O., Pollard, S., Parida, K.K., Liu, K., Gatti, P., Bansal, S., Flanagan, K., Chalk, J., Zhu, Z., Guerrier, R., Abdelazim, F., Zhu, B., Moltisanti, D., Wray, M., Doughty, H., Damen, D.: Hd-epic: A highly- detailed egocentric video dataset. In: CVPR. pp. 23901–23913 (June 2025) 5

    2025

  38. [38]

    Radford,A.,Kim,J.W.,Xu,T.,Brockman,G.,McLeavey,C.,Sutskever,I.:Robust speech recognition via large-scale weak supervision (2022) 9

    2022

  39. [39]

    Ren, W., Ma, W., Yang, H., Wei, C., Zhang, G., Chen, W.: Vamba: Understanding hour-long videos with hybrid mamba-transformers (2025) 9, 10, 13

    2025

  40. [40]

    Singh, H., Das, R.J., Han, M., Nakov, P., Laptev, I.: Malmm: Multi-agent large language models for zero-shot robotics manipulation (2025) 4

    2025

  41. [41]

    In: ACCV

    Sinha, S., Stergiou, A., Damen, D.: Every shot counts: Using exemplars for repe- tition counting in videos. In: ACCV. pp. 3056–3073 (December 2024) 5

    2024

  42. [42]

    arXiv preprint arXiv:2307.16449 (2023) 4

    Song, E., Chai, W., Wang, G., Zhang, Y., Zhou, H., Wu, F., Guo, X., Ye, T., Lu, Y., Hwang, J.N., et al.: Moviechat: From dense token to sparse memory for long video understanding. arXiv preprint arXiv:2307.16449 (2023) 4

    Pith/arXiv arXiv 2023

  43. [43]

    Wang, P., Bai, S., Tan, S., Wang, S., Fan, Z., Bai, J., Chen, K., Liu, X., Wang, J., Ge, W., Fan, Y., Dang, K., Du, M., Ren, X., Men, R., Liu, D., Zhou, C., Zhou, J., Lin, J.: Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution (2024) 4

    2024

  44. [44]

    In: Proc

    Wang,W.,He,Z.,Hong,W.,Cheng,Y.,Zhang,X.,Qi,J.,Ding,M.,Gu,X.,Huang, S., Xu, B., Dong, Y., Tang, J.: Lvbench: An extreme long video understanding benchmark. In: Proc. IEEE/CVF International Conference on Computer Vision (ICCV). pp. 22958–22967 (2025) 1, 3, 4, 5, 6, 7

    2025

  45. [45]

    In: Christodoulopoulos, C., Chakraborty, T., Rose, C., Peng, V

    Wang, X., Song, D., Chen, S., Chen, J., Cai, Z., Zhang, C., Sun, L., Wang, B.: LongLLaVA: Scaling multi-modal LLMs to 1000 images efficiently via a hybrid ar- chitecture. In: Christodoulopoulos, C., Chakraborty, T., Rose, C., Peng, V. (eds.) Findings of Empirical Methods in Natural Language Processing (EMNLP Find- ings). pp. 21419–21436 (Nov 2025) 9, 10, 13, 2

    2025

  46. [46]

    arXiv preprint arXiv:2501.12386 (2025) 9, 10, 13, 2

    Wang, Y., Li, X., Yan, Z., He, Y., Yu, J., Zeng, X., Wang, C., Ma, C., Huang, H., Gao, J., Dou, M., Chen, K., Wang, W., Qiao, Y., Wang, Y., Wang, L.: In- ternvideo2.5: Empowering video mllms with long and rich context modeling. arXiv preprint arXiv:2501.12386 (2025) 9, 10, 13, 2

    Pith/arXiv arXiv 2025

  47. [47]

    Wang, Y., Zhang, Z., McAuley, J., He, Z.: Lvchat: Facilitating long video compre- hension (2024) 4

    2024

  48. [48]

    Xiao, J., Shang, X., Yao, A., Chua, T.S.: Next-qa:next phase of question-answering to explaining temporal actions (2021) 1, 4

    2021

  49. [49]

    Xiaomi, L.C.T.: Mimo-vl technical report (2025) 9, 10, 13, 2

    2025

  50. [50]

    In: Proceedings of the AAAI Conference on Artificial Intelligence (AAAI)

    Xu, J., Mei, T., Yao, T., Zhang, Y.: Video question answering via gradually refined attention over appearance and motion. In: Proceedings of the AAAI Conference on Artificial Intelligence (AAAI). pp. 906–913 (2017) 1, 4

    2017

  51. [51]

    Ye, J., Xu, H., Liu, H., Hu, A., Yan, M., Qian, Q., Zhang, J., Huang, F., Zhou, J.: mplug-owl3: Towards long image-sequence understanding in multi-modal large language models (2024) 9, 10, 13, 2

    2024

  52. [52]

    arXiv preprint arXiv:2506.10857 (2025) 1, 4, 5 18 F

    Yu, J., Wu, Y., Chu, M., Ren, Z., Huang, Z., Chu, P., Zhang, R., He, Y., Li, Q., Li, S., et al.: Vrbench: A benchmark for multi-step reasoning in long narrative videos. arXiv preprint arXiv:2506.10857 (2025) 1, 4, 5 18 F. Tsuchiya et al

    Pith/arXiv arXiv 2025

  53. [53]

    Yu, Z., Xu, D., Yu, J., Yu, T., Zhao, Z., Zhuang, Y., Tao, D.: Activitynet-qa: A dataset for understanding complex web videos via question answering (2019) 4

    2019

  54. [54]

    Zhang, H., Li, X., Bing, L.: Video-llama: An instruction-tuned audio-visual lan- guage model for video understanding (2023) 4, 13

    2023

  55. [55]

    arXiv preprint arXiv:2408.05678 (2024) 9, 10, 13, 2

    Zhang, L., et al.: Videollama3: Advancing multimodal reasoning with long videos. arXiv preprint arXiv:2408.05678 (2024) 9, 10, 13, 2

    Pith/arXiv arXiv 2024

  56. [56]

    Zhang, P., Zhang, K., Li, B., Zeng, G., Yang, J., Zhang, Y., Wang, Z., Tan, H., Li, C., Liu, Z.: Long context transfer from language to vision (2024) 4, 9, 10, 13, 2

    2024

  57. [57]

    Zhang, Y., Li, B., Liu, h., Lee, Y.j., Gui, L., Fu, D., Feng, J., Liu, Z., Li, C.: Llava-next: A strong zero-shot video understanding model (2024) 4, 9, 10, 13, 2

    2024

  58. [58]

    arXiv preprint arXiv:2410.05352 (2024) 4

    Zhao, Y., Zhang, H., Li, C., Liu, P., Zhang, B., Yang, J., Wang, J., Lee, Y.J.: Llava- video: Learning to answer questions for long videos via iterative summarization. arXiv preprint arXiv:2410.05352 (2024) 4

    arXiv 2024

  59. [59]

    In: Proc

    Zhao, Z., Wang, L., Zhang, Y., Zhang, S., Ding, Z., Xing, Y., Zhang, J., Wang, Z., Gao, P., Shan, Y.: Longvideobench: A benchmark for long-form video understand- ing. In: Proc. AAAI Conference on Artificial Intelligence (AAAI). pp. 19672–19680 (2024) 1, 3, 4, 7

    2024

  60. [60]

    Zhao, Z., Lu, H., Huo, Y., Du, Y., Yue, T., Guo, L., Wang, B., Chen, W., Liu, J.: Needle in a video haystack: A scalable synthetic framework for benchmarking video mllms (2024) 5

    2024

  61. [61]

    [ts, te]

    Zhou, J., Shu, Y., Zhao, B., Wu, B., Xiao, S., Yang, X., Xiong, Y., Zhang, B., Huang, T., Liu, Z.: Mlvu: A comprehensive benchmark for multi-task long video understanding. arXiv preprint arXiv:2406.04264 (2024) 4 EC-Bench 1 Appendix In this Appendix, we provide supplementary details and additional analyses for EC-Bench. Section A presents the benchmark se...

    Pith/arXiv arXiv 2024

  62. [62]

    –Keep expressions concise—no full sentences

    Counting questions.Definition:Numerical answers such as counts or quantities.Rules: –Answer must be a number only (units optional). –Keep expressions concise—no full sentences. –Use commas if multiple values are needed

  63. [63]

    queries": [ {

    Enumeration questions.Definition:Listing multiple items or steps.Rules: –Items separated by commas. –Use “→” for ordered sequences. –Do not add explanations. Mandatory Global Requirements –Must require understanding of the entire video. –Must require visual and/or audio evidence. –Prefer questions that require multiple evidence clips. –Must remain fully o...

  64. [64]

    Counting Type Definition:Questions that require numerical answers such as frequency, number of people, or percentage of similar entities or events. Recording Rules: –Numbers only –No sentences—record only numbers –For multiple values: comma-separated (e.g., “2, 5, 7”) – 5-second rule: If the same subject reappears within 5 seconds, count it as a single oc...

  65. [65]

    →” –No explanations—items only –Omit ownership names (“A’s car

    Enumeration Type Definition:Questions that require listing multiple entities or procedures. Recording Rules: –List items separated by commas –For ordered (causal/procedural) sequences: connect using “→” –No explanations—items only –Omit ownership names (“A’s car”→“car”) Recording Examples: –“bat, glove, ball” –“Player A, Player B, Player C” –“forging→poli...

  66. [66]

    Carefully examine all provided video frames in chronological order

  67. [67]

    Use the audio transcription to understand context and dialog

  68. [68]

    Combine visual and audio information to generate accurate answers

  69. [69]

    Pay attention to temporal relationships between frames when counting or enumerating

  70. [70]

    results": [ {

    Identify precise video clips that support each answer Output Format Please provide answers in the following JSON format: { "results": [ { "query_id": 1, "answer": "accurate answer", "clips": [ ["00:04:12", "00:07:23"], ["00:12:12", "00:12:56"] ] } ] } Important Notes: –Choose the correct answer format: •Counting→numbers only •Enumeration→comma-separated i...

  71. [71]

    Enumerate every distinct instance relevant to the query

  72. [72]

    Describe each instance with distinguishing details (appearance, action, times- tamp hints)

  73. [73]

    Deduplicate if necessary and compute the final count

  74. [74]

    Link each instance to precise evidence clips

  75. [75]

    enumeration

    Do not guess—return only answers supported by provided data Output Format Return the answer using the following JSON structure: { "enumeration": [ "Instance description 1", "Instance description 2" ], "answer": "2", "clips": [ ["00:05:10", "00:05:35"], ["00:12:42", "00:13:05"] ] } Notes –enumeration: list of all counted instances –answer: final numeric co...