Pith. sign in

REVIEW 4 major objections 4 minor 50 references

NARU: A Benchmark for NARrative Evolution and Cultural Nuance Understanding in Japanese Extreme Long Video

T0 review · 4 major / 4 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Best AI model scores 76 percent on long Japanese video benchmark

desk verdict NARU is a genuinely useful new benchmark with a careful construction pipeline; the main soft spots are missing statistics (blind-solver convergence, inter-annotator agreement, error bars), but the core resource is solid and worth engaging with. read the letter →

arxiv 2608.13210 v1 pith:YCSO3PVF submitted 2026-08-13 cs.CV cs.AIcs.MM

Yuheng Huang , Jianlang Chen , Jiayang Song , Hua Qi , Aza Kai , Vincent Markert , Edison Marrese-Taylor , Jianjun Zhao
show 1 more author
Lei Ma
This is my paper · ORCID
classification cs.CVcs.AIcs.MM
keywords long-formvideounderstandingmultimodallargelanguagemodelsquestionansweringculturalnarrativeintelligenceJapanesehigh-contextcommunicationbenchmarkconstructionaizuchi
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper introduces NARU, a benchmark of 1,481 multiple-choice questions over 155 Japanese long-form videos totaling 146.8 hours, designed to test two abilities jointly: tracking a narrative as it evolves over hours and interpreting culturally implicit meaning such as 'reading the air' or the subtext of a polite offer. The authors argue that existing benchmarks keep these abilities separate, and that Japanese media is a strong testbed because meaning often depends on accumulated social context rather than literal content. To build the benchmark at scale, they use a hierarchical memory-based pipeline that chunks videos, annotates events, narratives, and cultural cues, generates questions with a multimodal model, and iteratively removes text-only shortcuts using a blind-solver loop, followed by two verification rounds with 68 native Japanese speakers. Evaluations across eight model configurations show the best model reaches 76.2 percent accuracy, while open-source models stay between 29.6 and 39.8 percent, with particular weakness in subtext interpretation and long-range thematic synthesis. If NARU is valid, it gives the field a metric for whether multimodal models genuinely integrate long-range, high-context video rather than retrieving isolated facts or exploiting option artifacts.

What carries the argument

The load-bearing mechanism is a hierarchical memory-based annotation pipeline that converts hours of video into structured, temporally connected evidence without requiring one model invocation to process the whole video. Videos are cut into five-minute chunks, each processed with a recap of prior chunks to maintain entity and event continuity; chunk records are merged into a global timeline, segmented into chapter-level units, and enriched by narrative and cultural annotation modules aligned with the nine-category taxonomy. Question generation is controlled by a Solver-Critic loop: a blind solver attempts each question with the video removed, a diagnostic agent identifies the shortcut (such as tone bias or process of elimination), and a revision agent rewrites the item until the blind solver approaches chance. The taxonomy itself, grounded in Event Segmentation Theory and high-context communication theory, defines what counts as narrative intelligence versus cultural understanding and shapes what evidence the annotation modules extract.

What would settle it

Run the final 1,481 items through the same blind solver with video access removed; if accuracy substantially exceeds 25 percent, text-only shortcuts remain in the benchmark. Independently, measure inter-annotator agreement on the culturally nuanced C.2/C.3 items; low agreement would show there is no single culturally correct answer for those questions.

Watch

Extended reading notes

Core claim

NARU's central claim is that long-form video understanding should be evaluated as the joint problem of maintaining narrative state and interpreting culturally situated implicit meaning, and that in Japanese long-form video this joint competence remains far from solved. The benchmark operationalizes the claim through nine categories: four narrative dimensions (character/entity evolution, sequential/topical flow, plot/conflict progression, idea/thematic development) and five cultural dimensions (aizuchi, 'reading the air,' subtext interpretation, cultural context recognition, sentiment analysis). The paper reports that the strongest evaluated model reaches 76.2 percent multiple-choice accuracy, that all models gain more from added frames on narrative than on cultural tasks, and that removing the multiple-choice options reverses the relative difficulty of narrative versus cultural questions. The intended conclusion is that current multimodal large language models still show substantial limitations in both long-range narrative integration and culturally grounded reasoning, and that NARU provides a systematic testing ground for improving them.

Load-bearing premise

NARU's validity depends on the blind-solver loop actually eliminating text-only shortcuts and on the two-stage native-speaker verification establishing a single correct answer for culturally nuanced items, yet the paper does not report final blind-solver accuracy or inter-annotator agreement.

Editorial extensions

If this is right

  • Current multimodal AI models, even the strongest evaluated, remain far from ceiling performance on NARU: the top model reaches 76.2 percent while open-source models hover near or below 40 percent, with some dipping below the 25 percent random baseline on narrative tracking and cultural tasks.
  • Narrative errors are largely caused by missing dispersed events, since denser frame sampling consistently raises narrative accuracy across all models, whereas cultural understanding improves little with more frames and appears bottlenecked by pragmatic reasoning and domain knowledge.
  • Multiple-choice evaluation can overestimate temporal sequence understanding: N.2 (sequential/topical flow) is the easiest narrative category in multiple-choice form but becomes the weakest for seven of eight models in open-ended form, indicating that option lists scaffold chronological reconstruction.
  • The dominant narrative difficulty shifts with model capability: weaker models fail most at low-level entity continuity, while stronger models struggle most at high-level thematic development, implying different bottlenecks along the capability spectrum.
  • Scores on culturally nuanced categories such as subtext interpretation remain low even for the leading model, suggesting that cultural understanding will not be fixed simply by longer context windows or more frames.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the construction pipeline generalizes, the same hierarchical-memory plus blind-solver-debiasing recipe could produce comparable benchmarks for other high-context cultures, enabling cross-cultural comparisons of multimodal models' pragmatic reasoning rather than treating Japanese as an isolated case.
  • The frame-sweep result implies that scaling context length and frame counts mainly helps narrative integration; improving cultural understanding may require training on explicitly annotated pragmatic and sociocultural cues rather than more raw video data.
  • The format-dependent reversal on sequential-flow questions suggests that benchmarks reporting only multiple-choice accuracy may overstate models' ability to reconstruct event order; adopting open-ended or ordering-free evaluation would give a more honest picture.
  • A testable extension would use NARU's per-category items to compare models of similar size but different pretraining data, isolating whether cultural failures reflect missing knowledge, weak pragmatic inference, or insufficient temporal integration.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper introduces NARU, a benchmark of 1,481 multiple-choice questions grounded in 155 Japanese long-form videos totaling 146.8 hours, organized into four narrative subcategories (N.1–N.4) and five cultural subcategories (C.1–C.5). The benchmark is built with a hierarchical, memory-based annotation pipeline that chunks videos, produces narrative and cultural annotations, and then generates MCQs with an iterative Solver-Critic debiasing loop, followed by two native-speaker verification stages involving 68 annotators. The authors evaluate eight MLLM configurations in both MCQ and open-ended formats, reporting that Gemini-3-Flash achieves the highest MCQ accuracy at 76.2%, while open-source models range from 29.6% to 39.8%. They also present a frame-sampling sweep and an open-ended F ActScore evaluation, concluding that current models show substantial limitations in long-range narrative integration and culturally grounded reasoning.

Significance. If the benchmark's validity claims hold, NARU addresses a real gap: existing long-video benchmarks mostly separate temporal/narrative reasoning from culturally situated implicit understanding, and few operate in high-context non-English media. The construction pipeline is carefully designed, the taxonomy is grounded in Event Segmentation Theory and high-context communication research, and the two-stage verification with 68 native-speaker annotators is a concrete strength. The open-ended evaluation with a human-judge validation of the automated judge is also a positive methodological step. However, the paper's central video-grounding claim depends on debiasing and verification evidence that is currently missing, so the benchmark's validity as a measure of video-grounded, culturally nuanced understanding is not yet fully established.

major comments (4)
  1. [III-C2, Step 2-2] The Solver-Critic debiasing loop is defined, but its stopping condition is never reported. The paper does not give the final Blind Solver Agent's accuracy on the final 1,481 items, per-category leakage rates, or the number of iterations relative to the iteration budget. Because the same model (Gemini 2.5 Pro) is used as blind solver, diagnostic agent, revision agent, and question generator, convergence for that model establishes self-consistency, not the absence of text-only shortcuts for the models actually evaluated in Table II, such as Gemini-3-Flash and Qwen3-VL-8B. The manuscript itself admits at the start of Sec. IV-D that 'multiple-choice options can inadvertently leak contextual hints,' so residual leakage in the primary MCQ format is an acknowledged risk. Please report the final blind-solver accuracy and per-category leakage rates; if blind accuracy remains above chance, remove or repair the leaked items and re-report Table II, because otherwise the central claim that NARU measures video-grounded understanding is unsupported.
  2. [III-D] The two-stage human verification is a strength, but no inter-annotator agreement statistics are reported for either the 40-annotator or 28-annotator stage, and the adjudication protocol is not described. This matters because the pipeline relies on labels that were demonstrably noisy: 177 label corrections in the first stage, 436 answer corrections in the second stage, and 107 annotator-authored answers. For culturally nuanced items, the claim of a single correct answer requires evidence of consensus (e.g., pairwise agreement, Cohen's kappa or Krippendorff's alpha, reported per category) and a documented rule for resolving disagreements; without this, NARU may encode one particular cultural interpretation rather than a verified ground truth.
  3. [IV-B, Table II] All model-comparison claims are presented without confidence intervals, significance tests, or multiple-comparison corrections. Per-category sample sizes are only 143–187 items, so statements such as 'the sole category where Gemini-3-Pro yields higher accuracy' and 'open-source models show virtually no difference between narrative and cultural dimensions' are not yet supported. Please report binomial confidence intervals for overall and per-category accuracy and use paired tests (e.g., McNemar) for model and category comparisons, including for the frame-sweep results in Fig. 3 and the open-ended results in Table III.
  4. [III-C and IV-B] Using Gemini 2.5 Pro as the question generator, annotator, and blind solver while evaluating Gemini-family models creates a distributional overlap that may inflate Gemini results relative to open-source models. This is not circularity in the ground-truth definition, because the labels are human-verified, but it is a correctness risk for the cross-family comparison. A concrete test would be to report model accuracy separately on items whose stems or answers were authored or substantially rewritten by human annotators versus items left unchanged, or to run the blind-solver loop with a non-Gemini model; either analysis would show whether the reported ranking is robust to generator-family effects.
minor comments (4)
  1. [Throughout] Please proofread for typographical errors, including 'T o' appearing in the abstract/footer text and 'question r answer pairs' in Sec. I.
  2. [Fig. 1] The Japanese gloss for 'kuuki wo yomu' appears garbled in the manuscript; ensure the Japanese script renders correctly in the camera-ready version.
  3. [III-C2] Define 'draws close to natural random chance' with a concrete threshold (e.g., within one percentage point of 25%) in the main text rather than only on the project website.
  4. [IV-D] Report the per-category distribution of the 500-question diagnostic subset and the atomic-fact decomposition statistics (e.g., average number of reference facts per answer) so the F ActScore recall comparison in Table III is interpretable.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: NARU's ground-truth answers are fixed by native-speaker verification and external evidence, not by the evaluated models.

full rationale

NARU's central derivation is benchmark construction followed by model evaluation, and the chain does not reduce to its inputs. The 1,481 final answer labels are established by two stages of native-Japanese expert verification (Sec. III-D), independent of the models scored in Table II; the paper explicitly reports 436 answer-label corrections and 161 repaired items, showing human adjudication, not model self-consistency, determines the keys. The Solver-Critic debiasing loop (Sec. III-C2, Step 2-2) is a quality-control filter whose output is itself re-verified by 28 human annotators, so a failure to report final blind-solver accuracy would be an evidence gap for the video-grounding claim, not a circular reduction: the benchmark's correctness does not by construction equal the blind solver's behavior. No fitted parameter is renamed as a prediction; the taxonomy (N.1-N.4, C.1-C.5) is imported from external theories (Event Segmentation Theory, Hall's high-context communication) and prior benchmarks, not derived from NARU's own scores. Self-citations appear only as a downstream-application reference [5] and the project website [19]; neither is load-bearing for the benchmark's validity. The acknowledged residual risk that MCQ options 'can inadvertently leak contextual hints' (Sec. IV-D) is a benchmark-quality concern, externalized to an open-ended reformulation and human checks, not evidence that the claimed evaluations are equivalent to the construction procedure. Overall, the derivation is self-contained against external benchmarks and human ground truth.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The benchmark construction introduces no new physical or conceptual entities beyond the taxonomy categories themselves. The main load-bearing prerequisites are the validity of the psychological and socio-linguistic theories used for the taxonomy, the faithfulness of MLLM annotations after human review, and the representativeness of the video sample.

free parameters (3)
  • cosine similarity threshold τ
    Chosen by hand for incremental semantic diversity sampling in Sec. III-B; controls how many videos are accepted and may affect coverage.
  • chunk duration = 5 minutes
    Fixed processing unit for chunking in Step 1-1; duration is a practical choice, not data-derived.
  • frame sampling rates = 0.25 FPS (Gemini); 64-128 frames (open-source)
    Evaluation settings that vary by model context window; these choices affect measured accuracy and are not part of benchmark construction.
assumptions (4)
  • domain assumption Event Segmentation Theory (EST) provides a valid decomposition of narrative understanding into the N.1-N.4 categories.
    Invoked in Sec. III-A to ground the narrative taxonomy; if EST is not the right model for video narrative, the taxonomy may mis-specify what is measured.
  • domain assumption The high-context communication framework (Hall) and the specific Japanese concepts (aizuchi, kuuki wo yomu, tatemae/honne) are meaningful, non-artifactual dimensions of cultural understanding.
    Used in Sec. III-A to define C.1-C.5; requires that these cultural constructs have stable interpretations that can be captured in MCQ form.
  • domain assumption MLLM-generated annotations, after native-speaker verification, are faithful to the source videos.
    The entire QA generation depends on annotations from Step 1-3; human verification only samples/checks QA items, not all underlying annotations, so annotation errors could propagate.
  • domain assumption The 155 selected videos are representative of context-rich Japanese long-form media.
    Selection relied on manual screening and embedding-based diversity sampling (Sec. III-B); any bias in selection limits generalization of the benchmark conclusions.

how reviews work

0 comments
Cite this review

Pith. "Pith review of NARU: A Benchmark for NARrative Evolution and Cultural Nuance Understanding in Japanese Extreme Long Video." pith.science (2026). https://pith.science/paper/YCSO3PVF

@misc{pith2026260813210,
  author       = {Pith},
  title        = {Pith review of: NARU: A Benchmark for NARrative Evolution and Cultural Nuance Understanding in Japanese Extreme Long Video},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/YCSO3PVF}},
  note         = {Machine review of arXiv:2608.13210}
}
read the original abstract

Long-form video understanding encompasses tasks that go beyond retrieving isolated events, including tracking an evolving narrative and interpreting social meaning that may remain implicit. However, existing benchmarks rarely evaluate these capabilities jointly, particularly in high-context, non-English media. To address this gap, we introduce NARU, a benchmark designed to evaluate Narrative evolution and Reasoning on cultural Understanding in Japanese long-form video. NARU consists of 1,481 questions grounded in 155 videos totaling 146.8 hours, spanning four narrative and five cultural dimensions. To construct the benchmark at this scale, we propose a hierarchical memory-based annotation pipeline that transforms raw video into structured event, narrative, and cultural annotations, then generates questions via task-oriented synthesis and iterative shortcut removal. The construction process includes two native-speaker verification stages involving 68 annotators. Evaluations across eight model configurations reveal substantial limitations in both long-range narrative integration and culturally grounded reasoning. By exposing these persistent gaps, NARU offers a systematic testing ground for developing MLLMs capable of reliably interpreting long-form, high-context video.

Figures

Figures reproduced from arXiv: 2608.13210 by the authors.

Figure 1
Figure 1. Example scenarios from NARU. for context-rich video understanding. Japan supports a substantial video-content ecosystem, with its domestic market projected to reach approximately 630 billion yen in fiscal year 2025 [18]. This scale motivates evaluating whether MLLMs can understand Japanese media. More importantly, Japanese communication is characterized by a high-context cultural style, in which meaning is often con… view at source ↗
Figure 2
Figure 2. Workflow overview of NARU. Video collection and filtering (Sec. III-B) select 155 long videos from the candidate set. A MLLM-centric pipeline (Sec. III-C1) is used to produce taxonomy-aligned evidence for narrative intelligence and cultural understanding annotation. Multiple-choice questions (Sec. III-C2) are subsequently generated and refined through a multi-agent pipeline. Verification (Sec. III-D) by 68 native Ja… view at source ↗
Figure 4
Figure 4. Changes in narrative and cultural accuracy between [PITH_FULL_IMAGE:figures/full_fig_p008_4.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

50 extracted references · 43 canonical work pages

  1. [1]

    Internvl3. 5: Advancing open-source multimodal models in versatility, reasoning, and efficiency,

    W. Wang, Z. Gao, L. Gu, H. Pu, L. Cui, X. Wei, Z. Liu, L. Jing, S. Ye, J. Shao et al. , “Internvl3. 5: Advancing open-source multimodal models in versatility, reasoning, and efficiency,” arXiv preprint arXiv:2508.18265 , 2025

    arXiv 2025

  2. [2]

    Qwen2. 5-vl technical report,

    S. Bai, K. Chen, X. Liu, J. Wang, W. Ge, S. Song, K. Dang, P. Wang, S. Wang, J. Tanget al., “Qwen2. 5-vl technical report,” arXiv preprint arXiv:2502.13923 , 2025. JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2026 10

    arXiv 2025

  3. [3]

    Videollama 3: Frontier multi- modal foundation models for image and video understanding,

    B. Zhang, K. Li, Z. Cheng, Z. Hu, Y. Yuan, G. Chen, S. Leng, Y. Jiang, H. Zhang, X. Li et al. , “Videollama 3: Frontier multi- modal foundation models for image and video understanding,” arXiv preprint arXiv:2501.13106 , 2025

    arXiv 2025

  4. [4]

    Video understanding with large language models: A survey,

    Y. Tang, J. Bi, S. Xu, L. Song, S. Liang, T. Wang, D. Zhang, J. An, J. Lin, R. Zhu et al. , “Video understanding with large language models: A survey,” IEEE Transactions on Circuits and Systems for Video Technology , 2025

    work page 2025

  5. [5]

    Open-domain video commentary generation,

    E. Marrese-Taylor, Y. Hamazono, T. Ishigaki, G. Topić, Y. Miyao, I. Kobayashi, and H. Takamura, “Open-domain video commentary generation,” in Proc. EMNLP . Abu Dhabi, United Arab Emirates: Association for Computational Linguistics, Dec. 2022, pp. 7326–7339. [Online]. A vailable: https://aclanthology.org/2022.emnlp-main.495/

    work page 2022

  6. [6]

    From seconds to hours: Reviewing multimodal large language models on comprehensive long video understanding,

    H. Zou, T. Luo, G. Xie, F. Lv, G. Wang, J. Chen, Z. Wang, H. Zhang, H. Zhang et al. , “From seconds to hours: Reviewing multimodal large language models on comprehensive long video understanding,” arXiv preprint arXiv:2409.18938 , 2024

    arXiv 2024

  7. [7]

    V2xum-llm: Cross-modal video summarization with temporal prompt instruction tuning,

    H. Hua, Y. Tang, C. Xu, and J. Luo, “V2xum-llm: Cross-modal video summarization with temporal prompt instruction tuning,” in Proc. AAAI, vol. 39, no. 4, 2025, pp. 3599–3607

    work page 2025

  8. [8]

    Moviechat: From dense token to sparse memory for long video understanding,

    E. Song, W. Chai, G. Wang, Y. Zhang, H. Zhou, F. Wu, H. Chi, X. Guo, T. Ye, Y. Zhang et al., “Moviechat: From dense token to sparse memory for long video understanding,” in Proc. CVPR, 2024, pp. 18 221–18 232

    work page 2024

Show all 50 references
  1. [9]

    Hermes: temporal-coherent long-form understand- ing with episodes and semantics,

    G. J. Faure, J.-F. Yeh, M.-H. Chen, H.-T. Su, S.-H. Lai, and W. H. Hsu, “Hermes: temporal-coherent long-form understand- ing with episodes and semantics,” in Proc. ICCV , 2025, pp. 22 911–22 921

    2025

  2. [10]

    Toward socially aware vision- language models: Evaluating cultural competence through mul- timodal story generation,

    A. Mukherjee and S. Ghosh, “Toward socially aware vision- language models: Evaluating cultural competence through mul- timodal story generation,” in Proc. ICCV, 2025, pp. 1491–1501

    2025

  3. [11]

    Memearena: Automating context-aware unbiased evaluation of harmfulness understanding for multimodal large language models,

    Z. Chen, H. Lin, K. Li, Z. Luo, Y. Deng, and J. Ma, “Memearena: Automating context-aware unbiased evaluation of harmfulness understanding for multimodal large language models,” in Proc. EMNLP, 2025, pp. 17 648–17 670

    2025

  4. [12]

    Multivent 2.0: A massive multilingual benchmark for event- centric video retrieval,

    R. Kriz, K. Sanders, D. Etter, K. Murray, C. Carpenter, H. Rec- knor, J. Guallar-Blasco, A. Martin, E. Yang, and B. Van Durme, “Multivent 2.0: A massive multilingual benchmark for event- centric video retrieval,” in Proc. CVPR, 2025, pp. 24 149–24 158

    2025

  5. [13]

    Longvideobench: A bench- mark for long-context interleaved video-language understand- ing,

    H. Wu, D. Li, B. Chen, and J. Li, “Longvideobench: A bench- mark for long-context interleaved video-language understand- ing,” Advances in Neural Information Processing Systems , vol. 37, pp. 28 828–28 857, 2024

    2024

  6. [14]

    Lvbench: An extreme long video understanding benchmark,

    W. Wang, Z. He, W. Hong, Y. Cheng, X. Zhang, J. Qi, M. Ding, X. Gu, S. Huang, B. Xu et al., “Lvbench: An extreme long video understanding benchmark,” in Proc. ICCV , 2025, pp. 22 958– 22 967

    2025

  7. [15]

    Vrbench: A benchmark for multi-step reasoning in long narrative videos,

    J. Yu, Y. Wu, M. Chu, Z. Ren, Z. Huang, P. Chu, R. Zhang, Y. He, Q. Li, S. Li et al., “Vrbench: A benchmark for multi-step reasoning in long narrative videos,” in Proc. ICCV , 2025, pp. 21 655–21 666

    2025

  8. [16]

    Storyvideoqa: Scaling deep video un- derstanding with a large-scale, multi-genre and auto-generated dataset,

    Z. Wu, Z. Liu, A. Chen, J. Zhang, R. Li, H. Ge, Z. Wang, C. Xiao, and C. Liang, “Storyvideoqa: Scaling deep video un- derstanding with a large-scale, multi-genre and auto-generated dataset,” International Journal of Computer Vision , vol. 134, no. 6, p. 308, 2026

    2026

  9. [17]

    A culturally-diverse multilin- gual multimodal video benchmark & model,

    B. S. Shafique, A. Vayani et al. , “A culturally-diverse multilin- gual multimodal video benchmark & model,” in Proc. EMNLP, 2025, pp. 20 009–20 033

    2025

  10. [18]

    Video content business market for fy2025,

    Yano Research Institute, “Video content business market for fy2025,” Sep. 2025. [Online]. A vailable: https://www. yanoresearch.com/en/press-release/show/press_id/3919

    2025

  11. [19]

    Website of this Paper, https://ma-labo.github.io/naru/

  12. [20]

    Video question answering via gradually refined attention over appearance and motion,

    D. Xu, Z. Zhao, J. Xiao, F. Wu, H. Zhang, X. He, and Y. Zhuang, “Video question answering via gradually refined attention over appearance and motion,” in ACM Multimedia, 2017

    2017

  13. [21]

    Next-qa: Next phase of question-answering to explaining temporal actions,

    J. Xiao, X. Shang, A. Yao, and T.-S. Chua, “Next-qa: Next phase of question-answering to explaining temporal actions,” in Proc. CVPR, 2021, pp. 9777–9786

    2021

  14. [22]

    Video-chatgpt: Towards detailed video understanding via large vision and lan- guage models,

    M. Maaz, H. Rasheed, S. Khan, and F. Khan, “Video-chatgpt: Towards detailed video understanding via large vision and lan- guage models,” in Proc. ACL (Volume 1: Long Papers) , 2024, pp. 12 585–12 602

    2024

  15. [23]

    Mvbench: A comprehensive multi- modal video understanding benchmark,

    K. Li, Y. Wang, Y. He et al., “Mvbench: A comprehensive multi- modal video understanding benchmark,” in Proc. CVPR, 2024, pp. 22 195–22 206

    2024

  16. [24]

    Mlvu: Benchmarking multi-task long video understanding,

    J. Zhou, Y. Shu et al. , “Mlvu: Benchmarking multi-task long video understanding,” in Proc. CVPR, 2025, pp. 13 691–13 701

    2025

  17. [25]

    Cinepile: A long video question answering dataset and benchmark,

    R. Rawal, K. Saifullah, M. Farré, R. Basri, D. Jacobs, G. Somepalli, and T. Goldstein, “Cinepile: A long video question answering dataset and benchmark,” arXiv preprint arXiv:2405.08813, 2024

    2024 arXiv

  18. [26]

    Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal llms in video analysis,

    C. Fu, Y. Dai, Y. Luo, L. Li, S. Ren et al. , “Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal llms in video analysis,” in Proc. CVPR, 2025, pp. 24 108–24 118

    2025

  19. [27]

    Scalelong: A multi-timescale benchmark for long video understanding,

    D. Ma, H. Yuan et al., “Scalelong: A multi-timescale benchmark for long video understanding,” in The Fourteenth International Conference on Learning Representations , 2026. [Online]. A vailable: https://openreview.net/forum?id=95sD6KKq51

    2026

  20. [28]

    Dramaqa: Character-centered video story understanding with hierarchical qa,

    S. Choi, K.-W. On, Y.-J. Heo, A. Seo et al. , “Dramaqa: Character-centered video story understanding with hierarchical qa,” in Proc. AAAI, vol. 35, no. 2, 2021, pp. 1166–1174

    2021

  21. [29]

    Scvbench: A benchmark with multi-turn dialogues for story-centric video understanding,

    S. You, B. Yuan, and B.-K. Bao, “Scvbench: A benchmark with multi-turn dialogues for story-centric video understanding,” in Proceedings of the Thirty-Fourth International Joint Conference on Artificial Intelligence , 2025, pp. 2287–2295

    2025

  22. [30]

    Narra- tivetrack: Evaluating video language models beyond the frame,

    H. Ha, J. Ge, B. Feng, K. Ma, and G. Chakraborty, “Narra- tivetrack: Evaluating video language models beyond the frame,” arXiv preprint arXiv:2601.01095 , 2026

    2026 arXiv

  23. [31]

    Narrative aligned long form video question answering,

    R. Jain, K. Doshi, B. Uzkent, and G. Kessler, “Narrative aligned long form video question answering,” in Proc. CVPR, 2026, pp. 8765–8774

    2026

  24. [32]

    Social-iq: A question answering benchmark for artificial social intelligence,

    A. Zadeh, M. Chan, P. P. Liang, E. Tong, and L.-P. Morency, “Social-iq: A question answering benchmark for artificial social intelligence,” in Proc. CVPR, 2019, pp. 8807–8817

    2019

  25. [33]

    Desiq: Towards an unbiased, challenging benchmark for social intelligence understanding,

    X.-Y. Guo, Y.-F. Li, and R. Haf, “Desiq: Towards an unbiased, challenging benchmark for social intelligence understanding,” in Proc. EMNLP, 2023, pp. 3169–3180

    2023

  26. [34]

    Benchmarking vision language models for cultural understanding,

    S. Nayak, K. Jain, R. Awal, S. Reddy, S. Van Steenkiste, L. A. Hendricks, K. Stańczak, and A. Agrawal, “Benchmarking vision language models for cultural understanding,” in Proc. EMNLP, 2024, pp. 5769–5790

    2024

  27. [35]

    Event segmentation,

    J. M. Zacks and K. M. Swallow, “Event segmentation,” Current directions in psychological science , vol. 16, no. 2, pp. 80–84, 2007

    2007

  28. [36]

    E. T. Hall, Beyond culture. Anchor, 1976

    1976

  29. [37]

    Nodding, aizuchi, and final particles in japanese conversation: How conversation reflects the ideology of communication and social relationships,

    S. Kita and S. Ide, “Nodding, aizuchi, and final particles in japanese conversation: How conversation reflects the ideology of communication and social relationships,” Journal of Pragmat- ics, vol. 39, no. 7, pp. 1242–1254, 2007

    2007

  30. [38]

    On back-channel behavior in japanese and english casual conversation,

    S. K. Maynard, “On back-channel behavior in japanese and english casual conversation,” 1986

    1986

  31. [39]

    Ego4d: Around the world in 3,000 hours of egocentric video,

    K. Grauman, A. Westbury et al. , “Ego4d: Around the world in 3,000 hours of egocentric video,” in Proc. CVPR, 2022, pp. 18 995–19 012

    2022

  32. [40]

    Egolife: Towards egocentric life assistant,

    J. Yang, S. Liu et al., “Egolife: Towards egocentric life assistant,” in Proc. CVPR. IEEE, 2025, pp. 28 885–28 900

    2025

  33. [41]

    Segmented discourse representa- tion theory: Dynamic semantics with discourse structure,

    A. Lascarides and N. Asher, “Segmented discourse representa- tion theory: Dynamic semantics with discourse structure,” in Computing meaning. Springer, 2007, pp. 87–124

    2007

  34. [42]

    Mirage: The illusion of visual understanding,

    M. Asadi, J. W. O’Sullivan, F. Cao, T. Nedaee, K. Rajabalifardi, F.-F. Li, E. Adeli, and E. Ashley, “Mirage: The illusion of visual understanding,” arXiv preprint arXiv:2603.21687 , 2026

    2026

  35. [43]

    Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities,

    Gemini Team, “Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities,” Google DeepMind, Tech. Rep., 2025. [Online]. A vailable: https://storage.googleapis. com/deepmind-media/gemini/gemini_v2_5_report.pdf

    2025

  36. [44]

    Gemini 3 flash model card,

    Google DeepMind, “Gemini 3 flash model card,”

  37. [45]

    Gemini 3 pro model card,

    ——, “Gemini 3 pro model card,” 2025. [Online]. A vailable: https://deepmind.google/models/model-cards/gemini-3-pro/

    2025

  38. [46]

    Qwen3.5,

    Qwen Team, “Qwen3.5,” Feb. 2026. [Online]. A vailable: https://qwen.ai/blog?id=qwen3.5

    2026

  39. [47]

    Qwen3-vl technical report,

    S. Bai, Y. Cai, R. Chen et al. , “Qwen3-vl technical report,” arXiv preprint arXiv:2511.21631 , 2025

    2025 arXiv

  40. [48]

    Minicpm-o-2.6 model card,

    OpenBMB, “Minicpm-o-2.6 model card,” https://huggingface. co/openbmb/MiniCPM-o-2_6, 2025, accessed: 2026-07-20

    2025

  41. [49]

    Factscore: Fine- grained atomic evaluation of factual precision in long form text generation,

    S. Min, K. Krishna, X. Lyu, M. Lewis, W.-t. Yih, P. W. Koh, M. Iyyer, L. Zettlemoyer, and H. Hajishirzi, “Factscore: Fine- grained atomic evaluation of factual precision in long form text generation,” in Proc. EMNLP, 2023

    2023

  42. [2025]

    A vailable: https://deepmind.google/models/ model-cards/gemini-3-flash/

    [Online]. A vailable: https://deepmind.google/models/ model-cards/gemini-3-flash/

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.