Pith. sign in

REVIEW 3 major objections 6 minor 1 cited by

The paper introduces NarrativeTrack, a benchmark claiming that multimodal models fail to maintain coherent entity identity across video scene changes, with GPT-4o at 72.27% and open models much lower.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-04 06:25 UTC pith:KYE7GBQY

load-bearing objection Useful diagnostic benchmark for entity tracking, but the Gemini-generated attribute ground truth is unvalidated and a prompt typo in §7.3 makes scene-change labels suspect. the 3 major comments →

arxiv 2601.01095 v4 pith:KYE7GBQY submitted 2026-01-03 cs.CV cs.LG

NarrativeTrack: Evaluating Entity-Centric Reasoning for Narrative Understanding

Hyeonjeong Ha , Jinjin Ge , Bo Feng , Kaixin Ma , Gargi Chakraborty This is my paper
classification cs.CV cs.LG
keywords video understandingmultimodal large language modelsentity-centric reasoningnarrative comprehensionbenchmarktemporal reasoningentity trackingcompositional reasoning progression
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper tries to establish that genuine narrative understanding in multimodal large language models (MLLMs) requires tracking entities—the people doing things—across time, not just recognizing static frames. To test this, it builds NarrativeTrack, a benchmark that decomposes videos into per-entity trajectories (bounding box, action, outfit, scene per timestamp) and questions models with a Compositional Reasoning Progression: entity existence, entity changes, and entity ambiguity. It evaluates 13 models and finds that even the strongest (GPT-4o at 72.27%) struggles with long-range identity, while open-source models drop sharply, revealing a trade-off between perceptual grounding and temporal coherence. The paper also shows that more frames do not help beyond 20, and models have a strong forward-only temporal bias. If correct, the benchmark provides a diagnostic framework for where entity-centric reasoning fails and motivates architectural changes like bidirectional temporal modeling.

Core claim

On the paper's own terms, the central discovery is that current MLLMs cannot maintain coherent entity representations across visual transitions and temporal gaps. NarrativeTrack operationalizes narrative understanding as entity-centric reasoning: models must answer questions about whether an entity persists, how its action, outfit, and scene change over time, and whether visually similar entities are distinct. Across all tested models, performance is far below ceiling, and the failure patterns are systematic: general-purpose open models ground perception well but lose temporal continuity; video-specialized open models capture continuity but hallucinate visual details; and even the proprietar

What carries the argument

The entity representation τ_ei: a per-timestamp tuple (timestamp, bounding box, action, scene, outfit) that binds identity to state across time. The Compositional Reasoning Progression (CRP) then turns this representation into questions of increasing difficulty—existence, changes, ambiguity—isolating whether a model fails at temporal persistence, grounded state transitions, or fine-grained disambiguation. The automated pipeline (ensemble detection, ReID clustering, face recognition, and Gemini-2.5-Pro attribute annotation) is what makes scalable construction possible; the QA templates are filled deterministically from the same metadata that defines ground truth.

Load-bearing premise

The ground-truth attributes (actions, outfits, scenes) that fill every question and answer are produced by Gemini-2.5-Pro's per-clip reading and were never checked against human-labeled attribute annotations; if that model mislabels what a person is doing or wearing, the benchmark's correct answers are wrong and the scores measure the annotator rather than the tested models.

What would settle it

Randomly sample about 100 video segments from NarrativeTrack, have multiple human annotators label the target entity's action, outfit, and scene per timestamp, and measure agreement with the Gemini-2.5-Pro-produced metadata. If human–machine agreement on attributes is markedly below the 96% human accuracy the paper reports for the final QA pairs, the ground-truth answers are not independently verified and the benchmark's model rankings could change under corrected labels.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • NarrativeTrack can be used to pinpoint which level of entity-centric reasoning a model fails at—existence, change, or ambiguity—giving a diagnostic breakdown rather than a single score.
  • Benchmark scores should not be treated as a saturated capability: even the strongest model leaves roughly 28% of questions unanswered correctly, so entity-centric narrative understanding is an open problem.
  • Adding more input frames does not improve entity tracking beyond about 20 frames, so progress will need better temporal-coherence mechanisms, not denser sampling.
  • The large forward/backward asymmetry implies that models trained on left-to-right sequences inherit a directional bias; fixing it will require bidirectional temporal modeling or contrastive reversal objectives.
  • Because removing visual input drops GPT-4o to near chance, the benchmark's questions genuinely require visual grounding, unlike many prior video benchmarks.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Since the attribute ground truth (action/outfit/scene) comes from an automated model without human validation, a portion of the benchmark's 'correct' answers may reflect that model's perception; human-labeling a random subset of segments would tell how much of the reported accuracies are about the tested models versus the annotator model.
  • The CRP structure could be reused as a training curriculum: ordering questions by existence → changes → ambiguity gives a natural difficulty progression for fine-tuning or reinforcement learning.
  • The entity-ambiguity dimension, where visually similar people must be disambiguated, is close to person re-identification; one could test whether explicit ReID features improve MLLM narrative reasoning, which the paper leaves to future work.
  • The forward/backward asymmetry parallels the 'reversal curse' in text LLMs; if the same bias appears in other video benchmarks, it suggests a general architectural limitation rather than a quirk of this dataset.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper introduces NARRATIVETRACK, a benchmark for evaluating entity-centric narrative understanding in multimodal large language models. A fully automated pipeline detects human entities, links them into trajectories via re-identification and human-consensus verification, and augments each trajectory with action, outfit, and scene attributes using Gemini-2.5-Pro. These structured representations are then used to instantiate a Compositional Reasoning Progression (CRP) spanning entity existence, entity changes, and entity ambiguity, with QA pairs generated programmatically from templates. The authors evaluate 12 open-source and 1 proprietary MLLM, reporting that GPT-4o achieves 72.27% accuracy while open-source models lag, and identify a trade-off between perceptual grounding and temporal coherence. Ablations show performance drops under text-only input and reversed frames, supporting the claim that the benchmark requires temporal grounding.

Significance. If the attribute layer is reliable, NARRATIVETRACK would be a valuable diagnostic resource: it is the first video benchmark to explicitly decompose narrative understanding into entity-centric CRP dimensions, it is scalable due to full automation, and the included ablations (text-only drop of 30.52%, reversed-frame collapse on ordering tasks) provide convincing evidence that the benchmark is not solvable by static cues. The detection/tracking validation on AVA (recall 0.848; 96.08% human agreement on identity filtering) and the QA review (Fleiss' κ = 0.767, 96% human accuracy after cleaning) demonstrate care in verifying the benchmark's basic integrity. The central weakness is that the action/outfit/scene attributes that ground every question and answer are produced solely by Gemini-2.5-Pro and have not been independently validated against human-labeled attribute ground truth; this is load-bearing for the paper's main claims.

major comments (3)
  1. [§3.1, Eq. (1), §3.3] The ground-truth attributes a_ij, o_ij, s_ij in Eq. (1) are produced by Gemini-2.5-Pro in the Contextual Recognition stage and are never validated against human-labeled attribute annotations. The AVA validation in §3.1 measures only detection and tracking quality, not the correctness of action/outfit/scene labels. The QA validity review in §3.3 checks that questions and answers are 'grounded to the target entity and its states', but it does not independently establish that the stated attribute categories are factually correct; the authors' manual refinement is not a blind, independent annotation. Since every template question and ground-truth answer is instantiated from these attributes, any systematic Gemini error propagates directly into the benchmark's 'correct' answers. Please report a human-annotator agreement study on the attribute labels themselves (e.g., per-attribute precision o
  2. [§7.3, Second Step, item 3] The contextual-recognition prompt contains an internal inconsistency: item 3 instructs the model to 'Determine whether the scene changes significantly ... only based on the provided outfit change description' — a copy-paste from item 2. If the pipeline was run with the prompt as printed, the scene-transition significance decisions that gate the scene-change dimension and the selection of videos for that dimension may be based on outfit descriptions rather than scene descriptions, corrupting the corresponding ground truth. Please clarify whether the deployed prompt differed from the appendix; if not, correct the prompt and re-generate or re-validate all affected scene-change QA pairs. This typo also underscores the need for independent attribute validation.
  3. [§3.1, §3.3] The benchmark composition is partially defined by the same model that generates the labels: 'main characters' are the top-4 ReID clusters, but the selection of videos with 'significant' attribute changes and visually similar entities is made by Gemini-2.5-Pro on the basis of its predicted attributes. Consequently, the distribution of CRP dimensions and question types inherits Gemini's perceptual biases, and the claimed progression of complexity (existence → changes → ambiguity) is not independently established. A concrete test would be to have human annotators classify a random sample of video/entity instances into the CRP dimensions and compare against the automated selection; reporting the agreement would calibrate how much of the model ranking is attributable to Gemini-dependent sampling rather than to the target narrative skill.
minor comments (6)
  1. [§3.1] The pipeline 'compromises three stages' should read 'comprises three stages'.
  2. [Table 2] Formatting issues: 'Boldand' is missing a space; the header 'OGP-' is undefined in the table caption; some row entries appear to have inconsistent spacing.
  3. [Fig. 1] Option D reads '22MPH' with missing spacing; please fix the figure text.
  4. [Table 10] The model name 'InterVL3-8B' is inconsistent with 'InternVL3-8B' used elsewhere.
  5. [Abstract / §4.2] The abstract reports Qwen2.5-VL-32B at 56.90%, while Table 2 reports 56.96%; please reconcile the numbers.
  6. [Throughout] Inconsistent spelling 'GPT4-o' versus 'GPT-4o'; please standardize.

Circularity Check

0 steps flagged

No significant circularity; the benchmark's QA labels are human-refined and the main MLLM-failure claims are empirical, not fitted. Minor self-citation is not load-bearing.

full rationale

No circular step in the paper's derivation chain. The ground-truth answers are generated by instantiating question templates over entity metadata produced by Gemini-2.5-Pro in §3.1 and §3.3, but the final 1,006 QA pairs received a triple-annotator validity review and a manual verification/refinement pass, and human accuracy on the cleaned benchmark is reported at 96%. The benchmark is therefore not closed under the annotation model alone; the accepted answer key is human-grounded at the QA level. Claims that MLLMs fail at entity tracking are empirical measurements of held-out models against that key, and no evaluated model (Gemini is not among the evaluated baselines) supplies the ground-truth attributes. Detection and tracking are externally validated on AVA (ensemble detection recall 0.848; 96.08% agreement with human track labels). The only self-citation is reference [13] (Feng et al., a co-author), used in §1 and §2.2 to support the premise that many existing video benchmarks can be solved without genuine temporal reasoning; this is not load-bearing because the paper independently demonstrates temporal grounding through its text-only and frame-reversal ablations in §9.4. Two validity caveats, which are correctness risks rather than circularity, are worth flagging: (a) the Gemini-produced attribute layer (actions, outfits, scenes) is not separately validated against human attribute labels, so systematic Gemini mislabeling would propagate into answers; and (b) Appendix §7.3, Second Step item 3 contains an apparent copy-paste error, instructing the model to 'Determine whether the scene changes significantly involving the target entity only based on the provided outfit change description.' If executed as printed, scene-transition decisions would be conditioned on outfit changes rather than scene changes. Neither caveat makes a claimed derivation reduce to its own inputs by construction, so circularity is minimal.

Axiom & Free-Parameter Ledger

4 free parameters · 5 axioms · 0 invented entities

The benchmark's central measurement depends on hand-set hyperparameters (top-4, confidence 0.3, IoU 0.5), on Gemini-2.5-Pro as the de facto annotator for attributes and video selection, and on an asserted but unvalidated complexity ordering. These are the 'unpaid' assumptions; the paper's own human checking covers QA consistency, not the underlying attribute ground truth.

free parameters (4)
  • top-4 main characters = 4
    Main characters are the four largest ReID embedding clusters; the number 4 is chosen by hand (§3.1).
  • detection confidence threshold = 0.3
    Detectron2 and Owlv2 detections kept above confidence 0.3 (§7.1).
  • IoU matching threshold = 0.5
    Boxes from the two detectors are merged when IoU ≥ 0.5; also used for trajectory matching (§3.1).
  • Gemini significance criteria = prompt-defined judgment
    Gemini-2.5-Pro decides whether action/outfit/scene transitions are 'significant' and whether similar-looking entities exist, with no calibrated threshold (§7.3 second step); these decisions select videos for the ambiguity and change dimensions (§3.1).
axioms (5)
  • domain assumption Recurrence correlates with narrative centrality
    The top-4 largest identity clusters are treated as main characters (§3.1 'top four selected as main characters under the assumption that recurrent presence correlates with narrative centrality').
  • domain assumption Restricting evaluation to human entities preserves narrative understanding
    Non-human entities are excluded to avoid ambiguity; the paper asserts humans are the primary narrative agents (§3).
  • ad hoc to paper Gemini-2.5-Pro's attribute predictions are accurate enough to serve as ground truth after filtering
    Actions, outfits, scenes are inferred by Gemini-2.5-Pro (§3.1) and become the fill-ins for QA templates; no direct human validation of these attributes is reported (§3.3).
  • domain assumption Off-the-shelf detection/ReID/face recognition maintain identity across long, unconstrained videos
    Tracking quality is validated only on AVA clips (short movie segments); long-form Video-MME/LVBench sources are not separately validated (§3.1).
  • ad hoc to paper The CRP ordering (existence → changes → ambiguity) reflects increasing narrative complexity
    The progression is asserted from cognitive/narratological citations (§3.2), not derived or calibrated.

pith-pipeline@v1.3.0-alltime-deepseek · 58 in / 14580 out tokens · 144681 ms · 2026-08-04T06:25:38.248880+00:00 · methodology

0 comments
read the original abstract

Multimodal large language models (MLLMs) have achieved impressive progress in vision-language reasoning, yet their ability to understand temporally unfolding narratives in videos remains underexplored. True narrative understanding requires grounding who is doing what, when, and where, maintaining coherent entity representations across dynamic visual and temporal contexts. We introduce NarrativeTrack, the first benchmark to evaluate narrative understanding in MLLMs through fine-grained entity-centric reasoning. Unlike existing benchmarks limited to short clips or coarse scene-level semantics, we decompose videos into constituent entities and examine their continuity via a Compositional Reasoning Progression (CRP), a structured evaluation framework that progressively increases narrative complexity across three dimensions: entity existence, entity changes, and entity ambiguity. CRP challenges models to advance from temporal persistence to contextual evolution and fine-grained perceptual reasoning. A fully automated entity-centric pipeline enables scalable extraction of temporally grounded entity representations, providing the foundation for CRP. Evaluations of state-of-the-art MLLMs reveal that models fail to robustly track entities across visual transitions and temporal dynamics, often hallucinating identity under context shifts. Open-source general-purpose MLLMs exhibit strong perceptual grounding but weak temporal coherence, while video-specific MLLMs capture temporal context yet hallucinate entities' contexts. These findings uncover a fundamental trade-off between perceptual grounding and temporal reasoning, indicating that narrative understanding emerges only from their integration. NarrativeTrack provides the first systematic framework to diagnose and advance temporally grounded narrative comprehension in MLLMs.

Figures

Figures reproduced from arXiv: 2601.01095 by Bo Feng, Gargi Chakraborty, Hyeonjeong Ha, Jinjin Ge, Kaixin Ma.

Figure 1
Figure 1. Figure 1: Examples of existing benchmark and NARRATIVETRACK. While existing benchmarks can often be answered from a single frame, ours requires reasoning by tracking entities over time. viewpoint changes, or scene transitions. Entities thus serve as the basic units of narrative structure, organizing events into meaningful temporal and causal relationships. Despite remarkable advances in multimodal large language mod… view at source ↗
Figure 2
Figure 2. Figure 2: Overview of Automated Entity-Centric Pipeline. [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Overview of NARRATIVETRACK. (a) Our benchmark is grounded in Compositional Reasoning Progression that introduces three levels of increasing complexity in entity-centric reasoning: entity existence, entity changes, and entity ambiguity. (b) The benchmark covers three question types: binary, multiple-choice (MC), and ordering, with diverse temporal scales from short to long (average of 55.3 seconds). context… view at source ↗
Figure 4
Figure 4. Figure 4: QA Generation & Qualitative Examples. QA pairs are constructed by instantiating CRP-aligned reasoning templates using extracted entity representations. Distractors include real ones from other entities in the same video and synthetic ones from different videos. beyond recognition. To probe temporal directional bias (§4.2), we define three reasoning patterns: (1) forward (tracking start → end), (2) backward… view at source ↗
Figure 5
Figure 5. Figure 5: Temporal Directional Bias of MLLMs. MLLMs encode temporal relations in a forward￾only manner and fail to generalize to reversed or bidirectional temporal contexts. (a) OGP-MLLMs. (b) OVS-MLLMs. (c) Proprietary MLLMs [PITH_FULL_IMAGE:figures/full_fig_p012_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Performance Across Track Types. OGP-MLLMs perform best on disappear cases, relying on static visual cues, whereas OVS- and proprietary MLLMs excel on reappear cases, reflecting stronger temporal integration but a higher tendency to hallucinate visual details. reasoning. The performance gap between forward and backward reasoning reaches 20.65%, 9.96%, and 17.55% for OGP-, OVS-, and proprietary MLLMs, respec… view at source ↗
Figure 8
Figure 8. Figure 8: Performance Across Frame Den￾sities. Scaling tem￾poral coverage does not guarantee perfor￾mance gain. Q. [Entity Existence] When does the person who appears at the beginning, speaking indoors while wearing a red hat in front of a patterned wall, appear? (a) At the start only, then disappears (b) At the start and the end, with an absence in between (c) Only at the end (d) None of the above Ground-Truth Answ… view at source ↗
Figure 9
Figure 9. Figure 9: Examples of Model Failure in NARRATIVETRACK.Video sources are extracted from AVA [16] and the output is generated from Video-LLaMA2-7B [8]. contrast, OVS-MLLMs achieve higher accuracy on reappear cases, reflecting stronger temporal integration but also a tendency to overpredict reappearances, often hallucinating entity reappearances or misattributing visual attributes. This indicates a trade-off: while OVS… view at source ↗
Figure 10
Figure 10. Figure 10: Examples of existing benchmarks in order of LongVideoBench [43], LVBench [40], NeXT-QA [45], PerceptionTest [32]. Existing benchmarks focused on semantic understanding that can be answered even from a single frame or scene, neglecting true temporal reasoning. 6.1 Video Sources of NARRATIVETRACK NARRATIVETRACK leverages three widely adapted video sources: AVA [16], Video￾MME [2], and LVBench [40]. The test… view at source ↗
Figure 11
Figure 11. Figure 11: Temporal Directional Bias in Agnostic Reasoning. 9.3 Ablation on Frame Density We further investigate the effect of frame density on reasoning performance across different dimensions in NARRATIVETRACK. For all models and reasoning types, per￾formance tends to generally increase as the number of input frames grows, peaking around 20 frames, but drops sharply beyond this threshold ( [PITH_FULL_IMAGE:figure… view at source ↗
Figure 12
Figure 12. Figure 12: Ablation study on frame density. These results demonstrate that NARRATIVETRACK effectively enforces temporally grounded reasoning and serves as a rigorous test of a model’s ability to reason over time with perceptual grounding. 9.5 Likelihood-based Evaluation While we minimize generation bias by balancing answer distributions, we introduce a likelihood-based evaluation (LE) inspired by StrictVLE [34]. LE … view at source ↗

Review history (2 revisions) →

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Towards High-Level Semantic Intelligence

    cs.AI 2026-07 conditional novelty 4.0

    A survey proposing that AI's next stage should be understood as High-Level Semantic Intelligence: mastering humor, sarcasm, metaphor, empathy, persuasion, and narrative across modalities.

Reference graph

Works this paper leans on

69 extracted references · 26 linked inside Pith · cited by 1 Pith paper

  1. [1]

    arXiv preprint arXiv:2303.08774 (2023)

    Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F.L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al.: Gpt-4 technical report. arXiv preprint arXiv:2303.08774 (2023)

    Pith/arXiv arXiv 2023

  2. [2]

    In: CVPR (2025)

    et al, F.: Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal llms in video analysis. In: CVPR (2025)

    2025

  3. [3]

    arXiv preprint arXiv:2408.03326 (2024)

    et al, L.: Llava-onevision: Easy visual task transfer. arXiv preprint arXiv:2408.03326 (2024)

    Pith/arXiv arXiv 2024

  4. [4]

    5-vl technical report

    Bai, S., Chen, K., Liu, X., Wang, J., Ge, W., Song, S., Dang, K., Wang, P., Wang, S., Tang, J., et al.: Qwen2. 5-vl technical report. arXiv preprint arXiv:2502.13923 (2025)

    Pith/arXiv arXiv 2025

  5. [5]

    a is b" fail to learn

    Berglund, L., Tong, M., Kaufmann, M., Balesni, M., Stickland, A.C., Korbak, T., Evans, O.: The reversal curse: Llms trained on "a is b" fail to learn "b is a" (2024), https://arxiv. org/abs/2309.12288

    Pith/arXiv arXiv 2024

  6. [6]

    Blume, A., Kim, J., Ha, H., Chatikyan, E., Jin, X., Nguyen, K.D., Peng, N., Chang, K.W., Hoiem, D., Ji, H.: Partonomy: Large multimodal models with part-level visual understanding (2025),https://arxiv.org/abs/2505.20759

    arXiv 2025

  7. [7]

    arXiv preprint arXiv:2407.06438 (2024)

    Chen, Y ., Wang, X., Peng, H., Ji, H.: Solo: A single transformer for scalable vision-language modeling. arXiv preprint arXiv:2407.06438 (2024)

    Pith/arXiv arXiv 2024

  8. [8]

    arXiv preprint arXiv:2406.07476 (2024)

    Cheng, Z., Leng, S., Zhang, H., Xin, Y ., Li, X., Chen, G., Zhu, Y ., Zhang, W., Luo, Z., Zhao, D., et al.: Videollama 2: Advancing spatial-temporal modeling and audio understanding in video-llms. arXiv preprint arXiv:2406.07476 (2024)

    Pith/arXiv arXiv 2024

  9. [9]

    Cho, J.H., Madotto, A., Mavroudi, E., Afouras, T., Nagarajan, T., Maaz, M., Song, Y ., Ma, T., Hu, S., Jain, S., Martin, M., Wang, H., Rasheed, H., Sun, P., Huang, P.Y ., Bolya, D., Ravi, N., Jain, S., Stark, T., Moon, S., Damavandi, B., Lee, V ., Westbury, A., Khan, S., Krähenbühl, P., Dollár, P., Torresani, L., Grauman, K., Feichtenhofer, C.: Perception...

    Pith/arXiv arXiv 2025

  10. [10]

    arXiv preprint arXiv:2507.06261 (2025)

    Comanici, G., Bieber, E., Schaekermann, M., Pasupat, I., Sachdeva, N., Dhillon, I., Blistein, M., Ram, O., Zhang, D., Rosen, E., et al.: Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities. arXiv preprint arXiv:2507.06261 (2025)

    Pith/arXiv arXiv 2025

  11. [11]

    Psychonomic bulletin & review23(6), 1713–1743 (2016)

    Cutting, J.E.: Narrative theory and the dynamics of popular movies. Psychonomic bulletin & review23(6), 1713–1743 (2016)

    2016

  12. [12]

    arXiv preprint arXiv:2510.09741 (2025)

    Dalal, D., Vashishtha, G., Mishra, U., Kim, J., Kanda, M., Ha, H., Lazebnik, S., Ji, H., Jain, U.: Constructive distortion: Improving mllms with attention-guided image warping. arXiv preprint arXiv:2510.09741 (2025)

    Pith/arXiv arXiv 2025

  13. [13]

    Feng, B., Lai, Z., Li, S., Wang, Z., Wang, S., Huang, P., Cao, M.: Breaking down video llm benchmarks: Knowledge, spatial perception, or true temporal understanding? arXiv preprint arXiv:2505.14321 (2025)

    Pith/arXiv arXiv 2025

  14. [14]

    arXiv preprint arXiv:2406.08656 (2024)

    Feng, W., Li, J., Saxon, M., Fu, T.j., Chen, W., Wang, W.Y .: Tc-bench: Benchmarking temporal compositionality in text-to-video and image-to-video generation. arXiv preprint arXiv:2406.08656 (2024)

    Pith/arXiv arXiv 2024

  15. [15]

    arXiv e-prints pp

    Feng, X., Yu, H., Wu, M., Hu, S., Chen, J., Zhu, C., Wu, J., Chu, X., Huang, K.: Narrlv: Towards a comprehensive narrative-centric evaluation for long video generation models. arXiv e-prints pp. arXiv–2507 (2025)

    2025

  16. [16]

    In: Proceedings of the IEEE conference on computer vision and pattern recognition

    Gu, C., Sun, C., Ross, D.A., V ondrick, C., Pantofaru, C., Li, Y ., Vijayanarasimhan, S., Toderici, G., Ricco, S., Sukthankar, R., et al.: Ava: A video dataset of spatio-temporally localized atomic visual actions. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 6047–6056 (2018)

    2018

  17. [17]

    arXiv preprint arXiv:2502.17832 (2025) 16 H

    Ha, H., Zhan, Q., Kim, J., Bralios, D., Sanniboina, S., Peng, N., Chang, K.W., Kang, D., Ji, H.: Mm-poisonrag: Disrupting multimodal rag with local and global poisoning attacks. arXiv preprint arXiv:2502.17832 (2025) 16 H. Ha et al

    Pith/arXiv arXiv 2025

  18. [18]

    In: International conference on machine learning

    Jia, C., Yang, Y ., Xia, Y ., Chen, Y .T., Parekh, Z., Pham, H., Le, Q., Sung, Y .H., Li, Z., Duerig, T.: Scaling up visual and vision-language representation learning with noisy text supervision. In: International conference on machine learning. pp. 4904–4916. PMLR (2021)

    2021

  19. [19]

    arXiv preprint arXiv:2402.16315 (2024)

    Kim, J., Ji, H.: Finer: Investigating and enhancing fine-grained visual concept recognition in large vision language models. arXiv preprint arXiv:2402.16315 (2024)

    Pith/arXiv arXiv 2024

  20. [20]

    arXiv preprint arXiv:1809.01696 (2018)

    Lei, J., Yu, L., Bansal, M., Berg, T.L.: Tvqa: Localized, compositional video question answer- ing. arXiv preprint arXiv:1809.01696 (2018)

    Pith/arXiv arXiv 2018

  21. [21]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Li, B., Ge, Y ., Ge, Y ., Wang, G., Wang, R., Zhang, R., Shan, Y .: Seed-bench: Benchmarking multimodal large language models. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 13299–13308 (2024)

    2024

  22. [22]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Li, K., Wang, Y ., He, Y ., Li, Y ., Wang, Y ., Liu, Y ., Wang, Z., Xu, J., Chen, G., Luo, P., et al.: Mvbench: A comprehensive multi-modal video understanding benchmark. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 22195–22206 (2024)

    2024

  23. [23]

    arXiv preprint arXiv:2005.00200 (2020)

    Li, L., Chen, Y .C., Cheng, Y ., Gan, Z., Yu, L., Liu, J.: Hero: Hierarchical encoder for video+ language omni-representation pre-training. arXiv preprint arXiv:2005.00200 (2020)

    Pith/arXiv arXiv 2005

  24. [24]

    In: European Conference on Computer Vision

    Li, Y ., Wang, C., Jia, J.: Llama-vid: An image is worth 2 tokens in large language models. In: European Conference on Computer Vision. pp. 323–340. Springer (2024)

    2024

  25. [25]

    arXiv preprint arXiv:2311.10122 (2023)

    Lin, B., Ye, Y ., Zhu, B., Cui, J., Ning, M., Jin, P., Yuan, L.: Video-llava: Learning united visual representation by alignment before projection. arXiv preprint arXiv:2311.10122 (2023)

    Pith/arXiv arXiv 2023

  26. [26]

    In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition

    Lin, J., Yin, H., Ping, W., Molchanov, P., Shoeybi, M., Han, S.: Vila: On pre-training for visual language models. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 26689–26699 (2024)

    2024

  27. [27]

    Advances in neural information processing systems36, 34892–34916 (2023)

    Liu, H., Li, C., Wu, Q., Lee, Y .J.: Visual instruction tuning. Advances in neural information processing systems36, 34892–34916 (2023)

    2023

  28. [28]

    arXiv preprint arXiv:2306.05424 (2023)

    Maaz, M., Rasheed, H., Khan, S., Khan, F.S.: Video-chatgpt: Towards detailed video under- standing via large vision and language models. arXiv preprint arXiv:2306.05424 (2023)

    Pith/arXiv arXiv 2023

  29. [29]

    Advances in Neural Information Processing Systems36, 46212–46244 (2023)

    Mangalam, K., Akshulakov, R., Malik, J.: Egoschema: A diagnostic benchmark for very long- form video language understanding. Advances in Neural Information Processing Systems36, 46212–46244 (2023)

    2023

  30. [30]

    Advances in Neural Information Processing Systems36, 72983–73007 (2023)

    Minderer, M., Gritsenko, A., Houlsby, N.: Scaling open-vocabulary object detection. Advances in Neural Information Processing Systems36, 72983–73007 (2023)

    2023

  31. [31]

    arXiv preprint arXiv:2502.19958 (2025)

    Niu, K., Yu, H., Zhao, M., Fu, T., Yi, S., Lu, W., Li, B., Qian, X., Xue, X.: Chatreid: Open- ended interactive person retrieval via hierarchical progressive tuning for vision language models. arXiv preprint arXiv:2502.19958 (2025)

    Pith/arXiv arXiv 2025

  32. [32]

    Advances in Neural Information Processing Systems36, 42748– 42761 (2023)

    Patraucean, V ., Smaira, L., Gupta, A., Recasens, A., Markeeva, L., Banarse, D., Koppula, S., Malinowski, M., Yang, Y ., Doersch, C., et al.: Perception test: A diagnostic benchmark for multimodal video models. Advances in Neural Information Processing Systems36, 42748– 42761 (2023)

    2023

  33. [33]

    In: International conference on machine learning

    Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al.: Learning transferable visual models from natural language supervision. In: International conference on machine learning. pp. 8748–8763. PmLR (2021)

    2021

  34. [34]

    In: Proceedings of the Computer Vision and Pattern Recognition Conference

    Saravanan, D., Gupta, V ., Singh, D., Khan, Z., Gandhi, V ., Tapaswi, M.: Velociti: Benchmark- ing video-language compositional reasoning with strict entailment. In: Proceedings of the Computer Vision and Pattern Recognition Conference. pp. 18914–18924 (2025)

    2025

  35. [35]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Song, E., Chai, W., Wang, G., Zhang, Y ., Zhou, H., Wu, F., Chi, H., Guo, X., Ye, T., Zhang, Y ., et al.: Moviechat: From dense token to sparse memory for long video understanding. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 18221–18232 (2024) NARRATIVETRACK17

    2024

  36. [36]

    In: Proceedings of the IEEE conference on computer vision and pattern recognition

    Tapaswi, M., Zhu, Y ., Stiefelhagen, R., Torralba, A., Urtasun, R., Fidler, S.: Movieqa: Un- derstanding stories in movies through question-answering. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 4631–4640 (2016)

    2016

  37. [37]

    URL https://arxiv

    Team, C.: Chameleon: Mixed-modal early-fusion foundation models, 2024. URL https://arxiv. org/abs/2405.098189(8) (2024)

    Pith/arXiv arXiv 2024

  38. [38]

    Advances in Neural Information Processing Systems 34, 200–212 (2021)

    Tsimpoukelli, M., Menick, J.L., Cabi, S., Eslami, S., Vinyals, O., Hill, F.: Multimodal few-shot learning with frozen language models. Advances in Neural Information Processing Systems 34, 200–212 (2021)

    2021

  39. [39]

    arXiv preprint arXiv:2409.12191 (2024)

    Wang, P., Bai, S., Tan, S., Wang, S., Fan, Z., Bai, J., Chen, K., Liu, X., Wang, J., Ge, W., et al.: Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution. arXiv preprint arXiv:2409.12191 (2024)

    Pith/arXiv arXiv 2024

  40. [40]

    arXiv preprint arXiv:2406.08035 (2024)

    Wang, W., He, Z., Hong, W., Cheng, Y ., Zhang, X., Qi, J., Gu, X., Huang, S., Xu, B., Dong, Y ., et al.: Lvbench: An extreme long video understanding benchmark. arXiv preprint arXiv:2406.08035 (2024)

    Pith/arXiv arXiv 2024

  41. [41]

    arXiv preprint arXiv:2409.18869 (2024)

    Wang, X., Zhang, X., Luo, Z., Sun, Q., Cui, Y ., Wang, J., Zhang, F., Wang, Y ., Li, Z., Yu, Q., et al.: Emu3: Next-token prediction is all you need. arXiv preprint arXiv:2409.18869 (2024)

    Pith/arXiv arXiv 2024

  42. [42]

    Wang, Z., Guo, X., Stoica, S., Xu, H., Wang, H., Ha, H., Chen, X., Chen, Y ., Yan, M., Huang, F., Ji, H.: Perception-aware policy optimization for multimodal reasoning (2025), https://arxiv.org/abs/2507.06448

    Pith/arXiv arXiv 2025

  43. [43]

    Advances in Neural Information Processing Systems37, 28828–28857 (2024)

    Wu, H., Li, D., Chen, B., Li, J.: Longvideobench: A benchmark for long-context interleaved video-language understanding. Advances in Neural Information Processing Systems37, 28828–28857 (2024)

    2024

  44. [44]

    Wu, Y ., Kirillov, A., Massa, F., Lo, W.Y ., Girshick, R.: Detectron2.https://github.com/ facebookresearch/detectron2(2019)

    2019

  45. [45]

    In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition

    Xiao, J., Shang, X., Yao, A., Chua, T.S.: Next-qa: Next phase of question-answering to explaining temporal actions. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 9777–9786 (2021)

    2021

  46. [46]

    In: Proceedings of the 25th ACM international conference on Multimedia

    Xu, D., Zhao, Z., Xiao, J., Wu, F., Zhang, H., He, X., Zhuang, Y .: Video question answering via gradually refined attention over appearance and motion. In: Proceedings of the 25th ACM international conference on Multimedia. pp. 1645–1653 (2017)

    2017

  47. [47]

    International Journal of Computer Vision pp

    Ye, M., Chen, S., Li, C., Zheng, W.S., Crandall, D., Du, B.: Transformer for object re- identification: A survey. International Journal of Computer Vision pp. 1–31 (2024)

    2024

  48. [48]

    arXiv preprint arXiv:2304.14178 (2023)

    Ye, Q., Xu, H., Xu, G., Ye, J., Yan, M., Zhou, Y ., Wang, J., Hu, A., Shi, P., Shi, Y ., et al.: mplug-owl: Modularization empowers large language models with multimodality. arXiv preprint arXiv:2304.14178 (2023)

    Pith/arXiv arXiv 2023

  49. [49]

    In: Proceedings of the AAAI Conference on Artificial Intelligence

    Yu, Z., Xu, D., Yu, J., Yu, T., Zhao, Z., Zhuang, Y ., Tao, D.: Activitynet-qa: A dataset for understanding complex web videos via question answering. In: Proceedings of the AAAI Conference on Artificial Intelligence. pp. 9127–9134. No. 01 (2019)

    2019

  50. [50]

    Psychological bulletin133(2), 273 (2007)

    Zacks, J.M., Speer, N.K., Swallow, K.M., Braver, T.S., Reynolds, J.R.: Event perception: a mind-brain perspective. Psychological bulletin133(2), 273 (2007)

    2007

  51. [51]

    arXiv preprint arXiv:2409.16597 (2024)

    Zhang, J., Jiao, Y ., Chen, S., Zhao, N., Tan, Z., Li, H., Chen, J.: Eventhallusion: Diagnosing event hallucinations in video llms. arXiv preprint arXiv:2409.16597 (2024)

    arXiv 2024

  52. [52]

    thinking with images

    Zheng, Z., Yang, M., Hong, J., Zhao, C., Xu, G., Yang, L., Shen, C., Yu, X.: Deepeyes: Incen- tivizing" thinking with images" via reinforcement learning. arXiv preprint arXiv:2505.14362 (2025)

    Pith/arXiv arXiv 2025

  53. [53]

    In: Proceedings of the IEEE/CVF international conference on computer vision

    Zhou, K., Yang, Y ., Cavallaro, A., Xiang, T.: Omni-scale feature learning for person re- identification. In: Proceedings of the IEEE/CVF international conference on computer vision. pp. 3702–3712 (2019)

    2019

  54. [54]

    same_identity

    Zhu, J., Wang, W., Chen, Z., Liu, Z., Ye, S., Gu, L., Tian, H., Duan, Y ., Su, W., Shao, J., et al.: Internvl3: Exploring advanced training and test-time recipes for open-source multimodal models. arXiv preprint arXiv:2504.10479 (2025) NARRATIVETRACK1 6 Existing Benchmarks Existing VideoLLM evaluation benchmarks mostly focus on semantic understanding, whe...

    Pith/arXiv arXiv 2025

  55. [55]

    Action --- one fine-grained action the person is performing

  56. [56]

    Outfit --- describe what the person is wearing (clothing type, color, accessories)

  57. [57]

    kitchen",

    Scene --- briefly describe the background environment or setting (e.g., "kitchen", " forest trail", "office room"). If there is no bounding box visible throughout the video, return the message`"INVALID"` for all three fields. ### Requirements: - Focus only on the person inside the green bounding box. - Use precise and visually grounded descriptions. - If ...

  58. [58]

    Determine whether the target entity shows significant action transitions only based on the provided action change description

  59. [59]

    Determine whether the target entity shows significant outfit transitions only based on the provided outfit change description

  60. [60]

    Determine whether the scene changes significantly involving the target entity only based on the provided outfit change description

  61. [61]

    Determine whether any similar-looking entity (i.e., someone with a similar outfit) appears or not in the video, based on the provided video

  62. [62]

    Describe the single action and outfit (e.g., clothes, color, accessories) of the other entities (not describe the changes) that are not highlighted by the bounding box, based on the provided video

  63. [63]

    Describe whether the target entity shows over three action transitions (e.g., talking -> walking -> talking -> crying) --- only based on the provided action change description

  64. [64]

    Describe whether the target entity shows over three outfit transitions (e.g., blue t- shirt -> white t-shirt -> pink dress -> black coat) --- only based on the provided outfit change description

  65. [65]

    sitting",

    Describe whether the target entity shows over three scene transitions (e.g., church -> stadium -> park -> indoor room) --- only based on the provided scene change description. In each case, return a binary decision as`true`or`false`, and provide a clear justification. If there is only a single element in the changes, you should return`false`for the corres...

  66. [66]

    Return only the raw JSON object; do NOT include any commentary, markdown, or explanation

  67. [67]

    significant_action_transition

    Your output should follow the below structured format (JSON): ```json {{ "significant_action_transition": true or false, "significant_scene_transition": true or false, "significant_outfit_transition": true or false, "similar_looking_existence": true or false, "justification": {{ "significant_action_transition": "Your explanation here.", "significant_scene...

  68. [68]

    Is grammatically correct

  69. [69]

    grammar" to

    Is easy to understand. While doing this, ensure the question's intent remains consistent with the given template. Rules: - If the question is grammatically correct, set "grammar" to "yes". - If the question is not grammatically correct, set "grammar" to a corrected version that preserves its meaning. - If the question is easy to understand, set "understan...