{"id":"e880b509-8c6d-41c1-b273-d1ce7823c74c","arxiv_id":"2601.01095","paper_version":4,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"NarrativeTrack shows that video AI models, including GPT-4o, falter at tracking a specific person across scene changes, outfit changes, and similar-looking characters in long videos.","lead":"NarrativeTrack is a new video benchmark that tests whether AI models can keep track of a specific person as scenes, clothing, and actions change and as look-alikes appear. It shows today's multimodal models struggle to maintain identity over time, even the best proprietary ones.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Ground-truth attributes come from Gemini-2.5-Pro with no independent human-labeled validation; a prompt typo in §7.3 adds internal evidence that this layer is unreliable.","rationale":"The reader's weakest assumption—that Gemini-generated attributes are unvalidated and could make the benchmark measure Gemini's perception—is the most load-bearing threat to the central claim. If the attribute layer is wrong, every downstream number is suspect, including the headline GPT-4o 72.27% score and the claim that models hallucinate identity. The paper's own pipeline evaluation validates detection and tracking, not the semantic attributes, and the QA validity review is not an independent attribute-labeling study. I also found a concrete internal prompt error in §7.3 that reinforces the concern: the scene-transition decision is instructed to be based on the outfit change description, which would directly corrupt scene metadata if followed.\n\nI do not think this warrants rejection. The tracking validation, the text-only ablation (30.52% drop), the reversed-video collapse on ordering (51.2% to 6.1%), and the 96% human accuracy after cleaning provide real evidence that the benchmark is not trivially solvable and that temporal order matters. Those results would survive even some attribute noise. The concern is that per-attribute correctness is unquantified and could change conclusions if Gemini is systematically wrong on actions, outfits, or scenes. That is an addressable validation gap, not a fatal flaw, so the paper should remain conditional pending an independent attribute-level audit.\n\nI agree with the reader's identification of the same weakest assumption. My contribution is to pinpoint the specific missing validation (per-attribute human agreement), the concrete prompt typo that makes the concern more acute, and a concrete test that would settle it.","tokens_in":29082,"tokens_out":5353,"duration_ms":58918,"concrete_test":"Sample ~200 NarrativeTrack QA pairs stratified by CRP dimension. Have at least three independent annotators, blinded to Gemini outputs and to the provided answers, label the target entity's action, outfit, scene, and identity continuity directly from the raw video clips for each queried segment, using the same segment boundaries as the pipeline. Compute per-attribute agreement between human labels and the Gemini-derived metadata, not just end-to-end QA accuracy. Then re-score the subset using human-corrected answers. If per-attribute agreement falls below ~90%, or if corrected-answer accuracy shifts by more than ~5 absolute points, the benchmark ground truth is not established and the reported model rankings should be recomputed. Separately, check whether the Appendix §7.3 scene-transition prompt was implemented as written; if so, regenerate the scene-change QA with the corrected prompt","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—that NarrativeTrack diagnoses entity-centric narrative understanding and that MLLMs fail at identity tracking—requires the ground-truth attributes (actions a_ij, outfits o_ij, scenes s_ij in Eq. 1) to be correct. These attributes are produced by Gemini-2.5-Pro in the Contextual Recognition stage (§3.1) and are then passed through templates that programmatically generate every question and answer (§3.3). The benchmark therefore has a single semantic source for its labels, and any systematic Gemini error propagates directly into the 'correct' answer.\n\nThe paper validates detection and tracking quantitatively (AVA recall 0.848; 96% tracking agreement, §3.1), and the QA validity review plus 96% human accuracy (§3.3) show the final questions are answerable. But none of these directly validates the attribute labels: the 100-item review checked 'grounded to the target entity and its states' without independently establishing that Gemini's action/outfit/scene categories are correct, and the authors' manual refinement is not an independent blind annotation. No per-attribute agreement with human labels is reported.\n\nThere is also a concrete internal inconsistency in the contextual-recognition prompt: Appendix §7.3, Second Step, item 3 instructs the model to 'Determine whether the scene changes significantly ... only based on the provided outfit change description'—a copy-paste from item 2. If the pipeline was run with that prompt as printed, scene-transition decisions are computed from the wrong input, contaminating the scene-change dimension and any QA using scene fields.\n\nConsequence: if Gemini mislabels non-trivially, human 96% accuracy could reflect agreement with flawed metadata, and the headline finding that models 'hallucinate identity under context shifts' could be an artifact of erroneous ground truth rather than a model deficiency.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces NARRATIVETRACK, a benchmark for evaluating entity-centric narrative understanding in multimodal large language models. A fully automated pipeline detects human entities, links them into trajectories via re-identification and human-consensus verification, and augments each trajectory with action, outfit, and scene attributes using Gemini-2.5-Pro. These structured representations are then used to instantiate a Compositional Reasoning Progression (CRP) spanning entity existence, entity changes, and entity ambiguity, with QA pairs generated programmatically from templates. The authors evaluate 12 open-source and 1 proprietary MLLM, reporting that GPT-4o achieves 72.27% accuracy while open-source models lag, and identify a trade-off between perceptual grounding and temporal coherence. Ablations show performance drops under text-only input and reversed frames, supporting the claim that the benchmark requires temporal grounding.","tokens_in":29430,"tokens_out":3114,"duration_ms":36363,"significance":"If the attribute layer is reliable, NARRATIVETRACK would be a valuable diagnostic resource: it is the first video benchmark to explicitly decompose narrative understanding into entity-centric CRP dimensions, it is scalable due to full automation, and the included ablations (text-only drop of 30.52%, reversed-frame collapse on ordering tasks) provide convincing evidence that the benchmark is not solvable by static cues. The detection/tracking validation on AVA (recall 0.848; 96.08% human agreement on identity filtering) and the QA review (Fleiss' κ = 0.767, 96% human accuracy after cleaning) demonstrate care in verifying the benchmark's basic integrity. The central weakness is that the action/outfit/scene attributes that ground every question and answer are produced solely by Gemini-2.5-Pro and have not been independently validated against human-labeled attribute ground truth; this is load-bearing for the paper's main claims.","major_comments":[{"comment":"The ground-truth attributes a_ij, o_ij, s_ij in Eq. (1) are produced by Gemini-2.5-Pro in the Contextual Recognition stage and are never validated against human-labeled attribute annotations. The AVA validation in §3.1 measures only detection and tracking quality, not the correctness of action/outfit/scene labels. The QA validity review in §3.3 checks that questions and answers are 'grounded to the target entity and its states', but it does not independently establish that the stated attribute categories are factually correct; the authors' manual refinement is not a blind, independent annotation. Since every template question and ground-truth answer is instantiated from these attributes, any systematic Gemini error propagates directly into the benchmark's 'correct' answers. Please report a human-annotator agreement study on the attribute labels themselves (e.g., per-attribute precision o","section":"§3.1, Eq. (1), §3.3"},{"comment":"The contextual-recognition prompt contains an internal inconsistency: item 3 instructs the model to 'Determine whether the scene changes significantly ... only based on the provided outfit change description' — a copy-paste from item 2. If the pipeline was run with the prompt as printed, the scene-transition significance decisions that gate the scene-change dimension and the selection of videos for that dimension may be based on outfit descriptions rather than scene descriptions, corrupting the corresponding ground truth. Please clarify whether the deployed prompt differed from the appendix; if not, correct the prompt and re-generate or re-validate all affected scene-change QA pairs. This typo also underscores the need for independent attribute validation.","section":"§7.3, Second Step, item 3"},{"comment":"The benchmark composition is partially defined by the same model that generates the labels: 'main characters' are the top-4 ReID clusters, but the selection of videos with 'significant' attribute changes and visually similar entities is made by Gemini-2.5-Pro on the basis of its predicted attributes. Consequently, the distribution of CRP dimensions and question types inherits Gemini's perceptual biases, and the claimed progression of complexity (existence → changes → ambiguity) is not independently established. A concrete test would be to have human annotators classify a random sample of video/entity instances into the CRP dimensions and compare against the automated selection; reporting the agreement would calibrate how much of the model ranking is attributable to Gemini-dependent sampling rather than to the target narrative skill.","section":"§3.1, §3.3"}],"minor_comments":[{"comment":"The pipeline 'compromises three stages' should read 'comprises three stages'.","section":"§3.1"},{"comment":"Formatting issues: 'Boldand' is missing a space; the header 'OGP-' is undefined in the table caption; some row entries appear to have inconsistent spacing.","section":"Table 2"},{"comment":"Option D reads '22MPH' with missing spacing; please fix the figure text.","section":"Fig. 1"},{"comment":"The model name 'InterVL3-8B' is inconsistent with 'InternVL3-8B' used elsewhere.","section":"Table 10"},{"comment":"The abstract reports Qwen2.5-VL-32B at 56.90%, while Table 2 reports 56.96%; please reconcile the numbers.","section":"Abstract / §4.2"},{"comment":"Inconsistent spelling 'GPT4-o' versus 'GPT-4o'; please standardize.","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":"The core idea is timely and the benchmark has solid engineering and useful ablations. The main risk is the single-source attribute ground truth; the prompt typo in §7.3 makes this risk concrete. I would support publication after the authors provide independent human validation of the attribute layer (not just QA answerability) and correct/re-run the affected pipeline stage. The requested validation is feasible within the manuscript's scope and does not require a fundamentally different benchmark."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: NarrativeTrack is a genuinely useful diagnostic benchmark for entity-centric tracking in MLLMs, and the compositional progression (existence, changes, ambiguity) is a good idea. But the ground-truth attributes that fill every question and answer come from Gemini-2.5-Pro with no independent human-labeled validation, and the prompt in §7.3 contains a copy-paste error that, if run as printed, computes scene-change decisions from the outfit description. Those two issues make the benchmark's headline findings conditional, not fatal.\n\nWhat it does well: the detection/tracking validation on AVA is solid (recall 0.848, 96% agreement with human labels on 1,108 detections). The QA design is thoughtful: balanced dimensions, real and synthetic distractors, and the ablations—text-only drops 30.5%, reversed video collapses ordering—show the benchmark is not answerable from static cues. The frame-density peak at 20 frames is an interesting empirical result. The directional bias (forward vs backward) is a natural extension of the Reversal Curse and worth reporting.\n\nThe soft spots, in order of severity. First, the attribute layer. Actions, outfits, and scenes in Eq. 1 are produced by Gemini-2.5-Pro in the Contextual Recognition stage; the QA templates then instantiate questions directly from those attributes. The 70% valid rate and subsequent manual refinement tell you the final questions are answerable, but they don't tell you whether Gemini's labels are correct. If Gemini systematically mislabels actions or outfits, the 'correct' answers are wrong and the benchmark measures Gemini's perception, not narrative understanding. You need a human-labeled attribute sample with per-attribute agreement. Second, the §7.3 prompt typo: item 3 says scene transitions should be determined 'only based on the provided outfit change description.' If the pipeline ran with that prompt, the scene-change dimension is contaminated. That's internal evidence the contextual-recognition step is not as reliable as the authors imply; it needs to be checked. Third, the abstract overclaims: it says neither scaling nor frame density improves narrative understanding, but the results show scaling helps within families (InternVL3-38B +6.66%, Video-LLaMA2-72B +5.66%) and only frame density peaks at 20. That's a contradiction, not a nuance. Fourth, 'fully automated' is misleading when every QA pair was manually refined. And the 'fundamental trade-off' between perceptual grounding and temporal reasoning is inferred from comparing OGP vs OVS model families, which differ in architecture and training data—that's a confounded comparison, so the trade-off claim is stronger than the evidence. The 'first benchmark' framing is roughly defensible against VELOCITI and PerceptionTest, though I'd soften 'first systematic framework' in the abstract.\n\nWho should read it: anyone building video QA benchmarks or studying entity tracking in MLLMs will get value from the CRP framework and the failure analyses. The paper deserves a serious referee—the benchmark is important enough and the issues are addressable—but it should not be accepted without the attribute validation and the prompt fix. If the authors add a human-labeled attribute sample and clarify the scaling statement, this could be a solid contribution.","headline":"Useful diagnostic benchmark for entity tracking, but the Gemini-generated attribute ground truth is unvalidated and a prompt typo in §7.3 makes scene-change labels suspect.","tokens_in":30003,"tokens_out":5328,"would_cite":false,"duration_ms":44370,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper introduces NarrativeTrack, a benchmark claiming that multimodal models fail to maintain coherent entity identity across video scene changes, with GPT-4o at 72.27% and open models much lower.","keywords":["video understanding","multimodal large language models","entity-centric reasoning","narrative comprehension","benchmark","temporal reasoning","entity tracking","compositional reasoning progression"],"falsifier":"Randomly sample about 100 video segments from NarrativeTrack, have multiple human annotators label the target entity's action, outfit, and scene per timestamp, and measure agreement with the Gemini-2.5-Pro-produced metadata. If human–machine agreement on attributes is markedly below the 96% human accuracy the paper reports for the final QA pairs, the ground-truth answers are not independently verified and the benchmark's model rankings could change under corrected labels.","tokens_in":28927,"feed_emoji":"🎬","tokens_out":4280,"duration_ms":376966,"temperature":0.7,"pith_summary":"The paper tries to establish that genuine narrative understanding in multimodal large language models (MLLMs) requires tracking entities—the people doing things—across time, not just recognizing static frames. To test this, it builds NarrativeTrack, a benchmark that decomposes videos into per-entity trajectories (bounding box, action, outfit, scene per timestamp) and questions models with a Compositional Reasoning Progression: entity existence, entity changes, and entity ambiguity. It evaluates 13 models and finds that even the strongest (GPT-4o at 72.27%) struggles with long-range identity, while open-source models drop sharply, revealing a trade-off between perceptual grounding and temporal coherence. The paper also shows that more frames do not help beyond 20, and models have a strong forward-only temporal bias. If correct, the benchmark provides a diagnostic framework for where entity-centric reasoning fails and motivates architectural changes like bidirectional temporal modeling.","feed_headline":"AI can't keep characters straight in videos, new test shows","feed_subtitle":"NarrativeTrack tests who-is-doing-what across scene changes; GPT-4o tops out at 72% and open-source models fall far behind.","key_machinery":"The entity representation τ_ei: a per-timestamp tuple (timestamp, bounding box, action, scene, outfit) that binds identity to state across time. The Compositional Reasoning Progression (CRP) then turns this representation into questions of increasing difficulty—existence, changes, ambiguity—isolating whether a model fails at temporal persistence, grounded state transitions, or fine-grained disambiguation. The automated pipeline (ensemble detection, ReID clustering, face recognition, and Gemini-2.5-Pro attribute annotation) is what makes scalable construction possible; the QA templates are filled deterministically from the same metadata that defines ground truth.","core_discovery":"On the paper's own terms, the central discovery is that current MLLMs cannot maintain coherent entity representations across visual transitions and temporal gaps. NarrativeTrack operationalizes narrative understanding as entity-centric reasoning: models must answer questions about whether an entity persists, how its action, outfit, and scene change over time, and whether visually similar entities are distinct. Across all tested models, performance is far below ceiling, and the failure patterns are systematic: general-purpose open models ground perception well but lose temporal continuity; video-specialized open models capture continuity but hallucinate visual details; and even the proprietar","pith_inferences":["Since the attribute ground truth (action/outfit/scene) comes from an automated model without human validation, a portion of the benchmark's 'correct' answers may reflect that model's perception; human-labeling a random subset of segments would tell how much of the reported accuracies are about the tested models versus the annotator model.","The CRP structure could be reused as a training curriculum: ordering questions by existence → changes → ambiguity gives a natural difficulty progression for fine-tuning or reinforcement learning.","The entity-ambiguity dimension, where visually similar people must be disambiguated, is close to person re-identification; one could test whether explicit ReID features improve MLLM narrative reasoning, which the paper leaves to future work.","The forward/backward asymmetry parallels the 'reversal curse' in text LLMs; if the same bias appears in other video benchmarks, it suggests a general architectural limitation rather than a quirk of this dataset."],"forward_implications":["NarrativeTrack can be used to pinpoint which level of entity-centric reasoning a model fails at—existence, change, or ambiguity—giving a diagnostic breakdown rather than a single score.","Benchmark scores should not be treated as a saturated capability: even the strongest model leaves roughly 28% of questions unanswered correctly, so entity-centric narrative understanding is an open problem.","Adding more input frames does not improve entity tracking beyond about 20 frames, so progress will need better temporal-coherence mechanisms, not denser sampling.","The large forward/backward asymmetry implies that models trained on left-to-right sequences inherit a directional bias; fixing it will require bidirectional temporal modeling or contrastive reversal objectives.","Because removing visual input drops GPT-4o to near chance, the benchmark's questions genuinely require visual grounding, unlike many prior video benchmarks."],"fun_headline_variants":["New benchmark shows AI loses track of who's who in videos","AI struggles to follow characters across video scenes, benchmark finds","Benchmark exposes AI's failure to track entities in video narratives","NarrativeTrack: AI models can't keep characters straight across time","Even best AI models fail to track characters in video narratives"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The ground-truth attributes (actions, outfits, scenes) that fill every question and answer are produced by Gemini-2.5-Pro's per-clip reading and were never checked against human-labeled attribute annotations; if that model mislabels what a person is doing or wearing, the benchmark's correct answers are wrong and the scores measure the annotator rather than the tested models.","fun_headline_variants_meta":{"raw":{"variants":["New benchmark shows AI loses track of who's who in videos","AI struggles to follow characters across video scenes, benchmark finds","Benchmark exposes AI's failure to track entities in video narratives","NarrativeTrack: AI models can't keep characters straight across time","Even best AI models fail to track characters in video narratives"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.0005,"raw_usage":{"total_tokens":2301,"prompt_tokens":779,"completion_tokens":1522,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":523,"completion_tokens_details":{"reasoning_tokens":1437}},"tokens_in":523,"tokens_out":1522,"duration_ms":11037,"temperature":1.0,"reasoning_tokens":1437,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-04T06:25:38.248880+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Randomly sample about 100 video segments from NarrativeTrack, have multiple human annotators label the target entity's action, outfit, and scene per timestamp, and measure agreement with the Gemini-2.5-Pro-produced metadata. If human–machine agreement on attributes is markedly below the 96% human accuracy the paper reports for the final QA pairs, the ground-truth answers are not independently verified and the benchmark's model rankings could change under corrected labels.","supporting_citations":[],"review_version":2}