Pith. sign in

REVIEW 2 major objections 18 references

Open-source Video-LLMs scoring 37–38% on long-form outfit-order questions are not tracking the named character.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-14 07:11 UTC pith:Q5XHWQXH

load-bearing objection Open-source Video-LLMs' 37–38% InfiniBench global-appearance scores are not character tracking: name swaps move answers only 4–31% of the time, mostly via gender, and open-ended collapses to zero fully correct. the 2 major comments →

arxiv 2607.11078 v1 pith:Q5XHWQXH submitted 2026-07-13 cs.CV cs.AI

Do Video-LLMs Actually Watch? Diagnosing Character-Tracking Failures in Long-Form Video

Mohammad Al-Ratrout , Shayla Sharmin , Aditya Raikwar , Roghayeh Leila Barmaki This is my paper
classification cs.CV cs.AI
keywords Video-LLMscharacter trackinglong-form videobenchmark diagnosticsname substitutionInfiniBenchoutfit changesmultimodal evaluation
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Benchmarks now score whether a Video-LLM can follow one named person across a full TV episode and list their outfit changes in order. Stronger open-source 7–8B models reach 37–38% on that task, but the paper asks whether the score comes from tracking the person or from something easier. Across three architecturally different models, swapping the name in the question while leaving the video and options fixed changes the answer only 4–31% of the time; the models move more on cross-gender swaps than same-gender ones and collapse when the same questions are asked open-endedly, with zero fully correct answers. Extra frames, oracle frames, and subtitles leave the identity failure untouched. A sympathetic reader cares because headline accuracy is being read as evidence of long-form character understanding that the models do not actually perform.

Core claim

The 37–38% accuracy of current open-source 7–8B Video-LLMs on InfiniBench global appearance does not come from tracking the named character. Name substitution (video and options held fixed) changes the answer letter in only 4–31% of cases; sensitivity is 13–28 points higher for cross-gender than same-gender swaps, showing only coarse gender cues; open-ended accuracy falls 18–25 points with 0 of 151 answers fully correct. Doubling frames raises accuracy without raising name sensitivity, so the bottleneck is binding visual content to the character-name token rather than how much video is seen.

What carries the argument

A nine-condition diagnostic protocol whose core is name substitution: replace the named character with another main-cast member while holding the episode video and multiple-choice options fixed, then measure how often the model’s answer letter changes. Gender decomposition of those swaps, open-ended re-asking, frame-count ablations, and collective accuracy complete the separation of aggregate score from genuine identity tracking.

Load-bearing premise

That a model truly conditioning on the named character would usually change its multiple-choice letter under name swap, even when the swapped character’s true outfit sequence is not among the original options.

What would settle it

Rewrite the option sets so the correct outfit sequence for the newly named character is always present among A–E, re-run the identical name-swap pairs, and check whether open-source models then change letter near 100% of the time; if they do, the original diagnostic overstated the identity failure.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Named-character video benchmarks should report name-substitution sensitivity next to accuracy, because the two can dissociate by large margins.
  • Fixed uniform-frame protocols need per-show annotation-window statistics; otherwise ~20-point accuracy swings can be sampling artifacts, not tracking differences.
  • Multiple-choice scores should be paired with open-ended variants or MC–OE gaps, which expose forced guessing rather than description ability.
  • Accuracy gains that are not accompanied by gains in name sensitivity are not improvements in character tracking.
  • Predicted-letter distributions should be reported, because strong letter bias can inflate scores independent of video content.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same visual-to-name binding failure is likely to affect any long-video task that requires following a specific person rather than detecting scene-level clothing or gender.
  • Diagnostic suites of this form could become routine for long-video benchmarks the way frame-shuffling tests became routine for temporal sensitivity.
  • Larger open-weight models may still fail same-gender name swaps even if their multiple-choice numbers climb, unless training explicitly targets identity binding.
  • Pretrained face recognition alone is unlikely to close the gap if models already confuse highly recognizable same-gender cast members under name swap.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 0 minor

Summary. The paper asks whether 37–38% accuracy of open-source 7–8B Video-LLMs on InfiniBench’s global appearance task reflects genuine tracking of a named character’s outfit sequence across a full TV episode. Using a nine-condition diagnostic protocol (C1–C9) on InternVL2-8B, Qwen2.5-VL-7B, LLaVA-NeXT-Video-7B, and Gemini 2.5 Flash as a partial frontier reference, the authors show that name substitution (video and options fixed) changes the answer letter only 4–31% of the time, with a 13–28 point same- vs cross-gender gap; open-ended accuracy falls 18–25 points with 0/151 fully correct open-source answers; collective accuracy is 0/20 BBT episodes; and frame-count, subtitle, and targeted-frame ablations leave name sensitivity unimproved. They conclude that models use coarse gender cues without binding visual identity to the named character, and release a diagnostic toolkit.

Significance. If the result holds, it is a useful diagnostic contribution for long-form video understanding: it shows that rising InfiniBench-style scores can be produced by position bias, option-set constraint, and category-level visual filters rather than character identity binding. Strengths include architecturally distinct open-source models, Wilson CIs, McNemar and chi-squared tests, Cohen’s h, gender decomposition, open-ended evaluation with human-validated LLM judging (κ=0.63), and a released toolkit. The work extends chart- and temporal-insensitivity diagnostics to character identity and gives concrete benchmark-design recommendations (report name-substitution sensitivity, MC–OE gaps, annotation-window statistics, letter distributions).

major comments (2)
  1. §4 C4 and §5.5 (Table 3): The name-substitution diagnostic treats any letter change under fixed original options as evidence of identity conditioning, even though the swapped character’s true outfit sequence need not appear among A–E. Letter change is therefore only a necessary, not sufficient, signal; letter bias (InternVL2 42% E, LLaVA 70% A) can further suppress observed sensitivity. The paper already notes the design and the gender split / open-ended results are independent of it, but the central claim would be stronger if the authors either (a) re-ran a subset with options rewritten for the swapped character or (b) quantified how often the original options remain plausible for the swap target, so readers can bound the interpretive softness.
  2. §3.2–3.3 and §7: Gemini 2.5 Flash is only a partial frontier reference (full-episode BBT, open-ended, collective accuracy, n=23 name swaps). The abstract and conclusion still use it as the upper-end contrast for the MC–OE gap and tracking failure. Either complete the name-substitution and gender-decomposition conditions for Gemini or clearly demote the frontier claims so that the open-source pattern is not over-generalized from an incomplete reference.

Circularity Check

0 steps flagged

No circularity: pure empirical diagnostic study with external labels and independent ablations; no derivation that reduces to its own inputs.

full rationale

This paper reports experimental measurements (name-substitution sensitivity, MC-vs-OE gaps, frame/subtitle ablations, collective accuracy, position bias) on three open-source Video-LLMs plus Gemini against InfiniBench/TVQA+ ground-truth labels and fixed video/option stimuli. Accuracy, sensitivity rates, and gender-decomposed deltas are computed directly from model outputs under controlled perturbations; nothing is fitted then re-presented as a prediction, no uniqueness theorem is imported, and no quantity is defined in terms of the quantity it is claimed to derive. Self-citations are ordinary (building on InfiniBench task definition and prior diagnostic style from chart/VQA papers) and not load-bearing for the central claim that 37–38% MC accuracy does not reflect character tracking. The derivation chain is simply “run models under nine conditions → observe low name sensitivity and large MC–OE drop,” which is self-contained and externally falsifiable. Score 0 is therefore the correct outcome.

Axiom & Free-Parameter Ledger

3 free parameters · 4 axioms · 1 invented entities

The central claim rests on standard evaluation assumptions and a small set of protocol choices, not on free physical constants or new ontological entities. Load-bearing premises are that InfiniBench/TVQA+ outfit orderings are correct enough for diagnostics, that letter change under name swap indicates identity conditioning, and that 16-frame open-source evaluation matches the intended use of these models. No graviton-like mediator is introduced; the “diagnostic protocol” is a measurement procedure, not a postulated physical entity.

free parameters (3)
  • uniform frame budget (default 16; ablations 8/32/64)
    Chosen to match InfiniBench open-source protocol rather than fit to maximize sensitivity; still a free protocol knob that interacts with episode length and annotation windows.
  • subtitle character cap (4000)
    Hand-chosen context limit for the subtitle ablation; could in principle change dialogue usefulness if raised.
  • open-ended scoring scale {0, 0.5, 1} via Claude judge
    Scoring rubric and judge model are design choices; human κ=0.63 validates directionally but leaves residual judge freedom.
axioms (4)
  • domain assumption InfiniBench global-appearance ground-truth outfit orderings (from filtered TVQA+ boxes) correctly identify unique outfit sequences per named character.
    All accuracy, OE, and collective-accuracy claims are scored against these labels (§3.1).
  • ad hoc to paper A model that conditions on character identity should change its multiple-choice letter under name substitution even when options stay fixed to the original character.
    Core interpretive rule of C4/§5.5; without it, low sensitivity does not imply identity failure.
  • domain assumption Uniform 16-frame sampling plus greedy decoding is a fair test of open-source Video-LLM character tracking as currently deployed/evaluated.
    Stated as matching InfiniBench open-source protocol (§3.3); frame ablations test robustness but native long-context open models are out of scope.
  • standard math Wilson CIs, McNemar, chi-squared, and Cohen’s h are appropriate for the reported paired accuracy and sensitivity comparisons.
    Used throughout §3.3 and Results for significance and effect size.
invented entities (1)
  • Nine-condition character-tracking diagnostic protocol (C1–C9) independent evidence
    purpose: Separate headline InfiniBench accuracy from identity binding, gender cues, option-set constraint, frame coverage, subtitles, and abstention.
    New measurement apparatus introduced by the paper; independent_evidence is true insofar as others can run the same conditions on new models and falsify the reported sensitivity rates.

pith-pipeline@v1.1.0-grok45 · 17908 in / 3257 out tokens · 40800 ms · 2026-07-14T07:11:58.623508+00:00 · methodology

0 comments
read the original abstract

Can a Video Large Language Model (Video-LLM) follow one person through a long video, keeping track of who they are well enough to report, in order, how their outfit changes across a full TV episode? Benchmarks increasingly score this kind of task, and the strongest open-source 7--8B models now reach 37--38% on InfiniBench's global appearance task, which asks exactly that. But does that score come from tracking the named character, or from something easier? We test this with a nine-condition diagnostic protocol applied to three architecturally distinct open-source Video-LLMs, with Gemini~2.5~Flash as a frontier reference, and find the accuracy does not come from character tracking. When we change the character named in the question to a different cast member, leaving the video and answer options untouched, the models change their answer only 4--31% of the time, so they are largely ignoring who the question asks about. Breaking that test down by the gender of the swapped name shows why: the models react more when the name is changed to a different-gender character than to a same-gender one (a 13--28 point gap), picking up coarse gender cues but unable to tell same-gender individuals apart. This shallow processing surfaces again when we drop the multiple-choice options and ask the same questions open-endedly: open-source accuracy drops 18--25 points, with none of 151 answers fully correct, versus a 12-point drop for Gemini. Further checks rule out the obvious innocent explanations, adding subtitles, using the most informative frames, or doubling the number of frames all leave character tracking unimproved, so the bottleneck is not how much video the model sees but how it ties that video to the person the question names. We release a diagnostic toolkit for auditing what such benchmark scores actually measure.

Figures

Figures reproduced from arXiv: 2607.11078 by Aditya Raikwar, Mohammad Al-Ratrout, Roghayeh Leila Barmaki, Shayla Sharmin.

Figure 1
Figure 1. Figure 1: Name substitution diagnostic. Same-gender (left): on Sheldon→Leonard, all three models keep the same letter, they do not condition on the name. Cross-gender (right): on Penny→Howard, two of three change letter but land on a wrong answer, a coarse gender cue, not identification. Bottom: sensitivity over n=124 BBT swap pairs by gender. Takeaway: models respond to gender, not identity, same-gender swaps move … view at source ↗
Figure 2
Figure 2. Figure 2: Annotation-structure confound across shows. Ac￾curacy lift from oracle frame selection versus average spacing between uniformly sampled frames. Longer videos have wider frame spacing under the same 16-frame budget, so oracle frames help more when uniform sampling misses annotated outfit-change windows. Dotted line: per-show mean lift. Takeaway: apparent per-show difficulty can shift by ∼20 Percentage Point… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

18 extracted references · 5 linked inside Pith

  1. [1]

    In- finibench: A comprehensive benchmark for large multimodal 8 models in very long video understanding

    Kirolos Ataallah, Chenhui Gou, Eslam Abdelrahman, Khushbu Pahwa, Jian Ding, and Mohamed Elhoseiny. In- finibench: A comprehensive benchmark for large multimodal 8 models in very long video understanding. InarXiv preprint arXiv:2406.19875, 2024. 2, 3, 4, 8

    arXiv 2024

  2. [2]

    InternVL2: Pushing open-source mul- timodal models to commercial levels

    Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. InternVL2: Pushing open-source mul- timodal models to commercial levels. InarXiv preprint arXiv:2410.16261, 2024. 3

    Pith/arXiv arXiv 2024

  3. [3]

    Routledge, 2 edition, 1988

    Jacob Cohen.Statistical Power Analysis for the Behavioral Sciences. Routledge, 2 edition, 1988. 3

    1988

  4. [4]

    Daniel Cores, Michael Dorkenwald, Manuel Mucientes, Cees G. M. Snoek, and Yuki M. Asano. TVBench: Re- designing video-language evaluation. InarXiv preprint arXiv:2410.07752, 2024. 2

    Pith/arXiv arXiv 2024

  5. [5]

    Hashimoto

    Yann Dubois, Balázs Galambosi, Percy Liang, and Tat- sunori B. Hashimoto. Length-controlled AlpacaEval: A sim- ple way to debias automatic evaluators. InarXiv preprint arXiv:2404.04475, 2024. 6

    Pith/arXiv arXiv 2024

  6. [6]

    Gemini 2.5: Pushing the frontier with ad- vanced reasoning, multimodality, long context, and next gen- eration agentic capabilities.arXiv preprint arXiv:2507.06261,

    Gemini Team. Gemini 2.5: Pushing the frontier with ad- vanced reasoning, multimodality, long context, and next gen- eration agentic capabilities.arXiv preprint arXiv:2507.06261,

    Pith/arXiv arXiv

  7. [7]

    What’s “up” with vision-language models? investigating their struggle with spatial reasoning

    Amita Kamath, Jack Hessel, and Kai-Wei Chang. What’s “up” with vision-language models? investigating their struggle with spatial reasoning. InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, 2023. 2

    2023

  8. [8]

    TVQA+: Spatio-temporal grounding for video question an- swering

    Jie Lei, Licheng Yu, Tamara Berg, and Mohit Bansal. TVQA+: Spatio-temporal grounding for video question an- swering. InProceedings of the 58th Annual Meeting of the Association for Computational Linguistics, 2020. 2

    2020

  9. [9]

    Evaluating object hallucination in large vision-language models

    Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. Evaluating object hallucination in large vision-language models. InProceedings of the 2023 Conference on Empirical Methods in Natural Language Pro- cessing, 2023. 2

    2023

  10. [10]

    Note on the sampling error of the difference between correlated proportions or percentages.Psychome- trika, 12:153–157, 1947

    Quinn McNemar. Note on the sampling error of the difference between correlated proportions or percentages.Psychome- trika, 12:153–157, 1947. 3

    1947

  11. [11]

    Plot twist: Multimodal models don’t comprehend simple chart details

    Yasaman Razeghi, Ishita Dasgupta, Fangyu Liu, Vinay Venkatesh Ramasesh, and Sameer Singh. Plot twist: Multimodal models don’t comprehend simple chart details. InFindings of the Association for Computational Linguistics: EMNLP 2024, pages 5922–5937, 2024. 2, 6, 7, 8

    2024

  12. [12]

    Qwen2.5-VL technical report

    Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Yang Fan, Kai Dang, Mengfei Du, Xuancheng Ren, Rui Men, Dayiheng Liu, Chang Zhou, Jingren Zhou, and Jun- yang Lin. Qwen2.5-VL technical report. InarXiv preprint arXiv:2502.13923, 2025. 3

    Pith/arXiv arXiv 2025

  13. [13]

    Probable inference, the law of succession, and statistical inference.Journal of the American Statistical Association, 22:209–212, 1927

    Edwin B Wilson. Probable inference, the law of succession, and statistical inference.Journal of the American Statistical Association, 22:209–212, 1927. 3

    1927

  14. [14]

    ChartInsights: Evaluating multimodal large language models for low-level chart question answering

    Yifan Wu, Lutao Yan, Leixian Shen, Yunhai Wang, Nan Tang, and Yuyu Luo. ChartInsights: Evaluating multimodal large language models for low-level chart question answering. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 12174–12200, 2024. 2

    2024

  15. [15]

    Can i trust your answer? visually grounded video question answering

    Junbin Xiao, Angela Yao, Yicong Li, and Tat-Seng Chua. Can i trust your answer? visually grounded video question answering. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024. 2

    2024

  16. [16]

    LLaV A- NeXT: A strong zero-shot video understanding model

    Yuanhan Zhang, Bo Li, haotian Liu, Yong jae Lee, Liangke Gui, Di Fu, Jiashi Feng, Ziwei Liu, and Chunyuan Li. LLaV A- NeXT: A strong zero-shot video understanding model. In Online Blog Post, 2024. 3

    2024

  17. [17]

    Large language models are not robust multiple choice selectors

    Chujie Zheng, Hao Zhou, Fandong Meng, Jie Zhou, and Minlie Huang. Large language models are not robust multiple choice selectors. InThe Twelfth International Conference on Learning Representations, 2024. 2

    2024

  18. [18]

    Xing, Hao Zhang, Joseph E

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonza- lez, and Ion Stoica. Judging LLM-as-a-judge with MT-Bench and Chatbot Arena. InAdvances in Neural Information Pro- cessing Systems (NeurIPS), Datasets and Benchmarks Track,