Pith. sign in

REVIEW 3 major objections 5 minor 2 cited by

Video-LevelGauge: Investigating Contextual Positional Bias in Large Video Language Models

T0 review · 3 major / 5 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Large video-language models answer identical probes differently depending on where the probe's content sits in the context, with most open-source models systematically favoring the head, neighboring content, or the middle, while commercial

desk verdict A useful benchmark for a real failure mode, held back by a leaderboard that rests on hand-set weights and no uncertainty quantification. read the letter →

arxiv 2508.19650 v3 pith:QZAIFGHX submitted 2025-08-27 cs.CV

Hou Xia , Zheren Fu , Fangcan Ling , Jiajun Li , Yi Tu , Zhendong Mao , Yongdong Zhang This is my paper
classification cs.CV
keywords positionalbiaslargevideolanguagemodelsunderstandingbenchmarkneedle-in-a-haystackcontextlengthmultimodalinterleavedmodelscalingLVLMevaluation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Video-LevelGauge claims that large video-language models (LVLMs) do not understand a video uniformly: ask the same question about the same clip, and the model's accuracy changes systematically with where that clip is placed in the input context. To establish this, the paper builds a benchmark of 438 videos yielding 1,177 multiple-choice and 120 open-ended probes that are validated to be unanswerable without actually seeing the visual content, inserts each probe at ten positions across several context types (multi-video, long-video, template, interleaved video-text), and measures accuracy at each position relative to standalone accuracy. Across 27 models the result is consistent: nearly every open-source model shows a stable positional signature — head preference, neighbor preference, or U-shaped curves — whereas commercial models, led by Gemini 2.5 Pro, stay essentially flat. The paper further finds that bias intensifies as context length grows, worsens in mixed text-video contexts, and shrinks as model size increases. This matters because existing benchmarks report only overall accuracy and cannot see these localized blind spots.

What carries the argument

The standardized probe — a short curated clip paired with a validated multi-choice question — is the load-bearing object. Each probe is constructed so that text-only models score near chance and single-frame inputs fall far below saturation, meaning accuracy tracks how thoroughly the model perceives the visual content. The probe is inserted at ten normalized positions into a controlled context, and its accuracy at each position is divided by standalone accuracy to form a relative score. Bias is then quantified two ways: statistically (position mean Pmean, range Pran, variance Pvar) and morphologically, by fitting linear and quadratic curves to the relative scores and classifying the shape in

What would settle it

Re-run the 27-model comparison with re-weighted composites (e.g., equal weights, or jackknife over positions) and with perturbed thresholds, and check whether Gemini 2.5 Pro remains the least-biased model and whether the 'stable' classifications persist; then re-test the top and bottom models with probes embedded in single continuous hour-long videos rather than nine-video concatenations and check whether the bias curves reproduce.

Watch

Extended reading notes

Core claim

The paper's central discovery is a measurement: for each of ten positions in a video context, compute a relative score — accuracy when the probe sits there divided by the model's standalone accuracy on the same probe. Positional bias is then the shape of that score curve, summarized by its mean, range, and variance and classified into five morphological patterns (stable, neighbor preference, head preference, lost in the middle, volatile) by polynomial fitting. Applied to 27 models, the curves are systematic and model-specific, not noise: most open-source models favor the head of the context, their neighboring content, or form a U-shape, while the commercial models evaluated stay nearly flat,

Load-bearing premise

The ranking that puts Gemini 2.5 Pro at the least-biased end depends on a composite metric whose weights (4, 3, 3) and shape-classification thresholds were chosen empirically without sensitivity analysis; change those and the ordering could change.

Editorial extensions

If this is right

  • Claims about long-video capability need re-checking: models trained specifically for long video, such as LongVA and LongVILA, still show marked positional bias, so their overall gains may reflect better visual perception rather than uniform sequence comprehension.
  • Because bias grows and changes shape with context length, compressing video tokens should help on two fronts at once — efficiency and positional consistency.
  • Interleaved video-text contexts trigger the worst bias, which recommends training on mixed-modal interleaved data and developing cross-modal context search as concrete mitigation paths.
  • Within a model family, larger variants show flatter bias curves and better handling of 'blind' positions, meaning part of scaling's benefit on long video is positional robustness rather than raw comprehension.
  • Reasoning or thinking modes reduce bias for the models that offer them (GLM-4.5V, MiMo-VL), suggesting test-time reasoning as an immediate, training-free mitigation lever.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • My read: the headline ordering of models rests on empirically chosen composite weights (CM = 4·Pran + 3·(100−Pmean) + 3·Pvar) and fit thresholds (MSE1 ≤ 3, |k| ≤ 0.5, MSE2 ≤ 2); the paper gives no sensitivity analysis, so the specific ranking — Gemini 2.5 Pro least biased — is the part most likely to shift under re-weighting.
  • The bias signatures are measured on concatenated multi-video contexts as the default; if real continuous long videos behave differently because of scene continuity or token deduplication, the magnitudes could change, though the paper reports comparable patterns for long-video and multi-video contexts.
  • The same probe methodology could be turned from a diagnostic into a training objective: penalize within-sequence variance of relative score directly, or ablate whether positional bias is inherited from the base LLM's positional encoding by comparing LVLMs sharing the same language backbone.
  • A testable extension is to subject the same probes to LVLMs with different frame-sampling strategies (uniform, keyframe, search-based) to separate sampling artifacts — the T-Star result suggests search algorithms can themselves introduce positional fluctuation.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. Video-LevelGauge introduces a benchmark for measuring contextual positional bias in large video-language models (LVLMs). The authors construct 438 videos, 1,177 multiple-choice questions, and 120 open-ended descriptive items, then insert each probe at ten positions within customized contexts (multi-video, long-video, template, interleaved video-text, etc.). They define a relative score, three statistical metrics (Pmean, Pran, Pvar), a morphological pattern classifier (MR), and a composite metric (CM). They evaluate 27 open-source and commercial LVLMs and report that commercial models, especially Gemini 2.5 Pro, show less positional bias, while open-source models exhibit head, neighbor, U-shaped, or volatile patterns. Additional experiments analyze the effects of context length, context type, and model scale.

Significance. If the ranking were statistically robust, this would be a valuable diagnostic benchmark and methodology. The probe validation is a clear strength: text-only inputs give near-random accuracy, and accuracy scales with frame count and model size, supporting the claim that the questions require genuine visual perception. The controlled insertion design avoids the common confound of varying question difficulty across positions. The public benchmark and the systematic evaluation of 27 models are also useful contributions. The main weakness is that the headline cross-model ordering is built on hand-set metric weights and thresholds without any uncertainty quantification; the comparative claims currently outrun the evidence.

major comments (3)
  1. [§3.4, Appendix A.5, Table 1] The headline ordering in Fig. 2 relies on CM = 4.0*Pran + 3.0*(100-Pmean) + 3.0*Pvar (Eq. 6), with weights selected empirically after inspection. Table 1 reports point estimates only. With 1,177 MCQA items across 10 positions, each position accuracy has a binomial standard error of about 1.2–1.5 percentage points; a Pran difference of 0.9 between Gemini 2.5 Pro (2.0) and GPT-4o-latest (2.9) is within sampling noise (difference ≈ 0.5 SE). The denominator S_meta is estimated from the same pool, adding correlated noise. Without bootstrap confidence intervals, permutation tests, or a weight-sensitivity sweep, the claim that Gemini 2.5 Pro is the least biased model is not statistically supported. This is load-bearing for the abstract and Fig. 2.
  2. [Appendix A.3, Eq. (4)–(5)] The morphological labels are assigned via thresholds (MSE1 ≤ 3, |k| ≤ 0.5, MSE2 ≤ 2) that are hand-set after viewing the fitted curves. These thresholds are not derived from any principled criterion, and the paper offers no evidence that the five-way classification is stable to small perturbations of the relative scores or to alternative fitting procedures. Because MR labels such as “—” are used to assert that a model has “minimal positional bias” (e.g., InternVL3-9B, GLM-4.5V, GPT-4o, Gemini 2.5 Pro), the classification must be shown to be robust. A threshold sweep should be reported.
  3. [§4.1, Table 1] Task-level conclusions such as “Qwen2.5-VL shows reduced positional bias on the OCR task” are based on Pran values computed from about 185 OCR items per position, giving a per-position standard error of roughly 3 percentage points. The reported OCR Pran of 8.1 for Qwen2.5-VL-7B is therefore indistinguishable from chance-level fluctuation. The exact number of questions used per cell should be stated, and all Pran/Pvar values should be accompanied by confidence intervals, or at least by sample sizes and a multiple-comparison caveat.
minor comments (5)
  1. [Appendix A.1] Typo: “The prompts used for for blind filtering” contains a duplicated “for”.
  2. [Fig. 7] The right-panel axis labels read “P 1 mean”; this should be “Pmean”.
  3. [§4.2, Table 1] The text states “parameter scale of 10.8 billion (108B)”, but 10.8 billion is 10.8B, not 108B. Table 1 lists GLM-4.5V as 108B. Please align the notation.
  4. [Eq. (1)] S_meta is used in the formula but defined only in the following sentence. Define it explicitly in the equation or its caption.
  5. [Fig. 2] The bar chart has no error bars and the plotted numbers are not defined in the caption; the reader must infer that they are CM values from Eq. (6).

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation: benchmark measurements are externally grounded; metric choices are analytic definitions, not fitted predictions.

full rationale

The paper makes no claim to derive a prediction from its inputs. It measures model accuracy on identical standardized probes inserted at different positions and summarizes those measurements. The relative score RSi = Si/Smeta is an explicit normalization defined by Eq. 1, not a parameter fitted to the conclusion. The statistical metrics Pmean, Pran, and Pvar are direct descriptive statistics of the measured response curve, and the composite metric CM = 4.0*Pran + 3.0*(100-Pmean) + 3.0*Pvar is a stated weighted summary whose weights are described as set empirically for comparability between components, not derived from the outcome being ranked. The morphological classification in Appendix A.3 applies explicit thresholds (MSE1 <= 3, |k| <= 0.5, MSE2 <= 2) to the same measured curves; these thresholds define the pattern labels rather than secretly reproducing the headline result. No load-bearing self-citation is present: the cited prior benchmarks and models are external, and no uniqueness theorem or ansatz is imported from the authors' own prior work. The probe construction is externally validated with text-only near-random performance and frame-count scaling (Fig. 5), which is an independent check rather than a circular support. Concerns about arbitrary weights, missing confidence intervals, threshold sensitivity, or the inclusion of the accuracy-like term (100-Pmean) in the composite metric are statistical-robustness and construct-validity questions, not circularity: different weights would yield a different numerical ranking, but that ranking is an explicit summary rule applied to measured data, not a derivation whose conclusion is already embedded in its inputs. Therefore the appropriate circularity finding is no significant circularity.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The paper relies on domain assumptions about the probe-insertion setup and on hand-chosen analysis parameters. It introduces no new physical entities or theoretical constructs. The main ledger content is the empirical freedom in metric aggregation and morphological labeling, which affects the reported model ranking rather than the raw measurements.

free parameters (2)
  • Composite metric weights (alpha1, alpha2, alpha3) = 4.0, 3.0, 3.0
    Appendix A.5: empirically set to ensure each component contributes comparably; no sensitivity analysis is provided, so cross-model rankings depend on these hand-chosen values.
  • Morphological recognition thresholds = MSE1 <= 3, |k| <= 0.5, MSE2 <= 2
    Appendix A.3: hand-chosen thresholds for classifying bias patterns into Stable, Neighbor, Head, U, and Volatile; no justification or stability check is given.
assumptions (4)
  • domain assumption Probe insertion preserves task-relevant content, so accuracy variation across positions isolates positional bias rather than content interaction effects.
    Sections 3.3 and 4.1: probes are inserted into contexts of nine unrelated videos; the paper assumes background content does not interact semantically with probe content in a position-dependent way.
  • domain assumption S_meta, the accuracy when the probe is shown alone, is a stable denominator for cross-model normalization in RS_i = S_i / S_meta.
    Equation (1): no confidence intervals or repeated-run variance are reported for S_meta; models with low S_meta may have noisy relative scores.
  • domain assumption Near-random text-only accuracy and scaling with frames and model size imply the questions are visual and sensitive to positional bias.
    Section 3.2.2, Probe Requirement Validation and Figure 5: validation is performed on only two probe models and does not prove that every question is position-sensitive.
  • domain assumption GPT-4 scoring of open-ended descriptions is a reliable proxy for description quality.
    Section 4.3.4 and Appendix Figure 19: no human agreement or calibration analysis is reported for the evaluator.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Video-LevelGauge: Investigating Contextual Positional Bias in Large Video Language Models." pith.science (2026). https://pith.science/paper/QZAIFGHX

@misc{pith2026250819650,
  author       = {Pith},
  title        = {Pith review of: Video-LevelGauge: Investigating Contextual Positional Bias in Large Video Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/QZAIFGHX}},
  note         = {Machine review of arXiv:2508.19650}
}
read the original abstract

Large video language models (LVLMs) have made notable progress in video understanding, spurring the development of corresponding evaluation benchmarks. However, existing benchmarks generally assess overall performance across entire video sequences, overlooking nuanced behaviors such as contextual positional bias, a critical yet under-explored aspect of LVLM performance. We present Video-LevelGauge, a dedicated benchmark designed to systematically assess positional bias in LVLMs. We employ standardized probes and customized contextual setups, allowing flexible control over context length, probe position, and contextual types to simulate diverse real-world scenarios. In addition, we introduce a comprehensive analysis method that combines statistical measures with morphological pattern recognition to characterize bias. Our benchmark comprises 438 manually curated videos spanning multiple types, yielding 1,177 high-quality multiple-choice questions and 120 open-ended questions, validated for their effectiveness in exposing positional bias. Based on these, we evaluate 27 state-of-the-art LVLMs, including both commercial and open-source models. Our findings reveal significant positional biases in many leading open-source models, typically exhibiting head or neighbor-content preferences. In contrast, commercial models such as Gemini2.5-Pro show impressive, consistent performance across entire video sequences. Further analyses on context length, context variation, and model scale provide actionable insights for mitigating bias and guiding model enhancement . https://github.com/Cola-any/Video-LevelGauge

Figures

Figures reproduced from arXiv: 2508.19650 by the authors.

Figure 1
Figure 1. (a) Large Video Language Models (LVLMs) suffer from [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Performance of state-of-the-art LVLMs on Video￾LevelGauge, evaluated by our composite metric, where higher values indicate more pronounced positional bias. Gemini 2.5 Pro [11] exhibits the least positional bias, followed by GLM￾4.5V [38], GPT-4o-latest [30], Doubao-Seed-1.6 [7], and others. soning and summarization. Notable benchmarks include MVBench [18], TempCompass [24], and MMVU [52] for short videos, as well as… view at source ↗
Figure 3
Figure 3. Overview of Video-LevelGauge, our benchmark for contextual positional bias in video understanding. It adopts a standardized probe and customized context paradigm, where crafted probes are inserted at varying positions within context. (a) Examples of standardized probes on six multi-choice question answering (MCQA) formatted evaluation tasks and one open-ended instructed description task. (b) Four customized context … view at source ↗
Figures from the paper (15 more)
Figure 5
Figure 5. Figure 5: Probe validation. (a) Compared to previous needle QAs [53], our questions yield near-random accuracy when using text-only inputs on Qwen2.5-VL-7B (LVLM) and InternVL3-8B (LVLM*). This indicates that our QAs are highly sensitive to the degree of visual perception, makin…
Figure 6
Figure 6. Figure 6: Effect of context length on positional bias. Each plot illustrates how positional bias manifests across different context lengths on a specific LVLM. The horizontal axis denotes the position of probes in the context, and the vertical axis shows the model’s relative sco…
Figure 7
Figure 7. Figure 7: Effect of context type on positional bias. For each model, we visualize the variation in relative scores across probe positions under six types of context (left), and quantify the severity of positional bias using the composite value of three proposed metrics (right), …
Figure 8
Figure 8. Figure 8: Effect of model size on positional bias. Larger variants exhibit more stable and superior performance across the entire se￾quence than their smaller counterparts. a wide range of capabilities. To investigate the effect of model size on positional bias, we conduct exper…
Figure 9
Figure 9. Figure 9: Illustration of positional bias of Qwen2.5-VL-7B across six video tasks. The horizontal axis represents the probe positions within [PITH_FULL_IMAGE:figures/full_fig_p013_9.png]
Figure 10
Figure 10. Figure 10: Effect of thinking mode on positional bias. Thinking [PITH_FULL_IMAGE:figures/full_fig_p013_10.png]
Figure 11
Figure 11. Figure 11: Examples of probes for six MCQA formatted evaluation tasks and one open-ended instructed description task. Each question is [PITH_FULL_IMAGE:figures/full_fig_p014_11.png]
Figure 12
Figure 12. Figure 12: Examples of probes for six MCQA formatted evaluation tasks and one open-ended instructed description task. Each question is [PITH_FULL_IMAGE:figures/full_fig_p014_12.png]
Figure 13
Figure 13. Figure 13: The collected videos are captioned frame-wise at 1 FPS using GPT-4o with this prompt. We instruct the model to generate [PITH_FULL_IMAGE:figures/full_fig_p015_13.png]
Figure 14
Figure 14. Figure 14: Based on detailed video captions and human-annotated protocols, GPT-4 is instructed, using this prompt, to generate 2–3 task [PITH_FULL_IMAGE:figures/full_fig_p016_14.png]
Figure 15
Figure 15. Figure 15: During QA refinement, the generated questions are submitted to GPT-4 for blind answering using the Blind Answer Prompt. [PITH_FULL_IMAGE:figures/full_fig_p017_15.png]
Figure 16
Figure 16. Figure 16: For MCQAs, we input the verified question-answer pairs into GPT-4 and use a specific prompt to generate three distractors. [PITH_FULL_IMAGE:figures/full_fig_p018_16.png]
Figure 17
Figure 17. Figure 17: Task definitions for six video understanding tasks. Additionally, descriptive questions are formulated as instructions that prompt [PITH_FULL_IMAGE:figures/full_fig_p018_17.png]
Figure 18
Figure 18. Figure 18: Three representative examples of filtered question–answer pairs are presented. The first example illustrates a question an [PITH_FULL_IMAGE:figures/full_fig_p019_18.png]
Figure 19
Figure 19. Figure 19: We use GPT-4 to evaluate model generated open-ended outputs by comparing them against annotations. [PITH_FULL_IMAGE:figures/full_fig_p019_19.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. M$^3$Eval: Multi-Modal Memory Evaluation through Cognitively-Grounded Video Tasks

    cs.CV 2026-06 unverdicted novelty 7.0 of 10

    M³Eval is a new cognitively-grounded benchmark that evaluates memory dimensions in multi-modal video models and reports consistent model weaknesses in disentanglement, interference, spatial-temporal grounding, and sym...

  2. DynaTok: Temporally Adaptive and Positional Bias-Aware Token Compression for Video-LLMs

    cs.CV 2026-05 unverdicted novelty 6.0 of 10

    DynaTok introduces temporally adaptive budget allocation with EMA memory and spatial selection with memory to compress video tokens, retaining over 95% accuracy at 90% reduction on VideoQA benchmarks.

Reference graph

Works this paper leans on

59 extracted references · 47 canonical work pages · cited by 2 Pith papers

  1. [1]

    Pixtral 12b

    Pravesh Agrawal, Szymon Antoniak, Emma Bou Hanna, Baptiste Bout, Devendra Chaplot, Jessica Chudnovsky, Diogo Costa, Baudouin De Monicault, Saurabh Garg, Theophile Gervet, et al. Pixtral 12b. arXiv preprint arXiv:2410.07073, 2024. 5

    arXiv 2024

  2. [2]

    Alibaba. Qvq-max. https://qwenlm.github.io/ zh/blog/qvq-max-preview/. 2025. 5, 6

    work page 2025

  3. [3]

    Claude-sonnet-4

    Anthropic. Claude-sonnet-4. https : / / www . anthropic.com/claude/sonnet. 2025. 6

    work page 2025

  4. [4]

    Minigpt4-video: Advancing multimodal llms for video understanding with interleaved visual-textual tokens,

    Kirolos Ataallah, Xiaoqian Shen, Eslam Abdelrahman, Es- sam Sleiman, Deyao Zhu, Jian Ding, and Mohamed El- hoseiny. Minigpt4-video: Advancing multimodal llms for video understanding with interleaved visual-textual tokens,

    work page

  5. [5]

    Qwen-vl: A versatile vision-language model for un- derstanding, localization, text reading, and beyond, 2023

    Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A versatile vision-language model for un- derstanding, localization, text reading, and beyond, 2023. 5, 6

    work page 2023

  6. [6]

    Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al. Qwen2. 5-vl technical report. arXiv preprint arXiv:2502.13923, 2025. 2, 5, 6, 7, 8, 13

    arXiv 2025

  7. [7]

    Doubao-seed-1.6

    Bytedance. Doubao-seed-1.6. https : / / seed . bytedance.com/zh/seed1_6. 2025. 2, 5, 6

    work page 2025

  8. [8]

    Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Jiaqi Wang, Yu Qiao, Dahua Lin, and Feng Zhao. Are we on the right way for evaluating large vision-language models? In Proceedings of the 38th Conference on Neural Information Processing Sys- tems (NeurIPS 2024), page 94237, 2024. 4

    work page 2024

Show all 59 references
  1. [9]

    LongVILA: Scaling long-context visual language models for long videos

    Yukang Chen, Fuzhao Xue, Dacheng Li, Qinghao Hu, Ligeng Zhu, Xiuyu Li, Yunhao Fang, Haotian Tang, Shang Yang, Zhijian Liu, Yihui He, Hongxu Yin, Pavlo Molchanov, Jan Kautz, Linxi Fan, Yuke Zhu, Yao Lu, and Song Han. LongVILA: Scaling long-context visual language models for lon...

    2025

  2. [10]

    Videollama 2: Advancing spatial-temporal modeling and audio understanding in video- llms, 2024

    Zesen Cheng, Sicong Leng, Hang Zhang, Yifei Xin, Xin Li, Guanzheng Chen, Yongxin Zhu, Wenqi Zhang, Ziyang Luo, Deli Zhao, and Lidong Bing. Videollama 2: Advancing spatial-temporal modeling and audio understanding in video- llms, 2024. 6

    2024

  3. [11]

    Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic ca- pabilities, 2025

    Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Asya Fadeeva, Zach Gleicher, Thang Luong, and Niket Kumar Bhumihar. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation age...

    2025

  4. [12]

    Video- mme: The first-ever comprehensive evaluation benchmark of multi-modal llms in video analysis

    Chaoyou Fu, Yuhan Dai, Yongdong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang, Chenyu Zhou, Yunhang Shen, Mengdan Zhang, Peixian Chen, Yanwei Li, Shao- hui Lin, Sirui Zhao, Ke Li, Tong Xu, Xiawu Zheng, En- hong Chen, Caifeng Shan, Ran He, and Xing Sun. Video- mme: The fir...

    2025

  5. [13]

    Ego4d: Around the world in 3,000 hours of egocentric video

    Kristen Grauman, Andrew Westbury, Eugene Byrne, Zachary Chavis, Antonino Furnari, Rohit Girdhar, Jackson Hamburger, Hao Jiang, Miao Liu, Xingyu Liu, et al. Ego4d: Around the world in 3,000 hours of egocentric video. In Proceedings of the IEEE/CVF Conference on Computer Vi- sio...

    2022

  6. [14]

    Ma-lmm: Memory-augmented large multimodal model for long-term video understanding

    Bo He, Hengduo Li, Young Kyun Jang, Menglin Jia, Xue- fei Cao, Ashish Shah, Abhinav Shrivastava, and Ser-Nam Lim. Ma-lmm: Memory-augmented large multimodal model for long-term video understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogni...

    2024

  7. [15]

    Scaling laws for neural language models

    Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361,

    2001 arXiv

  8. [16]

    Rush, Douwe Kiela, Matthieu Cord, and Victor Sanh

    Hugo Laurenc ¸on, Lucile Saulnier, L´eo Tronchon, Stas Bek- man, Amanpreet Singh, Anton Lozhkov, Thomas Wang, Siddharth Karamcheti, Alexander M. Rush, Douwe Kiela, Matthieu Cord, and Victor Sanh. Obelics: An open web- scale filtered dataset of interleaved image-text documents,

  9. [17]

    LLaV A-onevision: Easy visual task transfer

    Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Zi- wei Liu, and Chunyuan Li. LLaV A-onevision: Easy visual task transfer. Transactions on Machine Learning Research,

  10. [18]

    Mvbench: A comprehensive multi-modal video understand- ing benchmark

    Kunchang Li, Yali Wang, Yinan He, Yizhuo Li, Yi Wang, Yi Liu, Zun Wang, Jilan Xu, Guo Chen, Ping Luo, et al. Mvbench: A comprehensive multi-modal video understand- ing benchmark. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22195–...

    2024

  11. [19]

    Llama-vid: An image is worth 2 tokens in large language models

    Yanwei Li, Chengyao Wang, and Jiaya Jia. Llama-vid: An image is worth 2 tokens in large language models. In European Conference on Computer Vision, pages 323–340. Springer, 2024. 2, 6

    2024

  12. [20]

    Video-LLaV A: Learning united visual rep- 9 resentation by alignment before projection

    Bin Lin, Yang Ye, Bin Zhu, Jiaxi Cui, Munan Ning, Peng Jin, and Li Yuan. Video-LLaV A: Learning united visual rep- 9 resentation by alignment before projection. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 5971–5984, ...

    2024

  13. [21]

    Kangaroo: A powerful video-language model supporting long-context video input

    Jiajun Liu, Yibing Wang, Hanghang Ma, Xiaoping Wu, Xi- aoqi Ma, xiaoming Wei, Jianbin Jiao, Enhua Wu, and Jie Hu. Kangaroo: A powerful video-language model supporting long-context video input. arXiv preprint arXiv:2408.15542,

    arXiv

  14. [22]

    Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang

    Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. Lost in the middle: How language models use long contexts. Trans- actions of the Association for Computational Linguistics, 12: 157–173, 2024. 2, 5

    2024

  15. [23]

    Scaling laws of rope-based extrapola- tion, 2024

    Xiaoran Liu, Hang Yan, Shuo Zhang, Chenxin An, Xipeng Qiu, and Dahua Lin. Scaling laws of rope-based extrapola- tion, 2024. 6

    2024

  16. [24]

    TempCom- pass: Do video LLMs really understand videos? In Findings of the Association for Computational Linguistics: ACL 2024, pages 8731–8772, Bangkok, Thailand, 2024

    Yuanxin Liu, Shicheng Li, Yi Liu, Yuxiang Wang, Shuhuai Ren, Lei Li, Sishuo Chen, Xu Sun, and Lu Hou. TempCom- pass: Do video LLMs really understand videos? In Findings of the Association for Computational Linguistics: ACL 2024, pages 8731–8772, Bangkok, Thailand, 2024. Associ...

    2024

  17. [25]

    Nvila: Efficient frontier visual lan- guage models

    Zhijian Liu, Ligeng Zhu, Baifeng Shi, Zhuoyang Zhang, Yuming Lou, Shang Yang, Haocheng Xi, Shiyi Cao, Yuxian Gu, Dacheng Li, et al. Nvila: Efficient frontier visual lan- guage models. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 4122–4134, 2025. 6

    2025

  18. [26]

    Nee- dle in a video haystack: A scalable synthetic evaluator for video mllms

    Haoyu Lu, Yuqi Huo, Yifan Du, Zijia Zhao, Longteng Guo, Bingning Wang, Weipeng Chen, and Jing Liu. Nee- dle in a video haystack: A scalable synthetic evaluator for video mllms. In International Conference on Representation Learning, pages 99750–99782, 2025. 2, 3

    2025

  19. [27]

    Video-chatgpt: Towards detailed video un- derstanding via large vision and language models

    Muhammad Maaz, Hanoona Rasheed, Salman Khan, and Fahad Khan. Video-chatgpt: Towards detailed video un- derstanding via large vision and language models. In Pro- ceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages...

    2024

  20. [28]

    The serial position effect of free recall

    Bennet B Murdock Jr. The serial position effect of free recall. Journal of experimental psychology, 64(5):482, 1962. 2

    1962

  21. [29]

    Needle in the haystack for memory based large language models

    Elliot Nelson, Georgios Kollias, Payel Das, Subhajit Chaud- hury, and Soham Dan. Needle in the haystack for memory based large language models. arXiv preprint arXiv:2407.01437, 2024. 2, 3

    2024 arXiv

  22. [30]

    OpenAI. Gpt-4o. https://openai.com/zh-Hans- CN/index/hello-gpt-4o/. 2025. 1, 2, 4, 5, 6

    2025

  23. [31]

    Gpt-4 technical report, 2024

    OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, et al. Gpt-4 technical report, 2024. 6, 8

    2024

  24. [32]

    Too many frames, not all useful: Efficient strategies for long- form video QA

    Jongwoo Park, Kanchana Ranasinghe, Kumara Kahatapitiya, Wonjeong Ryu, Donghyun Kim, and Michael S Ryoo. Too many frames, not all useful: Efficient strategies for long- form video QA. In Workshop on Video-Language Models @ NeurIPS 2024, 2025. 2

    2024

  25. [33]

    Video-xl-2: Towards very long-video under- standing through task-aware kv sparsification.arXiv preprint arXiv:2506.19225, 2025

    Minghao Qin, Xiangrui Liu, Zhengyang Liang, Yan Shu, Huaying Yuan, Juenjie Zhou, Shitao Xiao, Bo Zhao, and Zheng Liu. Video-xl-2: Towards very long-video under- standing through task-aware kv sparsification.arXiv preprint arXiv:2506.19225, 2025. 2, 5, 6, 7, 8

    2025 arXiv

  26. [34]

    Video-xl: Extra-long vision language model for hour-scale video understanding, 2024

    Yan Shu, Zheng Liu, Peitian Zhang, Minghao Qin, Jun- jie Zhou, Zhengyang Liang, Tiejun Huang, and Bo Zhao. Video-xl: Extra-long vision language model for hour-scale video understanding, 2024. 6

    2024

  27. [35]

    Moviechat: From dense token to sparse memory for long video understanding

    Enxin Song, Wenhao Chai, Guanhong Wang, Yucheng Zhang, Haoyang Zhou, Feiyang Wu, Haozhe Chi, Xun Guo, Tian Ye, Yanting Zhang, et al. Moviechat: From dense token to sparse memory for long video understanding. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pa...

    2024

  28. [36]

    Real-world anomaly detection in surveillance videos

    Waqas Sultani, Chen Chen, and Mubarak Shah. Real-world anomaly detection in surveillance videos. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 6479–6488. IEEE, 2018. 4

    2018

  29. [37]

    Mimo-vl technical report, 2025

    Core Team, Zihao Yue, Zhenru Lin, Yifan Song, Weikun Wang, Shuhuai Ren, Shuhao Gu, Shicheng Li, Peidian Li, Liang Zhao, et al. Mimo-vl technical report, 2025. 2, 5, 6

    2025

  30. [38]

    Glm-4.5v and glm-4.1v-thinking: Towards versatile multimodal reasoning with scalable reinforcement learning, 2025

    V Team, Wenyi Hong, Wenmeng Yu, Xiaotao Gu, Guo Wang, Guobing Gan, Haomiao Tang, Jiale Cheng, Ji Qi, Junhui Ji, et al. Glm-4.5v and glm-4.1v-thinking: Towards versatile multimodal reasoning with scalable reinforcement learning, 2025. 2, 5, 6

    2025

  31. [39]

    Multimodal needle in a haystack: Benchmarking long-context capability of multimodal large language models

    Hengyi Wang, Haizhou Shi, Shiwei Tan, Weiyi Qin, Wenyuan Wang, Tunyu Zhang, Akshay Nambi, Tanuja Ganu, and Hao Wang. Multimodal needle in a haystack: Benchmarking long-context capability of multimodal large language models. In Proceedings of the 2025 Conference of the Nations ...

    2025

  32. [40]

    Lvbench: An extreme long video understanding benchmark, 2024

    Weihan Wang, Zehai He, Wenyi Hong, Yean Cheng, Xiao- han Zhang, Ji Qi, Xiaotao Gu, Shiyu Huang, Bin Xu, Yuxiao Dong, Ming Ding, and Jie Tang. Lvbench: An extreme long video understanding benchmark, 2024. 3, 7, 12

    2024

  33. [41]

    Needle in a multimodal haystack

    Weiyun Wang, Shuibo Zhang, Yiming Ren, Yuchen Duan, Tiantong Li, Shuo Liu, Mengkang Hu, Zhe Chen, Kaipeng Zhang, Lewei Lu, et al. Needle in a multimodal haystack. Advances in Neural Information Processing Systems , 37: 20540–20565, 2024. 3

    2024

  34. [42]

    Longvideobench: A benchmark for long-context interleaved video-language understanding

    Haoning Wu, Dongxu Li, Bei Chen, and Junnan Li. Longvideobench: A benchmark for long-context interleaved video-language understanding. Advances in Neural Informa- tion Processing Systems, 37:28828–28857, 2024. 2, 3

    2024

  35. [43]

    Mit- igating object hallucination via concentric causal atten- tion

    Yun Xing, Yiheng Li, Ivan Laptev, and Shijian Lu. Mit- igating object hallucination via concentric causal atten- tion. In Advances in Neural Information Processing Systems (NeurIPS) 37, page 96152, 2024. Poster Presentation. 5

    2024

  36. [44]

    Qwen3 technical report, 2025

    An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, 10 Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jia- long Tang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang...

    2025

  37. [45]

    Re-thinking temporal search for long-form video understanding

    Jinhui Ye, Zihan Wang, Haosen Sun, Keshigeyan Chan- drasegaran, Zane Durante, Cristobal Eyzaguirre, Yonatan Bisk, Juan Carlos Niebles, Ehsan Adeli, Li Fei-Fei, Jia- jun Wu, and Manling Li. Re-thinking temporal search for long-form video understanding. In Proceedings of the Com...

    2025

  38. [46]

    Lv-eval: A balanced long-context bench- mark with 5 length levels up to 256k

    Tao Yuan, Xuefei Ning, Dong Zhou, Zhijie Yang, Shiyao Li, Minghui Zhuang, Zheyue Tan, Zhuyu Yao, Dahua Lin, Boxun Li, et al. Lv-eval: A balanced long-context bench- mark with 5 length levels up to 256k. arXiv preprint arXiv:2402.05136, 2024. 3

    2024

  39. [47]

    Videorefer suite: Advancing spatial- temporal object understanding with video llm

    Yuqian Yuan, Hang Zhang, Wentong Li, Zesen Cheng, Bo- qiang Zhang, Long Li, Xin Li, Deli Zhao, Wenqiao Zhang, Yueting Zhuang, et al. Videorefer suite: Advancing spatial- temporal object understanding with video llm. In Proceed- ings of the Computer Vision and Pattern Recogniti...

    2025

  40. [48]

    Videollama 3: Frontier multi- modal foundation models for image and video understand- ing

    Boqiang Zhang, Kehan Li, Zesen Cheng, Zhiqiang Hu, Yuqian Yuan, Guanzheng Chen, Sicong Leng, Yuming Jiang, Hang Zhang, Xin Li, et al. Videollama 3: Frontier multi- modal foundation models for image and video understand- ing. arXiv preprint arXiv:2501.13106, 2025. 1, 5, 6

    2025 arXiv

  41. [49]

    A simple LLM framework for long-range video question-answering

    Ce Zhang, Taixi Lu, Md Mohaiminul Islam, Ziyang Wang, Shoubin Yu, Mohit Bansal, and Gedas Bertasius. A simple LLM framework for long-range video question-answering. In Proceedings of the 2024 Conference on Empirical Meth- ods in Natural Language Processing , pages 21715–21737,...

    2024

  42. [50]

    Long context transfer from language to vision

    Peiyuan Zhang, Kaichen Zhang, Bo Li, Guangtao Zeng, Jingkang Yang, Yuanhan Zhang, Ziyue Wang, Haoran Tan, Chunyuan Li, and Ziwei Liu. Long context transfer from language to vision. Transactions on Machine Learning Re- search, 2025. 2, 3, 5, 6, 7

    2025

  43. [51]

    Video instruction tuning with synthetic data

    Yuanhan Zhang, Jinming Wu, Wei Li, Bo Li, Zejun Ma, Zi- wei Liu, and Chunyuan Li. Video instruction tuning with synthetic data. arXiv preprint arXiv:2410.02713, 2024. 1, 5, 6, 8

    2024 arXiv

  44. [52]

    Mmvu: Measuring expert-level multi- discipline video understanding

    Yilun Zhao, Haowei Zhang, Lujing Xie, Tongyan Hu, Guo Gan, Yitao Long, Zhiyuan Hu, Weiyuan Chen, Chuhan Li, Zhijian Xu, et al. Mmvu: Measuring expert-level multi- discipline video understanding. In Proceedings of the Com- puter Vision and Pattern Recognition Conference , pages...

    2025

  45. [53]

    Mlvu: Benchmarking multi-task long video understanding

    Junjie Zhou, Yan Shu, Bo Zhao, Boya Wu, Zhengyang Liang, Shitao Xiao, Minghao Qin, Xi Yang, Yongping Xiong, Bo Zhang, Tiejun Huang, and Zheng Liu. Mlvu: Benchmarking multi-task long video understanding. In Pro- ceedings of the Computer Vision and Pattern Recognition Conference...

    2025

  46. [54]

    Internvl3: Exploring advanced training and test-time recipes for open-source multimodal models

    Jinguo Zhu, Weiyun Wang, Zhe Chen, Zhaoyang Liu, Shen- glong Ye, Lixin Gu, Hao Tian, Yuchen Duan, Weijie Su, Jie Shao, et al. Internvl3: Exploring advanced training and test-time recipes for open-source multimodal models. arXiv preprint arXiv:2504.10479, 2025. 2, 5, 6, 8, 13

    2025 arXiv

  47. [55]

    Detection and tracking meet drones challenge

    Pengfei Zhu, Longyin Wen, Dawei Du, Xiao Bian, Heng Fan, Qinghua Hu, and Haibin Ling. Detection and tracking meet drones challenge. IEEE Transactions on Pattern Anal- ysis and Machine Intelligence, 44(11):7380–7399, 2021. 4

    2021

  48. [56]

    Hlv-1k: A large-scale hour-long video benchmark for time- specific long video understanding, 2025

    Heqing Zou, Tianze Luo, Guiyang Xie, Victor Xiao Jie Zhang, Fengmao Lv, Guangcong Wang, Junyang Chen, Zhuochen Wang, Hansheng Zhang, and Huaijian Zhang. Hlv-1k: A large-scale hour-long video benchmark for time- specific long video understanding, 2025. 3 11 A. Appendix The Appe...

    2025

  49. [57]

    three chairs,

    Detailed Object Description Identify all distinct objects present (e.g., people, furniture, natural elements, text, etc.). For each object, specify: • Quantification: Exact or approximate number (e.g., "three chairs," "a single tree"). • Physical properties: Color (e.g., "maro...

  50. [58]

    the lamp is on top of the desk,

    Relationships Between Objects • Spatial relationships: Clearly define positions relative to one another (e.g., "the lamp is on top of the desk," "the person is standing to the left of the door," "the cup is beside the book"). Include distance cues where observable (e.g., "clos...

  51. [59]

    nearby" can be replaced with

    Language and Objectivity • Use clear, fluent, and precise natural language. Avoid ambiguous terms (e.g., "nearby" can be replaced with "immediately next to" if applicable). • Strictly base descriptions on what is visually verifiable. Do not include guesses about intent, contex...

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.