Pith. sign in

REVIEW 3 major objections 5 minor 93 references

The paper claims that current multimodal large language models, despite strong general video understanding, remain far below human performance at qualitative spatial-temporal reasoning from continuous visual cues, with the best model scorin

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 09:05 UTC pith:EKWVLUF5

load-bearing objection A carefully built dynamic-spatial-reasoning benchmark whose useful diagnostic numbers are undercut by an undocumented human baseline. the 3 major comments →

arxiv 2607.20868 v1 pith:EKWVLUF5 submitted 2026-07-23 cs.CV

ViSTR-Bench: Can MLLMs Reason from Continuous Visual Cues in Dynamic Scenes?

Han Li , Si Liu , Zehao Huang , Dongxin Lyu , Longfei Xu , Jiahui Fu , Daxin Tian , Yuliang Xiu
show 1 more author
Naiyan Wang
This is my paper
classification cs.CV
keywords multimodal large language modelsspatial-temporal reasoningvideo question answeringbenchmarkdynamic scenesqualitative evaluationoutcome predictionphysical dynamics
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

ViSTR-Bench asks whether multimodal large language models (MLLMs) can answer simple binary questions about dynamic scenes—whether a vehicle is moving faster, whether a basketball shot will go in, whether a Jenga tower will fall—using only the visual evidence available before the outcome is shown. The paper's central empirical claim is that current models largely cannot: the best system reaches 62.0% overall accuracy against a 50% random baseline and 57.9% frequency baseline, while human annotators score 91.0%. The paper argues this gap is not about recognizing objects or reading static geometry but about tracking temporal evidence, estimating motion states, extrapolating outcomes, and inferring latent physical dependencies. A sympathetic reader would care because these are the exact abilities needed for embodied AI, autonomous driving, and robotics, and because the benchmark is designed to be diagnostic rather than just competitive.

Core claim

ViSTR-Bench is a 1,340-item video question-answer benchmark organized into four reasoning dimensions—Motion Perception, Spatial Relations, Outcome Prediction, and Physical Dynamics—spanning 15 binary-choice subtasks. All items are qualitative (Yes/No or two-option choices), and for outcome tasks the videos are truncated at a manually chosen decision point so the answer cannot be read off the final frame. Evaluations across proprietary, open-source, and specialized spatial MLLMs show that the best model, GPT-5.4-thinking, reaches 62.0%, only 4.1 points above the frequency-based chance baseline and 29.0 points below the 91.0% human score. Error analysis of 600 incorrect predictions attributes

What carries the argument

The load-bearing instrument is the benchmark's construction pipeline: event localization to isolate single reasoning episodes, visual prompting (bounding boxes) to ground targets, outcome truncation at manually chosen decision points, and a four-criteria human quality control (visibility, temporal sufficiency, objectivity, non-triviality). This pipeline converts raw videos into binary qualitative questions that can only be answered by aggregating temporal evidence, and it defines the error taxonomy used to diagnose failures.

Load-bearing premise

The benchmark's central gap claim rests on the assumption that every truncated outcome-prediction and physical-dynamics clip has an unambiguous ground-truth answer that can be inferred from the retained pre-outcome footage; the paper does not report inter-annotator agreement, discarded-sample counts, or independent checks against label leakage (Sec. III-B, 'Outcome Truncation' and 'Human Quality Control').

What would settle it

Re-annotate a random sample of the outcome-prediction items by showing only the truncated pre-outcome clips to a fresh set of annotators and measuring agreement with the published labels; if agreement is below roughly 95% or if annotators cannot confidently infer the answers, the benchmark's 29-point human-model gap would be an artifact of label construction rather than a measure of reasoning ability.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • If the reported gap is accurate, current MLLMs cannot be relied upon for tasks that require anticipating physical outcomes from partial observations, such as driving or manipulation planning.
  • The failure distribution implies that improving target tracking and motion-state estimation is a more urgent bottleneck than improving object recognition.
  • Because specialized spatial MLLMs do not outperform general-purpose ones on ViSTR-Bench, static or geometry-centric spatial training does not transfer to dynamic reasoning.
  • Text-based chain-of-thought prompting is not sufficient to close the gap; gains are small and task-dependent.
  • Providing explicit task-relevant evidence (novel views, optical flow summaries) improves accuracy on specific tasks, suggesting input-centric and tool-augmented directions are promising.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the 29-point gap is real, then a model that couples a low-level tracker (optical flow, object permanence) with a language model might approach human performance on this benchmark without any new reasoning architecture—implying the bottleneck is perceptual evidence, not inference.
  • The benchmark's binary format may underestimate models that have partial knowledge; converting it to graded confidence or open-ended justification could separate 'knows the gist' from 'commits to the right answer.'
  • The truncation protocol suggests a natural training scheme: sample videos, truncate before outcomes, and supervise the model to predict the truncated outcome—this could produce a scalable self-supervised objective for temporal reasoning.
  • Because human accuracy reaches 100% on several subtasks (e.g., Rotation Direction, Interaction Direction, Fall Direction), those items may be easier than the average; a per-subtask analysis of where humans are also imperfect could refine the benchmark's difficulty calibration.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. This paper introduces ViSTR-Bench, a video question-answering benchmark with 1,340 binary-choice items across 15 subtasks organized into four dimensions: Motion Perception, Spatial Relations, Outcome Prediction, and Physical Dynamics. The authors collect videos from public datasets, web sources, and self-recorded clips; apply event localization, visual prompting, and outcome truncation; and then evaluate a broad set of proprietary, open-source, and specialized spatial MLLMs. The headline result is that the best model, GPT-5.4-thinking, achieves 62.0% overall accuracy versus 57.9% for the frequency-based chance baseline and 91.0% for human performance, yielding a 29.0-point human-model gap. The paper also studies text-based chain-of-thought prompting, visual input formats, a six-way error taxonomy, and pilot input/tool augmentation strategies.

Significance. If the benchmark and the human baseline are valid, ViSTR-Bench fills a real gap by focusing on qualitative, reasoning-oriented spatial-temporal understanding rather than static spatial attributes or quantitative prediction. The paper has several concrete strengths: the subtask counts sum exactly to 1,340; the reported weighted human mean matches the stated 91.0%; the inclusion of both random and frequency baselines is helpful; the benchmark is externally constructed with no fitted parameters; and the error analysis and pilot studies provide useful diagnostic signal. The central claim, however, depends critically on two pieces of evidence that are currently under-reported: the human evaluation protocol and the reliability of the outcome-truncation labels. Without those, the 29-point gap could be an artifact of information leakage rather than a measure of spatial-temporal reasoning ability.

major comments (3)
  1. [Sec. IV-A and Sec. IV-B (Table II)] The human evaluation is described in a single sentence: 'we conduct human evaluation to estimate human-level performance.' There is no annotator count, no description of instructions, no statement about whether evaluators saw the same truncated clips as the models, no blinding procedure, and no inter-annotator agreement. The headline 29.0-point gap (62.0% vs. 91.0%) is load-bearing for the paper's main claim. If the human evaluators saw full videos, or were the same expert annotators who selected the truncation points with knowledge of the outcome, the 91.0% figure is inflated and the gap is not a fair measure. The authors should report a full protocol and inter-annotator agreement, and ideally run a blind evaluation on the identical truncated inputs.
  2. [Sec. III-B, 'Outcome Truncation' and 'Human Quality Control'] The manual decision point for truncation is chosen by annotators who know the full-video outcome, and the quality-control stage reports no count of discarded samples, no inter-annotator agreement, and no independent check that the retained pre-outcome clip makes the ground-truth answer unambiguous. For outcome-prediction and physical-dynamics tasks, subtle outcome-revealing cues (e.g., a player's reaction, a ball's curve, a Jenga tower's tilt) may leak into the retained prefix. This is not circularity, but it is a label-validity risk. The authors should report the discard rate, have a separate group of annotators label the truncated clips blind to the outcome, and quantify agreement.
  3. [Table II and Sec. IV-B] No confidence intervals or significance tests are reported. Several subtasks have very small sample sizes (Knot Type n=29, Golf Shot n=53, Fall Direction n=46, Passage Feasibility n=55), and many per-subtask model accuracies are within a few points of 50%, so the per-subtask ranking claims are not statistically distinguishable. Even the overall claim that 'only three evaluated models outperform the frequency-based baseline' needs interval estimates; with n=1,340 the top-model difference of 4.1 points is likely significant, but the paper should demonstrate this rather than assert it. At minimum, report Clopper-Pearson intervals or bootstrap CIs for overall and per-subtask accuracy, and test against the frequency baseline.
minor comments (5)
  1. [Fig. 1] The figure panel labels 'Human Gap: 36.0%' and 'X' are confusing, and the 36.0% value does not match the 29.0-point gap quoted in Sec. IV-B. Please clarify whether the figure refers to a per-example illustration and correct the inconsistency.
  2. [Sec. III-B] The term 'expert annotators' is used without defining who they are, how many participated, or what expertise they had. Please provide this information, especially given the emphasis on 'objectivity' and 'sufficiency.'
  3. [Sec. IV-D] The diagnostic study on 20 Basketball Shot samples is labeled 'small,' which is appropriate, but the numbers should be presented with uncertainty (e.g., exact binomial CIs) and framed as anecdotal rather than as evidence of a general pattern.
  4. [Sec. III-B, 'Release and Leaderboard Protocol'] The paper reports results on the complete 1,340-item set while planning to release only 50% publicly. Please clarify that no model selection or hyperparameter tuning was performed on the private held-out portion, so that the leaderboard protocol cannot be gamed by information leakage from the public split.
  5. [Abstract] Minor grammar: 'a comprehensive four-dimensional evaluations' should be 'a comprehensive four-dimensional evaluation framework'; 'VisualSpatial-TemporalReasoningBenchmark' also needs spacing in the introduction.

Circularity Check

0 steps flagged

No significant circularity: ViSTR-Bench is an external benchmark with no fitted parameters or derivation chain; the 62.0% vs. 91.0% gap is an empirical measurement, not a result reduced from its own inputs.

full rationale

This paper introduces a new benchmark and reports measured performance of many third-party MLLMs against a human baseline. There is no derivation chain, no fitted parameter subsequently called a prediction, and no uniqueness theorem imported from the authors' prior work. The central claim (best model 62.0% vs. human 91.0%) is an empirical measurement over 1,340 manually annotated QA pairs and a large set of external models; the claim is not used to construct the labels. The only author-overlap items, GeoThinker and SpatialLadder, are evaluated as test subjects and perform poorly (52.8% and 47.8%), which is the opposite of what a self-serving construction would produce; they provide no evidence for the benchmark's validity. The legitimate concerns raised by the paper — undocumented human evaluation protocol, no inter-annotator agreement, no count of discarded samples, truncation decision points chosen by annotators who knew the full-video outcome, and no explicit leakage check for 'Outcome Truncation' samples — are measurement-validity and correctness risks, not circularity as defined here. To flag circularity I would need to exhibit a specific reduction (e.g., the human figure being derived from the construction criterion or the same annotators being re-used as the human baseline), which the paper's text does not establish. The paper even reports a small diagnostic study in Sec. IV-D where human accuracy rises monotonically with more video exposure while the tested model does not, which is consistent with the benchmark measuring information usage rather than a self-confirming loop. Under the hard rule that non-findings are expected and appropriate for benchmark/evaluation papers, I assign score 0.

Axiom & Free-Parameter Ledger

0 free parameters · 4 axioms · 0 invented entities

ViSTR-Bench is an empirical benchmark, so the ledger contains no fitted parameters and no invented physical entities. The load-bearing assumptions are about label validity, human annotation reliability, and the comparability of input formats across models.

axioms (4)
  • domain assumption Ground-truth labels for truncated outcome-prediction and physical-dynamics tasks are unambiguous and inferable from the retained pre-outcome clip.
    Invoked in Sec. III-B 'Outcome Truncation' and 'Human Quality Control'; no inter-annotator agreement or label-leakage check is reported.
  • domain assumption Expert annotators' non-triviality filtering reliably removes samples solvable from a single static frame or language priors.
    Invoked in Sec. III-B 'Human Quality Control'; the criteria are subjective and no discarded-sample statistics are given.
  • ad hoc to paper Uniform 16-frame sampling for image-only models preserves enough temporal evidence for fair comparison with native video models.
    Introduced in Sec. IV-A 'Input Protocol'; the paper later shows ordered frames barely outperform shuffled frames, so this assumption is active and possibly optimistic.
  • domain assumption The reported human evaluation (91.0%) is a representative upper bound on the benchmark.
    Stated in Sec. IV-A 'Baselines and Human Evaluation'; annotator count, expertise, and instructions are not specified.

pith-pipeline@v1.3.0-alltime-deepseek · 43381 in / 9122 out tokens · 96627 ms · 2026-08-01T09:05:11.369230+00:00 · methodology

0 comments
read the original abstract

Multimodal Large Language Models (MLLMs) have achieved remarkable success across diverse expert-level tasks, but they still struggle with fundamental abilities that humans naturally develop through continuous observation of the real world, such as spatial perception and dynamic reasoning. Recent studies have recognized this gap and introduced dedicated benchmarks to evaluate the spatial-temporal capabilities of MLLMs. However, existing benchmarks mostly focus on static scenes or require exact quantitative predictions, leaving intuitive reasoning from temporal cues largely underexplored. In this paper, we introduce the Visual Spatial-Temporal Reasoning Benchmark (ViSTR-Bench), a novel evaluation suite designed to systematically assess whether MLLMs can perform qualitative reasoning from continuous visual cues in dynamic scenes. Guided by the principles of temporal emphasis, reasoning orientation, and qualitative evaluation, ViSTR-Bench establishes a comprehensive four-dimensional evaluations covering Motion Perception, Spatial Relations, Outcome Prediction, and Physical Dynamics. The benchmark comprises 15 distinct subtasks and 1,340 high-quality video question-answer pairs spanning diverse tabletop, indoor, and outdoor scenarios. Extensive evaluations of a broad spectrum of state-of-the-art proprietary, open-source, and specialized spatial MLLMs reveal that, despite their strong general video understanding capabilities, current models still face substantial bottlenecks in complex spatial-temporal reasoning and remain far below human performance.

Figures

Figures reproduced from arXiv: 2607.20868 by Daxin Tian, Dongxin Lyu, Han Li, Jiahui Fu, Longfei Xu, Naiyan Wang, Si Liu, Yuliang Xiu, Zehao Huang.

Figure 1
Figure 1. Figure 1: Motivating examples and benchmark results. (a) Two representative examples from ViSTR-Bench, where underlined options indicate the correct answers. Although these questions are visually straightforward for humans, Claude [1] still produces incorrect answers, highlighting the difficulty of reasoning from continuous visual cues in dynamic scenes. (b) Quantitative evaluation results on ViSTR-Bench, showing th… view at source ↗
Figure 2
Figure 2. Figure 2: Representative task examples from ViSTR-Bench. Underlined options indicate the correct answers. Each task is formulated as a qualitative binary￾choice question that requires models to aggregate temporal evidence and reason about dynamic visual scenes. encoder initialized with geometric priors to capture 3D struc￾tural information. Spatial-SSRL [50] proposes a novel self￾supervised reinforcement learning pa… view at source ↗
Figure 3
Figure 3. Figure 3: Benchmark statistics. ViSTR-Bench consists of four task dimensions and 15 subtasks. TABLE I TASK-LEVEL DATA SOURCE STATISTICS OF VISTR-BENCH. FOR EACH SUBTASK, WE REPORT THE TOTAL NUMBER OF QA PAIRS AND THE NUMBER OF SAMPLES COLLECTED FROM PUBLIC DATASETS, CURATED WEB VIDEOS, AND SELF-COLLECTED RECORDINGS. Task Total Public Web Collected Motion Perception Vehicle Movement 112 112 0 0 Relative Velocity 84 8… view at source ↗
Figure 4
Figure 4. Figure 4: Benchmark construction pipeline. We construct ViSTR-Bench through a systematic multi-stage process, including data collection, video preprocessing, QA pair generation, and human quality control. This pipeline ensures that each sample contains sufficient temporal evidence, supports high-level reasoning, and has an unambiguous qualitative answer. that each sample requires temporal evidence, supports high￾lev… view at source ↗
Figure 5
Figure 5. Figure 5: Radar comparison across model groups. We visualize representative models from proprietary general-purpose MLLMs, open-source general-purpose MLLMs, and specialized spatial MLLMs across the 15 subtasks in ViSTR-Bench. Human performance is shown as a reference, highlighting the substantial gap between current MLLMs and human-level spatial-temporal reasoning. V2.5 [43], Qwen3.5 [45], and GLM-4.6V [47], we dir… view at source ↗
Figure 6
Figure 6. Figure 6: Representative examples for each primary error type. remain modest. Manual CoT achieves the best overall per￾formance, improving Direct Prompting by 1.6% and reaching 55.2%. By decomposing each task into task-specific reasoning steps, Manual CoT provides stronger guidance for grounding textual reasoning in relevant visual evidence. Nevertheless, the improvement is still moderate and highly task-dependent. … view at source ↗
Figure 7
Figure 7. Figure 7: Preliminary exploration on model improvement. We examine two complementary strategies for making task-relevant visual evidence more explicit. (a) For input-centric augmentation on Ego Motion, we reconstruct the scene using VGGT-Ω [90] and render auxiliary novel views to provide explicit 3D spatial evidence. (b) For tool-augmented reasoning on Interaction Direction, we use WAFT [91] to extract optical-flow … view at source ↗
Figure 8
Figure 8. Figure 8: Manual CoT prompt template for Vehicle Movement. Relative Velocity You are an expert in driving video analysis and motion comparison. I will provide you with a driving video. Your task is to determine which vehicle is moving faster: the vehicle in the green box or the vehicle in the blue box. Please base your decision on their motion over time, not just their positions in a single frame. Please observe the… view at source ↗
Figure 9
Figure 9. Figure 9: Manual CoT prompt template for Relative Velocity [PITH_FULL_IMAGE:figures/full_fig_p017_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Manual CoT prompt template for Rotation Direction. Ego Motion You are an expert in video-based spatial reasoning. I will provide you with a video recorded by a moving camera in an indoor scene. The target object is the {TARGET}. Your task is to determine the final position of the target object relative to the camera. The two candidate answers are: {OPTION_A} and {OPTION_B}. Please observe the video carefu… view at source ↗
Figure 11
Figure 11. Figure 11: Manual CoT prompt template for Ego Motion [PITH_FULL_IMAGE:figures/full_fig_p018_11.png] view at source ↗
Figure 12
Figure 12. Figure 12: Manual CoT prompt template for Passage Feasibility. Interaction Direction You are an expert in person-object interaction analysis. I will provide you with a video of an interaction between a person and {TARGET}. Your task is to determine the person’s movement direction relative to {TARGET}. The two candidate answers are: {OPTION_A} and {OPTION_B}. Please observe the video carefully, strictly follow the an… view at source ↗
Figure 13
Figure 13. Figure 13: Manual CoT prompt template for Interaction Direction [PITH_FULL_IMAGE:figures/full_fig_p019_13.png] view at source ↗
Figure 14
Figure 14. Figure 14: Manual CoT prompt template for Basketball Shot. Soccer Shot You are an expert in sports video analysis and physics trajectory prediction. I will provide you with a video of a soccer free kick. Your task is to predict whether the ball will go into the goal based on its trajectory, ignoring the goalkeeper and any defensive interference. Please observe the video carefully, strictly follow the analysis steps … view at source ↗
Figure 15
Figure 15. Figure 15: Manual CoT prompt template for Soccer Shot [PITH_FULL_IMAGE:figures/full_fig_p020_15.png] view at source ↗
Figure 16
Figure 16. Figure 16: Manual CoT prompt template for Golf Shot. Billiards Shot You are an expert in sports video analysis and trajectory-based motion prediction. I will provide you with a video of billiards. Your task is to predict whether the target ball will go into the pocket based on its current trajectory. Please observe the video carefully, strictly follow the analysis steps below, and output your final answer in a fixed… view at source ↗
Figure 17
Figure 17. Figure 17: Manual CoT prompt template for Billiards Shot [PITH_FULL_IMAGE:figures/full_fig_p021_17.png] view at source ↗
Figure 18
Figure 18. Figure 18: Manual CoT prompt template for Swimming Race. Fall Direction You are an expert in video-based human motion analysis. I will provide you with a video of a person falling. Your task is to determine the person’s motion direction or motion category. The two candidate answers are: {OPTION_A} and {OPTION_B}. Please observe the video carefully, strictly follow the analysis steps below, and output your final answ… view at source ↗
Figure 19
Figure 19. Figure 19: Manual CoT prompt template for Fall Direction [PITH_FULL_IMAGE:figures/full_fig_p022_19.png] view at source ↗
Figure 20
Figure 20. Figure 20: Manual CoT prompt template for Jenga Stability. Mikado Dependency You are an expert in physical interaction analysis and video-based contact reasoning. I will provide you with a video of a Mikado game. Your task is to predict whether the stick indicated by the pointing stick can be picked up without touching any other sticks. Please observe the video carefully, strictly follow the analysis steps below, an… view at source ↗
Figure 21
Figure 21. Figure 21: Manual CoT prompt template for Mikado Dependency [PITH_FULL_IMAGE:figures/full_fig_p023_21.png] view at source ↗
Figure 22
Figure 22. Figure 22: Manual CoT prompt template for Knot Type. id deface: 83 gpt 5.4 thinking gemini 3.1 pro preview claude ops 4.6 thinking : This is a driving video. Please determine whether the vehicle in the green box shows subtle movement during the video. Answer Yes or No. : The vehicle in the green box is driving forward normally, and its movement relative to the camera and the environment is clear and continuous throu… view at source ↗
Figure 23
Figure 23. Figure 23: Qualitative visualization for Vehicle Movement. We compare the outputs of GPT-5.4-thinking [40], Gemini-3.1-Pro-Preview [41], and Claude￾Opus-4.6-thinking [1]. Gray text denotes built-in thinking content [PITH_FULL_IMAGE:figures/full_fig_p024_23.png] view at source ↗
Figure 24
Figure 24. Figure 24: Qualitative visualization for Relative Velocity. We compare the outputs of GPT-5.4-thinking [40], Gemini-3.1-Pro-Preview [41], and Claude-Opus￾4.6-thinking [1]. Gray text denotes built-in thinking content [PITH_FULL_IMAGE:figures/full_fig_p025_24.png] view at source ↗
Figure 25
Figure 25. Figure 25: Qualitative visualization for Rotation Direction. We compare the outputs of GPT-5.4-thinking [40], Gemini-3.1-Pro-Preview [41], and Claude￾Opus-4.6-thinking [1]. Gray text denotes built-in thinking content [PITH_FULL_IMAGE:figures/full_fig_p026_25.png] view at source ↗
Figure 26
Figure 26. Figure 26: Qualitative visualization for Ego Motion. We compare the outputs of GPT-5.4-thinking [40], Gemini-3.1-Pro-Preview [41], and Claude-Opus-4.6- thinking [1]. Gray text denotes built-in thinking content [PITH_FULL_IMAGE:figures/full_fig_p027_26.png] view at source ↗
Figure 27
Figure 27. Figure 27: Qualitative visualization for Passage Feasibility. We compare the outputs of GPT-5.4-thinking [40], Gemini-3.1-Pro-Preview [41], and Claude￾Opus-4.6-thinking [1]. Gray text denotes built-in thinking content [PITH_FULL_IMAGE:figures/full_fig_p028_27.png] view at source ↗
Figure 28
Figure 28. Figure 28: Qualitative visualization for Interaction Direction. We compare the outputs of GPT-5.4-thinking [40], Gemini-3.1-Pro-Preview [41], and Claude￾Opus-4.6-thinking [1]. Gray text denotes built-in thinking content [PITH_FULL_IMAGE:figures/full_fig_p029_28.png] view at source ↗
Figure 29
Figure 29. Figure 29: Qualitative visualization for Basketball Shot. We compare the outputs of GPT-5.4-thinking [40], Gemini-3.1-Pro-Preview [41], and Claude-Opus￾4.6-thinking [1]. Gray text denotes built-in thinking content [PITH_FULL_IMAGE:figures/full_fig_p030_29.png] view at source ↗
Figure 30
Figure 30. Figure 30: Qualitative visualization for Soccer Shot. We compare the outputs of GPT-5.4-thinking [40], Gemini-3.1-Pro-Preview [41], and Claude-Opus-4.6- thinking [1]. Gray text denotes built-in thinking content. id deface: 395 gpt 5.4 thinking gemini 3.1 pro preview claude ops 4.6 thinking : This is a video of a golf shot. Please predict whether the golf ball will go into the hole based on its observed trajectory. A… view at source ↗
Figure 31
Figure 31. Figure 31: Qualitative visualization for Golf Shot. We compare the outputs of GPT-5.4-thinking [40], Gemini-3.1-Pro-Preview [41], and Claude-Opus-4.6- thinking [1]. Gray text denotes built-in thinking content [PITH_FULL_IMAGE:figures/full_fig_p031_31.png] view at source ↗
Figure 32
Figure 32. Figure 32: Qualitative visualization for Billiards Shot. We compare the outputs of GPT-5.4-thinking [40], Gemini-3.1-Pro-Preview [41], and Claude-Opus￾4.6-thinking [1]. Gray text denotes built-in thinking content [PITH_FULL_IMAGE:figures/full_fig_p032_32.png] view at source ↗
Figure 33
Figure 33. Figure 33: Qualitative visualization for Swimming Race. We compare the outputs of GPT-5.4-thinking [40], Gemini-3.1-Pro-Preview [41], and Claude-Opus￾4.6-thinking [1]. Gray text denotes built-in thinking content [PITH_FULL_IMAGE:figures/full_fig_p033_33.png] view at source ↗
Figure 34
Figure 34. Figure 34: Qualitative visualization for Fall Direction. We compare the outputs of GPT-5.4-thinking [40], Gemini-3.1-Pro-Preview [41], and Claude-Opus￾4.6-thinking [1]. Gray text denotes built-in thinking content [PITH_FULL_IMAGE:figures/full_fig_p034_34.png] view at source ↗
Figure 35
Figure 35. Figure 35: Qualitative visualization for Jenga Stability. We compare the outputs of GPT-5.4-thinking [40], Gemini-3.1-Pro-Preview [41], and Claude-Opus￾4.6-thinking [1]. Gray text denotes built-in thinking content [PITH_FULL_IMAGE:figures/full_fig_p035_35.png] view at source ↗
Figure 36
Figure 36. Figure 36: Qualitative visualization for Mikado Dependency. We compare the outputs of GPT-5.4-thinking [40], Gemini-3.1-Pro-Preview [41], and Claude￾Opus-4.6-thinking [1]. Gray text denotes built-in thinking content [PITH_FULL_IMAGE:figures/full_fig_p036_36.png] view at source ↗
Figure 37
Figure 37. Figure 37: Qualitative visualization for Knot Type. We compare the outputs of GPT-5.4-thinking [40], Gemini-3.1-Pro-Preview [41], and Claude-Opus-4.6- thinking [1]. Gray text denotes built-in thinking content [PITH_FULL_IMAGE:figures/full_fig_p037_37.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

93 extracted references · 21 linked inside Pith

  1. [1]

    Anthropic Model System Cards,

    Anthropic, “Anthropic Model System Cards,” https://www.anthropic.co m/system-cards, 2026

    2026

  2. [2]

    MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts,

    P. Lu, H. Bansal, T. Xia, J. Liu, C. Li, H. Hajishirzi, H. Cheng, K.-W. Chang, M. Galley, and J. Gao, “MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts,” inInternational Conference on Learning Representations, vol. 2024, 2024, pp. 23 439– 23 554

    2024

  3. [3]

    Measuring Multimodal Mathematical Reasoning with MATH-Vision Dataset,

    K. Wang, J. Pan, W. Shi, Z. Lu, H. Ren, A. Zhou, M. Zhan, and H. Li, “Measuring Multimodal Mathematical Reasoning with MATH-Vision Dataset,”Advances in Neural Information Processing Systems, vol. 37, pp. 95 095–95 169, 2024

    2024

  4. [4]

    MATHVERSE: Does Your Multi-modal LLM Truly See the Diagrams in Visual Math Problems?

    R. Zhang, D. Jiang, Y . Zhang, H. Lin, Z. Guo, P. Qiu, A. Zhou, P. Lu, K.-W. Chang, Y . Qiaoet al., “MATHVERSE: Does Your Multi-modal LLM Truly See the Diagrams in Visual Math Problems?” inEuropean Conference on Computer Vision, 2024, pp. 169–186

    2024

  5. [5]

    MMCode: Bench- marking Multimodal Large Language Models for Code Generation with Visually Rich Programming Problems,

    K. Li, Y . Tian, Q. Hu, Z. Luo, Z. Huang, and J. Ma, “MMCode: Bench- marking Multimodal Large Language Models for Code Generation with Visually Rich Programming Problems,” inFindings of the Association for Computational Linguistics: EMNLP 2024, 2024, pp. 736–783

    2024

  6. [6]

    Design2Code: Benchmarking Multimodal Code Generation for Automated Front-End Engineering,

    C. Si, Y . Zhang, R. Li, Z. Yang, R. Liu, and D. Yang, “Design2Code: Benchmarking Multimodal Code Generation for Automated Front-End Engineering,” inProceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), 2025, pp. 3956–3974

    2025

  7. [7]

    Emu3: Next-Token Prediction is All You Need,

    X. Wang, X. Zhang, Z. Luo, Q. Sun, Y . Cui, J. Wang, F. Zhang, Y . Wang, Z. Li, Q. Yuet al., “Emu3: Next-Token Prediction is All You Need,” arXiv preprint arXiv:2409.18869, 2024

    Pith/arXiv arXiv 2024

  8. [8]

    NExT-GPT: Any-to-Any Multimodal LLM,

    S. Wu, H. Fei, L. Qu, W. Ji, and T.-S. Chua, “NExT-GPT: Any-to-Any Multimodal LLM,” inForty-first International Conference on Machine Learning, 2024

    2024

  9. [9]

    MiniGPT-5: Interleaved Vision- and-Language Generation via Generative V okens,

    K. Zheng, X. He, and X. E. Wang, “MiniGPT-5: Interleaved Vision- and-Language Generation via Generative V okens,”arXiv preprint arXiv:2310.02239, 2023

    arXiv 2023

  10. [10]

    Introducing GPT-5,

    OpenAI, “Introducing GPT-5,” https://openai.com/index/introducing-g pt-5, 2025

    2025

  11. [11]

    GPQA: A Graduate-Level Google- Proof Q&A Benchmark,

    D. Rein, B. L. Hou, A. C. Stickland, J. Petty, R. Y . Pang, J. Dirani, J. Michael, and S. R. Bowman, “GPQA: A Graduate-Level Google- Proof Q&A Benchmark,”arXiv preprint arXiv:2311.12022, 2023

    Pith/arXiv arXiv 2023

  12. [12]

    Gemini Achieves Gold-Medal Level at the Interna- tional Collegiate Programming Contest World Finals,

    Google DeepMind, “Gemini Achieves Gold-Medal Level at the Interna- tional Collegiate Programming Contest World Finals,” https://deepmind .google/blog/gemini-achieves-gold-medal-level-at-the-international-col legiate-programming-contest-world-finals, 2025

    2025

  13. [13]

    LMDrive: Closed-Loop End-to-End Driving with Large Language Models,

    H. Shao, Y . Hu, L. Wang, G. Song, S. L. Waslander, Y . Liu, and H. Li, “LMDrive: Closed-Loop End-to-End Driving with Large Language Models,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024, pp. 15 120–15 130

    2024

  14. [14]

    DriveLM: Driving with Graph Visual Question Answering,

    C. Sima, K. Renz, K. Chitta, L. Chen, H. Zhang, C. Xie, J. Beißwenger, P. Luo, A. Geiger, and H. Li, “DriveLM: Driving with Graph Visual Question Answering,” inEuropean Conference on Computer Vision, 2024, pp. 256–274

    2024

  15. [15]

    DriveGPT4: Interpretable End-to-End Autonomous Driving Via Large Language Model,

    Z. Xu, Y . Zhang, E. Xie, Z. Zhao, Y . Guo, K.-Y . K. Wong, Z. Li, and H. Zhao, “DriveGPT4: Interpretable End-to-End Autonomous Driving Via Large Language Model,”IEEE Robotics and Automation Letters, vol. 9, no. 10, pp. 8186–8193, 2024

    2024

  16. [16]

    Open- VLA: An Open-Source Vision-Language-Action Model,

    M. J. Kim, K. Pertsch, S. Karamcheti, T. Xiao, A. Balakrishna, S. Nair, R. Rafailov, E. Foster, G. Lam, P. Sanketiet al., “Open- VLA: An Open-Source Vision-Language-Action Model,”arXiv preprint arXiv:2406.09246, 2024

    Pith/arXiv arXiv 2024

  17. [17]

    RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control,

    B. Zitkovich, T. Yu, S. Xu, P. Xu, T. Xiao, F. Xia, J. Wu, P. Wohlhart, S. Welker, A. Wahidet al., “RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control,” inConference on Robot Learning, 2023, pp. 2165–2183

    2023

  18. [18]

    Do As I Can, Not As I Say: Grounding Language in Robotic Affordances,

    M. Ahn, A. Brohan, N. Brown, Y . Chebotar, O. Cortes, B. David, C. Finn, C. Fu, K. Gopalakrishnan, K. Hausmanet al., “Do As I Can, Not As I Say: Grounding Language in Robotic Affordances,”arXiv preprint arXiv:2204.01691, 2022

    Pith/arXiv arXiv 2022

  19. [19]

    PaLM-E: An Embodied Multimodal Language Model,

    D. Driess, F. Xia, M. S. Sajjadi, C. Lynch, A. Chowdhery, B. Ichter, A. Wahid, J. Tompson, Q. Vuong, T. Yuet al., “PaLM-E: An Embodied Multimodal Language Model,”arXiv preprint arXiv:2303.03378, 2023

    Pith/arXiv arXiv 2023

  20. [20]

    OmniSpatial: Towards Comprehensive Spatial Reasoning Benchmark for Vision Language Models,

    M. Jia, Z. Qi, S. Zhang, W. Zhang, X. Yu, J. He, H. Wang, and L. Yi, “OmniSpatial: Towards Comprehensive Spatial Reasoning Benchmark for Vision Language Models,”arXiv preprint arXiv:2506.03135, 2025

    arXiv 2025

  21. [21]

    MMSI-Video-Bench: A Holistic Benchmark for Video- Based Spatial Intelligence,

    J. Lin, R. Xu, S. Zhu, S. Yang, P. Cao, Y . Ran, M. Hu, C. Zhu, Y . Xie, Y . Longet al., “MMSI-Video-Bench: A Holistic Benchmark for Video- Based Spatial Intelligence,”arXiv preprint arXiv:2512.10863, 2025

    arXiv 2025

  22. [22]

    SpaceR: Reinforcing MLLMs in Video Spatial Reasoning,

    K. Ouyang, Y . Liu, H. Wu, Y . Liu, H. Zhou, J. Zhou, F. Meng, and X. Sun, “SpaceR: Reinforcing MLLMs in Video Spatial Reasoning,” arXiv preprint arXiv:2504.01805, 2025

    Pith/arXiv arXiv 2025

  23. [23]

    Thinking in Space: How Multimodal Large Language Models See, Remember, and Recall Spaces,

    J. Yang, S. Yang, A. W. Gupta, R. Han, L. Fei-Fei, and S. Xie, “Thinking in Space: How Multimodal Large Language Models See, Remember, and Recall Spaces,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2025, pp. 10 632–10 643

    2025

  24. [24]

    Cambrian-S: Towards Spatial Supersensing in Video,

    S. Yang, J. Yang, P. Huang, E. L. Brown II, Z. Yang, Y . Yu, S. Tong, Z. Zheng, Y . Xu, M. Wanget al., “Cambrian-S: Towards Spatial Supersensing in Video,” inThe Fourteenth International Conference on Learning Representations, 2025. 13

    2025

  25. [25]

    From Flatland to Space: Teaching Vision-Language Models to Perceive and Reason in 3D,

    J. Zhang, Y . Chen, Y . Zhou, Y . Xu, Z. Huang, J. Mei, J. Chen, Y .- J. Yuan, X. Cai, G. Huanget al., “From Flatland to Space: Teaching Vision-Language Models to Perceive and Reason in 3D,”arXiv preprint arXiv:2503.22976, 2025

    arXiv 2025

  26. [26]

    Thinking in Dynamics: How Multimodal Large Language Models Perceive, Track, and Reason Dynamics in Physical 4D World,

    Y . Huang, K. Wen, R. Gao, D. Liu, Y . Lou, J. Wu, J. Xu, J. Zhang, Z. Yang, Y . Linet al., “Thinking in Dynamics: How Multimodal Large Language Models Perceive, Track, and Reason Dynamics in Physical 4D World,”arXiv preprint arXiv:2603.12746, 2026

    arXiv 2026

  27. [27]

    STI- Bench: Are MLLMs Ready for Precise Spatial-Temporal World Under- standing?

    Y . Li, Y . Zhang, T. Lin, X. Liu, W. Cai, Z. Liu, and B. Zhao, “STI- Bench: Are MLLMs Ready for Precise Spatial-Temporal World Under- standing?” inProceedings of the IEEE/CVF International Conference on Computer Vision, 2025, pp. 5622–5632

    2025

  28. [28]

    OST- Bench: Evaluating the Capabilities of MLLMs in Online Spatio-temporal Scene Understanding,

    J. Lin, C. Zhu, R. Xu, X. Mao, X. Liu, T. Wang, and J. Pang, “OST- Bench: Evaluating the Capabilities of MLLMs in Online Spatio-temporal Scene Understanding,”arXiv preprint arXiv:2507.07984, 2025

    arXiv 2025

  29. [29]

    VideoLoom: A Video Large Language Model for Joint Spatial-Temporal Understanding,

    J. Shi, J. Wang, Z. You, B. He, and Z. Wu, “VideoLoom: A Video Large Language Model for Joint Spatial-Temporal Understanding,” arXiv preprint arXiv:2601.07290, 2026

    arXiv 2026

  30. [30]

    MLLM-4D: Towards Visual-based Spatial-Temporal Intelligence,

    X. Yin, C. Li, J. Chang, C.-M. Pun, and X. Cun, “MLLM-4D: Towards Visual-based Spatial-Temporal Intelligence,”arXiv preprint arXiv:2603.00515, 2026

    arXiv 2026

  31. [31]

    DSI-Bench: A Benchmark for Dynamic Spatial Intelligence,

    Z. Zhang, Z. Wang, G. Zhang, W. Dai, Y . Xia, Z. Yan, M. Hong, and Z. Zhao, “DSI-Bench: A Benchmark for Dynamic Spatial Intelligence,” arXiv preprint arXiv:2510.18873, 2025

    arXiv 2025

  32. [32]

    Learning to Reason in 4D: Dynamic Spatial Understanding for Vision Language Models,

    S. Zhou, Y . Chen, Y . Ge, W. Huang, J. Lin, Y . Shan, and X. Qi, “Learning to Reason in 4D: Dynamic Spatial Understanding for Vision Language Models,”arXiv preprint arXiv:2512.20557, 2025

    arXiv 2025

  33. [33]

    VLM4D: Towards Spatiotemporal Awareness in Vision Language Models,

    S. Zhou, A. Vilesov, X. He, Z. Wan, S. Zhang, A. Nagachandra, D. Chang, D. Chen, X. E. Wang, and A. Kadambi, “VLM4D: Towards Spatiotemporal Awareness in Vision Language Models,” inProceedings of the IEEE/CVF International Conference on Computer Vision, 2025, pp. 8600–8612

    2025

  34. [34]

    Ego4D: Around the World in 3,000 Hours of Egocentric Video,

    K. Grauman, A. Westbury, E. Byrne, Z. Chavis, A. Furnari, R. Girdhar, J. Hamburger, H. Jiang, M. Liu, X. Liuet al., “Ego4D: Around the World in 3,000 Hours of Egocentric Video,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 18 995–19 012

    2022

  35. [35]

    ScanNet: Richly-Annotated 3D Reconstructions of Indoor Scenes,

    A. Dai, A. X. Chang, M. Savva, M. Halber, T. Funkhouser, and M. Nießner, “ScanNet: Richly-Annotated 3D Reconstructions of Indoor Scenes,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2017, pp. 5828–5839

    2017

  36. [36]

    ScanNet++: A High- Fidelity Dataset of 3D Indoor Scenes,

    C. Yeshwanth, Y .-C. Liu, M. Nießner, and A. Dai, “ScanNet++: A High- Fidelity Dataset of 3D Indoor Scenes,” inProceedings of the IEEE/CVF International Conference on Computer Vision, 2023, pp. 12–22

    2023

  37. [37]

    ARKitScenes: A Diverse Real- World Dataset For 3D Indoor Scene Understanding Using Mobile RGB- D Data,

    G. Baruch, Z. Chen, A. Dehghan, T. Dimry, Y . Feigin, P. Fu, T. Gebauer, B. Joffe, D. Kurz, A. Schwartzet al., “ARKitScenes: A Diverse Real- World Dataset For 3D Indoor Scene Understanding Using Mobile RGB- D Data,”arXiv preprint arXiv:2111.08897, 2021

    Pith/arXiv arXiv 2021

  38. [38]

    Scalability in Perception for Autonomous Driving: Waymo Open Dataset,

    P. Sun, H. Kretzschmar, X. Dotiwalla, A. Chouard, V . Patnaik, P. Tsui, J. Guo, Y . Zhou, Y . Chai, B. Caineet al., “Scalability in Perception for Autonomous Driving: Waymo Open Dataset,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020, pp. 2446–2454

    2020

  39. [39]

    Motion-X: A Large-scale 3D Expressive Whole-body Human Motion Dataset,

    J. Lin, A. Zeng, S. Lu, Y . Cai, R. Zhang, H. Wang, and L. Zhang, “Motion-X: A Large-scale 3D Expressive Whole-body Human Motion Dataset,”Advances in Neural Information Processing Systems, vol. 36, pp. 25 268–25 280, 2023

    2023

  40. [40]

    OpenAI GPT-5 System Card,

    A. Singh, A. Fry, A. Perelman, A. Tart, A. Ganesh, A. El-Kishky, A. McLaughlin, A. Low, A. Ostrow, A. Ananthramet al., “OpenAI GPT-5 System Card,”arXiv preprint arXiv:2601.03267, 2025

    Pith/arXiv arXiv 2025

  41. [41]

    Google DeepMind Model Cards,

    Google DeepMind, “Google DeepMind Model Cards,” https://deepmi nd.google/models/model-cards, 2026

    2026

  42. [42]

    Seed2.0 Model Card: Towards Intelligence Frontier for Real-World Complexity,

    ByteDance Seed Team, “Seed2.0 Model Card: Towards Intelligence Frontier for Real-World Complexity,” https://seed.bytedance.com/en/ seed2, 2026

    2026

  43. [43]

    Xiaomi MiMo-V2.5: A Leap in Agency and Multimodality,

    Xiaomi MiMo Team, “Xiaomi MiMo-V2.5: A Leap in Agency and Multimodality,” https://mimo.xiaomi.com/mimo-v2-5, 2026

    2026

  44. [44]

    LLaV A-OneVision-1.5: Fully Open Framework for Democratized Multimodal Training,

    X. An, Y . Xie, K. Yang, W. Zhang, X. Zhao, Z. Cheng, Y . Wang, S. Xu, C. Chen, D. Zhuet al., “LLaV A-OneVision-1.5: Fully Open Framework for Democratized Multimodal Training,”arXiv preprint arXiv:2509.23661, 2025

    Pith/arXiv arXiv 2025

  45. [45]

    Qwen3.5: Towards Native Multimodal Agents,

    Qwen Team, “Qwen3.5: Towards Native Multimodal Agents,” https: //qwen.ai/blog?id=qwen3.5, 2026

    2026

  46. [46]

    InternVL3.5: Advancing Open-Source Multi- modal Models in Versatility, Reasoning, and Efficiency,

    W. Wang, Z. Gao, L. Gu, H. Pu, L. Cui, X. Wei, Z. Liu, L. Jing, S. Ye, J. Shaoet al., “InternVL3.5: Advancing Open-Source Multi- modal Models in Versatility, Reasoning, and Efficiency,”arXiv preprint arXiv:2508.18265, 2025

    Pith/arXiv arXiv 2025

  47. [47]

    GLM-4.6V: Open Source Multimodal Models with Native Multi- modal Tool Use,

    Z.ai, “GLM-4.6V: Open Source Multimodal Models with Native Multi- modal Tool Use,” https://docs.z.ai/guides/vlm/glm-4.6v, 2025

    2025

  48. [48]

    Learning from Videos for 3D World: Enhancing MLLMs with 3D Vision Geometry Priors,

    D. Zheng, Y . Li, L. Wanget al., “Learning from Videos for 3D World: Enhancing MLLMs with 3D Vision Geometry Priors,”Advances in Neural Information Processing Systems, vol. 38, pp. 20 560–20 586, 2026

    2026

  49. [49]

    Spatial-MLLM: Boosting MLLM Capabilities in Visual-based Spatial Intelligence,

    D. Wu, F. Liu, Y .-H. Hung, and Y . Duan, “Spatial-MLLM: Boosting MLLM Capabilities in Visual-based Spatial Intelligence,”arXiv preprint arXiv:2505.23747, 2025

    Pith/arXiv arXiv 2025

  50. [50]

    Spatial-SSRL: Enhancing Spatial Understanding via Self- Supervised Reinforcement Learning,

    Y . Liu, B. Zhang, Y . Zang, Y . Cao, L. Xing, X. Dong, H. Duan, D. Lin, and J. Wang, “Spatial-SSRL: Enhancing Spatial Understanding via Self- Supervised Reinforcement Learning,”arXiv preprint arXiv:2510.27606, 2025

    arXiv 2025

  51. [51]

    Thinking with Geometry: Active Geometry Integration for Spatial Reasoning,

    H. Li, Q. Cao, T. Tang, K. Xiang, Z. Guo, J. Han, H. Xu, and X. Liang, “Thinking with Geometry: Active Geometry Integration for Spatial Reasoning,”arXiv preprint arXiv:2602.06037, 2026

    Pith/arXiv arXiv 2026

  52. [52]

    SEED-Bench: Benchmarking Multimodal Large Language Models,

    B. Li, Y . Ge, Y . Ge, G. Wang, R. Wang, R. Zhang, and Y . Shan, “SEED-Bench: Benchmarking Multimodal Large Language Models,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024, pp. 13 299–13 308

    2024

  53. [53]

    MMBench: Is Your Multi-modal Model an All- around Player?

    Y . Liu, H. Duan, Y . Zhang, B. Li, S. Zhang, W. Zhao, Y . Yuan, J. Wang, C. He, Z. Liuet al., “MMBench: Is Your Multi-modal Model an All- around Player?” inEuropean Conference on Computer Vision, 2024, pp. 216–233

    2024

  54. [54]

    MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilities,

    W. Yu, Z. Yang, L. Li, J. Wang, K. Lin, Z. Liu, X. Wang, and L. Wang, “MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilities,” inProceedings of the 41st International Conference on Machine Learning, 2024, pp. 57 730–57 754

    2024

  55. [55]

    MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI,

    X. Yue, Y . Ni, K. Zhang, T. Zheng, R. Liu, G. Zhang, S. Stevens, D. Jiang, W. Ren, Y . Sunet al., “MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024, pp. 9556–9567

    2024

  56. [56]

    MMBench-Video: A Long-Form Multi-Shot Benchmark for Holistic Video Understanding,

    X. Fang, K. Mao, H. Duan, X. Zhao, Y . Li, D. Lin, and K. Chen, “MMBench-Video: A Long-Form Multi-Shot Benchmark for Holistic Video Understanding,”Advances in Neural Information Processing Systems, vol. 37, pp. 89 098–89 124, 2024

    2024

  57. [57]

    Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysis,

    C. Fu, Y . Dai, Y . Luo, L. Li, S. Ren, R. Zhang, Z. Wang, C. Zhou, Y . Shen, M. Zhanget al., “Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysis,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2025, pp. 24 108–24 118

    2025

  58. [58]

    MVBench: A Comprehensive Multi-modal Video Understanding Benchmark,

    K. Li, Y . Wang, Y . He, Y . Li, Y . Wang, Y . Liu, Z. Wang, J. Xu, G. Chen, P. Luoet al., “MVBench: A Comprehensive Multi-modal Video Understanding Benchmark,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024, pp. 22 195–22 206

    2024

  59. [59]

    VI- TATECS: A Diagnostic Dataset for Temporal Concept Understanding of Video-Language Models,

    S. Li, L. Li, Y . Liu, S. Ren, Y . Liu, R. Gao, X. Sun, and L. Hou, “VI- TATECS: A Diagnostic Dataset for Temporal Concept Understanding of Video-Language Models,” inEuropean Conference on Computer Vision, 2024, pp. 331–348

    2024

  60. [60]

    Multi-modal Situated Reasoning in 3D Scenes,

    X. Linghu, J. Huang, X. Niu, X. Ma, B. Jia, and S. Huang, “Multi-modal Situated Reasoning in 3D Scenes,”Advances in Neural Information Processing Systems, vol. 37, pp. 140 903–140 936, 2024

    2024

  61. [61]

    TempCompass: Do Video LLMs Really Understand Videos?

    Y . Liu, S. Li, Y . Liu, Y . Wang, S. Ren, L. Li, S. Chen, X. Sun, and L. Hou, “TempCompass: Do Video LLMs Really Understand Videos?” inFindings of the Association for Computational Linguistics: ACL 2024, 2024, pp. 8731–8772

    2024

  62. [62]

    OpenEQA: Embodied Question Answering in the Era of Foundation Models,

    A. Majumdar, A. Ajay, X. Zhang, P. Putta, S. Yenamandra, M. Henaff, S. Silwal, P. Mcvay, O. Maksymets, S. Arnaudet al., “OpenEQA: Embodied Question Answering in the Era of Foundation Models,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024, pp. 16 488–16 498

    2024

  63. [63]

    EgoSchema: A Diagnostic Benchmark for Very Long-form Video Language Understanding,

    K. Mangalam, R. Akshulakov, and J. Malik, “EgoSchema: A Diagnostic Benchmark for Very Long-form Video Language Understanding,”Ad- vances in Neural Information Processing Systems, vol. 36, pp. 46 212– 46 244, 2023

    2023

  64. [64]

    Video-Bench: A Comprehensive Benchmark and Toolkit for Evaluating Video-based Large Language Models,

    M. Ning, B. Zhu, Y . Xie, B. Lin, J. Cui, L. Yuan, D. Chen, and L. Yuan, “Video-Bench: A Comprehensive Benchmark and Toolkit for Evaluating Video-based Large Language Models,”Computational Visual Media, 2025

    2025

  65. [65]

    TOMATO: Assessing Visual Temporal Reasoning Capabili- ties in Multimodal Foundation Models,

    Z. Shangguan, C. Li, Y . Ding, Y . Zheng, Y . Zhao, T. Fitzgerald, and A. Cohan, “TOMATO: Assessing Visual Temporal Reasoning Capabili- ties in Multimodal Foundation Models,” inThe Thirteenth International Conference on Learning Representations, 2024

    2024

  66. [66]

    Is A Picture Worth A Thousand Words? Delving Into Spatial Reasoning for 14 Vision Language Models,

    J. Wang, Y . Ming, Z. Shi, V . Vineet, X. Wang, Y . Li, and N. Joshi, “Is A Picture Worth A Thousand Words? Delving Into Spatial Reasoning for 14 Vision Language Models,”Advances in Neural Information Processing Systems, vol. 37, pp. 75 392–75 421, 2024

    2024

  67. [67]

    MM-Ego: Towards Building Egocentric Multimodal LLMs for Video QA,

    H. Ye, H. Zhang, E. Daxberger, L. Chen, Z. Lin, Y . Li, B. Zhang, H. You, D. Xu, Z. Ganet al., “MM-Ego: Towards Building Egocentric Multimodal LLMs for Video QA,” inThe Thirteenth International Conference on Learning Representations, 2024

    2024

  68. [68]

    LongVILA: Scaling Long-Context Visual Language Models for Long Videos,

    Y . Chen, F. Xue, D. Li, Q. Hu, L. Zhu, X. Li, Y . Fang, H. Tang, S. Yang, Z. Liuet al., “LongVILA: Scaling Long-Context Visual Language Models for Long Videos,”arXiv preprint arXiv:2408.10188, 2024

    Pith/arXiv arXiv 2024

  69. [69]

    LLaV A-OneVision: Easy Visual Task Transfer,

    B. Li, Y . Zhang, D. Guo, R. Zhang, F. Li, H. Zhang, K. Zhang, P. Zhang, Y . Li, Z. Liuet al., “LLaV A-OneVision: Easy Visual Task Transfer,” arXiv preprint arXiv:2408.03326, 2024

    Pith/arXiv arXiv 2024

  70. [70]

    VILA: On Pre-training for Visual Language Models,

    J. Lin, H. Yin, W. Ping, P. Molchanov, M. Shoeybi, and S. Han, “VILA: On Pre-training for Visual Language Models,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024, pp. 26 689–26 699

    2024

  71. [71]

    Long Context Transfer from Language to Vision,

    P. Zhang, K. Zhang, B. Li, G. Zeng, J. Yang, Y . Zhang, Z. Wang, H. Tan, C. Li, and Z. Liu, “Long Context Transfer from Language to Vision,” arXiv preprint arXiv:2406.16852, 2024

    Pith/arXiv arXiv 2024

  72. [72]

    LLaV A- Video: Video Instruction Tuning With Synthetic Data,

    Y . Zhang, J. Wu, W. Li, B. Li, Z. Ma, Z. Liu, and C. Li, “LLaV A- Video: Video Instruction Tuning With Synthetic Data,”arXiv preprint arXiv:2410.02713, 2024

    Pith/arXiv arXiv 2024

  73. [73]

    SpatialVLM: Endowing Vision-Language Models with Spatial Reasoning Capabilities,

    B. Chen, Z. Xu, S. Kirmani, B. Ichter, D. Sadigh, L. Guibas, and F. Xia, “SpatialVLM: Endowing Vision-Language Models with Spatial Reasoning Capabilities,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024, pp. 14 455–14 465

    2024

  74. [74]

    VLM-3R: Vision-Language Models Aug- mented with Instruction-Aligned 3D Reconstruction,

    Z. Fan, J. Zhang, R. Li, J. Zhang, R. Chen, H. Hu, K. Wang, H. Qu, S. Zhou, D. Wanget al., “VLM-3R: Vision-Language Models Aug- mented with Instruction-Aligned 3D Reconstruction,”arXiv preprint arXiv:2505.20279, 2025

    Pith/arXiv arXiv 2025

  75. [75]

    3DRS: MLLMs Need 3D-Aware Representation Supervision for Scene Understanding,

    X. Huang, J. Wu, Q. Xie, and K. Han, “3DRS: MLLMs Need 3D-Aware Representation Supervision for Scene Understanding,”arXiv preprint arXiv:2506.01946, 2025

    arXiv 2025

  76. [76]

    SpatialLadder: Progressive Training for Spatial Rea- soning in Vision-Language Models,

    H. Li, D. Li, Z. Wang, Y . Yan, H. Wu, W. Zhang, Y . Shen, W. Lu, J. Xiao, and Y . Zhuang, “SpatialLadder: Progressive Training for Spatial Rea- soning in Vision-Language Models,”arXiv preprint arXiv:2510.08531, 2025

    arXiv 2025

  77. [77]

    SpatialCoT: Advancing Spatial Reasoning through Coordinate Alignment and Chain-of-Thought for Embodied Task Planning,

    Y . Liu, D. Chi, S. Wu, Z. Zhang, Y . Hu, L. Zhang, Y . Zhang, S. Wu, T. Cao, G. Huanget al., “SpatialCoT: Advancing Spatial Reasoning through Coordinate Alignment and Chain-of-Thought for Embodied Task Planning,”arXiv preprint arXiv:2501.10074, 2025

    Pith/arXiv arXiv 2025

  78. [78]

    SoFar: Language-Grounded Orientation Bridges Spatial Reasoning and Object Manipulation,

    Z. Qi, W. Zhang, Y . Ding, R. Dong, X. Yu, J. Li, L. Xu, B. Li, X. He, G. Fanet al., “SoFar: Language-Grounded Orientation Bridges Spatial Reasoning and Object Manipulation,”arXiv preprint arXiv:2502.13143, 2025

    arXiv 2025

  79. [79]

    Visual Spatial Tuning,

    R. Yang, Z. Zhu, Y . Li, J. Huang, S. Yan, S. Zhou, Z. Liu, X. Li, S. Li, W. Wanget al., “Visual Spatial Tuning,”arXiv preprint arXiv:2511.05491, 2025

    arXiv 2025

  80. [80]

    Make Geometry Matter for Spatial Reasoning,

    S. Zhang, Q. Shen, S. Wang, T. Pan, and X. Wang, “Make Geometry Matter for Spatial Reasoning,”arXiv preprint arXiv:2603.26639, 2026

    arXiv 2026

Showing first 80 references.