Pith. sign in

REVIEW 3 major objections 4 minor 1 cited by

Temporal understanding in driving videos improves sharply when a vision-language model is handed a text summary of the ego vehicle's motion, and a new benchmark measures the gap to humans.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 18:24 UTC pith:QTV44Q44

load-bearing objection The TAD benchmark is a real contribution, but the headline TCogMap gain is likely inflated by oracle-like access to ego-action labels, and the STSBench claim is unsupported. the 3 major comments →

arxiv 2512.05277 v4 pith:QTV44Q44 submitted 2025-12-04 cs.CV cs.AI

From Segments to Scenes: Temporal Understanding for Agentic Autonomous Driving via Vision-Language Models

Kevin Cannons , Saeed Ranjbar Alvar , Mohammad Asiful Hossain , Ahmad Rezaei , Mohsen Gholami , Alireza Heidarikhazaei , Zhou Weimin , Yong Zhang
show 1 more author
Mohammad Akbari
This is my paper
classification cs.CV cs.AI
keywords TAD benchmarktemporal understandingautonomous drivingvision-language modelstemporal cognitive mapchain-of-thought reasoningNuScenesvehicle action recognition
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper argues that vision-language models (VLMs) fail at temporal reasoning in autonomous driving because they lack explicit access to ego-vehicle motion, and that this deficit is measurable and fixable. It introduces TAD, a benchmark of 5,861 question-answer pairs over 150 driving videos, spanning both short segment-level action recognition and longer scene-level temporal tasks. On TAD, state-of-the-art VLMs average roughly 52% accuracy while humans reach about 74.7%. The paper proposes two training-free interventions—Scene-CoT, a chain-of-thought reasoning wrapper, and TCogMap, a module that converts the ego vehicle's trajectory into a per-segment textual motion summary—and shows that TCogMap lifts average accuracy on TAD by up to 17.72 percentage points across several VLMs.

Core claim

The central claim is that temporal understanding in autonomous driving is a distinct, under-measured capability, and that a large part of the gap between VLMs and humans comes from missing explicit temporal structure rather than from missing visual information. The paper demonstrates this by building TAD, the first driving benchmark that mixes segment-level and scene-level temporal questions, and by showing that simply appending a textual temporal cognitive map of ego motion—derived from the vehicle's pose sequence, not from vision—improves accuracy substantially on both ego and non-ego questions. This works even when the VLM receives no additional frames beyond the baseline set, and it outp

What carries the argument

TCogMap's ego-vehicle motion classifier (Algorithm 1) is the load-bearing component. It consumes a sequence of ego poses (translation, quaternion, timestamp), computes finite-difference velocities, transforms them into the ego's local frame, and applies a hierarchical rule set with fixed thresholds to label each five-second segment as Stopped, Starting, Stopping, Straight, Turn left/right, or Change lane left/right. The resulting per-segment labels are formatted as strings like 'Motion summary for Frame1 to Frame8: The ego-vehicle is moving forward at a relatively constant speed' and concatenated into the VLM prompt. This converts high-frequency kinematic data into a compact, temporally inde

Load-bearing premise

The accuracy gains from TCogMap depend on empirically chosen thresholds in the motion classifier (e.g., 0.2 m/s for stopped, 10 degrees for turn, plus a 1.5x multiplier) that were not validated on a separate tuning set, so the reported improvements may be optimistic.

What would settle it

Re-run the TAD experiments with TCogMap after splitting the 150 videos into a tuning subset and a held-out test subset, and tune the six thresholds on the tuning subset only; if the average accuracy gain over baseline drops substantially (e.g., from 17.72% to near zero) or becomes inconsistent across models, the threshold setting is overfit to the benchmark.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • If TCogMap's gains hold, any existing driving VLM can improve its temporal reasoning by plugging in a trajectory-analysis module—no fine-tuning required, just a text prompt addition.
  • The TAD benchmark provides a reusable testbed for isolating temporal understanding from static scene understanding in driving, allowing future methods to be compared on a common ground.
  • Ego-motion summaries appear to supply contextual cues that also help non-ego object reasoning, suggesting that self-motion is a globally useful prior for scene dynamics.
  • The finding that raw ego pose text does not help as much as the classified summary implies that VLMs benefit from symbolic abstraction of kinematics, not just more data.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • A natural extension is to build similar temporal cognitive maps for surrounding vehicles—the paper mentions this as future work—and one would predict further gains on non-ego tasks if the map carries reliable object-level motion labels.
  • The threshold values in Algorithm 1 could likely be tuned per scene type or learned from data; if gains persist after proper held-out tuning, the method's practical value increases, but if not, the reported margins may shrink.
  • The benchmark's reliance on NuScenes' 20-second clips means it does not test very long-horizon temporal reasoning; a version with longer drives or event ordering across minutes might reveal different failure modes.
  • Scene-CoT's high inference cost (about 47 seconds per question) suggests that its benefits may be superseded by more efficient prompting or by TCogMap-style structured context in deployed systems.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper introduces TAD, a video QA benchmark for temporal understanding in autonomous driving, built on 150 NuScenes validation videos with 5,861 QA pairs across seven segment- and scene-level tasks and 4,481 vehicle-action annotations. The authors evaluate 9 closed- and open-source generalist and AD-specialist VLMs, reporting a large gap between model performance (~52% average) and human performance (74.72%). They also propose two training-free inference-time methods: Scene-CoT, a chain-of-thought video partitioning and LLM-QA pipeline, and TCogMap, which injects an ego-vehicle motion summary computed from NuScenes ego poses into the VLM prompt. The abstract claims average accuracy improvements on TAD of up to 17.72% and up to 10.35% on STSBench.

Significance. The TAD benchmark is a potentially valuable community resource: it is the first AD video benchmark combining segment-level action recognition and scene-level temporal tasks, it was constructed with human verification, and it includes a blind text-only control indicating that questions are not answerable from language alone. The release of data and evaluation code is a concrete strength. If the method-evaluation concerns below are addressed, the benchmark could support meaningful progress in VLM temporal reasoning for driving. The proposed methods, however, are currently presented as the paper's headline contribution, and the evidence for TCogMap in particular is confounded by oracle-like access to ground-truth ego trajectories.

major comments (3)
  1. [§4.2, §5.1, Table 3] The central claim that TCogMap improves temporal reasoning by up to 17.72% is not established by the current protocol. Algorithm 1 converts NuScenes ego poses into the same eight action categories used to construct TAD, using thresholds reported in §5.1 as 'empirically set' with no tuning protocol or held-out split. For the 53.7% of questions about the ego, and in particular the ego-only Action Duration (124 questions) and Relative Temporal Action Localization (92 questions) tasks, the motion summary supplied to the VLM is effectively a ground-truth action string. The 'baseline + textual ego pose' control in §5.1 does not remove the confound, because raw pose is not parsed into the benchmark's action vocabulary. A proper control would tune the thresholds on a held-out NuScenes split, report non-ego performance separately, and/or include a condition in which the motion summary comes from
  2. [Abstract; full text] The abstract states that the methods improve average accuracy by 'up to 10.35% on STSBench,' but no STSBench experiment, configuration, or numerical result appears anywhere in the main text or the supplementary material. STSBench is only cited in Table 1 and Related Work. This claim is unsupported as written and must either be removed or substantiated with an actual experimental comparison.
  3. [§5.3, Table 3] Scene-CoT is presented as a general improvement, but for the larger open-source models it decreases average accuracy: Qwen2.5-VL-32B drops from 55.20 to 48.78, InternVL3-14B from 52.35 to 51.81, and InternVL3-38B from 56.49 to 53.95. The text in §5.3 acknowledges that 'Scene-CoT does not appear to offer accuracy gains' for larger models, but the abstract and introduction present both proposed methods as improving average accuracy. The claims should be qualified to state that Scene-CoT helps only smaller models, while TCogMap provides more consistent gains, subject to the confound in the first major comment.
minor comments (4)
  1. [§5.3] The text reports a '17.74%' improvement for Qwen2.5-VL-7B, but Table 3 shows 62.18 - 44.46 = 17.72. Please make the numbers consistent.
  2. [Figure 4] The blind VLM test is performed only with InternVL3-8B. The conclusion that 'the benchmark questions cannot be answered without visual input' should be qualified as demonstrated for one model; running the blind condition for at least one additional model, particularly a closed-source one, would strengthen the claim.
  3. [§11.3, Table 10] The text states that Scene-CoT requires '42 seconds per question' for segment-description generation, while Table 10 lists an average inference time of 47.10 seconds for Scene-CoT. Clarify whether the 42 s figure excludes the LLM QA time and reconcile the numbers.
  4. [§3.1, §11.4] The empirical choices of the five-second segment length, 50 m distance cutoff, and 50% overlap are described only briefly, and the LLM selection for Scene-CoT is relegated to the supplement. Since these choices affect benchmark difficulty and method performance, a sentence in the main text summarizing the selection procedure would improve reproducibility.

Circularity Check

2 steps flagged

TCogMap's headline gain is partly circular: the injected ego-motion summary is built from the same trajectory source as TAD's ground-truth labels, so ego-task improvements are not independent evidence of VLM temporal understanding.

specific steps
  1. self definitional [Sec. 3.1 (Benchmark Annotations); Sec. 4.2 (TCogMap, Algorithm 1); Sec. 5.1 (Experimental Setup)]
    "Video segments with the corresponding bird's eye view (BEV) visualization, including the trajectories of ego and nearby vehicles, were provided to annotators. Annotators were then asked to label a vehicle's action corresponding to the dominant behavior during the video segment. ... TCogMap analyzes the sequence of ego car poses (translation, rotation, and timestamp, which are provided with NuScenes) in order to classify its motion. ... For the TCogMap method, the thresholds were empirically set to V_stat = 0.2 m/s, V_stopping = 1.0 m/s, Ψ_turn = 10.0 degrees, V_y,lc = 0.4 m/s, V_x,lc = 1.0 m/s"

    TAD's ground-truth action labels are assigned by annotators viewing BEV visualizations that include ego/nearby trajectories; TCogMap's motion classifier consumes exactly those NuScenes ego poses and, with empirically set thresholds, emits one of the same eight action categories (Algorithm 1). The classifier is therefore a reimplementation of the label source, not an independent VLM prediction. The claimed TCogMap gains on ego questions—including all of Action Duration and Relative Temporal Action Localization—receive the ground-truth action string as text input, so the improvement is forced by construction rather than measuring temporal understanding.

  2. fitted input called prediction [Sec. 4.2 (VLM-based QA); Table 6; Sec. 3.3 (Table 2 context)]
    "Crucially, to provide clear temporal context for the model, each motion summary m_i is formatted as a string that specifies the corresponding frames in the segment (e.g., 'Motion summary for Frame1 to Frame7: The ego-vehicle is stopped'). ... Which ego vehicle event happened earlier in the video? Respond with exactly one letter corresponding to the correct option. A. Stopping. B. Starting. Answer: B."

    Each TCogMap summary is formatted as 'Motion summary for Frame1 to Frame7: The ego-vehicle is stopped', so the model is given the exact per-segment ego-action labels used to construct the benchmark answers. For Action Duration and Relative Temporal Action Localization (which Table 2 shows only contain ego questions), the ordered frame-labeled summary directly contains the correct option; Table 6's example asks 'Which ego vehicle event happened earlier?' and the summary already lists the actions in order. The blind test confirms this: 'TCogMap Only setting yields higher accuracies in five tasks compared to Image Only', i.e., the text alone is often sufficient. Thus the reported up-to-17.72% TCogMap gain is partly an artifact of label injection, not evidence of improved VLM temporal reasonin

full rationale

TAD is a new benchmark on NuScenes with human annotation and human/chance calibration; that contribution is externally grounded and not circular. Scene-CoT is a CoT prompting method without injected labels; its mixed results are legitimate empirical findings. The circularity is concentrated in TCogMap, which is also the source of the paper's largest headline number (up to 17.72%). The same ego poses used to construct TAD's action ground truth (via BEV trajectory-based annotation) are fed through a rule-based classifier whose thresholds are disclosed only as 'empirically set' in Sec. 5.1, and whose output vocabulary exactly matches TAD's eight action categories. For ego-only tasks (Action Duration, Relative Temporal Action Localization) and much of the ego subset of Exact/MC Action Recognition and Temporal Action Localization, the injected per-segment summary is effectively an answer key. The baseline + raw ego pose control does not fix this: raw poses are not parsed into action labels, and the paper itself attributes TCogMap's advantage to converting poses into understandable labels. The 'TCogMap Only' blind test—five tasks above Image Only—confirms the text alone supplies the answers. No held-out tuning protocol is reported for the six thresholds, so the up-to-17.72pp gain cannot be separated from threshold/label-injection effects. The abstract's additional claim of up to 10.35% improvement on STSBench has no corresponding experiment in the full text, an unsupported assertion but not a circular one. Score 6 reflects partial circularity: the benchmark and Scene-CoT retain independent content, but the main TCogMap accuracy claim reduces partly by construction.

Axiom & Free-Parameter Ledger

8 free parameters · 6 axioms · 0 invented entities

The central claims rest on human-annotated ground truth, rule-based trajectory classification with hand-set thresholds, and template-generated QA. No new physical entities are postulated; the 'temporal cognitive map' is a representational artifact, not an invented physical object. The largest burden is the undisclosed tuning of TCogMap thresholds, which directly affects the method's headline gains.

free parameters (8)
  • V_stat = 0.2 m/s
    Threshold for classifying a segment as 'Stopped' (Algorithm 1, L9-11); empirically set in Section 5.1.
  • V_stopping = 1.0 m/s
    Speed threshold for 'Starting'/'Stopping' classification (Algorithm 1, L24-27); empirically set.
  • Ψ_turn = 10 degrees
    Yaw-change threshold for turn classification (Algorithm 1, L16-17); empirically set.
  • V_y,lc = 0.4 m/s
    Lateral velocity threshold for lane-change classification (Algorithm 1, L18-22); empirically set.
  • V_x,lc = 1.0 m/s
    Forward velocity threshold for lane-change classification (Algorithm 1, L18-22); empirically set.
  • 1.5× stopping-speed multiplier = 1.5
    Heuristic factor used to distinguish starting/stopping from straight (Algorithm 1, L24-27); chosen by hand.
  • Scene-CoT segment length / overlap / frames = 5s, 50%, 4 frames
    Video partitioning design choices (Section 4.1); affect Scene-CoT accuracy and cost.
  • Benchmark segment length and object distance cutoff = 5s, 50m
    Annotation design choices (Section 3.1); 'empirically observed/set' to balance action completeness and annotation effort.
axioms (6)
  • domain assumption NuScenes ego poses and timestamps are accurate enough for finite-difference velocity estimation
    Algorithm 1 (Section 4.2) computes velocities and yaw changes from raw poses; if pose noise is high, TCogMap labels are unreliable.
  • domain assumption The 8-category action taxonomy is complete and mutually unambiguous for AD temporal understanding
    Section 3.1 defines actions; human-level 81.7% on action recognition suggests taxonomy is usable but some confusion (e.g., stopped vs stopping) is acknowledged in Section 5.2.
  • domain assumption Template-generated QA with logical checks and human verification produces correct and unambiguous ground truth
    Section 3.2 describes the pipeline; no inter-annotator agreement or QA validation statistics are reported.
  • domain assumption Human performance on a random 10% subset is a reliable estimate of the human ceiling
    Section 5.2; small subset and no confidence intervals.
  • domain assumption VLM outputs are deterministic enough that single-run accuracy is comparable across configurations
    Table 3 reports one accuracy per configuration; no temperature/seed reporting or error bars.
  • domain assumption Cognitive maps improve VLM spatial reasoning and extend to temporal driving scenes
    TCogMap builds on [20,57] (Section 4.2); the blind test supports the map's usefulness but it is a prior-result dependency.

pith-pipeline@v1.3.0-alltime-deepseek · 24922 in / 13585 out tokens · 127769 ms · 2026-08-03T18:24:09.934571+00:00 · methodology

0 comments
read the original abstract

Vision-Language Models (VLMs) are increasingly deployed as the perception and reasoning backbone of autonomous agents acting in the wild, with autonomous driving (AD) being one of the most safety-critical instances. Reliable temporal understanding is essential for such agents to anticipate events, attribute causes, and act safely in dynamic environments, yet this remains a significant challenge even for state-of-the-art (SoTA) VLMs. Prior video benchmarks have emphasized other content (sports, cooking, etc.), yet no existing benchmark focuses exclusively on temporal understanding for both short- and long-form AD footage. To fill this gap, we present the Temporal Understanding in Autonomous Driving (TAD) benchmark, comprising nearly 6000 question-answer (QA) pairs across 7 tasks, and evaluate 9 closed- and open-source generalist as well as AD-specialist models. Current SoTA models perform substantially below human accuracy on TAD. To improve the temporal reasoning of VLM-based driving agents, we propose two novel training-free solutions: Scene-CoT, which uses Chain-of-Thought (CoT) reasoning, and TCogMap, which incorporates an ego-centric temporal cognitive map produced by a trajectory-analysis module that operates as an agentic tool around the VLM. Integrated with existing VLMs, our methods improve average accuracy on TAD by up to $17.72\%$ and by up to $10.35\%$ on STSBench. By introducing TAD, benchmarking SoTA models, and proposing effective enhancements, this work aims to catalyze further progress on temporal understanding for agentic AD systems operating in the wild. The benchmark and evaluation code are available at ${\href{https://huggingface.co/datasets/vbdai/TAD}{\text{Hugging Face}}}$ and ${\href{https://github.com/vbdi/tad_bench}{\text{GitHub}}}$, respectively.

Figures

Figures reproduced from arXiv: 2512.05277 by Ahmad Rezaei, Alireza Heidarikhazaei, Kevin Cannons, Mohammad Akbari, Mohammad Asiful Hossain, Mohsen Gholami, Saeed Ranjbar Alvar, Yong Zhang, Zhou Weimin.

Figure 1
Figure 1. Figure 1: Overview of the proposed TAD benchmark. Left: An example of labels and question types. Each segment covers approximately five seconds of a video from the NuScenes dataset. The labels consist of segment-level action annotations for the ego and other vehicles visible from the front camera. Two question types pertain to actions within these video segments, while the remaining five types relate to actions that… view at source ↗
Figure 2
Figure 2. Figure 2: TAD annotation and question generation pipeline. Boxes with green dash: Steps with human verification and quality assessment. [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Overview for the proposed TCogMap method. [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Blind test results for InternvL3-8B. Blind VLM Test: This ablation examines two questions regarding the TAD benchmark: 1) How much a model can infer from the question alone. 2) Whether temporal cogni￾tive maps are sufficient for temporal reasoning in isolation. As shown in [PITH_FULL_IMAGE:figures/full_fig_p008_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Distribution of the actions in the Exact Answer Action Recognition question type. Shorthand notations: Lane (L): change lane to left, Lane (R): change lane to right, Turn (L): Turn left, Turn (R): Turn right, and Straight: Straight, constant speed to be traveling “straight, constant speed”; whereas, non-ego vehicles are most often stopped. This result is quite intu￾itive, as the NuScenes data generally sho… view at source ↗
Figure 6
Figure 6. Figure 6: Word cloud for the questions in the TAD benchmark [PITH_FULL_IMAGE:figures/full_fig_p011_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Pictorial visualization example of the question types and format in the TAD benchmark. [PITH_FULL_IMAGE:figures/full_fig_p013_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Overview of the proposed Scene-CoT method. [PITH_FULL_IMAGE:figures/full_fig_p013_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Exact prompts and sample output with Scene-CoT for a question in TAD. [PITH_FULL_IMAGE:figures/full_fig_p014_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Data annotation pipeline [PITH_FULL_IMAGE:figures/full_fig_p019_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: Qualitative example for Multiple Choice Action Recognition task using the baseline (InternVL3-8B) and the proposed methods. Green text indicates a correct answer, red denotes an incorrect answer. 10 [PITH_FULL_IMAGE:figures/full_fig_p019_11.png] view at source ↗
Figure 12
Figure 12. Figure 12: Qualitative example for Exact Answer Action Recognition task using the baseline (InternVL3-8B) and proposed methods. Green text indicates a correct answer, red denotes an incorrect answer [PITH_FULL_IMAGE:figures/full_fig_p020_12.png] view at source ↗
Figure 13
Figure 13. Figure 13: Qualitative example for Action Duration task using the baseline (InternVL3-8B) and proposed methods. Green text indicates a correct answer, red denotes an incorrect answer. 11 [PITH_FULL_IMAGE:figures/full_fig_p020_13.png] view at source ↗
Figure 14
Figure 14. Figure 14: Qualitative example for Relative Temporal Action Localization task using the baseline (InternVL3-8B) and proposed methods. Green text indicates a correct answer, red denotes an incorrect answer. 12 [PITH_FULL_IMAGE:figures/full_fig_p021_14.png] view at source ↗
Figure 15
Figure 15. Figure 15: Prompts used for question answering. Note that for the baseline and TCogMap, these prompts are provided to the VLM as [PITH_FULL_IMAGE:figures/full_fig_p022_15.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. CPPO: Contrastive Perception Policy Optimization for VLM Agents

    cs.CV 2026-01 conditional novelty 6.0

    CPPO improves VLM agents by adding a contrastive perception loss to GRPO, applied only to entropy-selected image-dependent tokens, beating prior RL finetuning methods on visual reasoning benchmarks.

Reference graph

Works this paper leans on

61 extracted references · 11 linked inside Pith · cited by 1 Pith paper

  1. [1]

    Divprune: Diversity-based visual token pruning for large multimodal models

    Saeed Ranjbar Alvar, Gursimran Singh, Mohammad Akbari, and Yong Zhang. Divprune: Diversity-based visual token pruning for large multimodal models. InProceedings of the Computer Vision and Pattern Recognition Conference, pages 9392–9401, 2025. 8

    2025

  2. [2]

    Cosmos-reason1: From physical common sense to embodied reasoning.arXiv preprint arXiv:2503.15558, 2025

    Alisson Azzolini, Junjie Bai, Hannah Brandon, Jiaxin Cao, Prithvijit Chattopadhyay, Huayu Chen, Jinju Chu, Yin Cui, Jenna Diamond, Yifan Ding, et al. Cosmos-reason1: From physical common sense to embodied reasoning.arXiv preprint arXiv:2503.15558, 2025. 7, 8

    Pith/arXiv arXiv 2025

  3. [3]

    Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al. Qwen2. 5-vl technical report.arXiv preprint arXiv:2502.13923, 2025. 6, 7

    Pith/arXiv arXiv 2025

  4. [4]

    Activitynet: A large-scale video benchmark for human activity understanding

    Fabian Caba Heilbron, Victor Escorcia, Bernard Ghanem, and Juan Carlos Niebles. Activitynet: A large-scale video benchmark for human activity understanding. InProceed- ings of the ieee conference on computer vision and pattern recognition, pages 961–970, 2015. 2

    2015

  5. [5]

    nuscenes: A multi- modal dataset for autonomous driving

    Holger Caesar, Varun Bankiti, Alex H Lang, Sourabh V ora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Gi- ancarlo Baldan, and Oscar Beijbom. nuscenes: A multi- modal dataset for autonomous driving. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11621–11631, 2020. 2, 3

    2020

  6. [6]

    Temporalbench: Benchmarking fine- grained temporal understanding for multimodal video mod- els.arXiv preprint arXiv:2410.10818, 2024

    Mu Cai, Reuben Tan, Jianrui Zhang, Bocheng Zou, Kai Zhang, Feng Yao, Fangrui Zhu, Jing Gu, Yiwu Zhong, Yuzhang Shang, et al. Temporalbench: Benchmarking fine- grained temporal understanding for multimodal video mod- els.arXiv preprint arXiv:2410.10818, 2024. 2

    Pith/arXiv arXiv 2024

  7. [7]

    Automated evaluation of large vision-language models on self-driving corner cases

    Kai Chen, Yanze Li, Wenhua Zhang, Yanxin Liu, Pengxi- ang Li, Ruiyuan Gao, Lanqing Hong, Meng Tian, Xinhai Zhao, Zhenguo Li, et al. Automated evaluation of large vision-language models on self-driving corner cases. In2025 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pages 7817–7826. IEEE, 2025. 3

    2025

  8. [8]

    An image is worth 1/2 tokens after layer 2: Plug-and-play inference acceleration for large vision-language models

    Liang Chen, Haozhe Zhao, Tianyu Liu, Shuai Bai, Junyang Lin, Chang Zhou, and Baobao Chang. An image is worth 1/2 tokens after layer 2: Plug-and-play inference acceleration for large vision-language models. InEuropean Conference on Computer Vision, pages 19–35. Springer, 2024. 8

    2024

  9. [9]

    Timemarker: A versatile video-llm for long and short video understanding with superior temporal localiza- tion ability

    Shimin Chen, Xiaohan Lan, Yitian Yuan, Zequn Jie, and Lin Ma. Timemarker: A versatile video-llm for long and short video understanding with superior temporal localiza- tion ability. 2024. 2

    2024

  10. [10]

    Asynchronous large language model en- hanced planner for autonomous driving

    Yuan Chen, Zi-han Ding, Ziqin Wang, Yan Wang, Lijun Zhang, and Si Liu. Asynchronous large language model en- hanced planner for autonomous driving. InEuropean Con- ference on Computer Vision, pages 22–38. Springer, 2024. 2

    2024

  11. [11]

    Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities.arXiv preprint arXiv:2507.06261, 2025

    Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blis- tein, Ori Ram, Dan Zhang, Evan Rosen, et al. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities.arXiv preprint arXiv:2507.06261, 2025. 6, 7

    Pith/arXiv arXiv 2025

  12. [12]

    Lmdeploy: A toolkit for com- pressing, deploying, and serving llm.https://github

    LMDeploy Contributors. Lmdeploy: A toolkit for com- pressing, deploying, and serving llm.https://github. com/InternLM/lmdeploy, 2023. 2

    2023

  13. [13]

    Sketch, ground, and refine: Top-down dense video caption- ing

    Chaorui Deng, Shizhe Chen, Da Chen, Yuan He, and Qi Wu. Sketch, ground, and refine: Top-down dense video caption- ing. InProceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition, pages 234–243, 2021. 2

    2021

  14. [14]

    Holistic autonomous driving un- derstanding by bird’s-eye-view injected multi-modal large models

    Xinpeng Ding, Jianhua Han, Hang Xu, Xiaodan Liang, Wei Zhang, and Xiaomeng Li. Holistic autonomous driving un- derstanding by bird’s-eye-view injected multi-modal large models. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13668– 13677, 2024. 2

    2024

  15. [15]

    Extreme com- pression of large language models via additive quantization

    Vage Egiazarian, Andrei Panferov, Denis Kuznedelev, Elias Frantar, Artem Babenko, and Dan Alistarh. Extreme com- pression of large language models via additive quantization. arXiv preprint arXiv:2401.06118, 2024. 8

    Pith/arXiv arXiv 2024

  16. [16]

    Gptq: Accurate post-training quantization for generative pre-trained transformers.arXiv preprint arXiv:2210.17323, 2022

    Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. Gptq: Accurate post-training quantization for generative pre-trained transformers.arXiv preprint arXiv:2210.17323, 2022. 8

    Pith/arXiv arXiv 2022

  17. [17]

    Stsbench: A spatio-temporal scenario benchmark for multi-modal large language models in autonomous driving

    Christian Fruhwirth-Reisinger, Du ˇsan Mali´c, Wei Lin, David Schinagl, Samuel Schulter, and Horst Possegger. Stsbench: A spatio-temporal scenario benchmark for multi-modal large language models in autonomous driving. 2025. 3

    2025

  18. [18]

    Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal llms in video analysis

    Chaoyou Fu, Yuhan Dai, Yongdong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang, Chenyu Zhou, Yunhang Shen, Mengdan Zhang, et al. Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal llms in video analysis. InProceedings of the Computer Vision and Pattern Recognition Conference, pages 24108–24118, 2025. 2

    2025

  19. [19]

    Casp: Compression of large multimodal mod- els based on attention sparsity

    Mohsen Gholami, Mohammad Akbari, Kevin Cannons, and Yong Zhang. Casp: Compression of large multimodal mod- els based on attention sparsity. InProceedings of the Com- puter Vision and Pattern Recognition Conference, pages 9372–9381, 2025. 8

    2025

  20. [20]

    Spatial reasoning with vision-language models in ego-centric multi-view scenes.arXiv preprint arXiv:2509.06266, 2025

    Mohsen Gholami, Ahmad Rezaei, Zhou Weimin, Sitong Mao, Shunbo Zhou, Yong Zhang, and Mohammad Akbari. Spatial reasoning with vision-language models in ego-centric multi-view scenes.arXiv preprint arXiv:2509.06266, 2025. 3, 5

    arXiv 2025

  21. [21]

    Surds: Benchmarking spatial understand- ing and reasoning in driving scenarios with vision language models

    Xianda Guo, Ruijun Zhang, Yiqun Duan, Yuhang He, Dujun Nie, Wenke Huang, Chenming Zhang, Shuai Liu, Hao Zhao, and Long Chen. Surds: Benchmarking spatial understand- ing and reasoning in driving scenarios with vision language models. 2024. 3

    2024

  22. [22]

    Lita: Language instructed temporal-localization assistant

    De-An Huang, Shijia Liao, Subhashree Radhakrishnan, Hongxu Yin, Pavlo Molchanov, Zhiding Yu, and Jan Kautz. Lita: Language instructed temporal-localization assistant. In European Conference on Computer Vision, pages 202–218. Springer, 2024. 2, 6

    2024

  23. [23]

    Robotron- drive: All-in-one large multimodal model for autonomous driving

    Zhijian Huang, Chengjian Feng, Feng Yan, Baihui Xiao, Ze- qun Jie, Yujie Zhong, Xiaodan Liang, and Lin Ma. Robotron- drive: All-in-one large multimodal model for autonomous driving. InProceedings of the IEEE/CVF International Con- ference on Computer Vision, pages 8011–8021, 2025. 7, 8 14

    2025

  24. [24]

    Drivelmm-o1: A step-by-step reasoning dataset and large multimodal model for driving scenario understanding.arXiv preprint arXiv:2503.10621, 2025

    Ayesha Ishaq, Jean Lahoud, Ketan More, Omkar Thawakar, Ritesh Thawkar, Dinura Dissanayake, Noor Ahsan, Yuhao Li, Fahad Shahbaz Khan, Hisham Cholakkal, et al. Drivelmm-o1: A step-by-step reasoning dataset and large multimodal model for driving scenario understanding.arXiv preprint arXiv:2503.10621, 2025. 3

    Pith/arXiv arXiv 2025

  25. [25]

    Textual explanations for self-driving ve- hicles

    Jinkyu Kim, Anna Rohrbach, Trevor Darrell, John Canny, and Zeynep Akata. Textual explanations for self-driving ve- hicles. InProceedings of the European conference on com- puter vision (ECCV), pages 563–578, 2018. 2

    2018

  26. [26]

    Do you remember? dense video captioning with cross-modal memory retrieval

    Minkuk Kim, Hyeon Bae Kim, Jinyoung Moon, Jinwoo Choi, and Seong Tae Kim. Do you remember? dense video captioning with cross-modal memory retrieval. InProceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13894–13904, 2024. 2

    2024

  27. [27]

    Dense-captioning events in videos

    Ranjay Krishna, Kenji Hata, Frederic Ren, Li Fei-Fei, and Juan Carlos Niebles. Dense-captioning events in videos. In Proceedings of the IEEE international conference on com- puter vision, pages 706–715, 2017. 2

    2017

  28. [28]

    Seed-bench: Bench- marking multimodal large language models

    Bohao Li, Yuying Ge, Yixiao Ge, Guangzhi Wang, Rui Wang, Ruimao Zhang, and Ying Shan. Seed-bench: Bench- marking multimodal large language models. InProceedings of the IEEE/CVF Conference on Computer Vision and Pat- tern Recognition, pages 13299–13308, 2024. 2

    2024

  29. [29]

    Mvbench: A comprehensive multi-modal video understand- ing benchmark

    Kunchang Li, Yali Wang, Yinan He, Yizhuo Li, Yi Wang, Yi Liu, Zun Wang, Jilan Xu, Guo Chen, Ping Luo, et al. Mvbench: A comprehensive multi-modal video understand- ing benchmark. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22195– 22206, 2024. 2

    2024

  30. [30]

    Fine-grained evaluation of large vision-language mod- els in autonomous driving

    Yue Li, Meng Tian, Zhenyu Lin, Jiangtong Zhu, Dechang Zhu, Haiqiang Liu, Yueyi Zhang, Zhiwei Xiong, and Xinhai Zhao. Fine-grained evaluation of large vision-language mod- els in autonomous driving. InProceedings of the IEEE/CVF International Conference on Computer Vision, pages 9431– 9442, 2025. 2, 3

    2025

  31. [31]

    Sti-bench: Are mllms ready for precise spatial-temporal world understanding? 2025

    Yun Li, Yiming Zhang, Tao Lin, XiangRui Liu, Wenxiao Cai, Zheng Liu, and Bo Zhao. Sti-bench: Are mllms ready for precise spatial-temporal world understanding? 2025. 2, 3

    2025

  32. [32]

    Boosting multimodal large language models with visual to- kens withdrawal for rapid inference

    Zhihang Lin, Mingbao Lin, Luxi Lin, and Rongrong Ji. Boosting multimodal large language models with visual to- kens withdrawal for rapid inference. InProceedings of the AAAI Conference on Artificial Intelligence, pages 5334– 5342, 2025. 8

    2025

  33. [33]

    Tempcom- pass: Do video llms really understand videos? 2024

    Yuanxin Liu, Shicheng Li, Yi Liu, Yuxiang Wang, Shuhuai Ren, Lei Li, Sishuo Chen, Xu Sun, and Lu Hou. Tempcom- pass: Do video llms really understand videos? 2024. 2

    2024

  34. [34]

    Dolphins: Multimodal language model for driving

    Yingzi Ma, Yulong Cao, Jiachen Sun, Marco Pavone, and Chaowei Xiao. Dolphins: Multimodal language model for driving. InEuropean Conference on Computer Vision, pages 403–420. Springer, 2024. 2

    2024

  35. [35]

    Drama: Joint risk localization and captioning in driving

    Srikanth Malla, Chiho Choi, Isht Dwivedi, Joon Hee Choi, and Jiachen Li. Drama: Joint risk localization and captioning in driving. InProceedings of the IEEE/CVF winter confer- ence on applications of computer vision, pages 1043–1052,

  36. [36]

    Lingoqa: Visual question answering for autonomous driv- ing

    Ana-Maria Marcu, Long Chen, Jan H ¨unermann, Alice Karn- sund, Benoit Hanotte, Prajwal Chidananda, Saurabh Nair, Vijay Badrinarayanan, Alex Kendall, Jamie Shotton, et al. Lingoqa: Visual question answering for autonomous driv- ing. InEuropean Conference on Computer Vision, pages 252–269. Springer, 2024. 2, 3

    2024

  37. [37]

    Reason2drive: Towards interpretable and chain-based reasoning for au- tonomous driving

    Ming Nie, Renyuan Peng, Chunwei Wang, Xinyue Cai, Jianhua Han, Hang Xu, and Li Zhang. Reason2drive: Towards interpretable and chain-based reasoning for au- tonomous driving. InEuropean Conference on Computer Vision, pages 292–308. Springer, 2024. 2

    2024

  38. [38]

    Gpt-5.https://openai.com/gpt- 5/,

    OpenAI. Gpt-5.https://openai.com/gpt- 5/,

  39. [39]

    Vlp: Vision language planning for autonomous driving

    Chenbin Pan, Burhaneddin Yaman, Tommaso Nesti, Abhirup Mallik, Alessandro G Allievi, Senem Velipasalar, and Liu Ren. Vlp: Vision language planning for autonomous driving. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14760–14769, 2024. 2

    2024

  40. [40]

    Nuscenes-qa: A multi-modal visual ques- tion answering benchmark for autonomous driving scenario

    Tianwen Qian, Jingjing Chen, Linhai Zhuo, Yang Jiao, and Yu-Gang Jiang. Nuscenes-qa: A multi-modal visual ques- tion answering benchmark for autonomous driving scenario. InProceedings of the AAAI Conference on Artificial Intelli- gence, pages 4542–4550, 2024. 3

    2024

  41. [41]

    Rank2tell: A multimodal driving dataset for joint importance ranking and reasoning

    Enna Sachdeva, Nakul Agarwal, Suhas Chundi, Sean Roelofs, Jiachen Li, Mykel Kochenderfer, Chiho Choi, and Behzad Dariush. Rank2tell: A multimodal driving dataset for joint importance ranking and reasoning. InProceedings of the IEEE/CVF winter conference on applications of com- puter vision, pages 7513–7522, 2024. 3

    2024

  42. [42]

    Lmdrive: Closed-loop end-to-end driving with large language models

    Hao Shao, Yuxuan Hu, Letian Wang, Guanglu Song, Steven L Waslander, Yu Liu, and Hongsheng Li. Lmdrive: Closed-loop end-to-end driving with large language models. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15120–15130, 2024. 2

    2024

  43. [43]

    Video-xl: Extra-long vision language model for hour-scale video understanding

    Yan Shu, Zheng Liu, Peitian Zhang, Minghao Qin, Jun- jie Zhou, Zhengyang Liang, Tiejun Huang, and Bo Zhao. Video-xl: Extra-long vision language model for hour-scale video understanding. InProceedings of the Computer Vision and Pattern Recognition Conference, pages 26160–26169,

  44. [44]

    Drivelm: Driving with graph visual question answering

    Chonghao Sima, Katrin Renz, Kashyap Chitta, Li Chen, Hanxue Zhang, Chengen Xie, Jens Beißwenger, Ping Luo, Andreas Geiger, and Hongyang Li. Drivelm: Driving with graph visual question answering. InEuropean conference on computer vision, pages 256–274. Springer, 2024. 2, 3

    2024

  45. [45]

    Drivevlm: The convergence of autonomous driving and large vision-language models

    Xiaoyu Tian, Junru Gu, Bailin Li, Yicheng Liu, Yang Wang, Zhiyong Zhao, Kun Zhan, Peng Jia, Xianpeng Lang, and Hang Zhao. Drivevlm: The convergence of autonomous driving and large vision-language models. 2024. 2

    2024

  46. [46]

    Quip#: Even better llm quantization with hadamard incoherence and lattice code- books.arXiv preprint arXiv:2402.04396, 2024

    Albert Tseng, Jerry Chee, Qingyao Sun, V olodymyr Kuleshov, and Christopher De Sa. Quip#: Even better llm quantization with hadamard incoherence and lattice code- books.arXiv preprint arXiv:2402.04396, 2024. 8

    Pith/arXiv arXiv 2024

  47. [47]

    Grounded-videollm: Sharpening fine-grained tem- poral grounding in video large language models

    Haibo Wang, Zhiyang Xu, Yu Cheng, Shizhe Diao, Yu- fan Zhou, Yixin Cao, Qifan Wang, Weifeng Ge, and Lifu Huang. Grounded-videollm: Sharpening fine-grained tem- poral grounding in video large language models. 2024. 2 15

    2024

  48. [48]

    Omnidrive: A holistic vision-language dataset for au- tonomous driving with counterfactual reasoning

    Shihao Wang, Zhiding Yu, Xiaohui Jiang, Shiyi Lan, Min Shi, Nadine Chang, Jan Kautz, Ying Li, and Jose M Al- varez. Omnidrive: A holistic vision-language dataset for au- tonomous driving with counterfactual reasoning. InProceed- ings of the Computer Vision and Pattern Recognition Confer- ence, pages 22442–22452, 2025. 2

    2025

  49. [49]

    End-to-end dense video captioning with parallel decoding

    Teng Wang, Ruimao Zhang, Zhichao Lu, Feng Zheng, Ran Cheng, and Ping Luo. End-to-end dense video captioning with parallel decoding. InProceedings of the IEEE/CVF international conference on computer vision, pages 6847– 6857, 2021. 2

    2021

  50. [50]

    Chain-of-thought prompting elicits reasoning in large lan- guage models.Advances in neural information processing systems, 35:24824–24837, 2022

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large lan- guage models.Advances in neural information processing systems, 35:24824–24837, 2022. 4

    2022

  51. [51]

    Lan- guage prompt for autonomous driving

    Dongming Wu, Wencheng Han, Yingfei Liu, Tiancai Wang, Cheng-zhong Xu, Xiangyu Zhang, and Jianbing Shen. Lan- guage prompt for autonomous driving. InProceedings of the AAAI Conference on Artificial Intelligence, pages 8359– 8367, 2025. 3

    2025

  52. [52]

    Number it: Temporal grounding videos like flipping manga

    Yongliang Wu, Xinting Hu, Yuyang Sun, Yizhou Zhou, Wenbo Zhu, Fengyun Rao, Bernt Schiele, and Xu Yang. Number it: Temporal grounding videos like flipping manga. InProceedings of the Computer Vision and Pattern Recogni- tion Conference, pages 13754–13765, 2025. 2

    2025

  53. [53]

    Are vlms ready for autonomous driving? an empirical study from the reliability, data and metric perspectives

    Shaoyuan Xie, Lingdong Kong, Yuhao Dong, Chonghao Sima, Wenwei Zhang, Qi Alfred Chen, Ziwei Liu, and Liang Pan. Are vlms ready for autonomous driving? an empirical study from the reliability, data and metric perspectives. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 6585–6597, 2025. 3

    2025

  54. [54]

    Drivegpt4-v2: Harnessing large language model capabilities for enhanced closed-loop autonomous driving

    Zhenhua Xu, Yan Bai, Yujia Zhang, Zhuoling Li, Fei Xia, Kwan-Yee K Wong, Jianqiang Wang, and Hengshuang Zhao. Drivegpt4-v2: Harnessing large language model capabilities for enhanced closed-loop autonomous driving. InProceed- ings of the Computer Vision and Pattern Recognition Con- ference, pages 17261–17270, 2025. 2

    2025

  55. [55]

    Vid2seq: Large-scale pretraining of a vi- sual language model for dense video captioning

    Antoine Yang, Arsha Nagrani, Paul Hongsuck Seo, An- toine Miech, Jordi Pont-Tuset, Ivan Laptev, Josef Sivic, and Cordelia Schmid. Vid2seq: Large-scale pretraining of a vi- sual language model for dense video captioning. InProceed- ings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10714–10726, 2023. 2

    2023

  56. [56]

    An Yang, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoyan Huang, Jiandong Jiang, Jianhong Tu, Jian- wei Zhang, Jingren Zhou, et al. Qwen2. 5-1m technical re- port.arXiv preprint arXiv:2501.15383, 2025. 6

    Pith/arXiv arXiv 2025

  57. [57]

    Thinking in space: How mul- timodal large language models see, remember, and recall spaces

    Jihan Yang, Shusheng Yang, Anjali W Gupta, Rilyn Han, Li Fei-Fei, and Saining Xie. Thinking in space: How mul- timodal large language models see, remember, and recall spaces. InProceedings of the Computer Vision and Pattern Recognition Conference, pages 10632–10643, 2025. 5

    2025

  58. [58]

    Weakly- supervised video object grounding from text by loss weight- ing and object interaction.arXiv preprint arXiv:1805.02834,

    Luowei Zhou, Nathan Louis, and Jason J Corso. Weakly- supervised video object grounding from text by loss weight- ing and object interaction.arXiv preprint arXiv:1805.02834,

    Pith/arXiv arXiv

  59. [59]

    Towards automatic learning of procedures from web instructional videos

    Luowei Zhou, Chenliang Xu, and Jason Corso. Towards automatic learning of procedures from web instructional videos. InProceedings of the AAAI conference on artificial intelligence, 2018. 2

    2018

  60. [60]

    Streaming dense video captioning

    Xingyi Zhou, Anurag Arnab, Shyamal Buch, Shen Yan, Austin Myers, Xuehan Xiong, Arsha Nagrani, and Cordelia Schmid. Streaming dense video captioning. InProceedings of the IEEE/CVF Conference on Computer Vision and Pat- tern Recognition, pages 18243–18252, 2024. 2

    2024

  61. [61]

    Internvl3: Exploring advanced training and test-time recipes for open-source multimodal models.arXiv preprint arXiv:2504.10479, 2025

    Jinguo Zhu, Weiyun Wang, Zhe Chen, Zhaoyang Liu, Shen- glong Ye, Lixin Gu, Hao Tian, Yuchen Duan, Weijie Su, Jie Shao, et al. Internvl3: Exploring advanced training and test-time recipes for open-source multimodal models.arXiv preprint arXiv:2504.10479, 2025. 6, 7 16

    Pith/arXiv arXiv 2025