Pith. sign in

REVIEW 4 major objections 6 minor 2 cited by

TUMTraffic-VideoQA: A Benchmark for Unified Spatio-Temporal Video Understanding in Traffic Scenes

T0 review · 4 major / 6 minor · reviewed 2026-08-09 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper introduces a roadside video-language benchmark that unifies multi-choice video QA, referred object captioning, and spatio-temporal object grounding, and a fine-tuned baseline that clearly beats zero-shot open models.

desk verdict A genuinely new roadside video-language benchmark with an honest no-vision ablation, but the spatial-QA labels rest on an unvalidated pseudo-3D projection and the QA track leaks heavily to language priors. read the letter →

arxiv 2502.02449 v1 pith:BNSMCFEY submitted 2025-02-04 cs.CV

Xingcheng Zhou , Konstantinos Larintzakis , Hao Guo , Walter Zimmer , Mingyu Liu , Hu Cao , Jiajie Zhang , Venkatnarayanan Lakshminarasimhan
show 2 more authors
Leah Strand Alois C. Knoll
This is my paper · ORCID
classification cs.CV
keywords videoquestionansweringspatio-temporalgroundingreferredobjectcaptioningtrafficsceneunderstandingroadsidesurveillancevision-languagemodelsbenchmarkdatasetintelligenttransportationsystems
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper is trying to establish that a single benchmark can measure how well vision-language models understand roadside traffic videos in space and time. It introduces a dataset of 1,000 videos with 85,000 multiple-choice QA pairs, 2,300 referred-object captions, and 5,700 spatio-temporal grounding annotations, using a tuple object expression $(c, f_n, x, y)$ as a common interface. The paper also trains a unified baseline and reports that it reaches roughly 82% mean accuracy on multi-choice QA, far above the best zero-shot open model at about 56%, while positioning questions remain the hardest for every model. As a shared resource, it is intended to give intelligent transportation research a common testbed for fine-grained video reasoning under realistic roadside conditions.

What carries the argument

The load-bearing mechanism is the four-element tuple $(c, f_n, x, y)$ for referring to any object at any moment in a video: it gives one standardized language interface that the three benchmark tasks all consume. The benchmark's spatial QA labels are produced by projecting 2D detections to pseudo-3D positions under a planar ground assumption and binning relative directions into eight angular regions, so that positional relationships can be phrased as natural-language questions.

What would settle it

Take a random sample of validation videos, have independent human annotators label the relative spatial directions and the first and last visible frames of referred objects, and compare their labels with the benchmark's ground truth; if agreement is low, the reported accuracy and grounding errors do not measure true spatio-temporal understanding.

Watch

Extended reading notes

Core claim

The central claim is that this is the first roadside, third-person-perspective video-language benchmark to unify three tasks in one evaluation framework: multiple-choice question answering, referred object captioning, and spatio-temporal object grounding. Object references are written as tuples $(c, f_n, x, y)$, where $c$ is a persistent object identifier, $f_n$ is a normalized frame timestamp, and $(x,y)$ are normalized image coordinates, so the same object can be pointed to across the whole video and grounding is expressed as a start/end pair of tuples. The paper reports that its fine-tuned 7B baseline reaches about 81-82% mean accuracy on multi-choice QA versus at most 56% for zero-shot open-source vision-language models, cuts spatio-temporal error from roughly 0.8 down to about 0.24, and substantially improves captioning metrics, while positioning questions focusing on 3D spatial relationships remain the most difficult for all models tested. The dataset covers accidents, rescue operations, congestion, and adverse weather, and is released publicly.

Load-bearing premise

The benchmark scores are trustworthy only if the automatically produced labels are accurate: spatial answers come from projecting 2D detections to pseudo-3D positions under a planar ground assumption and binning directions into eight angles, while most questions and captions are written by a large language model with only selective human checks.

Editorial extensions

If this is right

  • A task-specific fine-tune on this benchmark currently gives a large boost over zero-shot open models, so a unified roadside video model needs in-domain training, not just a strong generic backbone.
  • Positioning questions are the hardest dimension for every model, so progress on the benchmark will depend on improving genuine 3D spatial reasoning rather than object recognition.
  • Multi-resolution visual token sampling improves multi-choice QA and captioning but slightly hurts spatio-temporal grounding, revealing a trade-off between frame-level detail and cross-frame consistency.
  • Increasing the number of input frames helps all tasks, with most of the gain coming by 11 frames and little additional gain at 101 frames, so efficient video token representation is a key bottleneck.
  • Because grounding is evaluated as start/end tuples with normalized coordinates, the benchmark directly measures cross-frame object association with a single numeric error, making it usable for tracking and grounding research.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the planar-ground pseudo-3D labels generalize, the same eight-angle binning pipeline could auto-generate spatial QA for other roadside camera networks without manual 3D annotation.
  • The tuple interface could become a common interchange format for referring to objects across video datasets in surveillance and embodied settings, not only traffic.
  • The no-visual-input ablation suggests part of the QA score may come from language priors; a counterfactual benchmark with balanced answer distributions would reveal how much of the apparent understanding is genuinely visual.
  • A testable extension is to replace the discrete 45-degree angular bins with continuous angles to see whether the remaining positioning errors come from bin granularity or from depth ambiguity in the pseudo-3D projection.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. This paper introduces TUMTraffic-VideoQA, a roadside traffic video-language benchmark built from 1,000 videos with 85,000 multiple-choice QA pairs, 2,300 referred-object captioning annotations, and 5,700 spatio-temporal grounding annotations. It proposes a tuple-based object expression (c, fn, x, y), three evaluation tasks (MQA, V-ROC, ST-OG), and a Qwen-2 based baseline, TUMTraffic-Qwen, with four visual token sampling strategies. Experiments report fine-tuned baseline results and zero-shot results for three open-source VLMs, along with ablations on frame count and token sampling.

Significance. Should the annotation pipeline be validated, the dataset would be a genuinely useful resource: it targets an under-served roadside perspective, unifies three tasks under one evaluation protocol, and comes with a public release and a reproducible baseline. The paper also contains a commendable no-vision ablation (Table 6) and an honest discussion of text-pattern exploitation. This transparency does not, however, dissolve the concern that the current MQA numbers substantially reflect language priors and that spatial ground truth rests on an unvalidated pseudo-3D projection.

major comments (4)
  1. [Appendix A.2, Eq. (8)] The spatial labels—both the eight relative-position classes in MQA and the references used in ST-OG—are derived from camera-based pseudo-3D coordinates, but the manuscript never states which 2D image point is projected (bounding-box center, bottom-center, or other), nor does it provide calibration reprojection error or a validation of the planar-ground assumption. For roadside cameras with nonzero pitch, projecting the box center instead of the ground-contact point produces a systematic, distance-dependent error that propagates into motion direction and all relative-position bins. Please specify the exact projection procedure, report intrinsic/extrinsic calibration accuracy, and include a quantitative agreement study between the automatically derived spatial labels and human-verified spatial relations on a labeled subset.
  2. [Section B.1, Table 6] The no-vision baseline reaches 70.18% (0.5B) and 73.32% (7B) positioning accuracy and 67.59% / 71.01% overall MQA accuracy with no video input. This shows that the MQA track is answerable to a large degree from the question text, option statistics, and template priors, so the current evaluation numbers do not isolate spatio-temporal visual understanding. The paper acknowledges the effect in the appendix, but the abstract and Section 5.1 still frame the benchmark primarily as a measure of spatio-temporal understanding. Please report per-type and per-difficulty gaps between no-vision and vision inputs, analyze answer/option distribution biases, and either add a suitable control protocol to the benchmark or temper the central claims so they are commensurate with the measured visual signal.
  3. [Section 3.1] QA pairs and captions are generated by GPT-4o-mini from templates, with only a 'selective quality evaluation' whose sample size, pass rates, and agreement statistics are not reported; the manual quality assurance step is described for metadata, not for the final QA or grounding labels. Without quantitative verification, label noise is uncontrolled and could inflate or deflate model scores in unknown ways. Please provide the number of QA pairs filtered, inter-annotator agreement on a random sample, per-question-type human error rates, and a clear description of the human verification protocol for the released annotations.
  4. [Appendix A.2, Eq. (8)] The relative-position bins are asymmetric in angular width: 'front left' and 'front right' each cover 60 degrees, while 'left' and 'right' cover only 30 degrees (and 'front' covers 30 degrees). This asymmetry is not justified in the text and can systematically skew the label distribution of positioning questions. Either justify the convention with reference to traffic-scene semantics or make the bins symmetric, and report the resulting label distribution per bin.
minor comments (6)
  1. [Table 1] Table 1 reports 87.3k QAs/captions while the abstract and Section 3.3 report 85k QA pairs; clarify that the 87.3k figure includes the 2.3k captioning annotations.
  2. [Eq. (7)] There is a mismatched parenthesis in Eq. (7) in the definition of p(X_a | S_v(Z_v), X_q); please fix the typesetting.
  3. [Section 4.1] The phrase 'The video is uniformly divided into 100 segments, including the first and last frames, resulting in a total of N = 101 frames' is confusing; 'sampled uniformly at 101 timestamps' would be clearer.
  4. [Table 5] METEOR and SPICE are reported only for the TUMTraffic-Qwen baseline, while open-source models show '-' without explanation; state whether these metrics were computed and omitted for space, or add them.
  5. [Section B.1, Table 6] The row label 'without' for frame count should be '0 frames' for readability.
  6. [Section 4.2] Training hyperparameters (batch size, learning rate schedule, warmup, number of steps, data augmentation) are not given; please add them for reproducibility.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the dataset labels, baseline training, and validation evaluation are decoupled, and the self-citations are context-only.

full rationale

The paper's central claim is the introduction of a new benchmark and a baseline model. The derivation chain is linear: roadside videos are collected; off-the-shelf detectors and trackers produce bounding boxes and trajectories; a pseudo-3D projection (Appendix A.2) converts these into positioning and motion labels; templates and GPT-4o-mini generate QA pairs; the TUMTraffic-Qwen baseline is trained on the train split and evaluated on a disjoint validation split. No parameter is fitted to the validation set and then reported as a prediction, and the tuple (c, fn, x, y) is an annotation/output format rather than a fitted quantity. The positioning ground truth is defined by the same pseudo-3D projection that generates it, but that is standard benchmark label construction, not a derivation of a result from its own input. The no-vision control in Table 6 is an honest analysis of text priors and does not indicate circularity; it is a validity concern about how much the MQA track measures visual reasoning. The self-citations [13], [34], [35] are surveys and position papers used for context and do not carry the benchmark claim; no uniqueness theorem or ansatz is imported from author-only prior work. The under-specified pseudo-3D projection is a correctness risk, not a circularity, because it concerns label trustworthiness rather than the logical derivation of predictions from fitted inputs.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

No physics or formal derivation is involved; the central product is an annotated dataset, so the ledger records annotation-pipeline assumptions and baseline hyperparameters rather than mathematical axioms. No invented entities: the tuple (c, fn, x, y) is an annotation format, not a new physical or theoretical construct.

free parameters (4)
  • Token pruning ratio r = 0.25
    Sets the fraction of visual tokens retained in MultiRes Token Pruning (Eq. 5); chosen to keep token counts comparable to spatial pooling, not derived from data.
  • Relative-position angular thresholds = 15, 75, 105, 165 degrees
    Eq. 8 defines ground-truth labels for Positioning questions; thresholds are hand-chosen and not validated against human judgments.
  • Input frame count N = 101
    Uniform sampling into 100 segments including first and last frames; Table 6 shows diminishing returns beyond 11 frames, so this is a design choice.
  • High and low resolution token grids = 27x27 and 14x14
    Table 2 uses these spatial token resolutions for SigLIP features; chosen for computational efficiency and not derived from task performance.
assumptions (5)
  • domain assumption Planar ground assumption for 2D-to-3D projection
    Appendix A.2 projects 2D detections to pseudo-3D positions on a ground plane using historical camera calibration; non-planar roads or calibration drift would make Positioning labels wrong.
  • domain assumption Off-the-shelf detectors and trackers are accurate enough to serve as ground truth metadata
    Section 3.1 uses YOLOv10, DETR, and open-vocabulary detectors to generate boxes and trajectories; only selective human QA is applied, so detector errors propagate into QA labels.
  • domain assumption LLM-generated QA pairs and captions are semantically correct after filtering
    Section 3.1 relies on GPT-4o-mini for QA generation and rephrasing and on large VLMs for appearance captions; quality evaluation is selective, not exhaustive.
  • domain assumption Weather and traffic record alignment via GPT-4o and embeddings is time-accurate
    Section 3.1 aligns video timestamps with historical weather and accident records; alignment errors would mislabel environmental conditions used in benchmark questions.
  • domain assumption Angle-based relative position labels are valid for moving objects
    Appendix A.2 Eq. 8 measures the angle between a moving object's motion direction and the vector to another object; for stationary, turning, or briefly occluded objects, the computed labels can be ambiguous.

how reviews work

0 comments
Cite this review

Pith. "Pith review of TUMTraffic-VideoQA: A Benchmark for Unified Spatio-Temporal Video Understanding in Traffic Scenes." pith.science (2026). https://pith.science/paper/BNSMCFEY

@misc{pith2026250202449,
  author       = {Pith},
  title        = {Pith review of: TUMTraffic-VideoQA: A Benchmark for Unified Spatio-Temporal Video Understanding in Traffic Scenes},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/BNSMCFEY}},
  note         = {Machine review of arXiv:2502.02449}
}
read the original abstract

We present TUMTraffic-VideoQA, a novel dataset and benchmark designed for spatio-temporal video understanding in complex roadside traffic scenarios. The dataset comprises 1,000 videos, featuring 85,000 multiple-choice QA pairs, 2,300 object captioning, and 5,700 object grounding annotations, encompassing diverse real-world conditions such as adverse weather and traffic anomalies. By incorporating tuple-based spatio-temporal object expressions, TUMTraffic-VideoQA unifies three essential tasks-multiple-choice video question answering, referred object captioning, and spatio-temporal object grounding-within a cohesive evaluation framework. We further introduce the TUMTraffic-Qwen baseline model, enhanced with visual token sampling strategies, providing valuable insights into the challenges of fine-grained spatio-temporal reasoning. Extensive experiments demonstrate the dataset's complexity, highlight the limitations of existing models, and position TUMTraffic-VideoQA as a robust foundation for advancing research in intelligent transportation systems. The dataset and benchmark are publicly available to facilitate further exploration.

Figures

Figures reproduced from arXiv: 2502.02449 by the authors.

Figure 1
Figure 1. TUMTraffic-VideoQA introduces a comprehensive benchmark for video-level traffic scene understanding. Our baseline model, [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Different methods for describing objects in images and [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. The workflow of the semi-automatic annotation pipeline for TUMTraffic-VideoQA generation, integrating external database, [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (22 more)
Figure 4
Figure 4. Figure 4: Statistical distributions of the dataset, including word [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]
Figure 5
Figure 5. Figure 5: Overview of the TUMTraffic-Qwen baseline model. [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]
Figure 6
Figure 6. Figure 6: Dataset distribution of video recordings by time, weather conditions, and perspectives. [PITH_FULL_IMAGE:figures/full_fig_p011_6.png]
Figure 7
Figure 7. Figure 7: Distributions of video recordings across time, weather conditions, and camera types in the dataset. [PITH_FULL_IMAGE:figures/full_fig_p011_7.png]
Figure 8
Figure 8. Figure 8: Illustration of the eight spatial regions used to cate [PITH_FULL_IMAGE:figures/full_fig_p012_8.png]
Figure 9
Figure 9. Figure 9: Results visualization for the open-source models and TUMTraffic-QWen baseline models on the Multi-Choice QA. [PITH_FULL_IMAGE:figures/full_fig_p013_9.png]
Figure 10
Figure 10. Figure 10: Illustration of cosine similarity-based token pruning, with dark-colored patches representing discarded tokens and preserved [PITH_FULL_IMAGE:figures/full_fig_p014_10.png]
Figure 11
Figure 11. Figure 11: The system prompt used in the experiments of TUMTraffic-VideoQA dataset. [PITH_FULL_IMAGE:figures/full_fig_p014_11.png]
Figure 12
Figure 12. Figure 12: Spatio-Temporal Object Grounding: A fire truck parked at the roadside. [PITH_FULL_IMAGE:figures/full_fig_p015_12.png]
Figure 13
Figure 13. Figure 13: Spatio-Temporal Object Grounding: A white car moving along a country road. [PITH_FULL_IMAGE:figures/full_fig_p015_13.png]
Figure 14
Figure 14. Figure 14: Spatio-Temporal Object Grounding: A white sedan in a nighttime scene. [PITH_FULL_IMAGE:figures/full_fig_p016_14.png]
Figure 15
Figure 15. Figure 15: Spatio-Temporal Object Grounding: A motorcycle moving through an intersection. [PITH_FULL_IMAGE:figures/full_fig_p016_15.png]
Figure 16
Figure 16. Figure 16: Referred Object Captioning Example: A partially occluded white van with a boxy shape. [PITH_FULL_IMAGE:figures/full_fig_p017_16.png]
Figure 17
Figure 17. Figure 17: Referred Object Captioning Example: A dark-purple-colored sedan from two perspectives. [PITH_FULL_IMAGE:figures/full_fig_p018_17.png]
Figure 18
Figure 18. Figure 18: Referred Object Captioning Example: A bus with a distinctive green roof. [PITH_FULL_IMAGE:figures/full_fig_p018_18.png]
Figure 20
Figure 20. Figure 20: Example Positioning question templates. {object id}, {object id 1}, and {object id 2} represent the objects being inquired about, {normalized frame} is a placeholder for a specific moment in the video duration, and {relative position} represents the relative position …
Figure 21
Figure 21. Figure 21: Example Counting question templates. {class name pl} is a placeholder for the plural form of the object class being inquired about, {object id} is a placeholder for the representation of the object being inquired about, {normalized frame} is a placeholder for a specif…
Figure 22
Figure 22. Figure 22: Example Motion question templates. {object id}, {object id 1}, and {object id 2} represent the objects being inquired about [PITH_FULL_IMAGE:figures/full_fig_p021_22.png]
Figure 23
Figure 23. Figure 23: Example Class question templates. {object id}, {object id 1}, and {object id 2} represent the objects being inquired about. Question Template Examples for Existence-Easy: Q:"Are there any {class name pl} visible in the video?" Q:"Are any {class name pl} present in the…
Figure 24
Figure 24. Figure 24: Example Existence question templates. {class name pl} is a placeholder for the plural form of the object class being inquired about, {object id} is a placeholder for the representation of the object being inquired about, {normalized frame} is a placeholder for a speci…
Figure 25
Figure 25. Figure 25: Example Spatio-Temporal Object Grounding question templates. [PITH_FULL_IMAGE:figures/full_fig_p023_25.png]
Figure 26
Figure 26. Figure 26: Example Referred Object Captioning question templates. [PITH_FULL_IMAGE:figures/full_fig_p023_26.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. InterAct-Video: Reasoning-Rich Video QA for Urban Traffic

    cs.CV 2025-07 conditional novelty 6.0 of 10

    A new traffic-intersection VideoQA benchmark containing roughly 28,800 human-verified GPT-seeded QA pairs, with evaluations showing fine-tuning improves three video-language models.

  2. STER-VLM: Spatio-Temporal With Enhanced Reference Vision-Language Models

    cs.CV 2025-08 conditional novelty 4.0 of 10

    STER-VLM decomposes traffic captions into spatial and temporal parts, selects a few informative frames, and adds 72B-model reference hints, yielding a small combined validation gain and a 55.655 AI City Challenge Trac...

Reference graph

Works this paper leans on

37 extracted references · 25 canonical work pages · cited by 2 Pith papers

  1. [1]

    Qwen-vl: A versatile vision-language model for un- derstanding, localization, text reading, and beyond

    Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A versatile vision-language model for un- derstanding, localization, text reading, and beyond. arXiv preprint arXiv:2308.12966, 2023. 7

    arXiv 2023

  2. [2]

    Lang, Sourabh V ora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Gi- ancarlo Baldan, and Oscar Beijbom

    Holger Caesar, Varun Bankiti, Alex H. Lang, Sourabh V ora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Gi- ancarlo Baldan, and Oscar Beijbom. nuscenes: A multi- modal dataset for autonomous driving. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020. 2

    work page 2020

  3. [3]

    Videollama 2: Advancing spatial-temporal modeling and audio understanding in video- llms, 2024

    Zesen Cheng, Sicong Leng, Hang Zhang, Yifei Xin, Xin Li, Guanzheng Chen, Yongxin Zhu, Wenqi Zhang, Ziyang Luo, Deli Zhao, and Lidong Bing. Videollama 2: Advancing spatial-temporal modeling and audio understanding in video- llms, 2024. 7

    work page 2024

  4. [4]

    Mevis: A large-scale benchmark for video segmentation with motion expressions, 2023

    Henghui Ding, Chang Liu, Shuting He, Xudong Jiang, and Chen Change Loy. Mevis: A large-scale benchmark for video segmentation with motion expressions, 2023. 3

    2023

  5. [5]

    Vision meets robotics: The KITTI dataset

    A Geiger, P Lenz, C Stiller, and R Urtasun. Vision meets robotics: The KITTI dataset. The International Journal of Robotics Research, 32(11):1231–1237, 2013. 2

    2013

  6. [6]

    Anirudh Goyal

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Ab- hinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, and et al. Anirudh Goyal. The llama 3 herd of models, 2024. 1

    work page 2024

  7. [7]

    Vtimellm: Empower llm to grasp video moments

    Bin Huang, Xin Wang, Hong Chen, Zihan Song, and Wenwu Zhu. Vtimellm: Empower llm to grasp video moments. In Proceedings of the IEEE/CVF Conference on Computer Vi- sion and Pattern Recognition, pages 14271–14280, 2024. 3

    2024

  8. [8]

    Described spatial-temporal video detection, 2024

    Wei Ji, Xiangyan Liu, Yingfei Sun, Jiajun Deng, You Qin, Ammar Nuwanna, Mengyao Qiu, Lina Wei, and Roger Zim- mermann. Described spatial-temporal video detection, 2024. 3

    work page 2024

Show all 37 references
  1. [9]

    Llava-onevision: Easy visual task transfer, 2024

    Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Zi- wei Liu, and Chunyuan Li. Llava-onevision: Easy visual task transfer, 2024. 5, 7

    2024

  2. [10]

    Univtg: Towards unified video- language temporal grounding

    Kevin Qinghong Lin, Pengchuan Zhang, Joya Chen, Shra- man Pramanick, Difei Gao, Alex Jinpeng Wang, Rui Yan, and Mike Zheng Shou. Univtg: Towards unified video- language temporal grounding. In 2023 IEEE/CVF Interna- tional Conference on Computer Vision (ICCV), pages 2782– 2792, 2023. 3

    2023

  3. [11]

    Visual instruction tuning, 2023

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning, 2023. 2

    2023

  4. [12]

    Llava-next: Im- proved reasoning, ocr, and world knowledge, 2024

    Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Llava-next: Im- proved reasoning, ocr, and world knowledge, 2024. 4

    2024

  5. [13]

    Mingyu Liu, Ekim Yurtsever, Jonathan Fossaert, Xingcheng Zhou, Walter Zimmer, Yuning Cui, Bare Luka Zagar, and Alois C. Knoll. A survey on autonomous driving datasets: Statistics, annotation quality, and a future outlook. IEEE Transactions on Intelligent Vehicles, pages 1–29, 2024. 2

    2024

  6. [14]

    Video token sparsification for efficient multimodal llms in autonomous driving, 2024

    Yunsheng Ma, Amr Abdelraouf, Rohit Gupta, Ziran Wang, and Kyungtae Han. Video token sparsification for efficient multimodal llms in autonomous driving, 2024. 6

    2024

  7. [15]

    Drama: Joint risk localization and captioning in driving

    Srikanth Malla, Chiho Choi, Isht Dwivedi, Joon Hee Choi, and Jiachen Li. Drama: Joint risk localization and captioning in driving. In Proceedings of the IEEE/CVF winter confer- ence on applications of computer vision , pages 1043–1052,

  8. [16]

    Lingoqa: Visual question answer- ing for autonomous driving

    Ana-Maria Marcu, Long Chen, Jan H ¨unermann, Alice Karn- sund, Benoit Hanotte, Prajwal Chidananda, Saurabh Nair, Vijay Badrinarayanan, Alex Kendall, Jamie Shotton, Elahe Arani, and Oleg Sinavski. Lingoqa: Visual question answer- ing for autonomous driving. In Computer Vision –...

    2024

  9. [17]

    Lama Ahmad

    OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, and et al. Lama Ahmad. Gpt-4 technical report, 2024. 4

    2024

  10. [18]

    Nuscenes-qa: A multi-modal visual ques- tion answering benchmark for autonomous driving scenario

    Tianwen Qian, Jingjing Chen, Linhai Zhuo, Yang Jiao, and Yu-Gang Jiang. Nuscenes-qa: A multi-modal visual ques- tion answering benchmark for autonomous driving scenario. In Proceedings of the AAAI Conference on Artificial Intelli- gence, pages 4542–4550, 2024. 2, 3, 5

    2024

  11. [19]

    Sai, Akash Kumar Mohankumar, and Mitesh M

    Ananya B. Sai, Akash Kumar Mohankumar, and Mitesh M. Khapra. A survey of evaluation metrics used for nlg systems. ACM Comput. Surv., 55(2), 2022. 5

    2022

  12. [20]

    Drivelm: Driving with graph visual question answering

    Chonghao Sima, Katrin Renz, Kashyap Chitta, Li Chen, Hanxue Zhang, Chengen Xie, Jens Beißwenger, Ping Luo, Andreas Geiger, and Hongyang Li. Drivelm: Driving with graph visual question answering. In Computer Vision – ECCV 2024: 18th European Conference, Milan, Italy, September ...

    2024

  13. [21]

    3d ques- tion answering for city scene understanding

    Penglei Sun, Yaoxian Song, Xiang Liu, Xiaofei Yang, Qiang Wang, Tiefeng Li, Yang Yang, and Xiaowen Chu. 3d ques- tion answering for city scene understanding. In Proceedings of the 32nd ACM International Conference on Multimedia , page 2156–2165, New York, NY , USA, 2024. Assoc...

    2024

  14. [22]

    Video understanding with large language models: A survey

    Yunlong Tang, Jing Bi, Siting Xu, Luchuan Song, Susan Liang, Teng Wang, Daoan Zhang, Jie An, Jingyang Lin, Rongyi Zhu, Ali V osoughi, Chao Huang, Zeliang Zhang, Feng Zheng, Jianguo Zhang, Ping Luo, Jiebo Luo, and Chen- liang Xu. Video understanding with large language models: ...

    2023

  15. [23]

    Human-centric spatio-temporal video grounding with visual transformers

    Zongheng Tang, Yue Liao, Si Liu, Guanbin Li, Xiaojie Jin, Hongxu Jiang, Qian Yu, and Dong Xu. Human-centric spatio-temporal video grounding with visual transformers. IEEE Transactions on Circuits and Systems for Video Tech- nology, 32(12):8238–8249, 2022. 3, 5

    2022

  16. [24]

    YOLOv10: Real-time end- to-end object detection

    Ao Wang, Hui Chen, Lihao Liu, Kai CHEN, Zijia Lin, Jun- gong Han, and Guiguang Ding. YOLOv10: Real-time end- to-end object detection. In The Thirty-eighth Annual Confer- ence on Neural Information Processing Systems, 2024. 4

    2024

  17. [25]

    Referring multi- object tracking

    Dongming Wu, Wencheng Han, Tiancai Wang, Xingping Dong, Xiangyu Zhang, and Jianbing Shen. Referring multi- object tracking. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14633– 14642, 2023. 2, 3

    2023

  18. [26]

    Language prompt for autonomous driving

    Dongming Wu, Wencheng Han, Tiancai Wang, Yingfei Liu, Xiangyu Zhang, and Jianbing Shen. Language prompt for autonomous driving. arXiv preprint arXiv:2309.04379,

    arXiv

  19. [27]

    General object foundation model for images and videos at scale

    Junfeng Wu, Yi Jiang, Qihao Liu, Zehuan Yuan, Xiang Bai, and Song Bai. General object foundation model for images and videos at scale. InProceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition (CVPR) , pages 3783–3795, 2024. 4

    2024

  20. [28]

    Universal instance perception as object discovery and retrieval

    Bin Yan, Yi Jiang, Jiannan Wu, Dong Wang, Zehuan Yuan, Ping Luo, and Huchuan Lu. Universal instance perception as object discovery and retrieval. In CVPR, 2023. 4

    2023

  21. [29]

    Qwen2 technical report

    An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jialong Tang, Jialin Wang, Jian Yang, Jianhong Tu, Jian- wei Zhang, Jianxin Ma, Jin Xu, Jingren Zhou, Jinze Bai, Jinzhe...

    2024 arXiv

  22. [30]

    Sigmoid loss for language image pre-training

    Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. In 2023 IEEE/CVF International Conference on Computer Vision (ICCV), pages 11941–11952, 2023. 5

    2023

  23. [31]

    Advancing its applications with llms: A sur- vey on traffic management, transportation safety, and au- tonomous driving

    Dingkai Zhang, Huanran Zheng, Wenjing Yue, and Xiaol- ing Wang. Advancing its applications with llms: A sur- vey on traffic management, transportation safety, and au- tonomous driving. In Rough Sets , pages 295–309, Cham,

  24. [32]

    Where does it exist: Spatio-temporal video grounding for multi-form sentences

    Zhu Zhang, Zhou Zhao, Yang Zhao, Qi Wang, Huasheng Liu, and Lianli Gao. Where does it exist: Spatio-temporal video grounding for multi-form sentences. In CVPR, 2020. 2, 3

    2020

  25. [33]

    Detrs beat yolos on real-time object detection

    Yian Zhao, Wenyu Lv, Shangliang Xu, Jinman Wei, Guanzhong Wang, Qingqing Dang, Yi Liu, and Jie Chen. Detrs beat yolos on real-time object detection. In 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 16965–16974, 2024. 4

    2024

  26. [34]

    Xingcheng Zhou and Alois C. Knoll. Gpt-4v as traffic assis- tant: An in-depth look at vision language model on complex traffic events, 2024. 2

    2024

  27. [35]

    Xingcheng Zhou, Mingyu Liu, Ekim Yurtsever, Bare Luka Zagar, Walter Zimmer, Hu Cao, and Alois C. Knoll. Vision language models in autonomous driving: A survey and out- look. IEEE Transactions on Intelligent Vehicles, pages 1–20,

  28. [36]

    What do you see to the {relative position} of {object id} at {normalized frame} of the video duration?

    Yunsong Zhou, Linyan Huang, Qingwen Bu, Jia Zeng, Tianyu Li, Hang Qiu, Hongzi Zhu, Minyi Guo, Yu Qiao, and Hongyang Li. Embodied understanding of driving scenarios. arXiv preprint arXiv:2403.04593, 2024. 2, 3, 5 TUMTraffic-VideoQA: Multi-Modal Benchmark for Spatial-Temporal Vi...

    2024 arXiv

  29. [2024]

    Springer Nature Switzerland. 2

Pith tools

Reviewed August 9, 2026 · model on record in the stance chip above.