Pith. sign in

REVIEW 3 major objections 4 minor 3 cited by

STSBench: A Spatio-temporal Scenario Benchmark for Multi-modal Large Language Models in Autonomous Driving

T0 review · 3 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Driving vision-language models fail at spatio-temporal scene understanding, a new 971-question benchmark argues.

desk verdict A well-built benchmark that fills a real gap, but the headline claim that driving VLMs lack spatio-temporal reasoning outruns the confounded evaluation. read the letter →

arxiv 2506.06218 v1 pith:RQAUSEHM submitted 2025-06-06 cs.CV

Christian Fruhwirth-Reisinger , Dušan Malić , Wei Lin , David Schinagl , Samuel Schulter , Horst Possegger This is my paper · ORCID
classification cs.CV
keywords autonomousdrivingvision-languagemodelsspatio-temporalreasoningscenariobenchmarkmultiple-choicequestionansweringnuScenesmulti-viewvideoend-to-end
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

STSBench is a framework that turns annotated driving datasets into multiple-choice benchmarks about traffic scenarios: it mines pre-defined events such as lane changes, overtakes, pedestrian crossings, and agent-to-agent interactions from ground-truth annotations, verifies them with a lightweight human interface, and generates questions. Applied to NuScenes, it yields STSnu, 971 human-verified questions across 43 scenarios in four categories: ego maneuvers, agent actions, ego-to-agent interactions, and agent-to-agent interactions. The paper's central claim is that current vision-language models, especially driving-expert VLMs trained for end-to-end driving, do not have reliable spatio-temporal understanding of dynamic traffic scenes. The evidence is that LLMs given ground-truth trajectories answer the same questions far more accurately than VLMs given raw images or multi-view video, and the largest failures occur where holistic spatio-temporal integration is required. If the paper is right, open-loop planning metrics on standard driving datasets substantially overstate what these models understand about the world.

What carries the argument

The load-bearing mechanism is the STSBench scenario catalog: 43 textual scenario definitions (e.g., lane change, overtake, wait for pedestrian to cross), each paired with negative scenarios that do not occur, so that every question has distractors requiring discrimination of close alternatives. Mining heuristics use ground-truth 3D bounding boxes, tracks, class labels, ego-motion, and HD maps to detect these patterns automatically; a lightweight human verification interface lets drivers confirm positives and reject false negatives; verified samples are then turned into multiple-choice questions of the form 'which of the following best describes ...' with one correct answer among at least four choices. The evaluation protocol compares three model families under adapted prompts: LLMs receive ground-truth trajectories, off-the-shelf VLMs receive single-view image sequences with camera metadata, and driving-expert VLMs receive full multi-view video, making the comparison hinge on how each model integrates spatial and temporal evidence.

What would settle it

Give the best driving-expert VLM perfect perception (for example, replace its visual input with ground-truth object tracks and rendered bounding boxes while keeping the language model frozen) and re-run STSnu; if its agent-to-agent accuracy jumps to the level of the trajectory-fed LLM, the paper's conclusion that the model lacks spatio-temporal reasoning would collapse, because the failure would trace to the vision-to-language interface rather than to reasoning.

Watch

Extended reading notes

Core claim

The paper's central claim is that spatio-temporal reasoning is the capability currently missing in driving vision-language models. Using STSnu, the paper shows that when an LLM is given perfect trajectories it can identify ego maneuvers and interactions at 57.08% average accuracy (GPT-4o), while the best driving-expert VLM (DriveMM) reaches only 39.51%; agent-to-agent interactions, where neither participant is the ego vehicle, are hardest for all models. The paper interprets this gap as evidence that visual models have not learned to jointly reason over spatially distributed multi-view inputs and temporally extended dynamics. It therefore argues that end-to-end driving models need architectural mechanisms that explicitly model spatio-temporal relationships, not just better perception heads or larger training sets.

Load-bearing premise

The evaluation assumes that the accuracy gap between LLMs fed ground-truth trajectories and VLMs fed raw images is caused by differences in spatio-temporal reasoning ability rather than by perception quality, prompt formatting, or model scale.

Editorial extensions

If this is right

  • Any claim that an end-to-end driving VLM understands a scene should be backed by interaction-level questions like those in STSnu; waypoint or ego-action accuracy alone is insufficient.
  • Training data and objectives for driving VLMs should explicitly include third-party interactions, since agent-to-agent scenarios are where all evaluated models drop hardest.
  • Injecting perception-derived trajectories or 3D object states into the language model may be a more direct route to spatio-temporal reasoning than asking the vision encoder to infer them from raw pixels.
  • STSBench can be re-instantiated on other datasets with ground-truth annotations, producing comparable interaction benchmarks across different sensor setups without per-dataset manual annotation.
  • Benchmark distractors should remain semantically close (e.g., overtake versus pass) because the results show models frequently confuse such distinctions.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A plausible extension is to use STSnu as a training signal: fine-tune a driving VLM with a trajectory-infusion module and re-measure; the paper's data predict a large gain if the gap is mainly perceptual rather than reasoning-based.
  • The multiple-choice format may reward elimination strategies rather than true identification; a follow-up variant that asks the model to justify its choice or to detect two simultaneous scenarios in one scene would test whether the apparent reasoning is robust.
  • Because NuScenes is recorded in Boston and Singapore and contains mostly lawful behavior, applying STSBench to more diverse or adversarial recordings could reveal even larger failures than the 57% ceiling suggests.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper introduces STSBench, a framework for automatically mining defined traffic scenarios (ego, agent, ego-to-agent, and agent-to-agent) from datasets with rich ground-truth annotations, together with a lightweight human-verification interface and automatic multiple-choice question generation. Applied to the NuScenes validation split, the framework yields STSnu, comprising 43 scenario types and 971 human-verified multiple-choice questions. The authors evaluate three LLMs, three off-the-shelf VLMs, and three driving-expert VLMs on STSnu, and report that LLMs given ground-truth trajectories substantially outperform the visual models. They interpret this as evidence that current driving VLMs lack spatio-temporal understanding of dynamic traffic scenes, particularly for third-party agent interactions.

Significance. The framework addresses a genuine evaluation gap: most existing driving-VLM benchmarks focus on single-image or monocular ego-centric tasks, whereas STSnu targets multi-view video, third-party interactions, and holistic spatio-temporal reasoning. The release of code and data, the reporting of inter-reviewer agreement (85.6% positive agreement and 20.8% disagreement on negatives), the analysis of multiple-choice letter distribution, and the ablations of query frame and chain-of-thought for OmniDrive are concrete strengths that make the benchmark resource potentially useful to the community. If the benchmark is adopted, the per-scenario results in Tables 8-11 will be a valuable reference. However, the paper's headline conclusion goes beyond what the current experimental design can establish, because the central comparison between LLMs and VLMs is confounded by differences in the information given to each model class, in prompt formatting, and in model scale.

major comments (3)
  1. [Section 4, Table 2, and Appendix D, Table 13] The claim that VLMs 'do not have a spatio-temporal understanding of dynamic traffic scenes' is not established by the reported comparison, because the LLM baseline receives ground-truth trajectories (GPS positions, LiDAR coordinates, velocities) while the VLMs receive raw images with per-model adapted prompts; the accuracy gap can therefore be explained by perception quality, prompt compatibility, or model scale rather than by reasoning ability alone. The authors' own control experiment in Table 13, which feeds ground-truth information to three expert VLMs, yields mixed results (DriveMM improves from 39.5 to 48.5, OmniDrive stays flat at 28.4, Senna collapses to 3.2), and it does not cover the off-the-shelf VLMs or the exact LLM prompt format, so it does not resolve the confound. I recommend adding an oracle-perception or text-input condition for the same VLMs (e.g., providing the same trajectory and GPS text used for the LLMs) and/or softening the conclusion to the supported claim that current driving VLMs perform poorly on these questions under their native input formats.
  2. [Section 4 and Tables 8-11] No confidence intervals or significance tests are reported, despite small per-scenario sample sizes (e.g., Table 8 shows scenario groups with 10-37 questions) and differences between model accuracies that are often small relative to the implied sampling noise (e.g., Table 2: InternVL 2.5 8B at 46.07% vs. DriveMM at 39.51% vs. OmniDrive at 29.33%). The phrase 'outperforms ... by a significant margin' in Section 4 requires a formal test, such as McNemar's test or bootstrap confidence intervals; without this, the model ranking and the 'critical shortcomings' narrative are not quantitatively supported.
  3. [Section 4 and Appendix A.4/E] The evaluation protocol adapts prompts and input formats per model 'in order to get better performance' (Appendix A.4), which makes cross-model accuracy differences at least partly attributable to how well each prompt matches the model's training distribution rather than to spatio-temporal reasoning. I recommend reporting results with a shared, minimally-adapted prompt as the primary protocol, with the per-model optimized prompts as a secondary analysis, and documenting the variance induced by prompt changes, for example by running each model with two or three variants on a subset of the benchmark.
minor comments (4)
  1. [Table 9 caption] The caption of Table 9 says 'for ego-to-agent scenarios', but the table lists single-agent scenarios such as jaywalking, walking, standing, crossing, left turn, and overtaking ego; the caption should read 'agent scenarios' to match the main text and the scenario categories.
  2. [Figures and Appendix E] There are several typos in the figures and appendix prompts, including 'U-Tuen' in Figure 2, 'caputred' in Figures 27-34, 'assistent' and 'specilized' in Figures 35-38, and 'whic is a pedestrian' in Figure 42; these should be corrected for a polished final version.
  3. [References] Reference [15] (HiLM-D) uses the same arXiv identifier 2308.12966 as reference [4] (Qwen-VL), which appears to be an incorrect duplicate; please verify and replace it with the correct identifier for the HiLM-D technical report.
  4. [Section 3.3] The verification protocol accepts positive samples by majority voting but keeps only negatives with full agreement across all reviewers; this asymmetry in quality thresholds is not analyzed, and a brief discussion of its potential effect on benchmark difficulty and on the false-negative rate would improve transparency.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: STSnu is built from NuScenes ground-truth annotations and human verification, and the model evaluations are independent of the benchmark construction.

full rationale

The paper's derivation chain is: define a scenario catalog, mine scenarios from NuScenes 3D tracks, ego-motion, and HD maps, verify them with human reviewers, generate fixed multiple-choice questions, and then evaluate models on those questions. The central findings (e.g., Table 2 showing that driving expert VLMs score poorly) are measurements on this externally constructed benchmark, not quantities fitted to any evaluated model. The LLM baselines receive ground-truth trajectories while VLMs receive images, so the accuracy gap may reflect perception quality, prompt adaptation, or model scale differences; however, that is an experimental confound relevant to the strength of the conclusion, not circularity, because no model output is used to define the questions or answers. The Appendix D experiment that provides ground-truth information to expert VLMs (Table 13) is an additional probe and shows mixed results, which further indicates the benchmark is not engineered to force a particular outcome. No self-citation is load-bearing, no uniqueness theorem is imported from prior work by the authors, and no ansatz is smuggled in via citation. The optional sub-sampling of scenarios based on occlusion and distance shapes benchmark difficulty, but that is a design choice rather than a circular step under the stated criteria.

Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

The benchmark depends on hand-set heuristic thresholds for scenario mining and sub-sampling, and on assumptions about the accuracy of NuScenes annotations and the validity of human verification. No invented entities are introduced. These choices are reasonable but should be disclosed in detail for full reproducibility.

free parameters (2)
  • Scenario mining heuristic thresholds (e.g., lane-crossing detection, speed change thresholds)
    The framework uses hand-defined thresholds to detect actions such as lane changes and accelerations (Section 3.1, step 2). Exact values are not given in the paper, so the benchmark's scenario pool depends on unspecified parameters.
  • Sub-sampling thresholds for occlusion rate, distance to ego, and spatial distribution
    In Section 3.3 and Appendix A.1, over-represented scenarios are sub-sampled based on these criteria, but the threshold values are not reported. This choice affects which samples enter the final 971 questions.
assumptions (3)
  • domain assumption NuScenes ground-truth annotations (3D boxes, tracks, ego motion, HD maps) are accurate and complete enough for scenario mining.
    The pipeline inherits the accuracy of NuScenes annotations (Section 3.1).
  • domain assumption Human reviewers can reliably distinguish true scenarios from false positives and negatives after brief training.
    Verification is used as the gold standard (Section 3.1, step 4; Section 3.3), with disagreements resolved by majority voting.
  • domain assumption Multiple-choice questions with one correct answer from a fixed scenario catalog are a valid test of spatio-temporal reasoning.
    The benchmark's construct validity rests on this assumption (Appendix A.4).

how reviews work

0 comments
Cite this review

Pith. "Pith review of STSBench: A Spatio-temporal Scenario Benchmark for Multi-modal Large Language Models in Autonomous Driving." pith.science (2026). https://pith.science/paper/RQAUSEHM

@misc{pith2026250606218,
  author       = {Pith},
  title        = {Pith review of: STSBench: A Spatio-temporal Scenario Benchmark for Multi-modal Large Language Models in Autonomous Driving},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/RQAUSEHM}},
  note         = {Machine review of arXiv:2506.06218}
}
read the original abstract

We introduce STSBench, a scenario-based framework to benchmark the holistic understanding of vision-language models (VLMs) for autonomous driving. The framework automatically mines pre-defined traffic scenarios from any dataset using ground-truth annotations, provides an intuitive user interface for efficient human verification, and generates multiple-choice questions for model evaluation. Applied to the NuScenes dataset, we present STSnu, the first benchmark that evaluates the spatio-temporal reasoning capabilities of VLMs based on comprehensive 3D perception. Existing benchmarks typically target off-the-shelf or fine-tuned VLMs for images or videos from a single viewpoint and focus on semantic tasks such as object recognition, dense captioning, risk assessment, or scene understanding. In contrast, STSnu evaluates driving expert VLMs for end-to-end driving, operating on videos from multi-view cameras or LiDAR. It specifically assesses their ability to reason about both ego-vehicle actions and complex interactions among traffic participants, a crucial capability for autonomous vehicles. The benchmark features 43 diverse scenarios spanning multiple views and frames, resulting in 971 human-verified multiple-choice questions. A thorough evaluation uncovers critical shortcomings in existing models' ability to reason about fundamental traffic dynamics in complex environments. These findings highlight the urgent need for architectural advances that explicitly model spatio-temporal reasoning. By addressing a core gap in spatio-temporal evaluation, STSBench enables the development of more robust and explainable VLMs for autonomous driving.

Figures

Figures reproduced from arXiv: 2506.06218 by the authors.

Figure 1
Figure 1. STSBench scenario categories. The benchmark covers common ego-vehicle (blue) actions, e.g., ego lane change (a) and interactions with agents (orange), e.g., ego overtaking agent (b), important for vehicle control. In addition, to test for complex spatio-temporal understanding, we evaluate agent actions, e.g., agent left turn (c), and interactions between agents, e.g., agent waiting for pedestrian to cross (d). VLMs,… view at source ↗
Figure 2
Figure 2. STSBench workflow. (Best viewed on screen) during training. As a result, they become tied to a fixed sensor setup for which they can reason accurately about object sizes and distances. This reliance on specific camera setups makes it difficult to evaluate a model trained on one dataset using the benchmark of another, unless domain adaptation or generalization strategies are employed. Therefore, it is essential to ha… view at source ↗
Figure 3
Figure 3. Scenario statistics. Number of mined scenarios in total (gray) and the remaining samples (green) after sub-sampling and verification. Scenarios with more than 50 samples (dashed red line) have been sub-sampled considering spatial distribution, occlusion, and distance to the ego-vehicle. such as overtaking, passing, following, or leading. The scenarios focus on agents in the immediate vicinity of the ego-vehicle and … view at source ↗
Figures from the paper (43 more)
Figure 4
Figure 4. Figure 4: Scenario distribution. Number of scenar￾ios per category. The scenario catalog of our STSnu bench￾mark comprises 43 different scenario descrip￾tions. Using this catalog, STSBench has au￾tomatically mined 4790 scenarios from 150 sequences of the NuScenes [6] validation …
Figure 5
Figure 5. Figure 5: Scenario statistics. Distribution of scenarios with agent involvement across camera views and on the x-y plane. A Benchmark details A.1 STSnu statistics Since driving scenes are dynamic environments in which both the ego-vehicle and other traffic participants are in mo…
Figure 6
Figure 6. Figure 6: Scenario catalog. Ego scenarios with assigned negative scenarios in red. 17 [PITH_FULL_IMAGE:figures/full_fig_p017_6.png]
Figure 7
Figure 7. Figure 7: Scenario catalog. Ego-Agent scenarios with assigned negative scenarios in red. 18 [PITH_FULL_IMAGE:figures/full_fig_p018_7.png]
Figure 8
Figure 8. Figure 8: Scenario catalog. Agent scenarios with assigned negative scenarios in red. 19 [PITH_FULL_IMAGE:figures/full_fig_p019_8.png]
Figure 9
Figure 9. Figure 9: Scenario catalog. Agent-Agent scenarios with assigned negative scenarios in red. 20 [PITH_FULL_IMAGE:figures/full_fig_p020_9.png]
Figure 10
Figure 10. Figure 10: Validation time. Comparison of the validation duration for each reviewer. A.3 Scenario verification Verification tool. Fast and simple verification requires a clean and versatile visualization interface paired with a simple input mask. In [PITH_FULL_IMAGE:figures/ful…
Figure 11
Figure 11. Figure 11: Verification tool for fast and simple verification (a) of mined traffic scenarios by inspect [PITH_FULL_IMAGE:figures/full_fig_p024_11.png]
Figure 12
Figure 12. Figure 12: Agent walk opposite. Accepted scenario sample after review, since the occlusion level has been rated difficult but reasonable. The bounding box is not visible for the tested models. (a) First frame (CAM_FRONT_RIGHT). (b) Last frame (CAM_BACK) [PITH_FULL_IMAGE:figures…
Figure 13
Figure 13. Figure 13: Ego-vehicle passes agent. The scenario sample was rejected after review since the ego-vehicle is not in the adjacent lane of the refereed agent. The bounding box is not visible for the tested models. (a) First frame (CAM_BACK). (b) Last frame (CAM_BACK) [PITH_FULL_IM…
Figure 14
Figure 14. Figure 14: Agent following ego-vehicle. Rejected scenario sample after review since there is another agent between the ego-vehicle and the refereed agent. The bounding box is not visible for the tested models. 25 [PITH_FULL_IMAGE:figures/full_fig_p025_14.png]
Figure 15
Figure 15. Figure 15: Agent jaywalking. Rejected scenario sample after review since there is no jaywalking. The bounding box is not visible for the tested models. (a) First frame (CAM_FRONT_LEFT). (b) 3D LiDAR scene [PITH_FULL_IMAGE:figures/full_fig_p026_15.png]
Figure 16
Figure 16. Figure 16: Agent right of agent. The scenario sample was rejected after review since there is another agent between the two involved agents. The bounding box is not visible for the tested models. (a) First frame (CAM_BACK_RIGHT). (b) Trajectory on map [PITH_FULL_IMAGE:figures/f…
Figure 17
Figure 17. Figure 17: Agent lane change. The scenario sample was rejected after review since no clear lane change was visible. The bounding box is not visible for the tested models. 26 [PITH_FULL_IMAGE:figures/full_fig_p026_17.png]
Figure 18
Figure 18. Figure 18: GPT-4o [46] reasoning for a jaywalking scenario. [PITH_FULL_IMAGE:figures/full_fig_p031_18.png]
Figure 19
Figure 19. Figure 19: GPT-4o [46] reasoning for an ego left turning scenario. [PITH_FULL_IMAGE:figures/full_fig_p032_19.png]
Figure 20
Figure 20. Figure 20: Distribution of individual letters that were the correct multiple-choice option, comparing [PITH_FULL_IMAGE:figures/full_fig_p033_20.png]
Figure 21
Figure 21. Figure 21: Distribution of individual letters that were the correct multiple-choice option (a), and the [PITH_FULL_IMAGE:figures/full_fig_p033_21.png]
Figure 22
Figure 22. Figure 22: Mean and variance of the DriveMMaccuracy across 5 independent evaluations, shown for [PITH_FULL_IMAGE:figures/full_fig_p035_22.png]
Figure 23
Figure 23. Figure 23: LLM Ego scenario prompt 36 [PITH_FULL_IMAGE:figures/full_fig_p036_23.png]
Figure 24
Figure 24. Figure 24: LLM Ego-to-Agent scenario prompt 37 [PITH_FULL_IMAGE:figures/full_fig_p037_24.png]
Figure 25
Figure 25. Figure 25: LLM Agent scenario prompt 38 [PITH_FULL_IMAGE:figures/full_fig_p038_25.png]
Figure 26
Figure 26. Figure 26: LLM Agent-to-Agent scenario prompt 39 [PITH_FULL_IMAGE:figures/full_fig_p039_26.png]
Figure 27
Figure 27. Figure 27: VLM InternVL 2.5 8B/1B Ego scenario prompt [PITH_FULL_IMAGE:figures/full_fig_p040_27.png]
Figure 28
Figure 28. Figure 28: VLM InternVL 2.5 8B/1B Ego-to-Agent scenario prompt [PITH_FULL_IMAGE:figures/full_fig_p041_28.png]
Figure 29
Figure 29. Figure 29: VLM InternVL 2.5 8B/1B Agent scenario prompt [PITH_FULL_IMAGE:figures/full_fig_p042_29.png]
Figure 30
Figure 30. Figure 30: VLM InternVL 2.5 8B/1B Agent-to-Agent scenario prompt [PITH_FULL_IMAGE:figures/full_fig_p043_30.png]
Figure 31
Figure 31. Figure 31: VLM Qwen 2.5 7B Ego scenario prompt 44 [PITH_FULL_IMAGE:figures/full_fig_p044_31.png]
Figure 32
Figure 32. Figure 32: VLM Qwen 2.5 7B Ego-to-Agent scenario prompt [PITH_FULL_IMAGE:figures/full_fig_p045_32.png]
Figure 33
Figure 33. Figure 33: VLM Qwen 2.5 7B Agent scenario prompt 46 [PITH_FULL_IMAGE:figures/full_fig_p046_33.png]
Figure 34
Figure 34. Figure 34: VLM Qwen 2.5 7B Agent-to-Agent scenario prompt [PITH_FULL_IMAGE:figures/full_fig_p047_34.png]
Figure 35
Figure 35. Figure 35: Senna-VLM [29] Ego scenario prompt 48 [PITH_FULL_IMAGE:figures/full_fig_p048_35.png]
Figure 36
Figure 36. Figure 36: Senna-VLM [29] Ego-to-Agent scenario prompt [PITH_FULL_IMAGE:figures/full_fig_p049_36.png]
Figure 37
Figure 37. Figure 37: Senna-VLM [29] Agent scenario prompt 50 [PITH_FULL_IMAGE:figures/full_fig_p050_37.png]
Figure 38
Figure 38. Figure 38: Senna-VLM [29] Agent-to-Agent scenario prompt [PITH_FULL_IMAGE:figures/full_fig_p051_38.png]
Figure 39
Figure 39. Figure 39: OmniDrive [60] Ego scenario prompt Multi-view Image Sequence: Prompt: You are a helpful traffic control expert specializing in the analysis and identification of temporal actions and maneuvers performed by various agents, as well as your own temporal actions and maneu…
Figure 40
Figure 40. Figure 40: OmniDrive [60] Ego-to-Agent scenario prompt [PITH_FULL_IMAGE:figures/full_fig_p052_40.png]
Figure 41
Figure 41. Figure 41: OmniDrive [60] Agent scenario prompt Multi-view Image Sequence: Prompt: You are a helpful traffic control expert specializing in the analysis and identification of temporal actions and maneuvers performed by various agents, as well as your own temporal actions and man…
Figure 42
Figure 42. Figure 42: OmniDrive [60] Agent-to-Agent scenario prompt [PITH_FULL_IMAGE:figures/full_fig_p053_42.png]
Figure 43
Figure 43. Figure 43: DriveMM [22] Ego scenario prompt Multi-view Video: Prompt: 1: <image> 2: <image> 3: <image> 4: <image> 5: <image> 6: <image>. These six images are the front view, front left view, front right view, back view, back left view and back right view of the ego vehicle. You …
Figure 44
Figure 44. Figure 44: DriveMM [22] Ego-to-Agent scenario prompt [PITH_FULL_IMAGE:figures/full_fig_p054_44.png]
Figure 45
Figure 45. Figure 45: DriveMM [22] Agent scenario prompt Multi-view Video: Prompt: 1: <image> 2: <image> 3: <image> 4: <image> 5: <image> 6: <image>. These six images are the front view, front left view, front right view, back view, back left view and back right view of the ego vehicle. Yo…
Figure 46
Figure 46. Figure 46: DriveMM [22] Agent-to-Agent scenario prompt [PITH_FULL_IMAGE:figures/full_fig_p055_46.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. STAR-VLM: Spatiotemporal Grounding Vision-Language Models for Motion and Velocity Estimation via Automotive Radar Supervision

    cs.CV 2026-08 conditional novelty 7.0 of 10

    Using automotive radar Doppler as supervision, STAR-VLM enables a vision-language model to estimate metric radial velocity and motion state of objects from video, outperforming zero-shot task-specific baselines on a n...

  2. ObsDriveBench: Benchmarking Multimodal Understanding under Adverse Weather with Observability Awareness

    cs.AI 2026-07 conditional novelty 6.0 of 10

    Real adverse-weather camera–LiDAR–radar MCQs expose VLM failures from observability estimation through spatial grounding to trajectory safety, partially mitigated by SFT+RL.

  3. CARLA-GS: Decoupling Representation, Reasoning, and Physics Simulation for Autonomous Driving Corner-Case Synthesis

    cs.RO 2026-07 conditional novelty 6.0 of 10

    CARLA-GS is a modular pipeline that uses an LLM for semantic trajectory planning, CARLA for physics execution, and 3D Gaussian Splatting for photorealistic rendering to synthesize autonomous driving corner cases.

Reference graph

Works this paper leans on

76 extracted references · 49 canonical work pages · cited by 3 Pith papers

  1. [1]

    LLaMA 3.2: Open Foundation and Instruction Models, 2024

    Meta AI. LLaMA 3.2: Open Foundation and Instruction Models, 2024. 9

    work page 2024

  2. [2]

    Flamingo: a Visual Language Model for Few-Shot Learning

    Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katie Millican, Malcolm Reynolds, Roman Ring, Eliza Rutherford, Serkan Cabi, Tengda Han, Zhitao Gong, Sina Samangooei, Marianne Monteiro, Jacob Menick, Sebastian Borgeaud, Andy Brock, Aida Nematzadeh, Sahand Sharifzadeh, Mikolaj Binkowski, R...

    work page 2022

  3. [3]

    CoVLA: Comprehensive Vision-Language-Action Dataset for Autonomous Driving

    Hidehisa Arai, Keita Miwa, Kento Sasaki, Yu Yamaguchi, Kohei Watanabe, Shunsuke Aoki, and Issei Yamamoto. CoVLA: Comprehensive Vision-Language-Action Dataset for Autonomous Driving. In WACV, 2024

    work page 2024

  4. [5]

    Qwen2.5-VL Technical Report

    Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Hang Zhang, Zhibo Yang, Haiyang Xu, and Junyang Lin. Qwen2.5-VL Technical Report. a...

    arXiv 2025

  5. [6]

    Lang, Sourabh V ora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom

    Holger Caesar, Varun Bankiti, Alex H. Lang, Sourabh V ora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom. nuScenes: A multimodal dataset for autonomous driving. In CVPR, 2020

    work page 2020

  6. [7]

    MAPLM: A Real-World Large-Scale Vision-Language Dataset for Map and Traffic Scene Understanding

    Xu Cao, Tong Zhou, Yunsheng Ma, Wenqian Ye, Can Cui, Kun Tang, Zhipeng Cao, Kaizhao Liang, Ziran Wang, James Rehg, and Chao Zheng. MAPLM: A Real-World Large-Scale Vision-Language Dataset for Map and Traffic Scene Understanding. In CVPR, 2024

    work page 2024

  7. [8]

    How far are we to gpt-4v? closing the gap to commercial multimodal models with open-source suites

    Zhe Chen, Weiyun Wang, Hao Tian, Shenglong Ye, Zhangwei Gao, Erfei Cui, Wenwen Tong, Kongzhi Hu, Jiapeng Luo, Zheng Ma, et al. How far are we to gpt-4v? closing the gap to commercial multimodal models with open-source suites. Science China Information Sciences , 67(12):220101, 2024

    2024

  8. [9]

    Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks

    Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. In CVPR, 2024

    2024

Show all 76 references
  1. [10]

    The Cityscapes Dataset for Semantic Urban Scene Understanding

    Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The Cityscapes Dataset for Semantic Urban Scene Understanding. In CVPR, 2016

    2016

  2. [11]

    InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning

    Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning. In NeurIPS, 2023

    2023

  3. [12]

    Deepseek-v3 technical report

    DeepSeek-AI. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437, 2024

    2024 arXiv

  4. [13]

    Smith, Hannaneh Hajishirzi, Ross Girshick, Ali Farhadi, and Aniruddha Kembhavi

    Matt Deitke, Christopher Clark, Sangho Lee, Rohun Tripathi, Yue Yang, Jae Sung Park, Mohammadreza Salehi, Niklas Muennighoff, Kyle Lo, Luca Soldaini, Jiasen Lu, Taira Anderson, Erin Bransom, Kiana Ehsani, Huong Ngo, YenSung Chen, Ajay Patel, Mark Yatskar, Chris Callison-Burch,...

    2024

  5. [14]

    Talk2Car: Taking control of your self-driving car

    Thierry Deruyttere, Simon Vandenhende, Dusan Grujicic, Luc Van Gool, and Marie-Francine Moens. Talk2Car: Taking control of your self-driving car. In EMNLP-IJCNLP, 2019. 10

    2019

  6. [15]

    HiLM-D: Towards High- Resolution Understanding in Multimodal Large Language Models for Autonomous Driving

    Xinpeng Ding, Jianhua Han, Hang Xu, Wei Zhang, and Xiaomeng Li. HiLM-D: Towards High- Resolution Understanding in Multimodal Large Language Models for Autonomous Driving. arXiv preprint arXiv:2308.12966, 2023

    2023 arXiv

  7. [16]

    Holistic Autonomous Driving Understanding by Bird’s-Eye-View Injected Multi-Modal Large Models

    Xinpeng Ding, Jinahua Han, Hang Xu, Xiaodan Laing, Xu Hang, Wei Zhang, and Xiaomeng Li. Holistic Autonomous Driving Understanding by Bird’s-Eye-View Injected Multi-Modal Large Models. In CVPR, 2024

    2024

  8. [17]

    CARLA: An Open Urban Driving Simulator

    Alexey Dosovitskiy, German Ros, Felipe Codevilla, Antonio Lopez, and Vladlen Koltun. CARLA: An Open Urban Driving Simulator. In CoRL, 2017

    2017

  9. [18]

    ORION: A Holistic End-to-End Autonomous Driving Framework by Vision-Language Instructed Action Generation

    Haoyu Fu, Diankun Zhang, Zongchuang Zhao, Jianfeng Cui, Dingkang Liang, Chong Zhang, Dingyuan Zhang, Hongwei Xie, Bing Wang, and Xiang Bai. ORION: A Holistic End-to-End Autonomous Driving Framework by Vision-Language Instructed Action Generation. arXiv preprint arXiv:2503.19755, 2025

    2025 arXiv

  10. [19]

    Are we ready for autonomous driving? The KITTI vision benchmark suite

    Andreas Geiger, Philip Lenz, and Raquel Urtasun. Are we ready for autonomous driving? The KITTI vision benchmark suite. In CVPR, 2012

    2012

  11. [20]

    DriveMLLM: A Benchmark for Spatial Understanding with Multimodal Large Language Models in Autonomous Driving

    Xianda Guo, Zhang Ruijun, Duan Yiqun, He Yuhang, Chenming Zhang, and Long Chen. DriveMLLM: A Benchmark for Spatial Understanding with Multimodal Large Language Models in Autonomous Driving. arXiv preprint arXiv:2411.13112, 2024

    2024 arXiv

  12. [21]

    Planning-oriented Autonomous Driving

    Yihan Hu, Jiazhi Yang, Li Chen, Keyu Li, Chonghao Sima, Xizhou Zhu, Siqi Chai, Senyao Du, Tianwei Lin, Wenhai Wang, Lewei Lu, Xiaosong Jia, Qiang Liu, Jifeng Dai, Yu Qiao, and Hongyang Li. Planning-oriented Autonomous Driving. In CVPR, 2023

    2023

  13. [22]

    Drivemm: All-in-one large multimodal model for autonomous driving

    Zhijian Huang, Chengjian Fen, Feng Yan, Baihui Xiao, Zequn Jie, Yujie Zhong, Xiaodan Liang, and Lin Ma. Drivemm: All-in-one large multimodal model for autonomous driving. arXiv preprint arXiv:2412.07689, 2024

    2024 arXiv

  14. [23]

    Making Large Language Models Better Planners with Reasoning-Decision Alignment

    Zhijian Huang, Tao Tang, Shaoxiang Chen, Sihao Lin, Zequn Jie, Lin Ma, Guangrun Wang, and Xiaodan Liang. Making Large Language Models Better Planners with Reasoning-Decision Alignment. In ECCV, 2024

    2024

  15. [24]

    Emma: End-to-end multimodal model for autonomous driving

    Jyh-Jing Hwang, Runsheng Xu, Hubert Lin, Wei-Chih Hung, Jingwei Ji, Kristy Choi, Di Huang, Tong He, Paul Covington, Benjamin Sapp, Yin Zhou, James Guo, Dragomir Anguelov, and Mingxing Tan. Emma: End-to-end multimodal model for autonomous driving. arXiv preprint arXiv:2410.23262, 2024

    arXiv

  16. [25]

    NuScenes-MQA: Integrated Evaluation of Captions and QA for Autonomous Driving Datasets Using Markup Annotations

    Yuichi Inoue, Yuki Yada, Kotaro Tanahashi, and Yu Yamaguchi. NuScenes-MQA: Integrated Evaluation of Captions and QA for Autonomous Driving Datasets Using Markup Annotations. In WACVW, 2024

    2024

  17. [26]

    Drivelmm-o1: A step-by-step reasoning dataset and large multimodal model for driving scenario understanding

    Ayesha Ishaq, Jean Lahoud, Ketan More, Omkar Thawakar, Ritesh Thawkar, Dinura Dis- sanayake, Noor Ahsan, Yuhao Li, Fahad Shahbaz Khan, Hisham Cholakkal, Ivan Laptev, Rao Muhammad Anwer, and Salman Khan. Drivelmm-o1: A step-by-step reasoning dataset and large multimodal model f...

    2025 arXiv

  18. [27]

    Bench2Drive: Towards Multi-Ability Benchmarking of Closed-Loop End-To-End Autonomous Driving

    Xiaosong Jia, Zhenjie Yang, Qifeng Li, Zhiyuan Zhang, and Junchi Yan. Bench2Drive: Towards Multi-Ability Benchmarking of Closed-Loop End-To-End Autonomous Driving. In NeurIPS, 2024

    2024

  19. [28]

    V AD: Vectorized Scene Representation for Efficient Autonomous Driving

    Bo Jiang, Shaoyu Chen, Qing Xu, Bencheng Liao, Jiajie Chen, Helong Zhou, Qian Zhang, Wenyu Liu, Chang Huang, and Xinggang Wang. V AD: Vectorized Scene Representation for Efficient Autonomous Driving. In ICCV, 2023

    2023

  20. [29]

    Senna: Bridging Large Vision-Language Models and End-to- End Autonomous Driving

    Bo Jiang, Shaoyu Chen, Bencheng Liao, Xingyu Zhang, Wei Yin, Qian Zhang, Chang Huang, Wenyu Liu, and Xinggang Wang. Senna: Bridging Large Vision-Language Models and End-to- End Autonomous Driving. arXiv preprint arXiv:2410.22313, 2024. 11

    arXiv

  21. [30]

    TOD3Cap: Towards 3D Dense Captioning in Outdoor Scenes

    Bu Jin, Yupeng Zheng, Pengfei Li, Weize Li, Yuhang Zheng, Sujie Hu, Xinyu Liu, Jinwei Zhu, Zhijie Yan, Haiyang Sun, Kun Zhan, Peng Jia, Xiaoxiao Long, Yilun Chen, and Hao Zhao. TOD3Cap: Towards 3D Dense Captioning in Outdoor Scenes. In ECCV, 2024

    2024

  22. [31]

    Textual Explana- tions for Self-Driving Vehicles

    Jinkyu Kim, Anna Rohrbach, Trevor Darrell, John Canny, and Zeynep Akata. Textual Explana- tions for Self-Driving Vehicles. In ECCV, 2018

    2018

  23. [32]

    Junnan Li, Dongxu Li, Caiming Xiong, and S. Hoi. BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation. In ICML, 2022

    2022

  24. [33]

    BLIP-2: bootstrapping language-image pre-training with frozen image encoders and large language models

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. BLIP-2: bootstrapping language-image pre-training with frozen image encoders and large language models. In ICML, 2023

    2023

  25. [34]

    Generative Planning with 3D-vision Language Pre-training for End-to-End Autonomous Driving

    Tengpeng Li, Hanli Wang, Xianfei Li, Wenlong Liao, Tao He, and Pai Peng. Generative Planning with 3D-vision Language Pre-training for End-to-End Autonomous Driving. arXiv preprint arXiv:2501.08861, 2025

    2025 arXiv

  26. [35]

    Automated Evaluation of Large Vision-Language Models on Self-driving Corner Cases

    Yanze Li, Wenhua Zhang, Kai Chen, Yanxin Liu, Pengxiang Li, Ruiyuan Gao, Lanqing Hong, Meng Tian, Xinhai Zhao, Zhenguo Li, et al. Automated Evaluation of Large Vision-Language Models on Self-driving Corner Cases. In WACV, 2025

    2025

  27. [36]

    BEVFormer: Learning Bird’s-Eye-View Representation from Multi-Camera Images via Spatiotemporal Transformers

    Zhiqi Li, Wenhai Wang, Hongyang Li, Enze Xie, Chonghao Sima, Tong Lu, Yu Qiao, and Jifeng Dai. BEVFormer: Learning Bird’s-Eye-View Representation from Multi-Camera Images via Spatiotemporal Transformers. In ECCV, 2022

    2022

  28. [37]

    Zhiqi Li, Zhiding Yu, Shiyi Lan, Jiahan Li, Jan Kautz, Tong Lu, and Jose M. Alvarez. Is Ego Status All You Need for Open-Loop End-to-End Autonomous Driving? . In CVPR, 2024

    2024

  29. [38]

    Visual Instruction Tuning

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual Instruction Tuning. In NeurIPS, 2023

    2023

  30. [39]

    Improved Baselines with Visual Instruction Tuning

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved Baselines with Visual Instruction Tuning. In CVPR, 2024

    2024

  31. [40]

    Llava-next: Improved reasoning, ocr, and world knowledge, 2024

    Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Llava-next: Improved reasoning, ocr, and world knowledge, 2024

    2024

  32. [41]

    Dolphins: Multimodal Language Model for Driving

    Yingzi Ma, Yulong Cao, Jiachen Sun, Marco Pavone, and Chaowei Xiao. Dolphins: Multimodal Language Model for Driving. In ECCV, 2024

    2024

  33. [42]

    DRAMA: Joint Risk Localization and Captioning in Driving

    Srikanth Malla, Chiho Choi, Isht Dwivedi, Joon Hee Choi, and Jiachen Li. DRAMA: Joint Risk Localization and Captioning in Driving. In WACV, 2023

    2023

  34. [43]

    One Million Scenes for Autonomous Driving: ONCE Dataset

    Jiageng Mao, Minzhe Niu, Chenhan Jiang, Xiaodan Liang, Yamin Li, Chaoqiang Ye, Wei Zhang, Zhenguo Li, Jie Yu, and Chunjing Xu. One Million Scenes for Autonomous Driving: ONCE Dataset. In NeurIPS, 2021

    2021

  35. [44]

    LingoQA: Video Question Answering for Autonomous Driving

    Ana-Maria Marcu, Long Chen, Jan Hünermann, Alice Karnsund, Benoit Hanotte, Prajwal Chidananda, Saurabh Nair, Vijay Badrinarayanan, Alex Kendall, Jamie Shotton, and Oleg Sinavski. LingoQA: Video Question Answering for Autonomous Driving. In ECCV, 2024

    2024

  36. [45]

    Reason2Drive: Towards Interpretable and Chain-based Reasoning for Autonomous Driving

    Ming Nie, Renyuan Peng, Chunwei Wang, Xinyue Cai, Jianhua Han, Hang Xu, and Li Zhang. Reason2Drive: Towards Interpretable and Chain-based Reasoning for Autonomous Driving. In ECCV, 2024

    2024

  37. [46]

    GPT-4 Technical Report

    OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Floren- cia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, and et al. GPT-4 Technical Report. arXiv preprint arXiv:2303.08774, 2024

    2024 arXiv

  38. [47]

    VLP: Vision Language Planning for Autonomous Driving

    Chenbin Pan, Burhaneddin Yaman, Tommaso Nesti, Abhirup Mallik, Alessandro G Allievi, Senem Velipasalar, and Liu Ren. VLP: Vision Language Planning for Autonomous Driving. In CVPR, 2024. 12

    2024

  39. [48]

    NuScenes-QA: A Multi-Modal Visual Question Answering Benchmark for Autonomous Driving Scenario

    Tianwen Qian, Jingjing Chen, Linhai Zhuo, Yang Jiao, and Yu-Gang Jiang. NuScenes-QA: A Multi-Modal Visual Question Answering Benchmark for Autonomous Driving Scenario. In AAAI, 2024

    2024

  40. [49]

    Learning Transferable Visual Models From Natural Language Supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning Transferable Visual Models From Natural Language Supervision. In ICML, 2021

    2021

  41. [50]

    Rerun: A Visualization SDK for Multimodal Data, 2024

    Rerun Development Team. Rerun: A Visualization SDK for Multimodal Data, 2024. URL https://www.rerun.io. Available from https://www.rerun.io/ and https://github.com/rerun- io/rerun

    2024

  42. [51]

    Rank2Tell: A Multimodal Driving Dataset for Joint Importance Ranking and Reasoning

    Enna Sachdeva, Nakul Agarwal, Suhas Chundi, Sean Roelofs, Jiachen Li, Mykel Kochender- fer, Chiho Choi, and Behzad Dariush. Rank2Tell: A Multimodal Driving Dataset for Joint Importance Ranking and Reasoning. In WACV, 2024

    2024

  43. [52]

    Waslander, Yu Liu, and Hong- sheng Li

    Hao Shao, Yuxuan Hu, Letian Wang, Guanglu Song, Steven L. Waslander, Yu Liu, and Hong- sheng Li. LMDrive: Closed-Loop End-to-End Driving with Large Language Models. In CVPR, 2024

    2024

  44. [53]

    DriveLM: Driving with Graph Visual Question Answering

    Chonghao Sima, Katrin Renz, Kashyap Chitta, Li Chen, Hanxue Zhang, Chengen Xie, Ping Luo, Andreas Geiger, and Hongyang Li. DriveLM: Driving with Graph Visual Question Answering. In ECCV, 2024

    2024

  45. [54]

    InsightDrive: Insight Scene Representation for End-to-End Autonomous Driving

    Ruiqi Song, Xianda Guo, Hangbin Wu, Qinggong Wei, and Long Chen. InsightDrive: Insight Scene Representation for End-to-End Autonomous Driving. arXiv preprint arXiv:2503.13047, 2025

    2025

  46. [55]

    Scalability in Perception for Autonomous Driving: Waymo Open Dataset

    Pei Sun, Henrik Kretzschmar, Xerxes Dotiwalla, Aurelien Chouard, Vijaysai Patnaik, Paul Tsui, James Guo, Yin Zhou, Yuning Chai, Benjamin Caine, et al. Scalability in Perception for Autonomous Driving: Waymo Open Dataset. In CVPR, 2020

    2020

  47. [56]

    NuScenes-SpatialQA: A Spatial Understanding and Reasoning Benchmark for Vision-Language Models in Autonomous Driving

    Kexin Tian, Jingrui Mao, Yunlong Zhang, Jiwan Jiang, Yang Zhou, and Zhengzhong Tu. NuScenes-SpatialQA: A Spatial Understanding and Reasoning Benchmark for Vision-Language Models in Autonomous Driving. arXiv preprint arXiv:2504.03164, 2025

    2025 arXiv

  48. [57]

    DriveVLM: The Convergence of Autonomous Driving and Large Vision-Language Models

    Xiaoyu Tian, Junru Gu, Bailin Li, Yicheng Liu, Zhiyong Zhao, Yang Wang, Kun Zhan, Peng Jia, Xianpeng Lang, and Hang Zhao. DriveVLM: The Convergence of Autonomous Driving and Large Vision-Language Models. In CoRL, 2024

    2024

  49. [58]

    Object Referring in Videos With Language and Human Gaze

    Arun Balajee Vasudevan, Dengxin Dai, and Luc Van Gool. Object Referring in Videos With Language and Human Gaze. In CVPR, 2018

    2018

  50. [59]

    Exploring Object- Centric Temporal Modeling for Efficient Multi-View 3D Object Detection

    Shihao Wang, Yingfei Liu, Tiancai Wang, Ying Li, and Xiangyu Zhang. Exploring Object- Centric Temporal Modeling for Efficient Multi-View 3D Object Detection. In ICCV, 2023

    2023

  51. [60]

    Shihao Wang, Zhiding Yu, Xiaohui Jiang, Shiyi Lan, Min Shi, Nadine Chang, Jan Kautz, Ying Li, and Jose M. Alvarez. OmniDrive: A holistic llm-agent framework for autonomous driving with 3d perception, reasoning and planning. arXiv preprint arXiv:2405.01533, 2024

    2024 arXiv

  52. [61]

    Chain-of-thought prompting elicits reasoning in large language models

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. NeurIPS, 2022

    2022

  53. [62]

    Argoverse 2: Next Generation Datasets for Self-driving Perception and Forecasting

    Benjamin Wilson, William Qi, Tanmay Agarwal, John Lambert, Jagjeet Singh, Siddhesh Khandelwal, Bowen Pan, Ratnesh Kumar, Andrew Hartnett, Jhony Kaesemodel Pontes, Deva Ramanan, Peter Carr, and James Hays. Argoverse 2: Next Generation Datasets for Self-driving Perception and Fo...

    2021

  54. [63]

    Katharina Winter, Mark Azer, and Fabian B. Flohr. BEVDriver: Leveraging BEV Maps in LLMs for Robust Closed-Loop Driving. arXiv preprint arXiv:2503.03074, 2025. 13

    2025 arXiv

  55. [64]

    Language Prompt for Autonomous Driving

    Dongming Wu, Wencheng Han, Tiancai Wang, Yingfei Liu, Xiangyu Zhang, and Jianbing Shen. Language Prompt for Autonomous Driving. In AAAI, 2025

    2025

  56. [65]

    Are VLMs Ready for Autonomous Driving? An Empirical Study from the Reliability, Data, and Metric Perspectives

    Shaoyuan Xie, Lingdong Kong, Yuhao Dong, Chonghao Sima, Wenwei Zhang, Qi Alfred Chen, Ziwei Liu, and Liang Pan. Are VLMs Ready for Autonomous Driving? An Empirical Study from the Reliability, Data, and Metric Perspectives. arXiv preprint arXiv:2501.04003, 2025

    2025 arXiv

  57. [66]

    Explainable Object-Induced Action Decision for Autonomous Vehicles

    Yiran Xu, Xiaoyin Yang, Lihang Gong, Hsuan-Chu Lin, Tz-Ying Wu, Yunsheng Li, and Nuno Vasconcelos. Explainable Object-Induced Action Decision for Autonomous Vehicles . In CVPR, 2020

    2020

  58. [67]

    Drivegpt4: Interpretable end-to-end autonomous driving via large language model

    Zhenhua Xu, Yujia Zhang, Enze Xie, Zhen Zhao, Yong Guo, Kwan-Yee K Wong, Zhenguo Li, and Hengshuang Zhao. Drivegpt4: Interpretable end-to-end autonomous driving via large language model. IEEE Robotics and Automation Letters , 2024

    2024

  59. [68]

    BDD100K: A Diverse Driving Dataset for Heterogeneous Multitask Learning

    Fisher Yu, Haofeng Chen, Xin Wang, Wenqi Xian, Yingying Chen, Fangchen Liu, Vashisht Madhavan, and Trevor Darrell. BDD100K: A Diverse Driving Dataset for Heterogeneous Multitask Learning. In CVPR, June 2020

    2020

  60. [69]

    Are Vision LLMs Road- Ready? A Comprehensive Benchmark for Safety-Critical Driving Video Understanding

    Tong Zeng, Longfeng Wu, Liang Shi, Dawei Zhou, and Feng Guo. Are Vision LLMs Road- Ready? A Comprehensive Benchmark for Safety-Critical Driving Video Understanding. arXiv preprint arXiv:2504.14526, 2025

    2025 arXiv

  61. [70]

    Sigmoid Loss for Language Image Pre-Training

    Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid Loss for Language Image Pre-Training. In ICCV, October 2023

    2023

  62. [71]

    Sce2DriveX: A Generalized MLLM Framework for Scene-to-Drive Learning, 2025

    Rui Zhao, Qirui Yuan, Jinyu Li, Haofeng Hu, Yun Li, Chengyuan Zheng, and Fei Gao. Sce2DriveX: A Generalized MLLM Framework for Scene-to-Drive Learning, 2025

    2025

  63. [72]

    Large Language Models Are Not Robust Multiple Choice Selectors

    Chujie Zheng, Hao Zhou, Fandong Meng, Jie Zhou, and Minlie Huang. Large Language Models Are Not Robust Multiple Choice Selectors. In ICLR, 2024

    2024

  64. [73]

    Doe-1: Closed-Loop Autonomous Driving with Large World Model

    Wenzhao Zheng, Zetian Xia, Yuanhui Huang, Sicheng Zuo, Jie Zhou, and Jiwen Lu. Doe-1: Closed-Loop Autonomous Driving with Large World Model. arXiv preprint arXiv: 2412.09627 , 2024

    2024 arXiv

  65. [74]

    HERMES: A Unified Self-Driving World Model for Simultaneous 3D Scene Understanding and Generation

    Xin Zhou, Dingkang Liang, Sifan Tu, Xiwu Chen, Yikang Ding, Dingyuan Zhang, Feiyang Tan, Hengshuang Zhao, and Xiang Bai. HERMES: A Unified Self-Driving World Model for Simultaneous 3D Scene Understanding and Generation. arXiv preprint arXiv:2501.14729 , 2025

    2025 arXiv

  66. [75]

    Xingcheng Zhou, Xuyuan Han, Feng Yang, Yunpu Ma, and Alois C. Knoll. OpenDriveVLA: Towards End-to-end Autonomous Driving with Large Vision Language Action Model. arXiv preprint arXiv:2503.23463, 2025

    2025

  67. [76]

    Embodied Understanding of Driving Scenarios

    Yunsong Zhou, Linyan Huang, Qingwen Bu, Jia Zeng, Tianyu Li, Hang Qiu, Hongzi Zhu, Minyi Guo, Yu Qiao, and Hongyang Li. Embodied Understanding of Driving Scenarios. In ECCV, 2024. 14 Appendix Table of Contents A Benchmark details 16 A.1 STSnu statistics . . . . . . . . . . . ....

    2024

  68. [77]

    Ego turning left,

    occlusion, 2) distance to the ego-vehicle, and 3) spatial distribution. Based on these criteria, we retain scenarios that are highly visible, occur in the near surrounding of the ego-vehicle, and are spatially well-distributed around it. The first criterion is straightforward ...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.