Pith. sign in

REVIEW 3 major objections 6 minor 17 cited by

Are VLMs Ready for Autonomous Driving? An Empirical Study from the Reliability, Data, and Metric Perspectives

T0 review · 3 major / 6 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read DriveBench shows that vision-language models for driving can produce plausible answers with no image at all, so current benchmark scores overstate their visual grounding.

desk verdict DriveBench is a useful, reusable warning about VLM overreliance on text and priors, but the curated subset and GPT-rubric scoring leave the magnitude of the problem less settled than the title implies. read the letter →

arxiv 2501.04003 v1 pith:7HFMFSIR submitted 2025-01-07 cs.CV cs.RO

Shaoyuan Xie , Lingdong Kong , Yuhao Dong , Chonghao Sima , Wenwei Zhang , Qi Alfred Chen , Ziwei Liu , Liang Pan This is my paper · ORCID
classification cs.CVcs.RO
keywords vision-languagemodelsautonomousdrivingvisualgroundingbenchmarkout-of-distributionrobustnesstext-onlyevaluationdatasetbiasGPT-based
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks whether vision-language models (VLMs) used for driving really ground their answers in what the camera sees. It builds DriveBench, a benchmark of 19,200 frames and 20,498 question-answer pairs evaluated under clean images, fifteen kinds of image corruption, and text-only inputs with no image at all. Across twelve VLMs, the paper finds that performance on text-only prompts stays close to performance on clean images, and that high accuracy can be achieved by repeating common answers such as 'Going Ahead'. The authors conclude that current VLMs often fabricate plausible driving responses from general knowledge and textual cues, and that existing accuracy, language, and GPT-based metrics systematically overstate their reliability. If true, this means benchmark results for VLM driving assistants cannot be trusted as evidence of visual grounding, and safety-critical deployment needs rethinking.

What carries the argument

The load-bearing object is DriveBench, a benchmark built by re-sampling DriveLM's nuScenes-based QA pairs to balance answer distributions and by filtering to instances GPT-4o can answer from a single frame, then testing every model under clean images, 15 synthesized corruptions (weather, sensor failures, blurs, transmission errors), and fully black text-only inputs. The critical mechanism is the text-only condition: comparing a model's answers with no visual input against its clean-image answers exposes how much of the apparent performance comes from language priors rather than vision. A second mechanism is the rubric-aware GPT evaluation, which the paper shows yields more discriminating scores when the prompt includes the question, task rubrics, and scene context rather than the answer and ground truth alone.

What would settle it

Take the DriveBench perception questions and replace the text-only condition with images whose pixels are randomly permuted or inverted, so no scene information remains but the question text is unchanged; if VLM accuracy stays at the clean-image level on these inputs, the claim that models lean on language priors is supported, whereas a drop to chance would falsify it. Alternatively, compute the always-'Going Ahead' accuracy on the balanced DriveBench behavior MCQs: if that baseline is near chance rather than above 70%, the dataset-imbalance explanation would need revision.

Watch

Extended reading notes

Core claim

DriveBench's central finding is that VLMs do not reliably ground driving answers in visual input: removing the image entirely leaves GPT scores nearly unchanged across tasks, with some models even scoring higher on text-only prompts, and accuracy on perception multiple-choice questions frequently stays at the same level with black images as with clean ones. The paper attributes this to learned priors, common-sense defaults such as 'Going Ahead', and textual cues embedded in the questions themselves, such as camera names and coordinates. Because existing datasets are imbalanced and current metrics reward fluent, template-like explanations over correct visual reasoning, these fabricated answers pass evaluation. The paper also finds that models are partly aware of corruptions when explicitly told about them, but do not volunteer that awareness or downgrade their confidence, and that driving-specific fine-tuning largely improves answer formatting rather than visual grounding.

Load-bearing premise

The benchmark selects test instances by keeping only those GPT-4o can answer correctly from a single frame, assuming that GPT-4o's success marks an instance where one frame carries enough visual information; if that filter inherits GPT-4o's own biases, then the measured differences between clean, corrupted, and text-only inputs do not cleanly measure visual grounding.

Editorial extensions

If this is right

  • VLM driving assistants that score well on existing benchmarks may be answering from language priors, so a high benchmark score alone is not evidence of visual understanding.
  • Robustness to corruption cannot be read from stable scores under corrupted images, because the same stability appears under text-only inputs, indicating the scores may not involve the image at all.
  • Evaluation protocols for driving VLMs should include text-only and black-image controls, balanced answer distributions, and task-specific rubrics with scene context.
  • Models' awareness of corruptions, when explicitly prompted, could be used to make them abstain or hedge in degraded conditions, improving reliability.
  • Fine-tuned driving VLMs can show inflated gains from answer-format matching, so the choice of metric changes conclusions about which model is best.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • My inference: the same blindfold test — comparing text-only against full-input performance — could be applied to any VLM claim in robotics, medical imaging, or other safety-critical domains, not just driving.
  • My inference: the paper's results imply a cheap diagnostic for future models: report the gap between clean, corrupted, and text-only accuracy, since a small gap means the model's answers are likely prior-driven.
  • My inference: because questions embed camera names and normalized coordinates, even 'text-only' inputs contain geometric cues; removing those cues by scrambling camera labels would isolate how much of the text-only performance comes from pure language priors versus coordinate heuristics.
  • My inference: a testable extension is to compute the majority-answer baseline ('always Going Ahead') on each balanced task; models near that baseline under corruption are best explained as prior-following rather than vision-based.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. DriveBench is an empirical benchmark study that evaluates 12 vision-language models (VLMs) on four driving-language tasks under clean, corrupted, and text-only inputs. The benchmark is built from DriveLM, rebalanced to counter the 78.6% 'Going Straight' bias, and supplemented with 15 corruption types. The paper reports three main findings: (1) VLMs often maintain comparable accuracy and GPT-scores under heavy corruption and even with no image at all, suggesting they rely on general knowledge and textual priors rather than visual grounding; (2) VLMs exhibit corruption awareness only when explicitly prompted; and (3) standard metrics (ROUGE/BLEU, and even naive GPT scores) are insensitive to fabrication, while rubric- and context-augmented GPT evaluation is more discriminative. The authors release the benchmark toolkit and dataset publicly.

Significance. If the findings hold, this is a valuable and timely empirical contribution: it provides a concrete, publicly available testbed with 17 input settings, introduces a text-only control that exposes a failure mode common to driving VLMs, and offers a transparent analysis of dataset imbalance (the 78.6% 'Going Straight' statistic) and metric pathologies. The paper's strengths include a large model sweep, a clear corruption taxonomy, a falsifiable text-only experimental design, and the public release of code and data. However, the central claim that 'VLMs can pass driving benchmarks without seeing the road' rests on the curated subset of DriveBench, and the curation procedure plus the evaluation methodology introduce load-bearing concerns that currently prevent the quantitative results from fully supporting the broad conclusions.

major comments (3)
  1. [Sec. 3.1 (Dataset curation)] The curation filter keeps only instances that 'GPT-4o can correctly interpret,' on the assumption that GPT-4o's success indicates sufficient single-frame visual information. This assumption is load-bearing for the central claim, but the paper's own Fig. 10 shows that models can answer questions from camera names and coordinates without any visual input. If GPT-4o's successes are partly driven by the same textual priors, the filter may preferentially retain textually guessable cases and remove the cases that genuinely require visual grounding. The comparable clean vs. text-only results in Tabs. 2 and 3 could then partly measure the guessability of the filtered subset rather than a general VLM property. Please provide a concrete test: evaluate the models on the filtered-out instances (where GPT-4o fails) and show whether text-only performance drops, or quantify the distribution of answer priors in the kept versus removed sets.
  2. [Tabs. 2 and 3] The main quantitative evidence for the 'comparable performance' claim is presented as single point estimates without error bars, confidence intervals, or significance tests. Several of the decisive differences are small (e.g., GPT-4o perception GPT score 35.37 clean vs. 36.48 text-only; accuracy 59.0 vs. 59.5 in Tab. 3). With roughly 200 perception MCQs and 200 behavior MCQs, these differences are plausibly within sampling noise. Please report bootstrap confidence intervals or per-keyframe variance and state which clean/text-only/corrupted comparisons are statistically distinguishable.
  3. [Appendix C.4 and Fig. 8 (Human evaluation)] The human evaluation is based on only 15 keyframes. The large human accuracy drops under corruption (e.g., perception 93.3 clean vs. 33.3 fog) are used to argue that the corruptions are genuinely impactful and that VLMs' lack of degradation is abnormal. With n=15, the standard error is large and the reported accuracy differences are fragile. Please either increase the human sample or report confidence intervals; without this, the human-vs-VLM contrast in Sec. 4.2.1 is not quantitatively supported.
minor comments (6)
  1. [A.3 and Tab. 6] The text in A.3 states that the curated dataset contains 1,461 questions, but Tab. 6 sums to 1,261 (400+61+600+200). The abstract reports 20,498 total QA pairs, which matches 19,237 robustness pairs plus 1,261, not 1,461. Please reconcile these numbers.
  2. [Tab. 3 caption] Tab. 3 reports only 8 of the 12 evaluated models; the selection criterion is stated only in the main text. Please add the criterion to the table caption for clarity.
  3. [Fig. 8] The y-axis label 'Performance Change' is ambiguous; please specify whether it is absolute accuracy change, relative change, or another quantity.
  4. [Tab. 14] There is a typo in the table title ('Predicion'), and the Qwen2VL7B and Qwen2VL72B rows are duplicated verbatim for the 7B model entries.
  5. [Abstract and Sec. 5] The abstract claims 'we propose refined evaluation metrics,' but the paper does not introduce a new metric; it advocates rubric- and context-augmented GPT scoring. Consider wording such as 'refined evaluation protocols' to avoid overclaiming.
  6. [Fig. 12] The caption of Fig. 12 should state the task and condition (prediction task, clean inputs) so the reader can interpret the correlation plots without consulting the main text.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity found: DriveBench's central comparison is an empirical measurement, not a derivation from fitted parameters or self-citation.

full rationale

The paper's main claim—that VLMs answer driving questions from textual or general-knowledge cues even without usable visual input—is supported by direct experimental contrasts between clean, corrupted, and text-only conditions across 12 models. No parameter is fitted to a subset and then renamed as a prediction; no equation is defined in terms of the target conclusion. The GPT-4o-based curation filter in Sec. 3.1 is a benchmark-construction choice aimed at removing temporal-context-dependent cases, and the paper's own Appendix E.2 acknowledges the resulting DriveLM-only scope, so the conclusion is not forced by an imported uniqueness claim or by a self-citation chain. Self-citations to DriveLM and to the authors' corruption-generation work supply datasets and tools, but the reliability conclusion is measured, not assumed. Hence no circular step can be exhibited with a specific reduction.

Assumptions & free parameters 1 free parameters · 4 assumptions · 0 invented entities

The central comparison (clean vs text-only) is an empirical measurement rather than a derivation, so there are no fitted constants. The main assumptions are about benchmark representativeness, corruption fidelity, and the validity of GPT-based scoring; these are stated or implicit in the construction sections.

free parameters (1)
  • Dataset curation filter based on GPT-4o correctness
    Section 3.1 retains instances that GPT-4o can correctly interpret and removes those that depend heavily on temporal context. This hand-chosen filter changes the difficulty of DriveBench and has no independent quantitative justification.
assumptions (4)
  • domain assumption DriveLM-nuScenes is representative of driving-with-language benchmarks
    Section 3.1 and Appendix B use DriveLM as the base dataset; the paper's conclusions about dataset imbalance and metric failure are drawn largely from this single benchmark, with BDD-X as a secondary check.
  • domain assumption The 15 corruption simulation algorithms approximate real-world sensor and weather degradations
    Appendix A.2 applies algorithms from prior robustness benchmarks, but the paper does not validate them against real degraded driving footage; the reliability conclusions depend on these corruptions being realistic.
  • ad hoc to paper GPT-4o correctness indicates sufficient single-frame visual information
    Section 3.1 filters the curated set to examples GPT-4o can interpret; the benchmark's ability to separate visual grounding from language priors depends on this premise.
  • domain assumption GPT-3.5-turbo rubric-based scores reflect human-preferred explanation quality
    Section 3.5 and Appendix C.3 use GPT-3.5 as the primary evaluator for most tables; the human study only checks MCQ accuracy on 15 keyframes, not the GPT score itself.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Are VLMs Ready for Autonomous Driving? An Empirical Study from the Reliability, Data, and Metric Perspectives." pith.science (2026). https://pith.science/paper/7HFMFSIR

@misc{pith2026250104003,
  author       = {Pith},
  title        = {Pith review of: Are VLMs Ready for Autonomous Driving? An Empirical Study from the Reliability, Data, and Metric Perspectives},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/7HFMFSIR}},
  note         = {Machine review of arXiv:2501.04003}
}
read the original abstract

Recent advancements in Vision-Language Models (VLMs) have sparked interest in their use for autonomous driving, particularly in generating interpretable driving decisions through natural language. However, the assumption that VLMs inherently provide visually grounded, reliable, and interpretable explanations for driving remains largely unexamined. To address this gap, we introduce DriveBench, a benchmark dataset designed to evaluate VLM reliability across 17 settings (clean, corrupted, and text-only inputs), encompassing 19,200 frames, 20,498 question-answer pairs, three question types, four mainstream driving tasks, and a total of 12 popular VLMs. Our findings reveal that VLMs often generate plausible responses derived from general knowledge or textual cues rather than true visual grounding, especially under degraded or missing visual inputs. This behavior, concealed by dataset imbalances and insufficient evaluation metrics, poses significant risks in safety-critical scenarios like autonomous driving. We further observe that VLMs struggle with multi-modal reasoning and display heightened sensitivity to input corruptions, leading to inconsistencies in performance. To address these challenges, we propose refined evaluation metrics that prioritize robust visual grounding and multi-modal understanding. Additionally, we highlight the potential of leveraging VLMs' awareness of corruptions to enhance their reliability, offering a roadmap for developing more trustworthy and interpretable decision-making systems in real-world autonomous driving contexts. The benchmark toolkit is publicly accessible.

Figures

Figures reproduced from arXiv: 2501.04003 by the authors.

Figure 1
Figure 1. Overview of DriveBench. Our benchmark evaluates the reliability and visual grounding of Vision-Language Models (VLMs) in autonomous driving across four mainstream driving tasks – perception, prediction, planning, and explanation – under a diverse spectrum of 17 settings (clean, corrupted, and text-only inputs). It includes 19, 200 frames and 20, 498 QA pairs spanning three question types: multiple-choice, open-ended… view at source ↗
Figure 2
Figure 2. Do VLMs provide reliable explanations based on vi￾sual cues in driving? We study this from perspectives on reliabil￾ity, data, and metrics. We find VLMs can fabricate quality answers to driving questions when visual information is absent. The fab￾ricated answers can bypass current metrics, even GPT scores, due to imbalance, lack of context dataset, and problematical evaluation protocols. Our observations challenge t… view at source ↗
Figure 3
Figure 3. The behavior distributions of steering and speed in DriveLM-nuScenes [59]. The majority actions of vehicle behav￾iors are “Going Ahead”, which has also been noted in [39]. 90.72 64.91 95.56 27.55 93.33 0 25 50 75 100 Human Random Clean No Pix. No Fea. Acc (%) (a) Perception Task 35.91 28.16 58.82 23.40 69.51 0 25 50 75 100 Human Random Clean No Pix. No Fea. Acc (%) (b) Behavior Task [PITH_FULL_IMAGE:figures/full_fi… view at source ↗
Figures from the paper (27 more)
Figure 4
Figure 4. Figure 4: The accuracy scores of perception and behavior tasks under different visual inputs. The results are from DriveLM￾Agent [59]. No Pix. and No Fea. denote zero image pixel and zero feature, respectively. cases [19, 51], which suggests that VLMs, trained on vast, diverse d…
Figure 6
Figure 6. Figure 6: The word cloud collected from the QA pairs in the proposed benchmark, highlighting the main focus on different au￾tonomous driving tasks in DriveBench. The larger the font size, the higher the frequency of occurrence. siderable difficulty in accurately interpreting cue…
Figure 7
Figure 7. Figure 7: GPT-4o failure cases. (a): GPT-4o reasons the moving status of the pedestrian by the moving position related to the frame, instead of the coordinate of the moving object itself, thus leading to wrong perception results. (b): The model struggles to distinguish the corre…
Figure 8
Figure 8. Figure 8: Illustration of performance degradation. After ap￾plying each corruption, we evaluate the MCQs accuracy changes compared with clean inputs. We observe that human performance largely decreases while most VLMs remain unchanged. sponses, thereby reducing the opaque “black…
Figure 9
Figure 9. Figure 9: Radar chart comparisons among different models. The performance for each input corruption type is averaged across all the 1, 261 questions spanning four different driving tasks. The evaluation metric used here is the GPT score. for models that exceed this random baseli…
Figure 10
Figure 10. Figure 10: Prediction spatial distribution of Qwen2-VL7B [67] under text-only prompts. The model can potentially “guess” the MCQ answers without visual information by leveraging text cues, e.g., camera and coordinate positions in the questions. tion: “Are driving VLMs aware of t…
Figure 12
Figure 12. Figure 12: Correlations when using different metrics. We study how well accuracy or ROUGE-L [40] matches the GPT scores for open-ended questions and multiple-choice questions (MCQs), re￾spectively. We find that ROUGE-L [40] fails to reflect semantic information (e.g., key object…
Figure 13
Figure 13. Figure 13: Comparisons among different evaluation types (rubric, question-aware, and context-aware). The GPT scores vary de￾pending on the rubric, question, and physical driving context. With more information added, the results become more distinguishable. Q: In this scenario, w…
Figure 14
Figure 14. Figure 14: Examples of GPT-4o [2] responses to four tasks and the corresponding evaluation results under the dark condition. We observe that GPT-4o [2] is aware of the low-light environment and can identify the bus and pedestrian from the image, showing resilience. 11 [PITH_FUL…
Figure 15
Figure 15. Figure 15: Examples of GPT-4o [2] responses to four tasks and the corresponding evaluation results under the motion blur condition. We observe that GPT-4o [2] are influenced by this type of corruption and tend to predict “driving fast” based on it. The example shows the potentia…
Figure 16
Figure 16. Figure 16: Examples of different VLM responses under the [PITH_FULL_IMAGE:figures/full_fig_p013_16.png]
Figure 17
Figure 17. Figure 17: Examples of different VLM responses under the [PITH_FULL_IMAGE:figures/full_fig_p014_17.png]
Figure 18
Figure 18. Figure 18: Examples of GPT evaluators with different information. The evaluator can revise the score and give a more accurate evaluation [PITH_FULL_IMAGE:figures/full_fig_p016_18.png]
Figure 19
Figure 19. Figure 19: BDD-X dataset [29]: detailed distribution of car actions. Only the actions with a frequency larger than 80 are visualized. The Stop actions and Turn actions are highlighted. We observe similar data distribution in balance as those in DriveLM [59], where turning action…
Figure 20
Figure 20. Figure 20: Inference system prompt. already included the visual description and moving status of important objects, we only prompt with PRED and GT with detailed rubrics. C.4. Human Evaluations In this subsection, we elaborate in more detail on how we conduct the human evaluatio…
Figure 21
Figure 21. Figure 21: Prediction spatial distributions from VLMs. The locations represent the object positions in the image within each camera, which [PITH_FULL_IMAGE:figures/full_fig_p020_21.png]
Figure 22
Figure 22. Figure 22: GPT evaluation prompts for MCQs in our benchmark. 21 [PITH_FULL_IMAGE:figures/full_fig_p021_22.png]
Figure 23
Figure 23. Figure 23: GPT evaluation prompts for Open-Ended Questions in our benchmark. 22 [PITH_FULL_IMAGE:figures/full_fig_p022_23.png]
Figure 24
Figure 24. Figure 24: Illustrative examples from our human evaluation interfaces. [PITH_FULL_IMAGE:figures/full_fig_p023_24.png]
Figure 25
Figure 25. Figure 25: Model performance comparisons using radar graphs. The performance for each input corruption type is averaged across all the 1, 261 questions spanning four different tasks using GPT scores. The gray dash line represents the performance of text-only input. We observe VL…
Figure 26
Figure 26. Figure 26: Example of Qwen2-VL72B for the Perception task. Under the text-only condition (right example), the model can fabricate answers based on the coordinate information in the question. The GPT evaluator can not reflect the fabricated answer even with detailed rubrics, resu…
Figure 27
Figure 27. Figure 27: Example of Qwen2-VL72B for the Prediction task. Under the snow condition (right example), the model can still observe some visible objects in the images. Since most objects are severely occluded, the third object noticed becomes more general: the building, compared to…
Figure 28
Figure 28. Figure 28: Example of Qwen2-VL72B for Planning task. Under the text-only condition (right example), the model fails to perceive the motorcycle to stop sign, giving their close distance on the image. When the image is absent, the model guesses the object to be a pedestrian or a s…
Figure 29
Figure 29. Figure 29: Example of GPT-4o for Perception task. Under the bit error condition (right example), the visual information for the target object is completely corrupted but the model can guess the answer correctly. However, the GPT evaluator here captures the nuance in the explanat…
Figure 30
Figure 30. Figure 30: Example of GPT-4o for Planning task. In both low-light (left example) and normal (right example) conditions, the ground truth is to turn left. However, in the left case, the vehicle is prepared to turn left. Turning right will cause a potential collision with the whit…
Figure 31
Figure 31. Figure 31: Examples of the GPT evaluator with different prompts. With the same question and answer, the GPT evaluator assigns different [PITH_FULL_IMAGE:figures/full_fig_p030_31.png]
Figure 32
Figure 32. Figure 32: Examples of the GPT evaluators with different information. We gradually add more information about the question and the [PITH_FULL_IMAGE:figures/full_fig_p031_32.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 17 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Steadily moving semi-infinite fracture in plane poroelasticity

    physics.geo-ph 2026-04 unverdicted novelty 7.0 of 10

    XEmbodied achieves SOTA on 18 embodied VQA benchmarks by fusing 3D geometric tokens and distilled physical cues into a 30B VLM with progressive curriculum training.

  2. STSBench: A Spatio-temporal Scenario Benchmark for Multi-modal Large Language Models in Autonomous Driving

    cs.CV 2025-06 conditional novelty 7.0 of 10

    A new benchmark, STSnu, uses 971 verified multiple-choice questions from NuScenes to test driving vision-language models' spatio-temporal reasoning, and shows they lag far behind text-only LLMs given perfect trajectories.

  3. MapTab: A Diagnostic Benchmark for Long-Horizon Multi-Criteria Multimodal Reasoning on Heterogeneous Topological Graphs

    cs.LG 2026-02 unverdicted novelty 6.0 of 10

    MapTab is a new multimodal benchmark with 328 images and nearly 200k queries that shows current MLLMs have substantial difficulty with multi-criteria route planning when visual and tabular information must be combined.

  4. DriveQA: Passing the Driving Knowledge Test

    cs.CV 2025-08 conditional novelty 6.0 of 10

    DriveQA is a new multimodal driving-knowledge benchmark showing that LLMs and MLLMs struggle with right-of-way, numerical traffic rules, and sign variations, with modest transfer gains to nuScenes and BDD.

  5. Monocular Semantic Scene Completion via Masked Recurrent Networks

    cs.CV 2025-07 conditional novelty 6.0 of 10

    Decomposing monocular semantic scene completion into a coarse stage plus a masked recurrent refinement network improves NYUv2 and SemanticKITTI completion and semantic IoU over prior monocular methods.

  6. Beyond One Shot, Beyond One Perspective: Cross-View and Long-Horizon Distillation for Better LiDAR Representations

    cs.CV 2025-07 conditional novelty 6.0 of 10

    LiMA distills long-term multi-camera image features into LiDAR backbones and reports consistent gains on segmentation and detection benchmarks.

  7. AD^2-Bench: A Hierarchical CoT Benchmark for MLLM in Autonomous Driving under Adverse Conditions

    cs.CV 2025-06 conditional novelty 6.0 of 10

    AD^2-Bench is a new adverse-weather driving benchmark with hierarchical chain-of-thought annotations and LLM-based quality metrics; 12 MLLMs all scored below 60%.

  8. Impromptu VLA: Open Weights and Open Data for Driving Vision-Language-Action Models

    cs.CV 2025-05 conditional novelty 6.0 of 10

    A new 80K-clip dataset of unstructured driving scenarios with Q&A annotations improves VLA performance on NeuroNCAP and nuScenes benchmarks.

  9. PixelThink: Towards Efficient Chain-of-Pixel Reasoning

    cs.CV 2025-05 conditional novelty 6.0 of 10

    A soft token-budget penalty driven by task difficulty and model confidence roughly halves the reasoning tokens used by segmentation MLLMs while slightly improving mask accuracy on ReasonSeg-derived benchmarks.

  10. Extending Large Vision-Language Model for Diverse Interactive Tasks in Autonomous Driving

    cs.CV 2025-05 conditional novelty 6.0 of 10

    DriveMonkey, which bridges a vision-language model to a pre-trained 3D detector with learnable queries, improves 3D visual grounding mAP by 9.86 percentage points over general LVLMs on the new NuInteract benchmark.

  11. Are Vision LLMs Road-Ready? A Comprehensive Benchmark for Safety-Critical Driving Video Understanding

    cs.CV 2025-04 conditional novelty 6.0 of 10

    DVBench introduces 10,000 expert-annotated questions on crash and near-crash driving videos and reports that no tested vision LLM exceeds 40 percent accuracy under its strict GroupEval scoring.

  12. Benchmarking the Robustness of Autonomous Driving to Environmental Illusions: A Lane Perception Perspective

    cs.CV 2026-07 conditional novelty 5.0 of 10

    Environmental illusions cause 5-7% accuracy drops in lane detection models and can trigger collisions in closed-loop simulation, with a proposed defense (MIDA) recovering ~4% robustness.

  13. RoadBench: Benchmarking MLLMs on Fine-Grained Spatial Understanding and Reasoning under Urban Road Scenarios

    cs.CV 2025-11 conditional novelty 5.0 of 10

    A new 9,121-case benchmark of road-marking tasks shows most multimodal LLMs perform near or below simple rule-based baselines in fine-grained urban spatial reasoning.

  14. RoboTron-Drive: All-in-One Large Multimodal Model for Autonomous Driving

    cs.CV 2024-12 conditional novelty 5.0 of 10

    One 8B multimodal model trained jointly on six driving datasets outperforms individual specialists on average and transfers zero-shot to three unseen driving benchmarks.

  15. A Survey on Vision-Language-Action Models for Autonomous Driving

    cs.CV 2025-06 conditional novelty 4.0 of 10

    A survey organizes vision-language-action models for autonomous driving into four stages, compares over 20 systems, and catalogs datasets, benchmarks, and open challenges.

  16. Chain-of-Thought for Autonomous Driving: A Comprehensive Survey and Future Prospects

    cs.RO 2025-05 conditional novelty 4.0 of 10

    A survey that classifies chain-of-thought methods for autonomous driving into modular, logical, and reflective pipelines, and proposes three evolutionary stages from direct prompting to reinforcement learning.

  17. Generative AI for Autonomous Driving: Frontiers and Opportunities

    cs.CV 2025-05 accept novelty 2.0 of 10

    A comprehensive, structured survey of generative AI for autonomous driving, covering model families, sensor modalities, real-world applications, and open research challenges.

Reference graph

Works this paper leans on

80 extracted references · 51 canonical work pages · cited by 17 Pith papers

  1. [1]

    Phi-3 technical report: A highly capable language model locally on your phone

    Marah Abdin, Sam Ade Jacobs, Ammar Ahmad Awan, Jyoti Aneja, Ahmed Awadallah, Hany Awadalla, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Harkirat Behl, et al. Phi-3 technical report: A highly capable language model locally on your phone. arXiv preprint arXiv:2404.14219, 2024. 2, 3, 7, 9, 10, 18, 20

    arXiv 2024

  2. [2]

    Gpt-4 technical report

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ah- mad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 ,

    arXiv

  3. [3]

    Mousumi Akter, Naman Bansal, and Shubhra Kanti Kar- maker. Revisiting automatic evaluation of extractive summa- rization task: Can we do better than rouge? In Findings of the Association for Computational Linguistics, pages 1547– 1560, 2022. 3

    work page 2022

  4. [4]

    Spice: Semantic propositional image cap- tion evaluation

    Peter Anderson, Basura Fernando, Mark Johnson, and Stephen Gould. Spice: Semantic propositional image cap- tion evaluation. In European Conference on Computer Vi- sion, pages 382–398, 2016. 3

    work page 2016

  5. [5]

    Qwen technical report

    Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xi- aodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, Binyuan Hui, Luo Ji, Mei Li, Junyang Lin, Runji Lin, Day- iheng Liu, Gao Liu, Chengqiang Lu, Keming Lu, Jianxin Ma, Rui Men, Xingzhang Ren, Xuancheng Ren, Chuanqi Tan, Sinan Tan, Jianhong Tu, Peng Wang, Shijie Wang, Wei Wang, Shengguang Wu, Benfe...

    arXiv 2023

  6. [6]

    Rt-2: Vision-language-action models transfer web knowledge to robotic control

    Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Xi Chen, Krzysztof Choromanski, Tianli Ding, Danny Driess, Avinava Dubey, Chelsea Finn, et al. Rt-2: Vision-language-action models transfer web knowledge to robotic control. arXiv preprint arXiv:2307.15818, 2023. 3

    arXiv 2023

  7. [7]

    nuscenes: A multi- modal dataset for autonomous driving

    Holger Caesar, Varun Bankiti, Alex H Lang, Sourabh V ora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Gi- ancarlo Baldan, and Oscar Beijbom. nuscenes: A multi- modal dataset for autonomous driving. In IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pages 11621–11631, 2020. 4

    work page 2020

  8. [8]

    Spatialvlm: Endow- ing vision-language models with spatial reasoning capabili- ties

    Boyuan Chen, Zhuo Xu, Sean Kirmani, Brain Ichter, Dorsa Sadigh, Leonidas Guibas, and Fei Xia. Spatialvlm: Endow- ing vision-language models with spatial reasoning capabili- ties. In IEEE/CVF Conference on Computer Vision and Pat- tern Recognition, pages 14455–14465, 2024. 3

    work page 2024

Show all 80 references
  1. [9]

    Automated evaluation of large vision-language models on self-driving corner cases

    Kai Chen, Yanze Li, Wenhua Zhang, Yanxin Liu, Pengxi- ang Li, Ruiyuan Gao, Lanqing Hong, Meng Tian, Xinhai Zhao, Zhenguo Li, Dit-Yan Yeung, Huchuan Lu, and Xu Jia. Automated evaluation of large vision-language models on self-driving corner cases. arXiv preprint arXiv:2404.10595,

    arXiv

  2. [10]

    Driving with llms: Fusing object-level vector modality for explainable autonomous driving

    Long Chen, Oleg Sinavski, Jan H ¨unermann, Alice Karnsund, Andrew James Willmott, Danny Birch, Daniel Maund, and Jamie Shotton. Driving with llms: Fusing object-level vector modality for explainable autonomous driving. In IEEE In- ternational Conference on Robotics and Automat...

    2024

  3. [11]

    Driving- gpt: Unifying driving world modeling and planning with multi-modal autoregressive transformers

    Yuntao Chen, Yuqi Wang, and Zhaoxiang Zhang. Driving- gpt: Unifying driving world modeling and planning with multi-modal autoregressive transformers. arXiv preprint arXiv:2412.18607, 2024. 3

    2024 arXiv

  4. [12]

    Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks

    Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, Bin Li, Ping Luo, Tong Lu, Yu Qiao, and Jifeng Dai. Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. arXiv prepri...

    2023 arXiv

  5. [13]

    How far are we to gpt-4v? closing the gap to commercial multimodal models with open-source suites

    Zhe Chen, Weiyun Wang, Hao Tian, Shenglong Ye, Zhang- wei Gao, Erfei Cui, Wenwen Tong, Kongzhi Hu, Jiapeng Luo, Zheng Ma, et al. How far are we to gpt-4v? closing the gap to commercial multimodal models with open-source suites. arXiv preprint arXiv:2404.16821, 2024. 2

    2024 arXiv

  6. [14]

    Drivelm: Driving with graph vi- sual question answering

    DriveLM contributors. Drivelm: Driving with graph vi- sual question answering. https : / / github . com / OpenDriveLab/DriveLM, 2023. 4, 7

    2023

  7. [15]

    Drive as you speak: Enabling human-like interac- tion with large language models in autonomous vehicles

    Can Cui, Yunsheng Ma, Xu Cao, Wenqian Ye, and Ziran Wang. Drive as you speak: Enabling human-like interac- tion with large language models in autonomous vehicles. In Proceedings of the IEEE/CVF Winter Conference on Appli- cations of Computer Vision, pages 902–909, 2024. 3

    2024

  8. [16]

    Talk2car: Taking control of your self-driving car

    Thierry Deruyttere, Simon Vandenhende, Dusan Grujicic, Luc Van Gool, and Marie-Francine Moens. Talk2car: Taking control of your self-driving car. In Conference on Empirical Methods in Natural Language Processing, pages 2088–2098,

  9. [17]

    Insight-v: Ex- ploring long-chain visual reasoning with multimodal large language models

    Yuhao Dong, Zuyan Liu, Hai-Long Sun, Jingkang Yang, Winston Hu, Yongming Rao, and Ziwei Liu. Insight-v: Ex- ploring long-chain visual reasoning with multimodal large language models. arXiv preprint arXiv:2411.14432 , 2024. 3, 18

    2024 arXiv

  10. [18]

    Out of the bleu: how should we assess quality of the code generation models? Journal of Systems and Software, 203:111741, 2023

    Mikhail Evtikhiev, Egor Bogomolov, Yaroslav Sokolov, and Timofey Bryksin. Out of the bleu: how should we assess quality of the code generation models? Journal of Systems and Software, 203:111741, 2023. 3

    2023

  11. [19]

    Data determines distributional robustness in contrastive language image pre-training (clip)

    Alex Fang, Gabriel Ilharco, Mitchell Wortsman, Yuhao Wan, Vaishaal Shankar, Achal Dave, and Ludwig Schmidt. Data determines distributional robustness in contrastive language image pre-training (clip). In International Conference on Machine Learning, pages 6216–6234. PMLR, 2022...

    2022

  12. [20]

    Drive like a human: Rethink- ing autonomous driving with large language models

    Daocheng Fu, Xin Li, Licheng Wen, Min Dou, Pinlong Cai, Botian Shi, and Yu Qiao. Drive like a human: Rethink- ing autonomous driving with large language models. In IEEE/CVF Winter Conference on Applications of Computer Vision, pages 910–919, 2024. 2, 3

    2024

  13. [21]

    Llama-adapter v2: Parameter-efficient vi- sual instruction model

    Peng Gao, Jiaming Han, Renrui Zhang, Ziyi Lin, Shijie Geng, Aojun Zhou, Wei Zhang, Pan Lu, Conghui He, Xi- angyu Yue, et al. Llama-adapter v2: Parameter-efficient vi- sual instruction model. arXiv preprint arXiv:2304.15010 ,

    arXiv

  14. [22]

    Frontiers: Can large lan- guage models capture human preferences? Marketing Sci- ence, 2024

    Ali Goli and Amandeep Singh. Frontiers: Can large lan- guage models capture human preferences? Marketing Sci- ence, 2024. 3

    2024

  15. [23]

    Drivemllm: A benchmark for spatial understanding with multimodal large language models in autonomous driving

    Xianda Guo, Ruijun Zhang, Yiqun Duan, Yuhang He, Chen- ming Zhang, Shuai Liu, and Long Chen. Drivemllm: A benchmark for spatial understanding with multimodal large language models in autonomous driving. arXiv preprint arXiv:2411.13112, 2024. 5

    2024 arXiv

  16. [24]

    Is your hd map constructor reliable under sensor corruptions? In Advances in Neural Informa- tion Processing Systems, 2024

    Xiaoshuai Hao, Mengchuan Wei, Yifan Yang, Haimei Zhao, Hui Zhang, Yi Zhou, Qiang Wang, Weiming Li, Lingdong Kong, and Jing Zhang. Is your hd map constructor reliable under sensor corruptions? In Advances in Neural Informa- tion Processing Systems, 2024. 5

    2024

  17. [25]

    Cogagent: A visual language model for gui agents

    Wenyi Hong, Weihan Wang, Qingsong Lv, Jiazheng Xu, Wenmeng Yu, Junhui Ji, Yan Wang, Zihan Wang, Yuxiao Dong, Ming Ding, et al. Cogagent: A visual language model for gui agents. InIEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14281–14290, 2024. 3

    2024

  18. [26]

    Planning-oriented autonomous driving

    Yihan Hu, Jiazhi Yang, Li Chen, Keyu Li, Chonghao Sima, Xizhou Zhu, Siqi Chai, Senyao Du, Tianwei Lin, Wen- hai Wang, et al. Planning-oriented autonomous driving. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 17853–17862, 2023. 2, 7

    2023

  19. [27]

    Survey of hallucination in natural language generation

    Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung. Survey of hallucination in natural language generation. ACM Computing Surveys, 55(12):1–38, 2023. 6, 10

    2023

  20. [28]

    Senna: Bridging large vision-language mod- els and end-to-end autonomous driving

    Bo Jiang, Shaoyu Chen, Bencheng Liao, Xingyu Zhang, Wei Yin, Qian Zhang, Chang Huang, Wenyu Liu, and Xing- gang Wang. Senna: Bridging large vision-language mod- els and end-to-end autonomous driving. arXiv preprint arXiv:2410.22313, 2024. 2, 6

    arXiv

  21. [29]

    Textual explanations for self-driving ve- hicles

    Jinkyu Kim, Anna Rohrbach, Trevor Darrell, John Canny, and Zeynep Akata. Textual explanations for self-driving ve- hicles. In European Conference on Computer Vision, pages 563–578, 2018. 2, 3, 5, 14, 17, 18

    2018

  22. [30]

    Robo3d: Towards robust and reliable 3d perception against corruptions

    Lingdong Kong, Youquan Liu, Xin Li, Runnan Chen, Wen- wei Zhang, Jiawei Ren, Liang Pan, Kai Chen, and Ziwei Liu. Robo3d: Towards robust and reliable 3d perception against corruptions. In IEEE/CVF International Conference on Computer Vision, pages 19994–20006, 2023. 3, 5, 16

    2023

  23. [31]

    Robodepth: Robust out-of-distribution depth estimation under corruptions

    Lingdong Kong, Shaoyuan Xie, Hanjiang Hu, Lai Xing Ng, Benoit Cottereau, and Wei Tsang Ooi. Robodepth: Robust out-of-distribution depth estimation under corruptions. Ad- vances in Neural Information Processing Systems, 36, 2024. 3, 16

    2024

  24. [32]

    Lingdong Kong, Shaoyuan Xie, Hanjiang Hu, Yaru Niu, Wei Tsang Ooi, Benoit R. Cottereau, Lai Xing Ng, Yuexin Ma, Wenwei Zhang, Liang Pan, Kai Chen, Ziwei Liu, We- ichao Qiu, Wei Zhang, Xu Cao, Hao Lu, Ying-Cong Chen, Caixin Kang, Xinning Zhou, Chengyang Ying, Wentao Shang, Xing...

    2024 arXiv

  25. [33]

    Gonzalez, Hao Zhang, and Ion Stoica

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Pro- ceedings of the ACM SIGOPS 29th Symposium on Operating Syste...

    2023

  26. [34]

    An environment for autonomous driving decision-making

    Edouard Leurent. An environment for autonomous driving decision-making. https://github.com/eleurent/ highway-env, 2018. 3

    2018

  27. [35]

    R-bench: Are your large multimodal model robust to real-world corruptions? arXiv preprint arXiv:2410.05474, 2024

    Chunyi Li, Jianbo Zhang, Zicheng Zhang, Haoning Wu, Yuan Tian, Wei Sun, Guo Lu, Xiaohong Liu, Xiongkuo Min, Weisi Lin, et al. R-bench: Are your large multimodal model robust to real-world corruptions? arXiv preprint arXiv:2410.05474, 2024. 4

    arXiv

  28. [36]

    Evaluating object hallucination in large vision-language models

    Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. Evaluating object hallucination in large vision-language models. In Conference on Empirical Methods in Natural Language Processing , pages 292–305,

  29. [37]

    Is your lidar placement optimized for 3d scene understanding? In Advances in Neural Information Process- ing Systems, 2024

    Ye Li, Lingdong Kong, Hanjiang Hu, Xiaohao Xu, and Xi- aonan Huang. Is your lidar placement optimized for 3d scene understanding? In Advances in Neural Information Process- ing Systems, 2024. 5

    2024

  30. [38]

    Bevformer: Learning bird’s-eye-view representation from multi-camera images via spatiotemporal transformers

    Zhiqi Li, Wenhai Wang, Hongyang Li, Enze Xie, Chong- hao Sima, Tong Lu, Yu Qiao, and Jifeng Dai. Bevformer: Learning bird’s-eye-view representation from multi-camera images via spatiotemporal transformers. In European Con- ference on Computer Vision, pages 1–18, 2022. 7

    2022

  31. [39]

    Is ego status all you need for open- loop end-to-end autonomous driving? In IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pages 14864–14873, 2024

    Zhiqi Li, Zhiding Yu, Shiyi Lan, Jiahan Li, Jan Kautz, Tong Lu, and Jose M Alvarez. Is ego status all you need for open- loop end-to-end autonomous driving? In IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pages 14864–14873, 2024. 2, 4, 17

    2024

  32. [40]

    Rouge: A package for automatic evaluation of summaries

    Chin-Yew Lin. Rouge: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pages 74–81, 2004. 3, 7, 9, 10, 14, 17, 19

    2004

  33. [41]

    Coarse corre- spondences elicit 3d spacetime understanding in multimodal language model

    Benlin Liu, Yuhao Dong, Yiqin Wang, Yongming Rao, Yan- song Tang, Wei-Chiu Ma, and Ranjay Krishna. Coarse corre- spondences elicit 3d spacetime understanding in multimodal language model. arXiv preprint arXiv:2408.00754, 2024. 3

    2024 arXiv

  34. [42]

    Visual instruction tuning

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. InAdvances in Neural Information Processing Systems, pages 34892–34916, 2023. 2, 3

    2023

  35. [43]

    Improved baselines with visual instruction tuning

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. 39 In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 26296–26306, 2024. 7, 9, 10, 18, 20

    2024

  36. [44]

    Llava-next: Im- proved reasoning, ocr, and world knowledge, 2024

    Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Llava-next: Im- proved reasoning, ocr, and world knowledge, 2024. 2, 3, 7, 9, 10, 13, 18

    2024

  37. [45]

    Aligning with hu- man judgement: The role of pairwise preference in large lan- guage model evaluators

    Yinhong Liu, Han Zhou, Zhijiang Guo, Ehsan Shareghi, Ivan Vuli´c, Anna Korhonen, and Nigel Collier. Aligning with hu- man judgement: The role of pairwise preference in large lan- guage model evaluators. arXiv preprint arXiv:2403.16950,

    arXiv

  38. [46]

    Oryx mllm: On-demand spatial-temporal understanding at arbitrary resolution

    Zuyan Liu, Yuhao Dong, Ziwei Liu, Winston Hu, Ji- wen Lu, and Yongming Rao. Oryx mllm: On-demand spatial-temporal understanding at arbitrary resolution. arXiv preprint arXiv:2409.12961, 2024. 7, 10

    2024 arXiv

  39. [47]

    Chain-of-spot: Interactive reasoning improves large vision-language models

    Zuyan Liu, Yuhao Dong, Yongming Rao, Jie Zhou, and Ji- wen Lu. Chain-of-spot: Interactive reasoning improves large vision-language models. arXiv preprint arXiv:2403.12966,

    arXiv

  40. [48]

    Dolphins: Multimodal language model for driving

    Yingzi Ma, Yulong Cao, Jiachen Sun, Marco Pavone, and Chaowei Xiao. Dolphins: Multimodal language model for driving. arXiv preprint arXiv:2312.00438, 2023. 2, 3, 6, 7, 10, 12, 18

    2023 arXiv

  41. [49]

    Drama: Joint risk localization and captioning in driving

    Srikanth Malla, Chiho Choi, Isht Dwivedi, Joon Hee Choi, and Jiachen Li. Drama: Joint risk localization and captioning in driving. In IEEE/CVF Winter Conference on Applications of Computer Vision, pages 1043–1052, 2023. 5

    2023

  42. [50]

    Gpt-driver: Learning to drive with gpt.arXiv preprint arXiv:2310.01415, 2023

    Jiageng Mao, Yuxi Qian, Junjie Ye, Hang Zhao, and Yue Wang. Gpt-driver: Learning to drive with gpt.arXiv preprint arXiv:2310.01415, 2023. 2, 6

    2023 arXiv

  43. [51]

    Quality not quantity: On the interaction between dataset design and robustness of clip

    Thao Nguyen, Gabriel Ilharco, Mitchell Wortsman, Se- woong Oh, and Ludwig Schmidt. Quality not quantity: On the interaction between dataset design and robustness of clip. Advances in Neural Information Processing Systems , 35:21455–21469, 2022. 4, 5

    2022

  44. [52]

    Foundation models for autonomous systems,

    OpenDriveLab. Foundation models for autonomous systems,

  45. [53]

    Vlp: Vision language planning for autonomous driving

    Chenbin Pan, Burhaneddin Yaman, Tommaso Nesti, Abhirup Mallik, Alessandro G Allievi, Senem Velipasalar, and Liu Ren. Vlp: Vision language planning for autonomous driving. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14760–14769, 2024. 2

    2024

  46. [54]

    Bleu: a method for automatic evaluation of machine translation

    Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation of machine translation. In Annual Meeting of the Association for Com- putational Linguistics, pages 311–318, 2002. 3, 7, 9, 14, 17

    2002

  47. [55]

    Nuscenes-qa: A multi-modal visual ques- tion answering benchmark for autonomous driving scenario

    Tianwen Qian, Jingjing Chen, Linhai Zhuo, Yang Jiao, and Yu-Gang Jiang. Nuscenes-qa: A multi-modal visual ques- tion answering benchmark for autonomous driving scenario. In AAAI Conference on Artificial Intelligence , pages 4542– 4550, 2024. 2, 3, 5, 12

    2024

  48. [56]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In International Con...

  49. [57]

    Rank2tell: A multimodal driving dataset for joint importance ranking and reasoning

    Enna Sachdeva, Nakul Agarwal, Suhas Chundi, Sean Roelofs, Jiachen Li, Mykel Kochenderfer, Chiho Choi, and Behzad Dariush. Rank2tell: A multimodal driving dataset for joint importance ranking and reasoning. In IEEE/CVF Winter Conference on Applications of Computer Vision , page...

    2024

  50. [58]

    Lmdrive: Closed-loop end-to-end driving with large language models

    Hao Shao, Yuxuan Hu, Letian Wang, Guanglu Song, Steven L Waslander, Yu Liu, and Hongsheng Li. Lmdrive: Closed-loop end-to-end driving with large language models. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15120–15130, 2024. 2, 6

    2024

  51. [59]

    Drivelm: Driving with graph visual question answering

    Chonghao Sima, Katrin Renz, Kashyap Chitta, Li Chen, Hanxue Zhang, Chengen Xie, Jens Beißwenger, Ping Luo, Andreas Geiger, and Hongyang Li. Drivelm: Driving with graph visual question answering. In European Conference on Computer Vision, pages 256–274, 2024. 2, 3, 4, 5, 6, 7, ...

    2024

  52. [60]

    Open-world ob- ject manipulation using pre-trained vision-language models

    Austin Stone, Ted Xiao, Yao Lu, Keerthana Gopalakrish- nan, Kuang-Huei Lee, Quan Vuong, Paul Wohlhart, Sean Kirmani, Brianna Zitkovich, Fei Xia, et al. Open-world ob- ject manipulation using pre-trained vision-language models. arXiv preprint arXiv:2303.00905, 2023. 3

    2023 arXiv

  53. [61]

    Scalability in perception for autonomous driving: Waymo open dataset

    Pei Sun, Henrik Kretzschmar, Xerxes Dotiwalla, Aurelien Chouard, Vijaysai Patnaik, Paul Tsui, James Guo, Yin Zhou, Yuning Chai, Benjamin Caine, et al. Scalability in perception for autonomous driving: Waymo open dataset. InIEEE/CVF Conference on Computer Vision and Pattern Rec...

    2020

  54. [62]

    Drivevlm: The convergence of autonomous driving and large vision-language models

    Xiaoyu Tian, Junru Gu, Bailin Li, Yicheng Liu, Yang Wang, Zhiyong Zhao, Kun Zhan, Peng Jia, Xianpeng Lang, and Hang Zhao. Drivevlm: The convergence of autonomous driving and large vision-language models. arXiv preprint arXiv:2402.12289, 2024. 2, 3, 5, 6, 12

    2024 arXiv

  55. [63]

    Does bleu score work for code migration? In IEEE/ACM International Conference on Program Compre- hension, pages 165–176, 2019

    Ngoc Tran, Hieu Tran, Son Nguyen, Hoan Nguyen, and Tien Nguyen. Does bleu score work for code migration? In IEEE/ACM International Conference on Program Compre- hension, pages 165–176, 2019. 3

    2019

  56. [64]

    How many unicorns are in this im- age? a safety evaluation benchmark for vision llms

    Haoqin Tu, Chenhang Cui, Zijun Wang, Yiyang Zhou, Bingchen Zhao, Junlin Han, Wangchunshu Zhou, Huaxiu Yao, and Cihang Xie. How many unicorns are in this im- age? a safety evaluation benchmark for vision llms. arXiv preprint arXiv:2311.16101, 2023. 6

    2023 arXiv

  57. [65]

    Cider: Consensus-based image description evalu- ation

    Ramakrishna Vedantam, C Lawrence Zitnick, and Devi Parikh. Cider: Consensus-based image description evalu- ation. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4566–4575, 2015. 3

    2015

  58. [66]

    Evaluation and analysis of hal- lucination in large vision-language models

    Junyang Wang, Yiyang Zhou, Guohai Xu, Pengcheng Shi, Chenlin Zhao, Haiyang Xu, Qinghao Ye, Ming Yan, Ji Zhang, Jihua Zhu, et al. Evaluation and analysis of hal- lucination in large vision-language models. arXiv preprint arXiv:2308.15126, 2023. 6

    2023 arXiv

  59. [67]

    Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution

    Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution. arXiv preprint arXiv:2409.12191, 2024. 2, 3, 7, 9, 10, 15, 18, 20

    2024 arXiv

  60. [68]

    40 Omnidrive: A holistic llm-agent framework for autonomous driving with 3d perception, reasoning and planning

    Shihao Wang, Zhiding Yu, Xiaohui Jiang, Shiyi Lan, Min Shi, Nadine Chang, Jan Kautz, Ying Li, and Jose M Alvarez. 40 Omnidrive: A holistic llm-agent framework for autonomous driving with 3d perception, reasoning and planning. arXiv preprint arXiv:2405.01533, 2024. 2, 6

    2024 arXiv

  61. [69]

    Dilu: A knowledge-driven approach to autonomous driving with large language models

    Licheng Wen, Daocheng Fu, Xin Li, Xinyu Cai, Tao Ma, Pinlong Cai, Min Dou, Botian Shi, Liang He, and Yu Qiao. Dilu: A knowledge-driven approach to autonomous driving with large language models. In International Conference on Learning Representations, 2024. 2, 3

    2024

  62. [70]

    Language prompt for autonomous driving

    Dongming Wu, Wencheng Han, Tiancai Wang, Yingfei Liu, Xiangyu Zhang, and Jianbing Shen. Language prompt for autonomous driving. arXiv preprint arXiv:2309.04379,

    arXiv

  63. [71]

    Benchmarking and im- proving bird’s eye view perception robustness in autonomous driving

    Shaoyuan Xie, Lingdong Kong, Wenwei Zhang, Jiawei Ren, Liang Pan, Kai Chen, and Ziwei Liu. Benchmarking and im- proving bird’s eye view perception robustness in autonomous driving. arXiv preprint arXiv:2405.17426, 2024. 3, 5, 10, 16

    2024 arXiv

  64. [72]

    Explain- able object-induced action decision for autonomous vehicles

    Yiran Xu, Xiaoyin Yang, Lihang Gong, Hsuan-Chu Lin, Tz- Ying Wu, Yunsheng Li, and Nuno Vasconcelos. Explain- able object-induced action decision for autonomous vehicles. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9523–9532, 2020. 2, 3, 5

    2020

  65. [73]

    Vlm-ad: End-to-end autonomous driving through vision-language model supervision

    Yi Xu, Yuxin Hu, Zaiwei Zhang, Gregory P Meyer, Siva Karthik Mustikovela, Siddhartha Srinivasa, Eric M Wolff, and Xin Huang. Vlm-ad: End-to-end autonomous driving through vision-language model supervision. arXiv preprint arXiv:2412.14446, 2024. 3

    2024 arXiv

  66. [74]

    Drivegpt4: Interpretable end-to-end autonomous driving via large language model

    Zhenhua Xu, Yujia Zhang, Enze Xie, Zhen Zhao, Yong Guo, Kwan-Yee K Wong, Zhenguo Li, and Hengshuang Zhao. Drivegpt4: Interpretable end-to-end autonomous driving via large language model. IEEE Robotics and Automation Let- ters, 9(10):8186–8193, 2024. 2, 3, 6

    2024

  67. [75]

    Octopus: Embodied vision-language programmer from environmental feedback

    Jingkang Yang, Yuhao Dong, Shuai Liu, Bo Li, Ziyue Wang, Haoran Tan, Chencheng Jiang, Jiamu Kang, Yuanhan Zhang, Kaiyang Zhou, et al. Octopus: Embodied vision-language programmer from environmental feedback. In European Conference on Computer Vision , pages 20–38. Springer,

  68. [76]

    A survey of large language models for autonomous driving

    Zhenjie Yang, Xiaosong Jia, Hongyang Li, and Junchi Yan. A survey of large language models for autonomous driving. arXiv preprint arXiv:2311.01043, 2023. 2, 3, 4

    2023 arXiv

  69. [77]

    Benchmarking the robustness of spatial-temporal models against corruptions

    Chenyu Yi, Siyuan Yang, Haoliang Li, Yap-peng Tan, and Alex Kot. Benchmarking the robustness of spatial-temporal models against corruptions. In Advances in Neural Informa- tion Processing Systems, 2021. 16

    2021

  70. [78]

    Bdd100k: A diverse driving dataset for heterogeneous multitask learning

    Fisher Yu, Haofeng Chen, Xin Wang, Wenqi Xian, Yingying Chen, Fangchen Liu, Vashisht Madhavan, and Trevor Dar- rell. Bdd100k: A diverse driving dataset for heterogeneous multitask learning. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2636–2645, 202...

    2020

  71. [2023]

    4, 5, 7, 8, 9, 10, 11, 12, 13, 15, 17, 19, 20

  72. [2024]

    Accessed: 2024-11-11. 4

    2024

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.