Pith. sign in

REVIEW 4 major objections 4 minor 1 cited by

AD^2-Bench: A Hierarchical CoT Benchmark for MLLM in Autonomous Driving under Adverse Conditions

T0 review · 4 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A new driving benchmark grades models on step-by-step reasoning and finds today's best score below 60 percent.

desk verdict Valuable adverse-weather CoT benchmark with a novel point-prompt design, but the GPT-4o-judged metrics need human validation before the headline claims can be trusted. read the letter →

arxiv 2506.09557 v1 pith:TGUTTEKZ submitted 2025-06-11 cs.CV cs.AI

Zhaoyang Wei , Chenhui Qiang , Bowen Jiang , Xumeng Han , Xuehui Yu , Zhenjun Han This is my paper
classification cs.CVcs.AI
keywords chain-of-thoughtreasoningmultimodallargelanguagemodelsautonomousdrivingadverseweathervisualquestionansweringbenchmarkatomicannotation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

AD2-Bench is introduced as the first benchmark for evaluating chain-of-thought (CoT) reasoning by multimodal large language models in autonomous driving under adverse weather and complex scenes. The paper argues that existing benchmarks mostly cover benign conditions and grade only final answers, so they miss where multi-step driving reasoning fails. To close that gap, the authors built roughly 10,000 real-world images with 70,000 question-answer pairs, of which 5,406 carry complete, manually annotated CoT chains in which each reasoning step is an atomic unit with its own ground truth. On this benchmark, twelve state-of-the-art models all score below 60 percent average accuracy, which the authors read as evidence that safer, interpretable end-to-end driving still needs substantial progress.

What carries the argument

The load-bearing mechanism is the hierarchical CoT pipeline, a fixed five-stage decomposition from weather and road conditions, to key vehicles, to vulnerable road users, to traffic rules, to a final driving suggestion, with each stage annotated as an atomic ground-truth unit. Around this pipeline the benchmark supplies three visual-prompt levels (region bounding boxes, point cues, and text-level references) to reduce vision-encoder failure in adverse conditions, and four CoT quality metrics—SACS, SLPS, ORCS, and DJSS—that use an LLM evaluator to score step accuracy, logical progression, overall coherence, and decision justification on a 1–10 scale. These metrics are what turn the dataset from a VQA collection into a reasoning-process benchmark.

What would settle it

Re-score a random subset of the CoT outputs with independent human experts using the same 1–10 rubrics and compare the per-model rankings to the GPT-4o-based ones; if the rankings diverge on model ordering or on the CoT-vs-no-CoT gap, the headline conclusions are an artifact of the chosen judge rather than a property of the models.

Watch

Extended reading notes

Core claim

The paper's central claim is that fine-grained CoT evaluation under adverse conditions exposes systematic weaknesses in current multimodal driving models that final-answer benchmarks hide. Treating each intermediate reasoning step as an atomic labeled unit lets the benchmark measure not just whether a model reaches the right answer but whether it perceives, understands, and reasons correctly along the way. The reported results show that no evaluated model surpasses 60 percent accuracy, that some models hallucinate hazards or miss key vehicles under rain and light fog, and that hierarchical CoT prompting lifts scores substantially for models trained with rationale-augmented data. The authors conclude that AD2-Bench provides a standardized platform for improving reliable, interpretable autonomous driving systems.

Load-bearing premise

The paper's quality scores, including rankings and the conclusion that CoT helps, depend on GPT-4o's judgments of reasoning correctness, progression, coherence, and justification, and the paper does not show that these judgments agree with human experts.

Editorial extensions

If this is right

  • Any multimodal model proposed for autonomous driving should be tested on adverse-weather and complex-scene reasoning, because benign-condition benchmarks overstate readiness.
  • The sub-60 percent ceiling on current models implies that perception, relation understanding, and decision reasoning under rain, fog, snow, and night remain open problems.
  • Hierarchical CoT prompting can add roughly 15–20 points for some models, so training with intermediate supervision and rationale-augmented data is a promising axis for improvement.
  • Since average step correctness tends to exceed whole-chain coherence, developers can localize failures to specific reasoning stages rather than treating a wrong final answer as a single error.
  • Point-level and region-level visual prompts help models notice occluded or distant hazards, offering a practical interface for steering attention during evaluation or deployment.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the four CoT metrics are re-scored by human experts on a random sample, the paper's model rankings and its claim that CoT improves quality might shift; a public human-agreement study would make the benchmark's conclusions independent of a single judge.
  • The benchmark's atomic-step design could transfer to other safety-critical vision-language domains such as robot navigation, medical imaging, or industrial inspection, where reasoning process matters as much as final output.
  • The observed gap between individual step accuracy and overall coherence suggests a testable hypothesis: models trained with explicit intermediate-supervision losses should close that gap, and AD2-Bench could serve as the evaluation instrument.
  • Extending the benchmark to video or temporal chains would test whether models can maintain coherent reasoning over time, not just within a single frame.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. AD^2-Bench is a proposed benchmark for evaluating chain-of-thought (CoT) reasoning in multimodal large language models (MLLMs) for autonomous driving under adverse weather and complex traffic conditions. The paper describes a dataset of roughly 10k real-world images, 70k QA pairs, and over 5.4k CoT instances with atomic step-level ground truth; introduces region-, point-, and text-level visual prompts; proposes four CoT quality metrics (SACS, SLPS, ORCS, DJSS) based on an LLM evaluator; and reports an evaluation of 12 open-source MLLMs, concluding that models generally score below 60% and that hierarchical CoT prompting improves performance (e.g., +20.2 for InternVL3).

Significance. If the annotation quality and evaluation metrics are validated, AD^2-Bench would fill a genuine gap in the autonomous-driving vision-language benchmark landscape: it combines adverse-condition imagery, hierarchical CoT annotations, and multi-level visual prompts in a single resource. The atomic annotation design, the breadth of subtasks (33), the detailed qualitative failure analysis, and the systematic comparison across 12 models are clear strengths. However, the paper's central quantitative claims currently rest on an unvalidated LLM-as-judge protocol and on internally inconsistent dataset counts and annotation descriptions. These issues are load-bearing for the benchmark's value, so the significance is not yet fully established.

major comments (4)
  1. [§3.3, Eqs. (1)-(4)] The four proposed metrics are all defined through an LLM evaluation function Eval_*, and Appendix A.4 states that GPT-4o is used as the evaluator for similarity between predictions and ground truth. The paper provides no human-agreement study, no score distributions, no alternative judge, and no error bars for these 1-10 scores. Because Table 3 and Section 4.2 use these scores to support the headline claims (models below 60%, CoT gains such as +20.2 for InternVL3), the results may reflect GPT-4o's rating preferences rather than reasoning quality. Please validate the metrics on a sample against human judgments, report Pearson or Kendall correlations with human scores, provide per-step score distributions, and show that model rankings are stable when a second LLM judge (e.g., Gemini or a Qwen model) is used.
  2. [Abstract, §1 vs. §3.2, Appendix A.13] The number of full CoT instances is inconsistent: the abstract and Introduction say "over 5.4k", Section 1 gives "5,406", Section 3.2 says a "dedicated 5,046-image CoT test suite", and Appendix A.13 repeats "over 5,400". The paper also reports 70K QA pairs and 10K images without defining how these counts decompose into the CoT test suite, validation sets, and perception/relation splits. Please reconcile these numbers and provide an exact split table; this is necessary for reproducibility and for interpreting the per-model scores.
  3. [§3.1, §3.2, Appendix A.13] The claims of "purely manual" CoT annotations and "All VQA and CoT steps are manually annotated by domain experts" conflict with the annotation pipeline described in Section 3.1, where Gemini 2.5-Pro, GPT-4o, and Qwen 2.5-Max score the atomic annotations and provide detailed evaluations, and Section 3.2, where Gemini 2.5-Pro post-processes manual VQA responses and generates multiple-choice counterparts. Moreover, despite claims of cross-validation by three or more experts, no inter-annotator agreement is reported. Please quantify the human versus LLM contribution to each annotation step and report agreement statistics (e.g., Cohen's kappa or Fleiss' kappa) for a sample of the CoT instances.
  4. [Abstract, §1, Table 2, Table 3] The headline claim that models score "below 60% accuracy" is ambiguous because Table 3 reports w/ CoT scores above 60 for several models (e.g., InternVL3 63.04, InternVL2.5 61.64), while Table 2's Avg-S column is below 60. Section 4.2's statement that direct reasoning scores are "typically below 40% (Tab. 2)" is also inconsistent with Table 3's w/o CoT values (e.g., InternVL3 42.21, Qwen2.5-VL 41.73). Please define the exact aggregate metric and the experimental setting behind each headline number, and clarify whether Table 2's "Event Res." column is the same as Table 3's "w/ CoT" column.
minor comments (4)
  1. [Figures 2, 3, 5; throughout] There are several typos and naming inconsistencies: "dementions" in Figure 2, "Pesestrain2" in Figure 3, "Intervl2.5" and "GroudTruth" in Figure 5, and inconsistent use of "ADBench" versus "AD^2-Bench" across the text and figure captions.
  2. [Table 6, Appendix A.5] Table 6 uses the header "OCR-SER" while the text and Appendix A.5 define Character Error Rate (CER); please align the notation to avoid confusion.
  3. [§3, §4] The paper calls the benchmark bilingual (Section 3) but reports only English-language evaluation results; please specify the language of the evaluated prompts or clarify how the bilingual release is realized.
  4. [NeurIPS Checklist, §5] The checklist states that anonymous URLs are provided, but no URLs appear in the main text; please add a data/code availability statement with concrete links and license information so that reproducibility claims can be checked.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the benchmark's empirical claims rest on manual annotations and external LLM-based scoring, not on definitions or self-citations that presuppose the results.

full rationale

AD^2-Bench is an evaluation benchmark rather than a derivation, and its central claims (models score below 60%; hierarchical CoT improves performance) are empirical measurements. The CoT metrics in Section 3.3 (SACS, SLPS, ORCS, DJSS) are defined through an LLM evaluation function, and Appendix A.4 states that GPT-4o is used to grade similarity between predictions and ground truth; however, this is an external scoring procedure, not a fitted parameter or a definition of the target quantity in terms of itself. The ground-truth annotations are described as manually produced and LLM-validated (Section 3.1, Appendix A.2), but the evaluated models are twelve different open-source MLLMs, so the reported scores are not equal to the benchmark's inputs by construction. The absence of a human-agreement study for the GPT-4o-based metrics is a legitimate validity risk, but under the stipulated criteria that circularity requires a specific reduction such as Eq. X = Eq. Y by construction, a fitted parameter renamed as prediction, or a load-bearing self-citation chain, no such step is present. There are also no self-citations invoked to establish uniqueness or to justify the core contribution. Accordingly, no circularity step is identified.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The benchmark's claims rest on several unverified domain assumptions, chiefly that expert annotators plus LLM checks yield accurate ground truth, that GPT-4o scores are valid proxies for human reasoning-quality judgments, that the curated adverse-weather images are representative and legally reusable, and that the 12 mid-size open-source models represent state-of-the-art MLLMs.

assumptions (4)
  • domain assumption Domain expert manual annotation plus LLM cross-checking yields accurate ground truth for driving CoT steps.
    The benchmark's quality claim rests on the reliability of expert annotators and LLM evaluators; no inter-annotator agreement statistics are reported.
  • domain assumption GPT-4o scoring accurately measures correctness, logical progression, coherence, and decision justification of MLLM CoT outputs.
    The proposed metrics SACS/SLPS/ORCS/DJSS are defined via an LLM evaluator; the paper provides no calibration against human judgments, so the validity of these scores is assumed.
  • domain assumption Images sourced from CODA, ACDC, DAWN, and web resources are representative of adverse weather and complex scenes, and web images are appropriately licensed.
    The dataset coverage claim depends on the provenance and labeling of these external sources; the paper does not provide a detailed breakdown of image sources or licenses in the main text.
  • domain assumption The evaluated 7-8B parameter models are representative of state-of-the-art MLLMs for autonomous driving reasoning.
    The claim that SOTA models score below 60% is based on only 12 open-source models of similar size; larger or closed-source models are not evaluated.

how reviews work

0 comments
Cite this review

Pith. "Pith review of AD^2-Bench: A Hierarchical CoT Benchmark for MLLM in Autonomous Driving under Adverse Conditions." pith.science (2026). https://pith.science/paper/TGUTTEKZ

@misc{pith2026250609557,
  author       = {Pith},
  title        = {Pith review of: AD^2-Bench: A Hierarchical CoT Benchmark for MLLM in Autonomous Driving under Adverse Conditions},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/TGUTTEKZ}},
  note         = {Machine review of arXiv:2506.09557}
}
read the original abstract

Chain-of-Thought (CoT) reasoning has emerged as a powerful approach to enhance the structured, multi-step decision-making capabilities of Multi-Modal Large Models (MLLMs), is particularly crucial for autonomous driving with adverse weather conditions and complex traffic environments. However, existing benchmarks have largely overlooked the need for rigorous evaluation of CoT processes in these specific and challenging scenarios. To address this critical gap, we introduce AD^2-Bench, the first Chain-of-Thought benchmark specifically designed for autonomous driving with adverse weather and complex scenes. AD^2-Bench is meticulously constructed to fulfill three key criteria: comprehensive data coverage across diverse adverse environments, fine-grained annotations that support multi-step reasoning, and a dedicated evaluation framework tailored for assessing CoT performance. The core contribution of AD^2-Bench is its extensive collection of over 5.4k high-quality, manually annotated CoT instances. Each intermediate reasoning step in these annotations is treated as an atomic unit with explicit ground truth, enabling unprecedented fine-grained analysis of MLLMs' inferential processes under text-level, point-level, and region-level visual prompts. Our comprehensive evaluation of state-of-the-art MLLMs on AD^2-Bench reveals accuracy below 60%, highlighting the benchmark's difficulty and the need to advance robust, interpretable end-to-end autonomous driving systems. AD^2-Bench thus provides a standardized evaluation platform, driving research forward by improving MLLMs' reasoning in autonomous driving, making it an invaluable resource.

Figures

Figures reproduced from arXiv: 2506.09557 by the authors.

Figure 1
Figure 1. Adverse conditions include adverse weather (up) and complex scenarios (down). [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. (a) Task Categories. Our benchmark spans 4 key dementions and 33 subtasks highly [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. To mitigate the challenges of vision encoder in adverse scenarios, we evaluate MLLM’s [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (8 more)
Figure 4
Figure 4. Figure 4: Hierarchical Chain-of-Thought (CoT) framework in AD [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]
Figure 5
Figure 5. Figure 5: Bad case: (a) In light fog conditions, the ambiguity caused by the non-dense fog leads [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: The visualization of Chain of Thought for MLLM. By breaking down unified scene [PITH_FULL_IMAGE:figures/full_fig_p023_6.png]
Figure 7
Figure 7. Figure 7: The visualization of Qwen2.5-VL and InternVL3’s perception and reasoning under adverse [PITH_FULL_IMAGE:figures/full_fig_p024_7.png]
Figure 8
Figure 8. Figure 8: The visualization of perception and reasoning of Qwen2.5-VL and InternVL3 in adverse [PITH_FULL_IMAGE:figures/full_fig_p025_8.png]
Figure 9
Figure 9. Figure 9: The visualization of changes in perception and reasoning under adverse weather conditions [PITH_FULL_IMAGE:figures/full_fig_p026_9.png]
Figure 10
Figure 10. Figure 10: The visualization of Basic Perception Tasks of Multimodal large language Models [PITH_FULL_IMAGE:figures/full_fig_p027_10.png]
Figure 11
Figure 11. Figure 11: The visualization of Advanced Perception Tasks of Multimodal large language Models [PITH_FULL_IMAGE:figures/full_fig_p028_11.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Deferred Exposure of Future Trajectories for Verifiable Reasoning in Autonomous Driving VLMs

    cs.AI 2026-08 conditional novelty 7.0 of 10

    Hiding future trajectory information until after a driving model forms its decision reduces rationalization and improves verifiable autonomous-driving reasoning in the proposed AD-MCQ and DEFT-RLVR framework.

Reference graph

Works this paper leans on

78 extracted references · 37 canonical work pages · cited by 1 Pith paper

  1. [1]

    Phi-3 technical report: A highly capable language model locally on your phone

    Marah Abdin, Sam Ade Jacobs, Ammar Ahmad Awan, Jyoti Aneja, Ahmed Awadallah, Hany Awadalla, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Harkirat Behl, et al. Phi-3 technical report: A highly capable language model locally on your phone. arXiv preprint arXiv:2404.14219, 2024

    arXiv 2024

  2. [2]

    A-bdd: Leveraging data augmentations for safe autonomous driving in adverse weather and lighting

    Felix Assion, Florens Gressner, Nitin Augustine, Jona Klemenc, Ahmed Hammam, Alexandre Krattinger, Holger Trittenbach, Anja Philippsen, and Sascha Riemer. A-bdd: Leveraging data augmentations for safe autonomous driving in adverse weather and lighting. arXiv preprint arXiv:2408.06071, 2024

    arXiv 2024

  3. [3]

    Qwen-vl: A versatile vision-language model for understanding, localization, text reading, and beyond

    Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A versatile vision-language model for understanding, localization, text reading, and beyond. arXiv preprint arXiv:2308.12966, 2023

    arXiv 2023

  4. [4]

    Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al. Qwen2. 5-vl technical report. arXiv preprint arXiv:2502.13923, 2025

    arXiv 2025

  5. [5]

    Rt-2: Vision-language-action models transfer web knowledge to robotic control

    Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Xi Chen, Krzysztof Choromanski, Tianli Ding, Danny Driess, Avinava Dubey, Chelsea Finn, et al. Rt-2: Vision-language-action models transfer web knowledge to robotic control. arXiv preprint arXiv:2307.15818, 2023

    arXiv 2023

  6. [6]

    nuscenes: A multimodal dataset for autonomous driving

    Holger Caesar, Varun Bankiti, Alex H Lang, Sourabh V ora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom. nuscenes: A multimodal dataset for autonomous driving. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11621–11631, 2020

    2020

  7. [7]

    Argoverse: 3d tracking and forecasting with rich maps

    Ming-Fang Chang, John Lambert, Patsorn Sangkloy, Jagjeet Singh, Slawomir Bak, Andrew Hartnett, De Wang, Peter Carr, Simon Lucey, Deva Ramanan, et al. Argoverse: 3d tracking and forecasting with rich maps. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 8748–8757, 2019

    work page 2019

  8. [8]

    Spa- tialvlm: Endowing vision-language models with spatial reasoning capabilities

    Boyuan Chen, Zhuo Xu, Sean Kirmani, Brain Ichter, Dorsa Sadigh, Leonidas Guibas, and Fei Xia. Spa- tialvlm: Endowing vision-language models with spatial reasoning capabilities. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14455–14465, 2024

    work page 2024

Show all 78 references
  1. [9]

    Occprophet: Pushing efficiency frontier of camera-only 4d occupancy forecasting with observer-forecaster-refiner framework

    Junliang Chen, Huaiyuan Xu, Yi Wang, and Lap-Pui Chau. Occprophet: Pushing efficiency frontier of camera-only 4d occupancy forecasting with observer-forecaster-refiner framework. arXiv preprint arXiv:2502.15180, 2025

    2025 arXiv

  2. [10]

    Automated evaluation of large vision-language models on self-driving corner cases

    Kai Chen, Yanze Li, Wenhua Zhang, Yanxin Liu, Pengxiang Li, Ruiyuan Gao, Lanqing Hong, Meng Tian, Xinhai Zhao, Zhenguo Li, et al. Automated evaluation of large vision-language models on self-driving corner cases. In 2025 IEEE/CVF Winter Conference on Applications of Computer V...

    2025

  3. [11]

    Janus-pro: Unified multimodal understanding and generation with data and model scaling

    Xiaokang Chen, Zhiyu Wu, Xingchao Liu, Zizheng Pan, Wen Liu, Zhenda Xie, Xingkai Yu, and Chong Ruan. Janus-pro: Unified multimodal understanding and generation with data and model scaling. arXiv preprint arXiv:2501.17811, 2025

    2025 arXiv

  4. [12]

    Drivinggpt: Unifying driving world modeling and planning with multi-modal autoregressive transformers

    Yuntao Chen, Yuqi Wang, and Zhaoxiang Zhang. Drivinggpt: Unifying driving world modeling and planning with multi-modal autoregressive transformers. arXiv preprint arXiv:2412.18607, 2024

    2024 arXiv

  5. [13]

    Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling

    Zhe Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Erfei Cui, Jinguo Zhu, Shenglong Ye, Hao Tian, Zhaoyang Liu, et al. Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling. arXiv preprint arXiv:2412.05271, 2024

    2024 arXiv

  6. [14]

    Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks

    Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. In Proceedings of the IEEE/CVF conference on computer vis...

    2024

  7. [15]

    Vision-based large-scale 3d semantic mapping for autonomous driving applications

    Qing Cheng, Niclas Zeller, and Daniel Cremers. Vision-based large-scale 3d semantic mapping for autonomous driving applications. In 2022 International Conference on Robotics and Automation (ICRA), pages 9235–9242. IEEE, 2022. 10

    2022

  8. [16]

    The cityscapes dataset for semantic urban scene understanding

    Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding. In Proceedings of the IEEE conference on computer vision and pattern reco...

    2016

  9. [17]

    Drivellm: Charting the path toward full autonomous driving with large language models

    Yaodong Cui, Shucheng Huang, Jiaming Zhong, Zhenan Liu, Yutong Wang, Chen Sun, Bai Li, Xiao Wang, and Amir Khajepour. Drivellm: Charting the path toward full autonomous driving with large language models. IEEE Transactions on Intelligent Vehicles, 9(1):1450–1464, 2024

    2024

  10. [18]

    Talk2car: Predicting physical trajectories for natural language commands

    Thierry Deruyttere, Dusan Grujicic, Matthew B Blaschko, and Marie-Francine Moens. Talk2car: Predicting physical trajectories for natural language commands. Ieee Access, 10:123809–123834, 2022

    2022

  11. [19]

    Insight-v: Exploring long-chain visual reasoning with multimodal large language models

    Yuhao Dong, Zuyan Liu, Hai-Long Sun, Jingkang Yang, Winston Hu, Yongming Rao, and Ziwei Liu. Insight-v: Exploring long-chain visual reasoning with multimodal large language models. arXiv preprint arXiv:2411.14432, 2024

    2024 arXiv

  12. [20]

    Drive like a human: Rethinking autonomous driving with large language models

    Daocheng Fu, Xin Li, Licheng Wen, Min Dou, Pinlong Cai, Botian Shi, and Yu Qiao. Drive like a human: Rethinking autonomous driving with large language models. In IEEE/CVF Winter Conference on Applications of Computer Vision, pages 910–919, 2024

    2024

  13. [21]

    Drivemllm: A benchmark for spatial understanding with multimodal large language models in autonomous driving

    Xianda Guo, Ruijun Zhang, Yiqun Duan, Yuhang He, Chenming Zhang, Shuai Liu, and Long Chen. Drivemllm: A benchmark for spatial understanding with multimodal large language models in autonomous driving. arXiv preprint arXiv:2411.13112, 2024

    2024 arXiv

  14. [22]

    A benchmark for anomaly segmentation

    Dan Hendrycks, Steven Basart, Mantas Mazeika, Mohammadreza Mostajabi, Jacob Steinhardt, and Dawn Song. A benchmark for anomaly segmentation. arXiv preprint arXiv:1911.11132, 2(7), 2019

    1911 arXiv

  15. [23]

    Cogagent: A visual language model for gui agents

    Wenyi Hong, Weihan Wang, Qingsong Lv, Jiazheng Xu, Wenmeng Yu, Junhui Ji, Yan Wang, Zihan Wang, Yuxiao Dong, Ming Ding, et al. Cogagent: A visual language model for gui agents. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14281–14290, 2024

    2024

  16. [24]

    Nuscenes-mqa: Integrated evaluation of captions and qa for autonomous driving datasets using markup annotations

    Yuichi Inoue, Yuki Yada, Kotaro Tanahashi, and Yu Yamaguchi. Nuscenes-mqa: Integrated evaluation of captions and qa for autonomous driving datasets using markup annotations. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 930–938, 2024

    2024

  17. [25]

    Dawn: vehicle detection in adverse weather nature dataset

    Mourad A Kenk and Mahmoud Hassaballah. Dawn: vehicle detection in adverse weather nature dataset. arXiv preprint arXiv:2008.05402, 2020

    2008 arXiv

  18. [26]

    Textual explanations for self-driving vehicles

    Jinkyu Kim, Anna Rohrbach, Trevor Darrell, John Canny, and Zeynep Akata. Textual explanations for self-driving vehicles. In ECCV, 2018

    2018

  19. [27]

    Llava-next-interleave: Tackling multi-image, video, and 3d in large multimodal models

    Feng Li, Renrui Zhang, Hao Zhang, Yuanhan Zhang, Bo Li, Wei Li, Zejun Ma, and Chunyuan Li. Llava-next-interleave: Tackling multi-image, video, and 3d in large multimodal models. arXiv preprint arXiv:2407.07895, 2024

    2024 arXiv

  20. [28]

    BLIP-2: Bootstrapping language-image pre- training with frozen image encoders and large language models

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. BLIP-2: Bootstrapping language-image pre- training with frozen image encoders and large language models. In International Conference on Machine Learning (ICML), 2023

    2023

  21. [29]

    Coda: A real-world road corner case dataset for object detection in autonomous driving

    Kaican Li, Kai Chen, Haoyu Wang, Lanqing Hong, Chaoqiang Ye, Jianhua Han, Yukuai Chen, Wei Zhang, Chunjing Xu, Dit-Yan Yeung, et al. Coda: A real-world road corner case dataset for object detection in autonomous driving. In European Conference on Computer Vision, pages 406–423...

    2022

  22. [30]

    V oxformer: Sparse voxel transformer for camera-based 3d semantic scene completion

    Yiming Li, Zhiding Yu, Christopher Choy, Chaowei Xiao, Jose M Alvarez, Sanja Fidler, Chen Feng, and Anima Anandkumar. V oxformer: Sparse voxel transformer for camera-based 3d semantic scene completion. In Proceedings of the IEEE/CVF conference on computer vision and pattern re...

    2023

  23. [31]

    Kitti-360: A novel dataset and benchmarks for urban scene understanding in 2d and 3d

    Yiyi Liao, Jun Xie, and Andreas Geiger. Kitti-360: A novel dataset and benchmarks for urban scene understanding in 2d and 3d. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(3):3292– 3310, 2022

    2022

  24. [32]

    Coarse correspondences elicit 3d spacetime understanding in multimodal language model

    Benlin Liu, Yuhao Dong, Yiqin Wang, Yongming Rao, Yansong Tang, Wei-Chiu Ma, and Ranjay Krishna. Coarse correspondences elicit 3d spacetime understanding in multimodal language model. arXiv preprint arXiv:2408.00754, 2024

    2024 arXiv

  25. [33]

    Improved baselines with visual instruction tuning

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 26296–26306, 2024. 11

    2024

  26. [34]

    Llava-next: Improved reasoning, ocr, and world knowledge, January 2024

    Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Llava-next: Improved reasoning, ocr, and world knowledge, January 2024

    2024

  27. [35]

    Visual instruction tuning

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In Advances in Neural Information Processing Systems, pages 34892–34916, 2023

    2023

  28. [36]

    Chain-of-spot: Interactive reasoning improves large vision-language models

    Zuyan Liu, Yuhao Dong, Yongming Rao, Jie Zhou, and Jiwen Lu. Chain-of-spot: Interactive reasoning improves large vision-language models. arXiv preprint arXiv:2403.12966, 2024

    2024 arXiv

  29. [37]

    Dolphins: Multimodal language model for driving

    Yingzi Ma, Yulong Cao, Jiachen Sun, Marco Pavone, and Chaowei Xiao. Dolphins: Multimodal language model for driving. arXiv preprint arXiv:2312.00438, 2023

    2023 arXiv

  30. [38]

    Drama: Joint risk localization and captioning in driving

    Srikanth Malla, Chiho Choi, Isht Dwivedi, Joon Hee Choi, and Jiachen Li. Drama: Joint risk localization and captioning in driving. In WACV, 2023

    2023

  31. [39]

    Lingoqa: Visual question answering for autonomous driving

    Ana-Maria Marcu, Long Chen, Jan Hünermann, Alice Karnsund, Benoit Hanotte, Prajwal Chidananda, Saurabh Nair, Vijay Badrinarayanan, Alex Kendall, Jamie Shotton, et al. Lingoqa: Visual question answering for autonomous driving. In European Conference on Computer Vision, pages 25...

    2024

  32. [40]

    Nuscenes-qa: A multi-modal visual question answering benchmark for autonomous driving scenario

    Tianwen Qian, Jingjing Chen, Linhai Zhuo, Yang Jiao, and Yu-Gang Jiang. Nuscenes-qa: A multi-modal visual question answering benchmark for autonomous driving scenario. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 4542–4550, 2024

    2024

  33. [41]

    Acdc: The adverse conditions dataset with corre- spondences for semantic driving scene understanding

    Christos Sakaridis, Dengxin Dai, and Luc Van Gool. Acdc: The adverse conditions dataset with corre- spondences for semantic driving scene understanding. In Proceedings of the IEEE/CVF international conference on computer vision, pages 10765–10775, 2021

    2021

  34. [42]

    Drivelm: Driving with graph visual question answering

    Chonghao Sima, Katrin Renz, Kashyap Chitta, Li Chen, Hanxue Zhang, Chengen Xie, Jens Beißwenger, Ping Luo, Andreas Geiger, and Hongyang Li. Drivelm: Driving with graph visual question answering. In European Conference on Computer Vision, pages 256–274, 2024

    2024

  35. [43]

    Open-world object manipulation using pre-trained vision-language models

    Austin Stone, Ted Xiao, Yao Lu, Keerthana Gopalakrishnan, Kuang-Huei Lee, Quan Vuong, Paul Wohlhart, Sean Kirmani, Brianna Zitkovich, Fei Xia, et al. Open-world object manipulation using pre-trained vision-language models. arXiv preprint arXiv:2303.00905, 2023

    2023 arXiv

  36. [44]

    Scalability in perception for autonomous driving: Waymo open dataset

    Pei Sun, Henrik Kretzschmar, Xerxes Dotiwalla, Aurelien Chouard, Vijaysai Patnaik, Paul Tsui, James Guo, Yin Zhou, Yuning Chai, Benjamin Caine, et al. Scalability in perception for autonomous driving: Waymo open dataset. In Proceedings of the IEEE/CVF conference on computer vi...

    2020

  37. [45]

    Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality

    The Vicuna Team. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality. https: //lmsys.org/blog/2023-03-30-vicuna/ . Accessed: 2023

    2023

  38. [46]

    Nuscenes- spatialqa: A spatial understanding and reasoning benchmark for vision-language models in autonomous driving

    Kexin Tian, Jingrui Mao, Yunlong Zhang, Jiwan Jiang, Yang Zhou, and Zhengzhong Tu. Nuscenes- spatialqa: A spatial understanding and reasoning benchmark for vision-language models in autonomous driving. arXiv preprint arXiv:2504.03164, 2025

    2025 arXiv

  39. [47]

    Drivevlm: The convergence of autonomous driving and large vision-language models

    Xiaoyu Tian, Junru Gu, Bailin Li, Yicheng Liu, Yang Wang, Zhiyong Zhao, Kun Zhan, Peng Jia, Xianpeng Lang, and Hang Zhao. Drivevlm: The convergence of autonomous driving and large vision-language models. arXiv preprint arXiv:2402.12289, 2024

    2024 arXiv

  40. [48]

    Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution

    Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution. arXiv preprint arXiv:2409.12191, 2024

    2024 arXiv

  41. [49]

    Chain-of-thought prompting elicits reasoning in large language models

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824–24837, 2022

    2022

  42. [50]

    Dilu: A knowledge-driven approach to autonomous driving with large language models

    Licheng Wen, Daocheng Fu, Xin Li, Xinyu Cai, Tao Ma, Pinlong Cai, Min Dou, Botian Shi, Liang He, and Yu Qiao. Dilu: A knowledge-driven approach to autonomous driving with large language models. In International Conference on Learning Representations, 2024

    2024

  43. [51]

    Language prompt for autonomous driving

    Dongming Wu, Wencheng Han, Tiancai Wang, Yingfei Liu, Xiangyu Zhang, and Jianbing Shen. Language prompt for autonomous driving. arXiv preprint arXiv:2309.04379, 2023

    2023 arXiv

  44. [52]

    Are vlms ready for autonomous driving? an empirical study from the reliability, data, and metric perspectives

    Shaoyuan Xie, Lingdong Kong, Yuhao Dong, Chonghao Sima, Wenwei Zhang, Qi Alfred Chen, Ziwei Liu, and Liang Pan. Are vlms ready for autonomous driving? an empirical study from the reliability, data, and metric perspectives. arXiv preprint arXiv:2501.04003, 2025. 12

    2025 arXiv

  45. [53]

    Autotrust: Benchmarking trustworthiness in large vision language models for autonomous driving

    Shuo Xing, Hongyuan Hua, Xiangbo Gao, Shenzhe Zhu, Renjie Li, Kexin Tian, Xiaopeng Li, Heng Huang, Tianbao Yang, Zhangyang Wang, et al. Autotrust: Benchmarking trustworthiness in large vision language models for autonomous driving. arXiv preprint arXiv:2412.15206, 2024

    2024

  46. [54]

    Vlm-ad: End-to-end autonomous driving through vision-language model supervision

    Yi Xu, Yuxin Hu, Zaiwei Zhang, Gregory P Meyer, Siva Karthik Mustikovela, Siddhartha Srinivasa, Eric M Wolff, and Xin Huang. Vlm-ad: End-to-end autonomous driving through vision-language model supervision. arXiv preprint arXiv:2412.14446, 2024

    2024 arXiv

  47. [55]

    Explainable object-induced action decision for autonomous vehicles

    Yiran Xu, Xiaoyin Yang, Lihang Gong, Hsuan-Chu Lin, Tz-Ying Wu, Yunsheng Li, and Nuno Vasconcelos. Explainable object-induced action decision for autonomous vehicles. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9523–9532, 2020

    2020

  48. [56]

    Wong, Zhenguo Li, and Hengshuang Zhao

    Zhenhua Xu, Yujia Zhang, Enze Xie, Zhen Zhao, Yong Guo, Kwan-Yee K. Wong, Zhenguo Li, and Hengshuang Zhao. Drivegpt4: Interpretable end-to-end autonomous driving via large language model. arXiv preprint arXiv:2310.01412, 2023

    arXiv

  49. [57]

    Don’t take things out of context: Attention intervention for enhancing chain-of-thought reasoning in large language models

    Shaotian Yan, Chen Shen, Wenxiao Wang, Liang Xie, Junjie Liu, and Jieping Ye. Don’t take things out of context: Attention intervention for enhancing chain-of-thought reasoning in large language models. arXiv preprint arXiv:2503.11154, 2025

    2025 arXiv

  50. [58]

    An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. 5 technical report. arXiv preprint arXiv:2412.15115, 2024

    2024 arXiv

  51. [59]

    Octopus: Embodied vision-language programmer from environmental feedback

    Jingkang Yang, Yuhao Dong, Shuai Liu, Bo Li, Ziyue Wang, Haoran Tan, Chencheng Jiang, Jiamu Kang, Yuanhan Zhang, Kaiyang Zhou, et al. Octopus: Embodied vision-language programmer from environmental feedback. In European Conference on Computer Vision, pages 20–38. Springer, 2025

    2025

  52. [60]

    Minicpm-v: A gpt-4v level mllm on your phone

    Yuan Yao, Tianyu Yu, Ao Zhang, Chongyi Wang, Junbo Cui, Hongji Zhu, Tianchi Cai, Haoyu Li, Weilin Zhao, Zhihui He, et al. Minicpm-v: A gpt-4v level mllm on your phone. arXiv preprint arXiv:2408.01800, 2024

    2024 arXiv

  53. [61]

    Bdd100k: A diverse driving dataset for heterogeneous multitask learning

    Fisher Yu, Haofeng Chen, Xin Wang, Wenqi Xian, Yingying Chen, Fangchen Liu, Vashisht Madhavan, and Trevor Darrell. Bdd100k: A diverse driving dataset for heterogeneous multitask learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pag...

    2020

  54. [62]

    completion

    Jinguo Zhu, Weiyun Wang, Zhe Chen, Zhaoyang Liu, Shenglong Ye, Lixin Gu, Yuchen Duan, Hao Tian, Weijie Su, Jie Shao, et al. Internvl3: Exploring advanced training and test-time recipes for open-source multimodal models. arXiv preprint arXiv:2504.10479, 2025. 13 A Appendix A.1 ...

    2025 arXiv

  55. [63]

    Guidelines: • The answer NA means that the abstract and introduction do not include the claims made in the paper

    Claims Question: Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope? Answer: [Yes] Justification: The abstract and introduction provide a comprehensive overview of the background and the motivation of this study, eff...

  56. [64]

    Limitations

    Limitations Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes] Justification: The paper provides the limitation discussion in Appendix A.13. Guidelines: • The answer NA means that the paper has no limitation while the answer No...

  57. [65]

    Moreover, it covers detailed ablative studies and qualitative Studies, ensuring completeness and accuracy in the theoretical presentation

    Theory assumptions and proofs Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? 29 Answer: [Yes] Justification: The method in this article includes theoretical formulation. Moreover, it covers detailed...

  58. [66]

    Guidelines: • The answer NA means that the paper does not include experiments

    Experimental result reproducibility Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and da...

  59. [67]

    After the acceptance, we will make our codes publicly available

    Open access to data and code Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material? Answer: [Yes] Justification: We provide the anonymous UR...

  60. [68]

    Guidelines: • The answer NA means that the paper does not include experiments

    Experimental setting/details Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results? Answer: [Yes] Justification: We give the experimental sett...

  61. [69]

    In addition, we do not have enough compute resources to report error bars for each experiment

    Experiment statistical significance Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments? Answer: [No] Justification: Generally, papers in our field do not include error...

  62. [70]

    Guidelines: • The answer NA means that the paper does not include experiments

    Experiments compute resources Question: For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments? Answer: [Yes] Justification: We give the information...

  63. [71]

    Guidelines: • The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics

    Code of ethics Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines? Answer: [Yes] Justification: After carefully reviewing the referenced document, we certify that our research...

  64. [72]

    This study only serves as a evaluation benchmark, which is not directly applicable to practical scenarios until now

    Broader impacts Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed? Answer: [NA] 32 Justification: The paper primarily focuses on vision question answer evaluation benchmark using publicly available dat...

  65. [73]

    These datasets have been extensively used in the computer vision community and have undergone comprehensive safety risk assessments

    Safeguards Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [NA] Justification: The proposed be...

  66. [74]

    We also mark that our evaluation codes are based on the ms-swift (version 3.4.1) library

    Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected? Answer: [Yes] Justification: In the paper, we...

  67. [75]

    After completing the review process, we will make the code and data publicly available to the community

    New assets Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets? Answer: [Yes] Justification: We have provided the URLs of the code and data, along with detailed usage instructions in our paper. After completin...

  68. [76]

    Guidelines: • The answer NA means that the paper does not involve crowdsourcing nor research with human subjects

    Crowdsourcing and research with human subjects Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)? A...

  69. [77]

    All experiments were conducted using codes and GPU servers

    Institutional review board (IRB) approvals or equivalent for research with human subjects 34 Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approv...

  70. [78]

    Declaration of LLM usage Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the ...

    2025

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.