Pith. sign in

REVIEW 4 major objections 4 minor 5 cited by

WiseAD: Knowledge Augmented End-to-End Autonomous Driving with Vision-Language Model

T0 review · 4 major / 4 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Adding driving knowledge to a vision-language model cuts critical accidents in closed-loop driving, raising the driving score by 11.9%.

desk verdict Plausible recipe paper, but the knowledge-augmentation claim is confounded by an unaccounted inference-time prompt; needs controlled ablations and data release. read the letter →

arxiv 2412.09951 v2 pith:KJHBQE6H submitted 2024-12-13 cs.CV

Songyan Zhang , Wenhui Huang , Zihui Gao , Hao Chen , Chen Lv This is my paper · ORCID
classification cs.CV
keywords vision-languagemodelautonomousdrivingclosed-loopevaluationknowledgetrajectoryplanningCARLAsimulatorjointtrainingzero-shot
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

WiseAD is a vision-language model for autonomous driving that is jointly trained on both driving knowledge question-answer data and trajectory planning data. The paper claims that expanding the depth and breadth of this fundamental driving knowledge—covering scene understanding, risk analysis, action justification, and suggestions—consistently improves closed-loop driving performance in the CARLA simulator. Compared with training only on the CARLA planning data, the knowledge-augmented joint training improves the driving score by 11.9% and route completion by 12.4%, with notably fewer critical accidents such as collisions and red-light violations. The authors also report state-of-the-art zero-shot driving performance and superior knowledge evaluation results over several larger VLMs.

What carries the argument

The key machinery is joint learning on a mixed dataset of driving knowledge QA pairs (LingoQA, DRAMA) and trajectory planning data from CARLA, in roughly equal proportions, on top of a pre-trained MobileVLM backbone with a frozen CLIP ViT encoder and a learnable projector. The trajectory is represented textually as a sequence of waypoints, and an attention-prefix prompt at inference explicitly cues the model to apply its learned knowledge. This mechanism allows the model to learn both 'how' to plan (imitation of the autopilot) and 'why' (the reasoning behind driving decisions), and the authors argue that this rationale-aligned training is what reduces accidents.

What would settle it

A controlled experiment that fixes the inference prompt, the total training data volume, and the dataset composition, while varying only whether the QA pairs contain semantically meaningful driving knowledge versus shuffled or templated non-driving text, would settle whether the gains are knowledge-driven. If a model trained on non-driving QA pairs plus the same prompt achieves the same driving score and accident reduction, the knowledge claim fails.

Watch

Extended reading notes

Core claim

The central claim is that injecting explicit, structured driving knowledge into a vision-language model through joint training with planning data makes the model a safer and more competent end-to-end driver. The discovery is not just that a VLM can answer driving questions, but that the learned knowledge transfers to the trajectory planning head: as the authors add knowledge domains (LingoQA for reasoning and perception, then DRAMA suggestions, then DRAMA risk analysis), the CARLA closed-loop driving score and route completion rise monotonically, while critical accident counts drop. This is evidenced by Table 2, where the final configuration reaches a 69.88 driving score and 93.79 route completion, up from 62.46 and 83.47 for the Carla-only baseline. The paper further shows that a simple attention-prefix prompt, 'Pay attention to your surroundings and do not break traffic rules,' is a key trigger for leveraging this knowledge, contributing a 3-point driving-score gain (Table 6).

Load-bearing premise

That the observed closed-loop driving improvements come from the semantic content of the added driving knowledge, rather than from confounds such as the inference-time attention prompt, increased data diversity, or the regularization effect of joint training.

Editorial extensions

If this is right

  • If the claim holds, knowledge-augmented joint training becomes a simple, scalable recipe for improving closed-loop driving safety in VLM-based planners, without changing the network architecture.
  • The monotonic improvement with added knowledge breadth suggests a positive scaling law: adding more diverse driving knowledge domains should further reduce accidents and improve route completion.
  • The attention-prefix prompt result implies that textual prompting can act as a switch to activate learned knowledge, which could be a general mechanism for steering VLM behavior in deployment.
  • The zero-shot generalization to BDD-X, DriveLM, and HAD indicates that the learned knowledge is transferable across datasets and task formats, supporting the idea that the driver is not overfitting to a single simulator.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper's causal attribution is correlational: joint training changes several factors at once (data diversity, instruction-following, and the inference prompt), so the observed gains might partly come from better instruction following or regularization, not from the semantic content of driving knowledge. A controlled experiment that holds the prompt constant and varies only the knowledge content
  • A testable extension is to measure whether the knowledge transfer holds under domain shift beyond CARLA, e.g., in a different simulator or in real-world closed-loop settings, where the gap between simulation and reality could be large.
  • The attention-prefix ablation suggests that the prompt itself contributes a large portion of the driving-score gain; an analysis of what the prompt changes internally (e.g., attention maps or hidden states) could reveal whether it actually retrieves risk knowledge or merely biases output formatting.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper proposes WiseAD, a vision-language model for end-to-end autonomous driving that is jointly trained on CARLA trajectory data and driving-knowledge QA datasets (LingoQA and DRAMA). The model outputs textual waypoints for closed-loop control and is evaluated on the CARLA Town05 benchmark plus in-domain and out-of-domain knowledge QA sets (BDDX, DriveLM, HAD). The central claims are that expanding driving knowledge improves closed-loop driving, with an 11.9% driving-score gain and a 12.4% route-completion gain over a Carla-only baseline, while also reducing critical accidents, and that the model achieves state-of-the-art zero-shot closed-loop performance.

Significance. The question investigated is meaningful: whether explicit driving-knowledge training can improve closed-loop trajectory planning in a vision-language model. The paper has real strengths: it uses external benchmarks (BDDX, DriveLM, HAD) for knowledge evaluation, which reduces circularity; it includes closed-loop evaluation in CARLA with standard metrics; and it reports a practical joint-training recipe that avoids the catastrophic forgetting observed in sequential fine-tuning. The main trend in Table 2—that adding DRAMA suggestion/risk data improves driving and reduces accidents—is plausible and worth pursuing. However, the headline causal claim is not yet established because the reported comparisons conflate the knowledge intervention with an inference-time prompt and with increased auxiliary training data.

major comments (4)
  1. [Sec. 4.4, Table 2 and Sec. 4.6, Table 6] The central causal claim is confounded by the inference-time attention prefix. The headline comparison is between a Carla-only baseline (DS 62.46, RC 83.47) and the full knowledge-augmented model (DS 69.88, RC 93.79), but the paper never states whether the baseline was evaluated with or without the prefix introduced in Sec. 3.2. Table 6 shows that removing the prefix from the full model degrades DS from 69.88 to 66.89 and RC from 93.79 to 85.35. If the baseline was evaluated without the prefix, the residual knowledge-only contribution is +4.43 DS and +1.88 RC, not the advertised 11.9% and 12.4%. The authors must report the Carla-only baseline under the same prompt condition and, ideally, add a full-model row without the prefix to Table 2.
  2. [Sec. 4.4, Table 2] The joint-training design changes multiple factors at once: the addition of LingoQA and DRAMA QA data, the data-mixing ratio, and the auxiliary task losses. There is no control condition that isolates semantic driving knowledge from generic multi-task regularization or instruction following. A control trained on an equal-size non-driving QA dataset, or on LingoQA/DRAMA with shuffled answers, is needed to support the abstract's attribution of driving improvements to 'fundamental driving knowledge' rather than to increased data diversity or auxiliary-task regularization.
  3. [Sec. 4.4, Table 2] The accident counts are non-monotonic and no uncertainty estimates are given. Adding LingoQA alone increases red-light infractions from 2.60 to 3.79 and vehicle collisions from 2.35 to 5.60; the authors attribute this to extended driving distances, but no per-route exposure measure or variance is reported. Without multiple random seeds or confidence intervals, the claim of a 'significant drop in all critical driving accidents' in the final row is not statistically supported.
  4. [Sec. 4.5, Table 5] The closed-loop comparison against TransFuser, NEAT, Roach, ST-P3, and VAD is uncontrolled: these methods use different input views (multiview or BEV), different training data regimes, and different model architectures. The 'state-of-the-art' and 'best zero-shot' claims therefore cannot be established from Table 5. A matched comparison—same routes, same sensor input configuration, and comparable training-data budget—or an explicit statement of the evaluation protocol and its differences is required.
minor comments (4)
  1. [Section heading] The heading 'Methodolgy' in Section 3 is a typo and should read 'Methodology'.
  2. [Table 4 and Table 5] Column headers such as 'Route compl' and 'Infrac.' should be spelled out or defined in the caption, and the units of infraction counts should be clarified.
  3. [Sec. 3.2] The dataset name is written inconsistently as 'BDDX' and 'BDD-X'; please choose one form and use it throughout.
  4. [References] References [21] and [22] appear to cite the same arXiv paper; please check and remove the duplicate.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the closed-loop and knowledge results are empirical comparisons against external benchmarks, and the attention-prompt concern is an attribution confound rather than a constructional circularity.

full rationale

WiseAD's central claims are empirical rather than derived: closed-loop driving performance is measured on CARLA Town05 against external leaderboard-style metrics (DS, RC, IS), and knowledge acquisition is measured on external or held-out QA datasets (LingoQA validation, BDDX, DriveLM, HAD). No equation in the paper defines a predicted quantity in terms of a fitted input, and no fitted parameter is renamed as a prediction. The joint-training comparisons in Table 2 vary multiple factors (knowledge data, data diversity, task mixture), but this is a confound in causal attribution, not a circular reduction. The closest self-referential move is the attention-prefix discussion: Sec. 3.2 calls the prefix 'a trigger to leverage the learned knowledge' and Sec. 4.6 interprets the prompt-ablation drop as validating knowledge alignment. That interpretation is stronger than the ablation alone supports, because the drop could also reflect generic instruction-following or regularization, but the measured outcome (routing completion and driving score) is not defined in terms of 'driving knowledge' and no derivation equates the prompt effect with the knowledge-construct by construction. The paper also does not rely on load-bearing self-citations or imported uniqueness theorems. Accordingly, the paper is self-contained with respect to circularity concerns, and the residual issues belong to experimental validity rather than circular reasoning.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

The central claim rests on several domain assumptions about the validity of the QA datasets as driving knowledge, the CARLA simulator as a testbed, and the VLM architecture. It also depends on hand-chosen hyperparameters such as the inference attention prompt and data mix, which materially affect the reported gains.

free parameters (5)
  • Inference attention prefix prompt = "Pay attention to your surroundings and do not break traffic rules."
    Hand-crafted prompt used only at inference, not in training; ablation shows it contributes +2.99 driving score and +8.44 route completion (Table 6). It is a tunable component of the central result.
  • Data mixing ratio (Carla sampling multiplier) = Carla sampled twice per epoch
    Chosen to balance knowledge and planning data (Sec 4.1). The ratio is a hand-picked hyperparameter that affects the knowledge/planning trade-off.
  • Number of frames per clip = 5
    Adopted from LingoQA default; temporal context size affects both QA and trajectory predictions.
  • Peak learning rates = 4e-5 (epoch 1), 1e-5 (epoch 2)
    Standard training hyperparameters; not derived from first principles and may affect convergence and final performance.
  • Random sampling seed for 500-item evaluation subsets = not reported
    Evaluation subsets for BDDX, DriveLM, HAD are randomly sampled without a specified seed, so results are not exactly reproducible.
assumptions (5)
  • domain assumption MobileVLM provides a sufficiently strong general visual-linguistic foundation.
    The paper builds on MobileVLM's pretrained weights and instruction tuning (Sec 4.2). If this foundation is weak, the improved performance may not transfer.
  • domain assumption LingoQA and DRAMA contain relevant, correctly labeled driving knowledge.
    The paper treats these QA datasets as ground-truth driving knowledge without auditing label quality (Sec 3.2).
  • domain assumption CARLA closed-loop metrics are a valid proxy for autonomous driving safety and performance.
    All closed-loop claims are evaluated in CARLA Town05; no real-world validation is presented (Sec 4.4).
  • domain assumption The PID controller can accurately track textual waypoints from the VLM.
    The planned waypoints are generated as text and converted to control commands (Sec 3.1); any parsing or tracking errors would affect metrics.
  • domain assumption The rule-based autopilot trajectories used for training are expert-level demonstrations.
    The model imitates an autopilot agent; biased or suboptimal autopilot behavior would be learned as 'correct' driving (Sec 4.1).

how reviews work

0 comments
Cite this review

Pith. "Pith review of WiseAD: Knowledge Augmented End-to-End Autonomous Driving with Vision-Language Model." pith.science (2026). https://pith.science/paper/KJHBQE6H

@misc{pith2026241209951,
  author       = {Pith},
  title        = {Pith review of: WiseAD: Knowledge Augmented End-to-End Autonomous Driving with Vision-Language Model},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/KJHBQE6H}},
  note         = {Machine review of arXiv:2412.09951}
}
read the original abstract

The emergence of general human knowledge and impressive logical reasoning capacity in rapidly progressed vision-language models (VLMs) have driven increasing interest in applying VLMs to high-level autonomous driving tasks, such as scene understanding and decision-making. However, an in-depth study on the relationship between knowledge proficiency, especially essential driving expertise, and closed-loop autonomous driving performance requires further exploration. In this paper, we investigate the effects of the depth and breadth of fundamental driving knowledge on closed-loop trajectory planning and introduce WiseAD, a specialized VLM tailored for end-to-end autonomous driving capable of driving reasoning, action justification, object recognition, risk analysis, driving suggestions, and trajectory planning across diverse scenarios. We employ joint training on driving knowledge and planning datasets, enabling the model to perform knowledge-aligned trajectory planning accordingly. Extensive experiments indicate that as the diversity of driving knowledge extends, critical accidents are notably reduced, contributing 11.9% and 12.4% improvements in the driving score and route completion on the Carla closed-loop evaluations, achieving state-of-the-art performance. Moreover, WiseAD also demonstrates remarkable performance in knowledge evaluations on both in-domain and out-of-domain datasets.

Figures

Figures reproduced from arXiv: 2412.09951 by the authors.

Figure 1
Figure 1. An overview of the proposed WiseAD, a specialized vision-language model for end-to-end autonomous driving with extensive fundamental driving knowledge. Given a clip of the video sequence, our WiseAD is capable of answering various driving-related questions and performing knowledge-augmented trajectory planning according to the target waypoint. tential risk analysis, driving action reasoning, driving action suggestio… view at source ↗
Figure 2
Figure 2. The framework of the WiseAD. Our model is built upon the MobileVLM and takes video sequences and textual prompts as input. The output for corresponding answers is unified into the linguistic expression to leverage the logical reasoning capability in vision-language models. Fv ∈ R T ×Nv×Dv are projected to modality-aligned visual tokens Hv ∈ R T × Nv 4 ×Dl where Dv and Dl denote hid￾den dimension of ViT and MobileLLa… view at source ↗
Figure 3
Figure 3. Qualitative comparison with the InternVL-8B for driving knowledge evaluation. [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 5 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Deferred Exposure of Future Trajectories for Verifiable Reasoning in Autonomous Driving VLMs

    cs.AI 2026-08 conditional novelty 7.0 of 10

    Hiding future trajectory information until after a driving model forms its decision reduces rationalization and improves verifiable autonomous-driving reasoning in the proposed AD-MCQ and DEFT-RLVR framework.

  2. GeoWorldAD: Geometry World Action Model for Autonomous Driving

    cs.RO 2026-07 conditional novelty 6.0 of 10

    Grounding an autonomous-driving action model in ego-aligned multi-scale 3D geometry and latent future-geometry tokens improves NAVSIM closed-loop PDMS/EPDMS over prior geometry- and world-model-based planners.

  3. Extending Large Vision-Language Model for Diverse Interactive Tasks in Autonomous Driving

    cs.CV 2025-05 conditional novelty 6.0 of 10

    DriveMonkey, which bridges a vision-language model to a pre-trained 3D detector with learnable queries, improves 3D visual grounding mAP by 9.86 percentage points over general LVLMs on the new NuInteract benchmark.

  4. DSDrive: Distilling Large Language Model for Lightweight End-to-End Autonomous Driving with Unified Reasoning and Planning

    cs.RO 2025-05 conditional novelty 6.0 of 10

    A 1B-parameter model distilled from a large vision-language model's reasoning, with waypoints shared between reasoning and planning heads, matches a 7B system on several CARLA routes.

  5. Chain-of-Thought for Autonomous Driving: A Comprehensive Survey and Future Prospects

    cs.RO 2025-05 conditional novelty 4.0 of 10

    A survey that classifies chain-of-thought methods for autonomous driving into modular, logical, and reflective pipelines, and proposes three evolutionary stages from direct prompting to reinforcement learning.

Reference graph

Works this paper leans on

45 extracted references · 27 canonical work pages · cited by 5 Pith papers

  1. [1]

    com/apolloauto/apollo, note = Accessed: 2019-02-

    Baidu Apollo team (2017), Apollo: Open Source Au- tonomous Driving, howpublished = https://github. com/apolloauto/apollo, note = Accessed: 2019-02-

    work page 2017

  2. [2]

    Covla: Comprehensive vision-language-action dataset for autonomous driving

    Hidehisa Arai, Keita Miwa, Kento Sasaki, Yu Yamaguchi, Kohei Watanabe, Shunsuke Aoki, and Issei Yamamoto. Covla: Comprehensive vision-language-action dataset for autonomous driving. arXiv preprint arXiv:2408.10845, 2024. 3

    arXiv 2024

  3. [3]

    Qwen-vl: A versatile vision-language model for understand- ing, localization, text reading, and beyond

    Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A versatile vision-language model for understand- ing, localization, text reading, and beyond. arXiv preprint arXiv:2308.12966, 2023. 1

    arXiv 2023

  4. [4]

    nuscenes: A multimodal dataset for autonomous driving

    Holger Caesar, Varun Bankiti, Alex H Lang, Sourabh V ora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Gi- ancarlo Baldan, and Oscar Beijbom. nuscenes: A multimodal dataset for autonomous driving. In Proc. IEEE Conf. Comp. Vis. Patt. Recogn., pages 11621–11631, 2020. 3

    work page 2020

  5. [5]

    Internvl: Scaling up vision foundation mod- els and aligning for generic visual-linguistic tasks

    Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. Internvl: Scaling up vision foundation mod- els and aligning for generic visual-linguistic tasks. In Proc. IEEE Conf. Comp. Vis. Patt. Recogn., pages 24185–24198,

    work page

  6. [6]

    Neat: Neural attention fields for end-to-end autonomous driving

    Kashyap Chitta, Aditya Prakash, and Andreas Geiger. Neat: Neural attention fields for end-to-end autonomous driving. In Proc. IEEE Conf. Comp. Vis. Patt. Recogn., pages 15793– 15803, 2021. 7

    work page 2021

  7. [7]

    Mobilevlm: A fast, reproducible and strong vision language assistant for mobile devices

    Xiangxiang Chu, Limeng Qiao, Xinyang Lin, Shuang Xu, Yang Yang, Yiming Hu, Fei Wei, Xinyu Zhang, Bo Zhang, Xiaolin Wei, et al. Mobilevlm: A fast, reproducible and strong vision language assistant for mobile devices. arXiv preprint arXiv:2312.16886, 2023. 1

    arXiv 2023

  8. [8]

    Mobilevlm v2: Faster and stronger baseline for vision language model

    Xiangxiang Chu, Limeng Qiao, Xinyu Zhang, Shuang Xu, Fei Wei, Yang Yang, Xiaofei Sun, Yiming Hu, Xinyang Lin, Bo Zhang, et al. Mobilevlm v2: Faster and stronger baseline for vision language model. arXiv preprint arXiv:2402.03766,

    arXiv

Show all 45 references
  1. [9]

    The cityscapes dataset for semantic urban scene understanding

    Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding. In Proc. IEEE Conf. Comp. Vis. Patt. Recogn., pages 3213–3223, 2016. 3

    2016

  2. [10]

    Talk2car: Taking con- trol of your self-driving car.arXiv preprint arXiv:1909.10838,

    Thierry Deruyttere, Simon Vandenhende, Dusan Grujicic, Luc Van Gool, and Marie-Francine Moens. Talk2car: Taking con- trol of your self-driving car.arXiv preprint arXiv:1909.10838,

    1909 arXiv

  3. [11]

    Carla: An open urban driving simulator

    Alexey Dosovitskiy, German Ros, Felipe Codevilla, Antonio Lopez, and Vladlen Koltun. Carla: An open urban driving simulator. In Conference on robot learning , pages 1–16. PMLR, 2017. 4, 5, 6

    2017

  4. [12]

    Vision meets robotics: The kitti dataset

    Andreas Geiger, Philip Lenz, Christoph Stiller, and Raquel Urtasun. Vision meets robotics: The kitti dataset. The Inter- national Journal of Robotics Research, 32(11):1231–1237,

  5. [13]

    St-p3: End-to-end vision-based au- tonomous driving via spatial-temporal feature learning

    Shengchao Hu, Li Chen, Penghao Wu, Hongyang Li, Junchi Yan, and Dacheng Tao. St-p3: End-to-end vision-based au- tonomous driving via spatial-temporal feature learning. In Proc. Eur. Conf. Comp. Vis., pages 533–549, 2022. 1, 7, 8

    2022

  6. [14]

    Planning-oriented autonomous driv- ing

    Yihan Hu, Jiazhi Yang, Li Chen, Keyu Li, Chonghao Sima, Xizhou Zhu, Siqi Chai, Senyao Du, Tianwei Lin, Wenhai Wang, Lewei Lu, Xiaosong Jia, Qiang Liu, Jifeng Dai, Yu Qiao, and Hongyang Li. Planning-oriented autonomous driv- ing. In Proc. IEEE Conf. Comp. Vis. Patt. Recogn. , p...

    2023

  7. [15]

    Vad: Vectorized scene representation for efficient autonomous driving

    Bo Jiang, Shaoyu Chen, Qing Xu, Bencheng Liao, Jiajie Chen, Helong Zhou, Qian Zhang, Wenyu Liu, Chang Huang, and Xinggang Wang. Vad: Vectorized scene representation for efficient autonomous driving. In Proc. IEEE Int. Conf. Comp. Vis., pages 8306–8316, 2023. 1, 7, 8

    2023

  8. [16]

    Adapt: Action-aware driving caption transformer

    Bu Jin, Xinyu Liu, Yupeng Zheng, Pengfei Li, Hao Zhao, Tong Zhang, Yuhang Zheng, Guyue Zhou, and Jingjing Liu. Adapt: Action-aware driving caption transformer. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pages 7554–7561. IEEE, 2023. 3

    2023

  9. [17]

    Textual explanations for self-driving vehicles

    Jinkyu Kim, Anna Rohrbach, Trevor Darrell, John Canny, and Zeynep Akata. Textual explanations for self-driving vehicles. In Proc. Eur. Conf. Comp. Vis., pages 563–578, 2018. 3, 4, 5

    2018

  10. [18]

    Grounding human-to-vehicle advice for self- driving vehicles

    Jinkyu Kim, Teruhisa Misu, Yi-Ting Chen, Ashish Tawari, and John Canny. Grounding human-to-vehicle advice for self- driving vehicles. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 10591– 10599, 2019. 3, 4, 5

    2019

  11. [19]

    Knowledge representation and reasoning

    Hector J Levesque. Knowledge representation and reasoning. Annual Review of Computer Science, 1(1):255–287, 1986. 1

    1986

  12. [20]

    Towards knowledge-driven autonomous driving

    Xin Li, Yeqi Bai, Pinlong Cai, Licheng Wen, Daocheng Fu, Bo Zhang, Xuemeng Yang, Xinyu Cai, Tao Ma, Jianfei Guo, et al. Towards knowledge-driven autonomous driving. arXiv preprint arXiv:2312.04316, 2023. 3

    2023 arXiv

  13. [22]

    Automated evaluation of large vision- language models on self-driving corner cases

    Yanze Li, Wenhua Zhang, Kai Chen, Yanxin Liu, Pengxiang Li, Ruiyuan Gao, Lanqing Hong, Meng Tian, Xinhai Zhao, Zhenguo Li, et al. Automated evaluation of large vision- language models on self-driving corner cases. arXiv preprint arXiv:2404.10595, 2024. 3

    2024 arXiv

  14. [23]

    Visual instruction tuning

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36, 2024. 1, 3, 6, 7

    2024

  15. [24]

    Video swin transformer

    Ze Liu, Jia Ning, Yue Cao, Yixuan Wei, Zheng Zhang, Stephen Lin, and Han Hu. Video swin transformer. In Pro- ceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 3202–3211, 2022. 3

    2022

  16. [25]

    Decoupled weight decay regularization

    I Loshchilov. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101, 2017. 6 9

    2017 arXiv

  17. [26]

    Deepseek-vl: towards real-world vision-language understanding

    Haoyu Lu, Wen Liu, Bo Zhang, Bingxuan Wang, Kai Dong, Bo Liu, Jingxiang Sun, Tongzheng Ren, Zhuoshu Li, Yaofeng Sun, et al. Deepseek-vl: towards real-world vision-language understanding. arXiv preprint arXiv:2403.05525, 2024. 1, 7

    2024 arXiv

  18. [27]

    Drama: Joint risk localization and captioning in driving

    Srikanth Malla, Chiho Choi, Isht Dwivedi, Joon Hee Choi, and Jiachen Li. Drama: Joint risk localization and captioning in driving. In Proc. Winter Conf. on Appl. of Comp. Vis. , pages 1043–1052, 2023. 2, 3, 4, 5

    2023

  19. [28]

    Lingoqa: Video question answering for autonomous driving

    Ana-Maria Marcu, Long Chen, Jan H¨unermann, Alice Karn- sund, Benoit Hanotte, Prajwal Chidananda, Saurabh Nair, Vijay Badrinarayanan, Alex Kendall, Jamie Shotton, et al. Lingoqa: Video question answering for autonomous driving. arXiv preprint arXiv:2312.14115, 2023. 1, 2, 3, 4, 5, 6

    2023 arXiv

  20. [29]

    Bleu: a method for automatic evaluation of machine translation

    Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pages 311–318,

  21. [30]

    Multi- modal fusion transformer for end-to-end autonomous driving

    Aditya Prakash, Kashyap Chitta, and Andreas Geiger. Multi- modal fusion transformer for end-to-end autonomous driving. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 7077–7087, 2021. 7

    2021

  22. [31]

    Nuscenes-qa: A multi-modal visual question answering benchmark for autonomous driving scenario

    Tianwen Qian, Jingjing Chen, Linhai Zhuo, Yang Jiao, and Yu-Gang Jiang. Nuscenes-qa: A multi-modal visual question answering benchmark for autonomous driving scenario. In Proc. AAAI Conf. Artificial Intell., pages 4542–4550, 2024. 3

    2024

  23. [32]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In Proc. Int. Conf. Learn. Representations, pages 8...

    2021

  24. [33]

    Toward driving scene understanding: A dataset for learning driver behavior and causal reasoning

    Vasili Ramanishka, Yi-Ting Chen, Teruhisa Misu, and Kate Saenko. Toward driving scene understanding: A dataset for learning driver behavior and causal reasoning. In Proc. IEEE Conf. Comp. Vis. Patt. Recogn., pages 7699–7707, 2018. 3

    2018

  25. [34]

    Safety-enhanced autonomous driving using inter- pretable sensor fusion transformer

    Hao Shao, Letian Wang, Ruobing Chen, Hongsheng Li, and Yu Liu. Safety-enhanced autonomous driving using inter- pretable sensor fusion transformer. In Conference on Robot Learning, pages 726–737. PMLR, 2023. 1, 4

    2023

  26. [35]

    Reasonnet: End-to-end driv- ing with temporal and global reasoning

    Hao Shao, Letian Wang, Ruobing Chen, Steven L Waslander, Hongsheng Li, and Yu Liu. Reasonnet: End-to-end driv- ing with temporal and global reasoning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 13723–13733, 2023

    2023

  27. [36]

    Lmdrive: Closed-loop end-to-end driving with large language models

    Hao Shao, Yuxuan Hu, Letian Wang, Guanglu Song, Steven L Waslander, Yu Liu, and Hongsheng Li. Lmdrive: Closed-loop end-to-end driving with large language models. InProc. IEEE Conf. Comp. Vis. Patt. Recogn., pages 15120–15130, 2024. 1, 3, 4, 5

    2024

  28. [37]

    Drivelm: Driving with graph visual ques- tion answering

    Chonghao Sima, Katrin Renz, Kashyap Chitta, Li Chen, Hanxue Zhang, Chengen Xie, Ping Luo, Andreas Geiger, and Hongyang Li. Drivelm: Driving with graph visual ques- tion answering. arXiv preprint arXiv:2312.14150, 2023. 3, 4, 5

    2023 arXiv

  29. [38]

    Scalability in perception for autonomous driving: Waymo open dataset

    Pei Sun, Henrik Kretzschmar, Xerxes Dotiwalla, Aurelien Chouard, Vijaysai Patnaik, Paul Tsui, James Guo, Yin Zhou, Yuning Chai, Benjamin Caine, et al. Scalability in perception for autonomous driving: Waymo open dataset. In Proc. IEEE Conf. Comp. Vis. Patt. Recogn., pages 2446...

    2020

  30. [39]

    Drivevlm: The convergence of autonomous driving and large vision-language models

    Xiaoyu Tian, Junru Gu, Bailin Li, Yicheng Liu, Chenxu Hu, Yang Wang, Kun Zhan, Peng Jia, Xianpeng Lang, and Hang Zhao. Drivevlm: The convergence of autonomous driving and large vision-language models. arXiv preprint arXiv:2402.12289, 2024. 2, 3

    2024 arXiv

  31. [40]

    Llama: Open and efficient foundation language models

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Mar- tinet, Marie-Anne Lachaux, Timoth ´ee Lacroix, Baptiste Rozi`ere, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. 2023. 3

    2023

  32. [41]

    Cider: Consensus-based image description evalu- ation

    Ramakrishna Vedantam, C Lawrence Zitnick, and Devi Parikh. Cider: Consensus-based image description evalu- ation. In Proc. IEEE Conf. Comp. Vis. Patt. Recogn., pages 4566–4575, 2015. 6

    2015

  33. [42]

    Drivemlm: Aligning multi-modal large language models with behavioral planning states for autonomous driving

    Wenhai Wang, Jiangwei Xie, ChuanYang Hu, Haoming Zou, Jianan Fan, Wenwen Tong, Yang Wen, Silei Wu, Hanming Deng, Zhiqi Li, et al. Drivemlm: Aligning multi-modal large language models with behavioral planning states for autonomous driving. arXiv preprint arXiv:2312.09245, 2023. 3

    2023

  34. [43]

    Drivegpt4: Interpretable end-to-end autonomous driving via large language model

    Zhenhua Xu, Yujia Zhang, Enze Xie, Zhen Zhao, Yong Guo, Kwan-Yee K Wong, Zhenguo Li, and Hengshuang Zhao. Drivegpt4: Interpretable end-to-end autonomous driving via large language model. IEEE Robotics and Automation Letters,

  35. [44]

    Rag-driver: Gen- eralisable driving explanations with retrieval-augmented in- context learning in multi-modal large language model

    Jianhao Yuan, Shuyang Sun, Daniel Omeiza, Bo Zhao, Paul Newman, Lars Kunze, and Matthew Gadd. Rag-driver: Gen- eralisable driving explanations with retrieval-augmented in- context learning in multi-modal large language model. arXiv preprint arXiv:2402.10828, 2024. 3

    2024

  36. [45]

    End-to-end urban driving by imitating a re- inforcement learning coach

    Zhejun Zhang, Alexander Liniger, Dengxin Dai, Fisher Yu, and Luc Van Gool. End-to-end urban driving by imitating a re- inforcement learning coach. In Proceedings of the IEEE/CVF international conference on computer vision, pages 15222– 15232, 2021. 7

    2021

  37. [46]

    Embodied understanding of driving scenarios

    Yunsong Zhou, Linyan Huang, Qingwen Bu, Jia Zeng, Tianyu Li, Hang Qiu, Hongzi Zhu, Minyi Guo, Yu Qiao, and Hongyang Li. Embodied understanding of driving scenarios. arXiv preprint arXiv:2403.04593, 2024. 3, 6 10

    2024 arXiv

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.