Pith. sign in

REVIEW 3 major objections 5 minor 81 references

BEV features give LLMs 46% better cross-view spatial reasoning; distilling LLM semantics into the same BEV then lifts closed-loop driving safety by 29%.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-15 13:43 UTC pith:ILP3AL57

load-bearing objection Solid representation study plus a practical distillation recipe that lifts closed-loop safety; the causal isolation of the LLM teacher is incomplete but the empirical gains are real and worth engaging. the 3 major comments →

arxiv 2603.06576 v2 pith:ILP3AL57 submitted 2026-03-06 cs.CV cs.AIcs.LGcs.RO

BEVLM: Distilling Semantic Knowledge from LLMs into Bird's-Eye View Representations

Thomas Monninger , Shaoyuan Xie , Qi Alfred Chen , Sihao Ding This is my paper
classification cs.CV cs.AIcs.LGcs.RO
keywords autonomous drivinglarge language modelsbird's-eye viewsemantic distillationend-to-end planningspatial reasoningvision-language models
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Current LLM-based driving systems feed each camera view and frame as separate tokens, which wastes compute and breaks the geometric consistency needed for 3-D spatial reasoning. Bird's-eye-view (BEV) grids already fuse multi-view geometry into one compact map, yet they are trained only on detection-style labels and therefore lack the open-world semantics of foundation vision-language models. This paper shows that a simple projector lets an LLM read BEV tokens as fluently as a detection head, and that those tokens already outperform multi-view image tokens by a large margin on cross-view questions. Building on that result, the authors freeze the LLM and use visual-question-answering as a teacher signal to distill semantic knowledge back into the BEV encoder while regularizing with ordinary object detection. The resulting semantic-aware BEV improves both LLM scene understanding and closed-loop end-to-end planning, especially in safety-critical corner cases. A sympathetic reader cares because safer behavior under rare events remains the practical bottleneck for deploying autonomous driving systems.

Core claim

A spatially consistent BEV feature map is a superior input representation for LLMs in multi-view driving scenes, raising object-centric cross-view accuracy by 46% over independent multi-view image tokens. Freezing the LLM and distilling its semantic knowledge into the BEV encoder via VQA further produces a representation that, when used by an end-to-end planner, raises the NeuroNCAP closed-loop safety score by 29% and lowers collision rate.

What carries the argument

BEVLM: a lightweight MLP projector that maps a max-pooled BEV grid into LLM token space, followed by representation distillation in which the frozen LLM supplies a fixed semantic teacher manifold through VQA losses while joint detection preserves geometric structure.

Load-bearing premise

One epoch of distillation from a human-curated driving VQA set under a frozen LLM injects the right safety semantics into the BEV encoder without destroying the geometric structure needed for planning.

What would settle it

Train an identical BEV encoder using only perception VQA questions (or with the teacher LLM weights randomly scrambled) and measure NeuroNCAP score and impact velocity; if the safety gains disappear or reverse relative to the full Behavior+Planning distillation, the claim that LLM semantic knowledge is the causal driver fails.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • LLMs can answer panoramic spatial questions more accurately when given one fused BEV token map instead of separate multi-view image tokens.
  • End-to-end planners that inherit a semantically distilled BEV encoder produce lower-severity collisions and higher NeuroNCAP scores in closed-loop safety-critical re-simulations.
  • Behavior and planning VQA subsets contribute more of the safety gain than perception or prediction subsets alone.
  • Scaling the frozen teacher LLM from 1B to 8B further improves the distilled BEV's closed-loop planning metrics.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same distillation recipe should transfer to other BEV encoder families without redesigning the downstream planner heads.
  • Replacing human-curated DriveLM questions with large-scale synthetic or VLM-generated VQA could remove the labor bottleneck and test whether the method scales with teacher data volume.
  • Treating BEV as a fixed semantic manifold suggests future vision-language-action controllers could natively operate on BEV grids rather than raw multi-view tokens.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper argues that Bird’s-Eye View (BEV) features are a superior unified input for LLM spatial reasoning in autonomous driving compared with independent multi-view image tokens, and that semantic knowledge can be distilled from frozen LLMs into BEV encoders via VQA. Empirically, BEV tokens improve object-centric MCQ accuracy on Ego3D by 46.0% over perspective tokens (and match a 10× larger ViT). The authors then freeze an LLM teacher, train a BEV encoder jointly on DriveLM VQA + detection for one epoch, and show that the resulting encoder, when plugged into UniAD, raises the NeuroNCAP closed-loop safety score from 2.10 to 2.71 (≈29%) while lowering collision rate from 0.62 to 0.55. Ablations examine pooling, VQA subsets, and teacher scale (1B vs 8B).

Significance. If the causal claims hold, the work supplies a practical bridge between geometrically structured BEV pipelines (still the workhorse of production AD stacks) and the semantic richness of foundation VLMs. The representation study cleanly isolates the BEV fusion step and is valuable on its own; a verified distillation recipe that measurably improves closed-loop safety on NeuroNCAP would be of immediate interest to both the end-to-end planning and VLA communities. Strengths include multi-VLM consistency, full class-wise tables, VQA-subset and pooling ablations, and qualitative roll-outs that illustrate anticipatory behavior.

major comments (3)
  1. §4.1–5.3 and Table 4: the central safety claim (NeuroNCAP 2.10→2.71) attributes gains to “LLM semantic distillation.” After detection pre-training the BEV encoder is unfrozen for one epoch of joint VQA+detection loss, then re-frozen. The sole control is a pure-detection baseline that never unfreezes the encoder. Consequently any improvement could arise from additional gradient steps on the BEV features (or from the detection regularizer) rather than alignment to the frozen LLM’s semantic manifold. A matched-compute detection-only continuation for the same number of steps, or a randomly-initialized decoder of identical capacity, is required to isolate the claimed mechanism.
  2. Abstract vs. §5 / Table 4: the abstract states gains “across UniAD and VAD, with gains of up to 28.2%,” yet the body reports only UniAD numbers (and quotes 29.0%). Either VAD results must be added with the same protocol or the abstract claim must be corrected; the present discrepancy undermines reproducibility of the headline number.
  3. §4.1 Eq. (1) and the “semantic manifold” framing: the distillation objective is ordinary next-token CE under a frozen LLM; the latent-alignment interpretation is post-hoc. Without an analysis showing that the BEV features actually move toward the teacher’s token space (e.g., cosine similarity to LLM embeddings of safety concepts, or probing for “blocked lane” / “unsafe velocity” before vs. after distillation), the claim that high-level semantics rather than generic multi-task regularization are transferred remains unproven.
minor comments (5)
  1. Abstract / body number inconsistency: 46.0% vs. “46 %”, 28.2% vs. 29.0%; unify all reported percentages.
  2. Table 1 / Appendix B: majority-class and linear-probe baselines are useful; please also report the exact spatial mask construction for the linear probe so the experiment is fully reproducible.
  3. Fig. 4 and Appendix D: velocity profiles are informative; adding the corresponding NeuroNCAP impact-velocity numbers next to each qualitative example would strengthen the link to the quantitative score.
  4. §3.2 / Table 3: token counts (I_ViT 4 608, I_UniAD 2 250, B_UniAD 2 500) are close but not identical; a short note on whether padding or truncation was applied would remove a minor confound.
  5. Appendix E limitations: the authors correctly flag the human-curated DriveLM dependency and single BEV architecture; elevating a short version of this discussion into the main text would help readers gauge generality.

Circularity Check

0 steps flagged

No circularity: empirical representation comparisons and distillation gains measured on external benchmarks (Ego3D, NeuroNCAP) with no by-construction reductions.

full rationale

The paper's load-bearing claims are purely empirical. The 46% cross-view accuracy gain (Tab. 3) is a direct head-to-head measurement of BUniAD vs. IViT/IUniAD tokens on the external Ego3D object-centric split under identical LLM and projector training; nothing is fitted then re-reported as a prediction. The 29% NeuroNCAP safety improvement (Tab. 4) is likewise an external closed-loop evaluation after one-epoch joint VQA+detection fine-tuning of a frozen-LLM student BEV encoder (Eq. 1 uses ordinary next-token CE as proxy; the LLM remains frozen and is never trained on NeuroNCAP). The VQA-subset ablation (Tab. 6) and downsampling ablation (Tab. 5) are ordinary controls, not self-definitional. Self-citations to UniAD/BEVFormer supply the shared encoder architecture and baseline, which is standard reuse rather than a load-bearing uniqueness theorem or ansatz smuggled from the same authors. No parameter is fitted to a subset and then called a prediction of a related quantity; no result reduces by construction to its inputs. The derivation chain is therefore self-contained against external data.

Axiom & Free-Parameter Ledger

3 free parameters · 4 axioms · 1 invented entities

The central empirical claims rest on standard deep-learning training assumptions, the geometric fidelity of the UniAD/BEVFormer BEV encoder, and the premise that DriveLM VQA questions form a useful semantic teacher. Free parameters are ordinary hyper-parameters; no new physical constants or particles are introduced. The invented entity is the BEVLM distillation procedure itself.

free parameters (3)
  • distillation epochs / loss weights = 1 epoch, equal weights
    1 epoch of joint VQA+detection with equal loss weights is chosen by hand; longer or re-weighted schedules are not explored for the final E2E numbers.
  • BEV downsampling method and resolution = max-pool 50×50
    Max-pool to 50×50 (2 500 tokens) selected after ablation; other pool/conv choices give similar but not identical numbers.
  • learning rates and optimizer schedule = 1e-3 / 1e-6
    Projector LR 1e-3, encoder fine-tune 1e-6, AdamW + cosine; standard but still free choices that affect final metrics.
axioms (4)
  • domain assumption A frozen LLM’s next-token prediction loss on VQA is a faithful proxy for the ideal semantic token embeddings required by the teacher manifold.
    Stated in Sec. 4.1 as the justification for representation distillation; never independently verified outside the DriveLM distribution.
  • domain assumption Joint object-detection supervision is sufficient to prevent catastrophic forgetting of geometric structure during semantic distillation.
    Invoked in Sec. 4.1 and training protocol; only one epoch of distillation is used, so the assumption is lightly tested.
  • domain assumption NeuroNCAP’s NeRF-based closed-loop scenarios are a valid proxy for real-world safety-critical driving.
    Used as the primary safety metric in Sec. 5.3; the paper itself notes the open-/closed-loop gap.
  • standard math Standard transformer next-token prediction and MLP projection mathematics hold.
    Background for the projector and LLM loss.
invented entities (1)
  • BEVLM / semantic-distilled BEV representation no independent evidence
    purpose: Unified intermediate that is both spatially consistent and aligned to an LLM’s semantic space so that downstream planners inherit open-world knowledge.
    The core methodological contribution; its utility is demonstrated only inside the paper’s own experiments.

pith-pipeline@v1.1.0-grok45 · 29003 in / 3088 out tokens · 30948 ms · 2026-07-15T13:43:14.353899+00:00 · methodology

0 comments
read the original abstract

The integration of Large Language Models (LLMs) into autonomous driving has attracted growing interest for their strong reasoning and semantic understanding abilities, which are essential for handling complex decision-making and long-tail scenarios. However, existing methods typically feed LLMs with tokens from multi-view and multi-frame images independently, leading to redundant computation and limited spatial consistency. This separation in visual processing hinders accurate 3D spatial reasoning and fails to maintain geometric coherence across views. On the other hand, Bird's-Eye View (BEV) representations learned from geometrically annotated tasks (e.g., object detection) provide spatial structure but lack the semantic richness of foundation vision encoders. To bridge this gap, we propose BEVLM, a framework that connects a spatially consistent and semantically distilled BEV representation with LLMs. Through extensive experiments, we show that BEVLM enables LLMs to reason more effectively in cross-view driving scenes, improving accuracy by 46.0%, by leveraging BEV features as unified inputs. Furthermore, by distilling semantic knowledge from LLMs into BEV representations, BEVLM significantly improves closed-loop end-to-end driving performance in safety-critical scenarios across UniAD and VAD, with gains of up to 28.2%.

Figures

Figures reproduced from arXiv: 2603.06576 by Qi Alfred Chen, Shaoyuan Xie, Sihao Ding, Thomas Monninger.

Figure 1
Figure 1. Figure 1: Representation Comparison: Left: Vision encoders can leverage widely available semantically rich image-text data, but process multi-view images indepen￾dently. Center: Bird’s-Eye View (BEV) encoders provide a spatially consistent scene representation, but are limited to geometrically annotated data. Right (ours): We propose the semantic distillation from LLMs to BEV encoders to build a semantic￾enhanced an… view at source ↗
Figure 2
Figure 2. Figure 2: Representation Study. We compare between (1) IViT, visual tokens extracted from the Vision Transformer of the original VLM; (2) IUniAD, visual tokens from the backbone before the BEV fusion; and (3) BUniAD, BEV tokens produced from the same backbone after the BEV fusion. The input language question is the same, but not visualized here for simplicity. head, where answers are derived by matching class, movin… view at source ↗
Figure 3
Figure 3. Figure 3: BEV Semantic Distillation: We distill the knowledge from the language model to the BEV representations by using Visual Question Answering (VQA) tasks while regularizing BEV spatial structure using the original object detection tasks. Specifically, the MCQ accuracy is improved by 46.0 %, and the L1 error is de￾creased by 27.8 %. Additionally, BUniAD shows comparable performance to IViT w/ ft. even though th… view at source ↗
Figure 4
Figure 4. Figure 4: Qualitative NeuroNCAP Results. Two representative closed-loop plan￾ning scenarios are presented for comparison between the baseline and semantically distilled models. The distilled model demonstrates improved decision-making under safety-critical scenarios, successfully performing a safe right turn in corner case 1 and an evasive lane change to the free right lane in corner case 2 to avoid potential colli￾… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

81 extracted references · 1 canonical work pages

  1. [1]

    arXiv preprint arXiv:2303.08774 (2023)

    Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F.L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al.: GPT-4 Technical Report. arXiv preprint arXiv:2303.08774 (2023)

    Pith/arXiv arXiv 2023

  2. [2]

    In: Proceedings of the IEEE International Con- ference on Computer Vision (ICCV) (2015)

    Antol, S., Agrawal, A., Lu, J., Mitchell, M., Batra, D., Zitnick, C.L., Parikh, D.: VQA: Visual Question Answering. In: Proceedings of the IEEE International Con- ference on Computer Vision (ICCV) (2015)

    2015

  3. [3]

    In: Proceedings of the acl work- shop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization

    Banerjee, S., Lavie, A.: METEOR: An Automatic Metric for MT Evaluation with Improved Correlation with Human Judgments. In: Proceedings of the acl work- shop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization. pp. 65–72 (2005)

    2005

  4. [4]

    arXiv Preprint arXiv:2507.19370 (2025)

    Brandstaetter, F., Schuetz, E., Winter, K., Flohr, F.: BEV-LLM: Leveraging Mul- timodal BEV Maps for Scene Captioning in Autonomous Driving. arXiv Preprint arXiv:2507.19370 (2025)

    Pith/arXiv arXiv 2025

  5. [5]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2020)

    Caesar, H., Bankiti, V., Lang, A.H., Vora, S., Liong, V.E., Xu, Q., Krishnan, A., Pan, Y., Baldan, G., Beijbom, O.: nuScenes: A Multimodal Dataset for Au- tonomous Driving. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2020)

    2020

  6. [6]

    In: Conference on Robot Learning (CoRL) (2025)

    Cao, W., Hallgarten, M., Li, T., Dauner, D., Gu, X., Wang, C., Miron, Y., Aiello, M., Li, H., Gilitschenski, I., Ivanovic, B., Pavone, M., Geiger, A., Chitta, K.: Pseudo-Simulation for Autonomous Driving. In: Conference on Robot Learning (CoRL) (2025)

    2025

  7. [7]

    In: IEEE Symposium on Security and Privacy (SP)

    Cao, Y., Wang, N., Xiao, C., Yang, D., Fang, J., Yang, R., Chen, Q.A., Liu, M., Li, B.: Invisible for Both Camera and LiDAR: Security of Multi-Sensor Fusion Based Perception in Autonomous Driving Under Physical-World Attacks. In: IEEE Symposium on Security and Privacy (SP). pp. 176–194. IEEE (2021)

    2021

  8. [8]

    In: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

    Chen, B., Xu, Z., Kirmani, S., Ichter, B., Sadigh, D., Guibas, L., Xia, F.: Spa- tialVLM: Endowing Vision-Language Models with Spatial Reasoning Capabilities. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 14455–14465 (2024) 16 T. Monninger, S. Xie, Q. A. Chen, and S. Ding

    2024

  9. [9]

    arXiv preprint arXiv:2412.18607 (2024)

    Chen, Y., Wang, Y., Zhang, Z.: DrivingGPT: Unifying Driving World Model- ing and Planning with Multi-Modal Autoregressive Transformers. arXiv preprint arXiv:2412.18607 (2024)

    Pith/arXiv arXiv 2024

  10. [10]

    In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2017)

    Chollet, F.: Xception: Deep Learning with Depthwise Separable Convolutions. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2017)

    2017

  11. [11]

    arXiv Preprint arXiv:2507.06261 (2025)

    Comanici, G., Bieber, E., Schaekermann, M., Pasupat, I., Sachdeva, N., Dhillon, I., Blistein,M.,Ram,O.,Zhang,D.,Rosen,E.,etal.:Gemini2.5:PushingtheFrontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities. arXiv Preprint arXiv:2507.06261 (2025)

    Pith/arXiv arXiv 2025

  12. [12]

    In: Conference on Robot Learn- ing

    Dauner, D., Hallgarten, M., Geiger, A., Chitta, K.: Parting with Misconceptions About Learning-Based Vehicle Motion Planning. In: Conference on Robot Learn- ing. pp. 1268–1281. PMLR (2023)

    2023

  13. [13]

    Advances in Neural Information Processing Systems37, 28706–28719 (2024)

    Dauner, D., Hallgarten, M., Li, T., Weng, X., Huang, Z., Yang, Z., Li, H., Gilitschenski, I., Ivanovic, B., Pavone, M., et al.: NavSim: Data-Driven Non- Reactive Autonomous Vehicle Simulation and Benchmarking. Advances in Neural Information Processing Systems37, 28706–28719 (2024)

    2024

  14. [14]

    In: NeurIPS 2025 Workshop on Regulatable ML (2025)

    Ding, S., Vasa, S., Ramadwar, A.: Explanation-Driven Counterfactual Testing for Faithfulness in Vision-Language Model Explanations. In: NeurIPS 2025 Workshop on Regulatable ML (2025)

    2025

  15. [15]

    arXiv Preprint arXiv:2010.11929 (2020)

    Dosovitskiy, A.: An Image Is Worth 16x16 Words: Transformers for Image Recog- nition at Scale. arXiv Preprint arXiv:2010.11929 (2020)

    Pith/arXiv arXiv 2010

  16. [16]

    In: Proceedings of the IEEE/CVF Winter Con- ference on Applications of Computer Vision (WACV) (2022)

    Fadadu, S., Pandey, S., Hegde, D., Shi, Y., Chou, F.C., Djuric, N., Vallespi- Gonzalez, C.: Multi-View Fusion of Sensor Data for Improved Perception and Prediction in Autonomous Driving. In: Proceedings of the IEEE/CVF Winter Con- ference on Applications of Computer Vision (WACV) (2022)

    2022

  17. [17]

    arXiv preprint arXiv:2505.15925 (2025)

    Feng, B., Mei, Z., Li, B., Ost, J., Girgis, R., Majumdar, A., Heide, F.: VERDI: VLM-Embedded Reasoning for Autonomous Driving. arXiv preprint arXiv:2505.15925 (2025)

    Pith/arXiv arXiv 2025

  18. [18]

    arXiv preprint arXiv:2503.19755 (2025)

    Fu,H.,Zhang,D.,Zhao,Z.,Cui,J.,Liang,D.,Zhang,C.,Zhang,D.,Xie,H.,Wang, B., Bai, X.: Orion: A Holistic End-to-End Autonomous Driving Framework by Vision-Language Instructed Action Generation. arXiv preprint arXiv:2503.19755 (2025)

    Pith/arXiv arXiv 2025

  19. [19]

    arXiv preprint arXiv:2509.06266 (2025)

    Gholami,M.,Rezaei,A.,Weimin,Z.,Mao,S.,Zhou,S.,Zhang,Y.,Akbari,M.:Spa- tial Reasoning with Vision-Language Models in Ego-Centric Multi-View Scenes. arXiv preprint arXiv:2509.06266 (2025)

    arXiv 2025

  20. [20]

    Guo, Z., Gubernatorov, K., Asfaw, S., Yagudin, Z., Tsetserukou, D.: VDT-Auto: End-to-EndAutonomousDrivingwithVLM-GuidedDiffusionTransformers.arXiv preprint arXiv:2502.20108 (2025)

    Pith/arXiv arXiv 2025

  21. [21]

    In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2016)

    He, K., Zhang, X., Ren, S., Sun, J.: Deep Residual Learning for Image Recogni- tion. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2016)

    2016

  22. [22]

    In: Proceedings of the IEEE/CVF International Con- ference on Computer Vision (ICCV)

    Heo, B., Kim, J., Yun, S., Park, H., Kwak, N., Choi, J.Y.: A Comprehensive Over- haul of Feature Distillation. In: Proceedings of the IEEE/CVF International Con- ference on Computer Vision (ICCV). pp. 1921–1930 (2019)

    1921

  23. [23]

    arXiv preprint arXiv:1503.02531 (2015)

    Hinton, G., Vinyals, O., Dean, J.: Distilling the Knowledge in a Neural Network. arXiv preprint arXiv:1503.02531 (2015)

    Pith/arXiv arXiv 2015

  24. [24]

    In: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2023) BEVLM 17

    Hu,Y.,Yang,J.,Chen,L.,Li,K.,Sima,C.,Zhu,X.,Chai,S.,Du,S.,Lin,T.,Wang, W., et al.: Planning-Oriented Autonomous Driving. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2023) BEVLM 17

    2023

  25. [25]

    arXiv Preprint arXiv:2112.11790 (2021)

    Huang, J., Huang, G., Zhu, Z., Ye, Y., Du, D.: BEVDet: High-Performance Multi- Camera 3D Object Detection in Bird’s-Eye-View. arXiv Preprint arXiv:2112.11790 (2021)

    Pith/arXiv arXiv 2021

  26. [26]

    In: European Conference on Computer Vision (ECCV)

    Huang, Z., Tang, T., Chen, S., Lin, S., Jie, Z., Ma, L., Wang, G., Liang, X.: Making Large Language Models Better Planners with Reasoning-Decision Alignment. In: European Conference on Computer Vision (ECCV). Springer (2024)

    2024

  27. [27]

    Transactions on Machine Learning Research (2025),https://openreview.net/forum?id=kH3t5lmOU8

    Hwang, J.J., Xu, R., Lin, H., Hung, W.C., Ji, J., Choi, K., Huang, D., He, T., Cov- ington,P.,Sapp,B.,Zhou,Y.,Guo,J.,Anguelov,D.,Tan,M.:EMMA:End-to-End Multimodal Model for Autonomous Driving. Transactions on Machine Learning Research (2025),https://openreview.net/forum?id=kH3t5lmOU8

    2025

  28. [28]

    Advances in Neu- ral Information Processing Systems37, 819–844 (2024)

    Jia, X., Yang, Z., Li, Q., Zhang, Z., Yan, J.: Bench2Drive: Towards Multi-Ability Benchmarking of Closed-Loop End-to-End Autonomous Driving. Advances in Neu- ral Information Processing Systems37, 819–844 (2024)

    2024

  29. [29]

    arXiv preprint arXiv:2410.22313 (2024)

    Jiang, B., Chen, S., Liao, B., Zhang, X., Yin, W., Zhang, Q., Huang, C., Liu, W., Wang, X.: Senna: Bridging Large Vision-Language Models and End-to-End Autonomous Driving. arXiv preprint arXiv:2410.22313 (2024)

    Pith/arXiv arXiv 2024

  30. [30]

    In: IEEE/CVF International Conference on Computer Vision (ICCV) (2023)

    Jiang,B.,Chen,S.,Xu,Q.,Liao,B.,Chen,J.,Zhou,H.,Zhang,Q.,Liu,W.,Huang, C., Wang, X.: VAD: Vectorized Scene Representation for Efficient Autonomous Driving. In: IEEE/CVF International Conference on Computer Vision (ICCV) (2023)

    2023

  31. [31]

    In: IEEE/CVF International Conference on Computer Vision (ICCV) (2023)

    Kirillov, A., Mintun, E., Ravi, N., Mao, H., Rolland, C., Gustafson, L., Xiao, T., Whitehead, S., Berg, A.C., Lo, W.Y., et al.: Segment Anything. In: IEEE/CVF International Conference on Computer Vision (ICCV) (2023)

    2023

  32. [32]

    In: IEEE/CVF International Conference on Computer Vision (ICCV) (2023)

    Kong, L., Liu, Y., Li, X., Chen, R., Zhang, W., Ren, J., Pan, L., Chen, K., Liu, Z.: Robo3D: Towards Robust and Reliable 3D Perception Against Corruptions. In: IEEE/CVF International Conference on Computer Vision (ICCV) (2023)

    2023

  33. [33]

    In: Inter- national Conference on Machine Learning (ICML)

    Li, J., Li, D., Xiong, C., Hoi, S.: BLIP: Bootstrapping Language-Image Pre- Training for Unified Vision-Language Understanding and Generation. In: Inter- national Conference on Machine Learning (ICML). PMLR (2022)

    2022

  34. [34]

    In: European Conference on Computer Vision (ECCV)

    Li, Z., Wang, W., Li, H., Xie, E., Sima, C., Lu, T., Qiao, Y., Dai, J.: BEVFormer: Learning Bird’s-Eye-View Representation from Multi-Camera Images via Spa- tiotemporal Transformers. In: European Conference on Computer Vision (ECCV). Springer (2022)

    2022

  35. [35]

    Li, Z., Yu, Z., Lan, S., Li, J., Kautz, J., Lu, T., Alvarez, J.M.: Is Ego Status All You Need for Open-Loop End-to-End Autonomous Driving? In: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2024)

    2024

  36. [36]

    In: The Eleventh International Conference on Learning Representations (ICLR) (2023)

    Liao, B., Chen, S., Wang, X., Cheng, T., Zhang, Q., Liu, W., Huang, C.: MapTR: Structured Modeling and Learning for Online Vectorized HD Map Construction. In: The Eleventh International Conference on Learning Representations (ICLR) (2023)

    2023

  37. [37]

    In: Text Summarization Branches Out

    Lin, C.Y.: ROUGE: A Package for Automatic Evaluation of Summaries. In: Text Summarization Branches Out. pp. 74–81 (2004)

    2004

  38. [38]

    In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2017)

    Lin, T.Y., Dollár, P., Girshick, R., He, K., Hariharan, B., Belongie, S.: Feature Pyramid Networks for Object Detection. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2017)

    2017

  39. [39]

    Advances in Neural Information Processing Systems36(2023)

    Liu, H., Li, C., Wu, Q., Lee, Y.J.: Visual Instruction Tuning. Advances in Neural Information Processing Systems36(2023)

    2023

  40. [40]

    In: 2023 IEEE International Conference on Robotics and Automation (ICRA)

    Liu, Z., Tang, H., Amini, A., Yang, X., Mao, H., Rus, D.L., Han, S.: BEVFusion: Multi-Task Multi-Sensor Fusion with Unified Bird’s-Eye View Representation. In: 2023 IEEE International Conference on Robotics and Automation (ICRA). IEEE (2023) 18 T. Monninger, S. Xie, Q. A. Chen, and S. Ding

    2023

  41. [41]

    In: European Conference on Computer Vision (ECCV)

    Ljungbergh, W., Tonderski, A., Johnander, J., Caesar, H., Åström, K., Fels- berg, M., Petersson, C.: NeuroNCap: Photorealistic Closed-Loop Safety Testing for Autonomous Driving. In: European Conference on Computer Vision (ECCV). Springer (2024)

    2024

  42. [42]

    arXiv preprint arXiv:1711.05101 (2017)

    Loshchilov, I., Hutter, F.: Decoupled Weight Decay Regularization. arXiv preprint arXiv:1711.05101 (2017)

    Pith/arXiv arXiv 2017

  43. [43]

    Lu, H., Liu, W., Zhang, B., Wang, B., Dong, K., Liu, B., Sun, J., Ren, T., Li, Z., Yang, H., Sun, Y., Deng, C., Xu, H., Xie, Z., Ruan, C.: DeepSeek-VL: Towards Real-World Vision-Language Understanding (2024)

    2024

  44. [44]

    IEEE Transac- tions on Pattern Analysis and Machine Intelligence (2024)

    Ma, Y., Wang, T., Bai, X., Yang, H., Hou, Y., Wang, Y., Qiao, Y., Yang, R., Manocha, D., Zhu, X.: Vision-Centric BEV Perception: A Survey. IEEE Transac- tions on Pattern Analysis and Machine Intelligence (2024)

    2024

  45. [45]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2024)

    Min, C., Zhao, D., Xiao, L., Zhao, J., Xu, X., Zhu, Z., Jin, L., Li, J., Guo, Y., Xing, J., et al.: DriveWorld: 4D Pre-Trained Scene Understanding via World Models for Autonomous Driving. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2024)

    2024

  46. [46]

    arXiv preprint arXiv:2503.13430 (2025)

    Monninger, T., Anwar, M.Z., Antol, S., Staab, S., Ding, S.: AugMapNet: Improving Spatial Latent Structure via BEV Grid Augmentation for Enhanced Vectorized Online HD Map Construction. arXiv preprint arXiv:2503.13430 (2025)

    arXiv 2025

  47. [47]

    In: 2025 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)

    Monninger, T., Zhang, Z., Mo, Z., Anwar, M.Z., Staab, S., Ding, S.: MapDiffusion: Generative diffusion for vectorized online hd map construction and uncertainty estimation in autonomous driving. In: 2025 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). pp. 4099–4106. IEEE (2025)

    2025

  48. [48]

    IEEE Transactions on Intelligent Transportation Systems22(7), 4316–4336 (2020)

    Muhammad, K., Ullah, A., Lloret, J., Del Ser, J., De Albuquerque, V.H.C.: Deep Learning for Safe Autonomous Driving: Current Challenges and Future Directions. IEEE Transactions on Intelligent Transportation Systems22(7), 4316–4336 (2020)

    2020

  49. [49]

    In: IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition (CVPR) (2024)

    Pan, C., Yaman, B., Nesti, T., Mallik, A., Allievi, A.G., Velipasalar, S., Ren, L.: VLP: Vision Language Planning for Autonomous Driving. In: IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition (CVPR) (2024)

    2024

  50. [50]

    In: Annual Meeting of the Association for Computational Linguistics

    Papineni, K., Roukos, S., Ward, T., Zhu, W.J.: BLEU: A Method for Automatic Evaluation of Machine Translation. In: Annual Meeting of the Association for Computational Linguistics. pp. 311–318 (2002)

    2002

  51. [51]

    In: AAAI Conference on Artificial Intelligence

    Qian, T., Chen, J., Zhuo, L., Jiao, Y., Jiang, Y.G.: nuScenes-QA: A Multi-Modal Visual Question Answering Benchmark for Autonomous Driving Scenario. In: AAAI Conference on Artificial Intelligence. vol. 38, pp. 4542–4550 (2024)

    2024

  52. [52]

    In: International Conference on Machine Learning (ICML)

    Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., Krueger, G., Sutskever, I.: Learning Transferable Visual Models from Natural Language Supervision. In: International Conference on Machine Learning (ICML). PMLR (2021)

    2021

  53. [53]

    In: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

    Renz, K., Chen, L., Arani, E., Sinavski, O.: SimLingo: Vision-Only Closed-Loop Autonomous Driving with Language-Action Alignment. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 11993–12003 (2025)

    2025

  54. [54]

    In: 30th USENIX Security Symposium (USENIX Security) (2021)

    Sato, T., Shen, J., Wang, N., Jia, Y., Lin, X., Chen, Q.A.: Dirty Road Can Attack: SecurityofDeepLearningBasedAutomatedLaneCenteringUnderPhysical-World Attack. In: 30th USENIX Security Symposium (USENIX Security) (2021)

    2021

  55. [55]

    In: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2024)

    Shao, H., Hu, Y., Wang, L., Song, G., Waslander, S.L., Liu, Y., Li, H.: LMDrive: Closed-Loop End-to-End Driving with Large Language Models. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2024)

    2024

  56. [56]

    In: European Conference on Computer Vision (ECCV)

    Sima, C., Renz, K., Chitta, K., Chen, L., Zhang, H., Xie, C., Beißwenger, J., Luo, P., Geiger, A., Li, H.: DriveLM: Driving with Graph Visual Question Answering. In: European Conference on Computer Vision (ECCV). Springer (2024) BEVLM 19

    2024

  57. [57]

    Tian, X., Gu, J., Li, B., Liu, Y., Wang, Y., Zhao, Z., Zhan, K., Jia, P., Lang, X., Zhao, H.: DriveVLM: The Convergence of Autonomous Driving and Large Vision- LanguageModels.In:ConferenceonRobotLearning.pp.4698–4726.PMLR(2024)

    2024

  58. [58]

    In: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

    Vedantam, R., Zitnick, C.L., Parikh, D.: CIDEr: Consensus-Based Image Descrip- tion Evaluation. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 4566–4575 (2015)

    2015

  59. [59]

    In: ISOC Network and Distributed Systems Security (NDSS) Symposium (2022)

    Wan,Z.,Shen,J.,Chuang,J.,Xia,X.,Garcia,J.,Ma,J.,Chen,Q.A.:TooAfraidto Drive: Systematic Discovery of Semantic DoS Vulnerability in Autonomous Driv- ing Planning Under Physical-World Attacks. In: ISOC Network and Distributed Systems Security (NDSS) Symposium (2022)

    2022

  60. [60]

    In: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2025)

    Wang, S., Yu, Z., Jiang, X., Lan, S., Shi, M., Chang, N., Kautz, J., Li, Y., Alvarez, J.M.: OmniDrive: A Holistic Vision-Language Dataset for Autonomous Driving with Counterfactual Reasoning. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2025)

    2025

  61. [61]

    arXiv preprint arXiv:2511.00088 (2025)

    Wang, Y., Luo, W., Bai, J., Cao, Y., Che, T., Chen, K., Chen, Y., Diamond, J., Ding, Y., Ding, W., et al.: Alpamayo-R1: Bridging Reasoning and Action Pre- diction for Generalizable Autonomous Driving in the Long Tail. arXiv preprint arXiv:2511.00088 (2025)

    Pith/arXiv arXiv 2025

  62. [62]

    arXiv Preprint arXiv:2503.03074 (2025)

    Winter, K., Azer, M., Flohr, F.B.: BEVDriver: Leveraging BEV Maps in LLMs for Robust Closed-Loop Driving. arXiv Preprint arXiv:2503.03074 (2025)

    Pith/arXiv arXiv 2025

  63. [63]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2020)

    Wu, P., Chen, S., Metaxas, D.N.: MotionNet: Joint Perception and Motion Predic- tion for Autonomous Driving Based on Bird’s Eye View Maps. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2020)

    2020

  64. [64]

    In: IEEE/CVF International Conference on Computer Vision (ICCV) (October 2025)

    Xie, S., Kong, L., Dong, Y., Sima, C., Zhang, W., Chen, Q.A., Liu, Z., Pan, L.: Are VLMs Ready for Autonomous Driving? An Empirical Study from the Relia- bility, Data, and Metric Perspectives. In: IEEE/CVF International Conference on Computer Vision (ICCV) (October 2025)

    2025

  65. [65]

    IEEE Transactions on Pattern Analysis and Machine Intelligence47(5) (2025)

    Xie, S., Kong, L., Zhang, W., Ren, J., Pan, L., Chen, K., Liu, Z.: Benchmarking and Improving Bird’s Eye View Perception Robustness in Autonomous Driving. IEEE Transactions on Pattern Analysis and Machine Intelligence47(5) (2025). https://doi.org/10.1109/TPAMI.2025.3535960

    work page doi:10.1109/tpami.2025.3535960 2025

  66. [66]

    Transactions on Machine Learning Research (2024)

    Xie, S., Li, Z., Wang, Z., Xie, C.: On the Adversarial Robustness of Camera-Based 3D Object Detection. Transactions on Machine Learning Research (2024)

    2024

  67. [67]

    In: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2025)

    Xie, Y., Xu, R., He, T., Hwang, J.J., Luo, K., Ji, J., Lin, H., Chen, L., Lu, Y., Leng, Z., et al.: S4-Driver: Scalable Self-Supervised Driving Multimodal Large Language Model with Spatio-Temporal Visual Representation. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2025)

    2025

  68. [68]

    arXiv preprint arXiv:2412.14446 (2024)

    Xu, Y., Hu, Y., Zhang, Z., Meyer, G.P., Mustikovela, S.K., Srinivasa, S., Wolff, E.M., Huang, X.: VLM-AD: End-to-End Autonomous Driving Through Vision- Language Model Supervision. arXiv preprint arXiv:2412.14446 (2024)

    Pith/arXiv arXiv 2024

  69. [69]

    IEEE Robotics and Automation Letters9(10) (2024)

    Xu, Z., Zhang, Y., Xie, E., Zhao, Z., Guo, Y., Wong, K.Y.K., Li, Z., Zhao, H.: DriveGPT4: Interpretable End-to-End Autonomous Driving via Large Language Model. IEEE Robotics and Automation Letters9(10) (2024)

    2024

  70. [70]

    arXiv Preprint arXiv:2505.09388 (2025)

    Yang, A., Li, A., Yang, B., Zhang, B., Hui, B., Zheng, B., Yu, B., Gao, C., Huang, C., Lv, C., et al.: Qwen3 Technical Report. arXiv Preprint arXiv:2505.09388 (2025)

    Pith/arXiv arXiv 2025

  71. [71]

    Monninger, S

    Yang, A., Yang, B., Zhang, B., Hui, B., Zheng, B., Yu, B., Li, C., Liu, D., Huang, F., Wei, H., Lin, H., Yang, J., Tu, J., Zhang, J., Yang, J., Yang, J., Zhou, J., Lin, J., Dang, K., Lu, K., Bao, K., Yang, K., Yu, L., Li, M., Xue, M., Zhang, P., Zhu, Q., Men, R., Lin, R., Li, T., Tang, T., Xia, T., Ren, X., Ren, X., Fan, Y., Su, 20 T. Monninger, S. Xie, Q...

    Pith/arXiv arXiv 2025

  72. [72]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2024)

    Yang, H., Zhang, S., Huang, D., Wu, X., Zhu, H., He, T., Tang, S., Zhao, H., Qiu, Q., Lin, B., et al.: UniPAD: A Universal Pre-Training Paradigm for Autonomous Driving. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2024)

    2024

  73. [73]

    arXiv preprint arXiv:2311.01043 (2023)

    Yang, Z., Jia, X., Li, H., Yan, J.: LLM4Drive: A Survey of Large Language Models for Autonomous Driving. arXiv preprint arXiv:2311.01043 (2023)

    Pith/arXiv arXiv 2023

  74. [74]

    arXiv preprint arXiv:2505.17685 (2025)

    Zeng, S., Chang, X., Xie, M., Liu, X., Bai, Y., Pan, Z., Xu, M., Wei, X., Guo, N.: FutureSightDrive: Thinking Visually with Spatio-Temporal COT for Autonomous Driving. arXiv preprint arXiv:2505.17685 (2025)

    Pith/arXiv arXiv 2025

  75. [75]

    In: IEEE/CVF International Conference on Computer Vision (ICCV) (2023)

    Zhai, X., Mustafa, B., Kolesnikov, A., Beyer, L.: Sigmoid Loss for Language Im- age Pre-Training. In: IEEE/CVF International Conference on Computer Vision (ICCV) (2023)

    2023

  76. [76]

    arXiv Preprint arXiv:2205.09743 (2022)

    Zhang, Y., Zhu, Z., Zheng, W., Huang, J., Huang, G., Zhou, J., Lu, J.: BEV- erse: Unified Perception and Prediction in Bird’s-Eye-View for Vision-Centric Au- tonomous Driving. arXiv Preprint arXiv:2205.09743 (2022)

    Pith/arXiv arXiv 2022

  77. [77]

    In: IEEE/CVF International Conference on Com- puter Vision (ICCV) (October 2025)

    Zhou, X., Liang, D., Tu, S., Chen, X., Ding, Y., Zhang, D., Tan, F., Zhao, H., Bai, X.: HERMES: A Unified Self-Driving World Model for Simultaneous 3D Scene Understanding and Generation. In: IEEE/CVF International Conference on Com- puter Vision (ICCV) (October 2025)

    2025

  78. [78]

    arXiv preprint arXiv:2503.23463 (2025)

    Zhou, X., Han, X., Yang, F., Ma, Y., Knoll, A.C.: OpenDriveVLA: Towards End- to-End Autonomous Driving with Large Vision Language Action Model. arXiv preprint arXiv:2503.23463 (2025)

    arXiv 2025

  79. [79]

    arXiv preprint arXiv:2506.13757 (2025)

    Zhou, Z., Cai, T., Zhao, S.Z., Zhang, Y., Huang, Z., Zhou, B., Ma, J.: AutoVLA: A Vision-Language-Action Model for End-to-End Autonomous Driving with Adap- tive Reasoning and Reinforcement Fine-Tuning. arXiv preprint arXiv:2506.13757 (2025)

    Pith/arXiv arXiv 2025

  80. [80]

    In: The Twelfth International Conference on Learning Representations (ICLR) (2024)

    Zhu, D., Chen, J., Shen, X., Li, X., Elhoseiny, M.: MiniGPT-4: Enhancing Vision- Language Understanding with Advanced Large Language Models. In: The Twelfth International Conference on Learning Representations (ICLR) (2024)

    2024

Showing first 80 references.