REVIEW 4 major objections 5 minor 3 cited by
X-Driver: Explainable Autonomous Driving with Vision-Language Models
T0 review · 4 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash
Pith's one-line read A vision-language model that reasons step by step outperforms prior closed-loop driving systems while explaining its decisions.
desk verdict A plausible but under-evidenced LLaVA+CoT driving pipeline claims SOTA on Bench2Drive, yet the table only compares against UniAD with mismatched training frames and no error bars. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The core mechanism is a LLaVA-style multimodal language model—a ViT image encoder aligned with a large language model—fine-tuned so that perception, prediction, and control are generated as one autoregressive sequence. Rather than tokenizing images with VQ-VAE, the paper feeds the model a continuous low-dimensional ViT feature map, arguing that this preserves small distant objects such as traffic lights. Chain-of-thought prompting decomposes the driving task into four subtasks (object detection, traffic light status, traffic sign recognition, lane detection), and the model outputs intermediate reasoning tokens before emitting waypoints and a driving command. The autoregressive factorization, $P(Y \mid (T, I)) = \prod_{t} P(y_t \mid y_{<t}, (T, I))$, lets all of these outputs share one probability model, which is what makes the reasoning and the decision jointly trainable and jointly interpretable.
What would settle it
Run X-Driver on the same Bench2Drive routes with a VQ-VAE-discrete encoding of the image stream while keeping all chain-of-thought prompts and training data identical; if success on routes that require stopping at distant red lights does not drop, the paper's claimed information-preservation mechanism is not load-bearing.
Extended reading notes
Core claim
The paper's central claim is that adding chain-of-thought supervision to a vision-language model yields a closed-loop driving policy that outperforms the current SOTA, UniAD, on the Bench2Drive benchmark while remaining explainable. In the closed-loop experiments, the CoT version achieves a driving score of 51.7 and a success rate of 18.1% on the 220-scenario split, and 57.8 and 24.0% on the 50-scenario split, compared with UniAD's 45.9/17.5% and 49.1/22.0%. In open-loop tests, it reports a 3D object-recognition IoU of 0.724 at sample level and 0.806 at box level, and waypoint errors of ADE 0.679 m at 0.5 s rising to 2.472 m at 3 s. The interpretation is that explicit step-by-step reasoning mitigates the hallucinations that otherwise cause MLLM drivers to misread scenes or miss hazards.
Load-bearing premise
The claimed closed-loop advantage rests on the untested assumption that a continuous ViT image encoder preserves distant traffic-light information that a VQ-VAE encoder would lose, so the chain-of-thought reasoning can still detect red lights far ahead.
Editorial extensions
If this is right
- On the 220-route Bench2Drive split, the CoT system scores 51.7 on the driving score and 18.1% success, ahead of UniAD's 45.9 and 17.5%.
- On the 50-route split, it reaches 57.8 and 24.0%, against UniAD's 49.1 and 22.0%, so the improvement persists when the evaluation is shorter.
- Because the model emits intermediate reasoning tokens, every braking or turning decision can be inspected as a chain of perceived objects, traffic-light states, signs, and lanes before the final waypoint.
- The autoregressive formulation means perception, prediction, and control share a single sequence, so the same framework can be extended to other instruction-following tasks without changing the model architecture.
Reading between the lines
- The headline comparison trains X-Driver on 3 million frames and UniAD on 2 million, so part of the margin may be attributable to data scale rather than the CoT mechanism; a same-data ablation would isolate the reasoning contribution.
- If the continuous-encoding claim survives controlled tests, it suggests a design rule for MLLM driving systems: avoid discrete visual tokens whenever distant small objects such as traffic lights or pedestrians are safety-critical.
- The four-subtask decomposition creates a natural error audit: log each reasoning stage and see whether collisions trace to perception failures, rule-misreading, or trajectory errors, which could target future data collection.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper presents X-Driver, a LLaVA-based vision-language model for closed-loop autonomous driving that uses chain-of-thought (CoT) reasoning to decompose driving into perception, prediction, and decision subtasks. The method employs a continuous ViT image encoder, autoregressive multimodal fusion, and supervised fine-tuning with CoT prompts. The authors report open-loop object detection and waypoint prediction results on Bench2Drive, and closed-loop CARLA results in which they claim to surpass the current state of the art (UniAD). The paper argues that CoT provides interpretability and improves closed-loop performance.
Significance. If the closed-loop results were properly supported, X-Driver would be a useful step toward interpretable, MLLM-based closed-loop driving, and the qualitative CoT-vs-auxiliary comparison in Fig. 4 is a suggestive demonstration of the behavioral effect of reasoning. However, the current evidence is insufficient: the SOTA claim rests on a single mismatched baseline, the open-loop waypoint statement is internally inconsistent, and a key encoding assumption is unverified. The paper does not provide code, model weights, or machine-checked proofs; its contribution is primarily an empirical architecture proposal, so the empirical support is the crux.
major comments (4)
- [§4.3, Table 4] The central claim of surpassing current SOTA is supported only by a comparison against UniAD [4], a 2023 non-MLLM planner. Table 4 omits the recent closed-loop MLLM/CoT baselines cited in the related work (LMDrive [11], DriveCoT [15], EMMA [5], CoT-Drive [7]), and it compares with UniAD trained on 2M Bench2Drive frames while X-Driver uses 3M frames. The observed gaps (e.g., driving score 51.7 vs 45.9 on bench2drive220) could therefore be due to data quantity rather than the proposed architecture; no error bars or statistical significance tests are provided. To support the abstract's absolute SOTA claim, the authors should compare against current MLLM/CoT baselines under matched training data and report run-to-run variance.
- [§4.2, Table 3 and text] The claim that the model 'achieves prediction accuracy of less than 1% for future trajectories' is incorrect: the reported ADE values are 0.679 m (0.5s), 0.837 m (1s), 1.128 m (2s), and 1.488 m (3s), and FDE is 2.472 m, none of which are percentages. This statement should be corrected with explicit units and a proper interpretation, preferably with a comparison to existing waypoint predictors, so that the open-loop evaluation supports rather than undermines the paper's claims.
- [§3.2] The paper asserts that continuous ViT encoding preserves distant traffic-light information that VQ-VAE encoding would lose, and that this preservation is important for CoT reasoning ('when a traffic light appears in the distant scene, the use of VQ-VAE encoding can result in the loss of crucial information'). No experiment, ablation, or quantitative evidence supports this assertion. Since the closed-loop performance and the claimed advantage of the architecture depend on this property, please provide a perception-level experiment or an ablation across encoders.
- [§4.3, Table 4] The CoT row in Table 4 is labeled 'Ours(CoT: 3 million frames, auxiliary task version, split into 4 subtasks)', which conflates CoT with an auxiliary-task decomposition. The w/o CoT row is not described as having those auxiliary subtasks, so the comparison may not isolate CoT. To claim that CoT reasoning is the cause of improvement, the authors should clarify the exact training differences between the two variants and ensure that CoT is the only manipulated variable.
minor comments (5)
- [Abstract and Section 1] The abstract and introduction misuse the citation 'multi-modal large language models[2]' (bracket notation) and contain informal phrases such as 'And we already finished closed-loop experiments' in the contributions list; these should be cleaned up.
- [Section 4.2] The sentence 'From this table, we can see that my object recognition method achieved a remarkably high accuracy' uses the first person 'my' and is unmeasured; please rephrase objectively.
- [General presentation] Figure cross-references are incomplete (e.g., 'As illustrated in 1', 'As illustrated in 3'), and some figure captions like 'Fig. 2. examples of Training Process' are truncated or lack periods.
- [Table 3] The column headers of Table 3 are ambiguous: it is unclear which numeric values are ADE and which are FDE under each time horizon; please restructure the table so that each metric has a clear column.
- [Section 4.3] The closed-loop evaluation reports only aggregate Driving Score and Success Rate; a per-route breakdown would help identify where the CoT version gains or loses, especially given the small aggregate differences.
Circularity Check
No circular reasoning found; the derivation is self-contained and evaluation is against an external benchmark.
full rationale
X-Driver's derivation is a standard supervised fine-tuning recipe: a pretrained LLaVA backbone is prompted with chain-of-thought templates, trained on Bench2Drive-derived inputs and labels, and then evaluated closed-loop in CARLA on Bench2Drive metrics. No equation in the paper defines the model's output in terms of the reported result, no fitted parameter is renamed as a prediction, and no load-bearing claim rests on a self-citation by the present authors. The continuous-ViT-versus-VQ-VAE argument in Section 3.2 is an architectural assumption, not a circular step. The closed-loop comparison in Table 4 is limited because it uses only UniAD as a baseline and unequal training-frame counts, but that is an empirical-validity concern about the 'SOTA' claim, not circularity. The paper's claims could be weakened by missing baselines or unverified assumptions, but the derivation chain does not reduce to its own inputs.
Assumptions & free parameters
assumptions (5)
- standard math The autoregressive factorization in Eq. (2) correctly models the decision sequence.
- domain assumption CARLA simulation is a valid proxy for real-world driving performance.
- domain assumption The CoT training data described only as 'meticulously crafted' is of sufficient quality and coverage.
- domain assumption The ViT-based continuous encoder preserves distant traffic light information.
- domain assumption The UniAD baseline scores were obtained under comparable evaluation conditions.
Cite this review
Pith. "Pith review of X-Driver: Explainable Autonomous Driving with Vision-Language Models." pith.science (2026). https://pith.science/paper/P57KCFHA
@misc{pith2026250505098,
author = {Pith},
title = {Pith review of: X-Driver: Explainable Autonomous Driving with Vision-Language Models},
year = {2026},
howpublished = {\url{https://pith.science/paper/P57KCFHA}},
note = {Machine review of arXiv:2505.05098}
}
read the original abstract
End-to-end autonomous driving has advanced significantly, offering benefits such as system simplicity and stronger driving performance in both open-loop and closed-loop settings than conventional pipelines. However, existing frameworks still suffer from low success rates in closed-loop evaluations, highlighting their limitations in real-world deployment. In this paper, we introduce X-Driver, a unified multi-modal large language models(MLLMs) framework designed for closed-loop autonomous driving, leveraging Chain-of-Thought(CoT) and autoregressive modeling to enhance perception and decision-making. We validate X-Driver across multiple autonomous driving tasks using public benchmarks in CARLA simulation environment, including Bench2Drive[6]. Our experimental results demonstrate superior closed-loop performance, surpassing the current state-of-the-art(SOTA) while improving the interpretability of driving decisions. These findings underscore the importance of structured reasoning in end-to-end driving and establish X-Driver as a strong baseline for future research in closed-loop autonomous driving.
Figures
Figures from the paper (1 more)
Forward citations
Cited by 3 Pith papers
-
SpaceDrive: Infusing Spatial Awareness into VLM-based Autonomous Driving
SpaceDrive replaces textual coordinate tokens with shared 3D positional encodings in a VLM driving planner, achieving state-of-the-art open-loop planning on nuScenes and 78.02 Driving Score on Bench2Drive.
-
A Survey of Multi-sensor Fusion Perception for Embodied AI: Background, Methods, Challenges and Prospects
A task-agnostic survey of multi-sensor fusion perception methods for embodied AI, covering multi-modal, multi-agent, time-series, and multimodal large language model fusion.
-
Chain-of-Thought for Autonomous Driving: A Comprehensive Survey and Future Prospects
A survey that classifies chain-of-thought methods for autonomous driving into modular, logical, and reflective pipelines, and proposes three evolutionary stages from direct prompting to reinforcement learning.
Reference graph
Works this paper leans on
-
[4]
In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition
Hu, Y., Yang, J., Chen, L., Li, K., Sima, C., Zhu, X., Chai, S., Du, S., Lin, T., Wang, W., et al.: Planning-oriented autonomous driving. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 17853–17862 (2023)
2023
-
[11]
In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition
Shao, H., Hu, Y., Wang, L., Song, G., Waslander, S.L., Liu, Y., Li, H.: Lmdrive: Closed-loop end-to- end driving with large language models. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 15120–15130 (2024)
work page 2024
-
[15]
arXiv preprint arXiv:2403.16996 (2024)
Wang, T., Xie, E., Chu, R., Li, Z., Luo, P.: Drivecot: Integrating chain-of-thought reasoning with end-to-end driving. arXiv preprint arXiv:2403.16996 (2024)
arXiv 2024
-
[5]
arXiv preprint arXiv:2410.23262 (2024)
Hwang, J.J., Xu, R., Lin, H., Hung, W.C., Ji, J., Choi, K., Huang, D., He, T., Covington, P., Sapp, B., et al.: Emma: End-to-end multimodal model for autonomous driving. arXiv preprint arXiv:2410.23262 (2024)
arXiv 2024
-
[7]
arXiv preprint arXiv:2503.07234 (2025)
Liao, H., Kong, H., Wang, B., Wang, C., Ye, W., He, Z., Xu, C., Li, Z.: Cot-drive: Efficient mo- tion forecasting for autonomous driving with llms and chain-of-thought prompting. arXiv preprint arXiv:2503.07234 (2025)
arXiv 2025
-
[1]
Advances in neural information processing systems35, 23716–23736 (2022)
Alayrac, J.B., Donahue, J., Luc, P., Miech, A., Barr, I., Hasson, Y., Lenc, K., Mensch, A., Millican, K., Reynolds, M., et al.: Flamingo: a visual language model for few-shot learning. Advances in neural information processing systems35, 23716–23736 (2022)
2022
-
[2]
In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision
Cui, C., Ma, Y., Cao, X., Ye, W., Zhou, Y., Liang, K., Chen, J., Lu, J., Yang, Z., Liao, K.D., et al.: A survey on multimodal large language models for autonomous driving. In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. pp. 958–979 (2024)
work page 2024
-
[3]
Advances in neural information processing systems33, 6840–6851 (2020)
Ho, J., Jain, A., Abbeel, P.: Denoising diffusion probabilistic models. Advances in neural information processing systems33, 6840–6851 (2020)
2020
Show all 18 references
-
[6]
arXiv preprint arXiv:2406.03877 (2024)
Jia, X., Yang, Z., Li, Q., Zhang, Z., Yan, J.: Bench2drive: Towards multi-ability benchmarking of closed-loop end-to-end autonomous driving. arXiv preprint arXiv:2406.03877 (2024)
2024 arXiv
-
[8]
In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition
Liu, H., Li, C., Li, Y., Lee, Y.J.: Improved baselines with visual instruction tuning. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 26296–26306 (2024)
2024
-
[9]
arXiv preprint arXiv:2412.02025 (2024)
Luo, X., Ding, F., Song, Y., Zhang, X., Loo, J.: Pkrd-cot: A unified chain-of-thought prompting for multi-modal large language models in autonomous driving. arXiv preprint arXiv:2412.02025 (2024)
2024 arXiv
-
[10]
In: International conference on machine learning
Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al.: Learning transferable visual models from natural language supervision. In: International conference on machine learning. pp. 8748–8763. PmLR (2021)
2021
-
[12]
Advances in neural information processing systems30(2017)
Van Den Oord, A., Vinyals, O., et al.: Neural discrete representation learning. Advances in neural information processing systems30(2017)
2017
-
[13]
Advances in neural information processing systems30(2017)
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, L., Polosukhin, I.: Attention is all you need. Advances in neural information processing systems30(2017)
2017
-
[14]
Ieee Access (2024)
Vemprala, S.H., Bonatti, R., Bucker, A., Kapoor, A.: Chatgpt for robotics: Design principles and model abilities. Ieee Access (2024)
2024
-
[16]
arXiv preprint arXiv:2502.14917 (2025)
Zhao, R., Yuan, Q., Li, J., Hu, H., Li, Y., Zheng, C., Gao, F.: Sce2drivex: A generalized mllm framework for scene-to-drive learning. arXiv preprint arXiv:2502.14917 (2025)
2025 arXiv
-
[17]
arXiv preprint arXiv:2408.11039 (2024)
Zhou, C., Yu, L., Babu, A., Tirumala, K., Yasunaga, M., Shamis, L., Kahn, J., Ma, X., Zettlemoyer, L., Levy, O.: Transfusion: Predict the next token and diffuse images with one multi-modal model. arXiv preprint arXiv:2408.11039 (2024)
2024 arXiv
-
[18]
arXiv preprint arXiv:2402.02205 (2024)
Zhou, X., Knoll, A.C.: Gpt-4v as traffic assistant: an in-depth look at vision language model on complex traffic events. arXiv preprint arXiv:2402.02205 (2024)
2024 arXiv
Reviewed August 15, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.