DriVLM: Domain Adaptation of Vision-Language Models in Autonomous Driving

Chang D. Yoo; Xuran Zheng

REVIEW 4 major objections 4 minor 18 references

DriVLM: Domain Adaptation of Vision-Language Models in Autonomous Driving

T0 review · 4 major / 4 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A 2B-parameter vision-language model, adapted to autonomous driving purely by reformatting tasks as visual questions, achieves a 0.191 CIDEr score on the DriveLM benchmark.

desk verdict Applies someone else's open-source model to DriveLM without citing it, and the single reported number is unverifiable as written; a desk reject rather than a referee assignment. read the letter →

arxiv 2501.05081 v1 pith:ZQJG4NCS submitted 2025-01-09 cs.LG

Xuran Zheng , Chang D. Yoo This is my paper · ORCID

classification cs.LG

keywords domainadaptationvision-languagemodelssmall-scalemultimodalLLMsautonomousdrivingvisualquestionansweringinstructiontuningDriveLMCIDErevaluation

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to show that a small-scale multimodal language model can be moved into autonomous driving without heavy architectural surgery, by re-expressing perception, prediction, and planning tasks as visual question answering and fine-tuning all parameters. It reports that the 2B-parameter Mini-InternVL variant scores 0.191 CIDEr, 0.763 accuracy, and 0.762 BLEU-1 on the DriveLM-nuScenes 1.1 validation set, a result the paper takes as evidence of successful domain adaptation. The 1B variant also adapts, reaching 0.167 CIDEr. The practical stake is that researchers and developers with limited compute can still apply multimodal language models to driving, provided the base model has a strong small vision encoder and a unified prompt-level transfer format.

What carries the argument

The machinery is the VQA-style instruction-tuning format combined with the Mini-InternVL architecture. Classification becomes multiple-choice questions; localization uses <ref></ref> name tags and <box></box> coordinate tags; region-aware tasks put a box around the target region; and multi-view driving images are resized into 448x448 tiles, given a thumbnail for global context, and labeled with camera positions. The model itself is a compact vision-language stack whose 300M vision encoder is initialized from a web-scale image-text encoder and improved by distillation, connected through an MLP projector to a small language model, then fully fine-tuned on a mix of driving and general multimodal data. This unified reformatting is what carries the domain transfer, because it lets one training procedure handle perception, prediction, and planning without per-task architecture changes.

What would settle it

A reader could settle the claim by fine-tuning Mini-InternVL-2B on DriveLM in three variants: with the full VQA reformatting, with plain caption-style prompts, and with no general-data mixing; if the 0.191 CIDEr changes by a large margin across variants, the reported score is an artifact of undocumented recipe choices rather than evidence for the unified transfer scheme.

Watch

Extended reading notes

Core claim

On its own terms, the paper's central claim is that Mini-InternVL, a family of 1B, 2B, and 4B parameter multimodal models built from an InternViT-300M encoder, an MLP projector, and small pre-trained language backbones, transfers to autonomous driving through a single recipe: reformat every driving task into VQA prompts, give the model multi-view images with camera labels and dynamic resolution, and fully fine-tune with a balanced mix of general multimodal data. Evaluated on the DriveLM-nuScenes 1.1 validation split, the 2B model scores 0.763 accuracy, 0.762 BLEU-1, and 0.191 CIDEr, matching the 0.18 CIDEr of the listed MTMM entry and exceeding the accuracy of the listed MMFM_AD entry; the 1B variant reaches 0.167 CIDEr. The paper takes this as evidence that small-scale MLLMs can be competitive in a specialized domain when equipped with a strong small vision encoder and a unified instruction-tuning format.

Load-bearing premise

The load-bearing premise is that re-expressing all driving tasks as VQA prompts and fine-tuning with an unstated balanced mix of general and driving data is sufficient for domain transfer: nothing in the paper isolates the effect of the prompt format, the data ratio, or the base checkpoint.

Editorial extensions

If this is right

The reported CIDEr of 0.191 for the 2B model tops the 0.18 CIDEr of the listed MTMM entry, suggesting a small model can match or exceed a specialized baseline on DriveLM.
The 1B variant also adapts successfully with 0.701 accuracy, 0.736 BLEU-1, and 0.167 CIDEr, indicating the recipe is not tied to one model size.
Because the transfer scheme is just prompt reformatting plus full fine-tuning, the same training run can cover perception, prediction, and planning without changing the model architecture.
The small scale and open-source framing lower the compute barrier to applying multimodal language models in autonomous driving research.
Mixing general multimodal data with driving data during fine-tuning is presented as necessary to preserve generalization while gaining domain-specific performance.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

My extension: hold the 2B checkpoint fixed and vary only the general-to-driving data ratio; a wide plateau of ratios reproducing 0.191 CIDEr would show the recipe is robust, while a sharp peak would mean the score depends on an undocumented setting.
My extension: the same VQA reformatting could be carried to other ego-centric domains such as robotics manipulation or navigation, where a single prompt scheme would avoid per-task architectural work.
My extension: if the scaling trend holds, the 4B Mini-InternVL variant, which the paper does not evaluate on DriveLM, should push CIDEr above 0.191 under the same fine-tuning setup; checking this would test the generality of the claimed adaptation.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Quick take: this is a thin workshop write-up that takes the existing Mini-InternVL model (OpenGVLab's work, uncited) and fine-tunes it on DriveLM. The only new numbers are two rows in Table 1. The rest of the paper re-describes Mini-InternVL's architecture and two-stage training as if it were the authors' own contribution. The conclusion literally says 'we introduce Mini-InternVL.' That is a provenance problem, not a style problem.

What's worth keeping: the paper does call attention to a genuinely useful data point — a 2B-parameter model can reach CIDEr 0.191 on DriveLM-nuScenes-val with a simple VQA-reformatting recipe, which is cheap compared to larger driving MLLMs. If the recipe were fully specified and the base checkpoint clearly identified, this would be a useful benchmark entry for the small-MLLM-for-driving crowd.

Where it falls apart: reproducibility. Section 3.3 gives exactly one table row per model, with no standard deviations, no training hyperparameters, no data split details, and no code. Section 3.2 mentions mixing 'a balanced proportion of general multimodal data' but doesn't say what that proportion is. Without that ratio, without the prompt templates, and without a zero-shot run of the original Mini-InternVL checkpoint on DriveLM-val, the 0.191 CIDEr is not interpretable as a domain-adaptation result — it could be mostly the base model's prior knowledge. The paper's own limitation sentence about 'constraints of the training data and evaluation metrics' is honest but doesn't fix the missing details.

The comparison table is also shaky: MTMM and Team NVIDIA numbers are taken from other papers with no protocol alignment, and the paper doesn't say whether its evaluation matches theirs.

Net: the central claim ('excellent result') is unsupported as written. The paper is not ready for peer review in its current form; a serious editor would desk-reject it on the grounds that the method is prior art and the single new number is unverifiable. A revision could add the missing details, cite Mini-InternVL properly, and rename the model to DriVLM consistently — then it might be a legitimate workshop contribution. As is, I wouldn't cite it and wouldn't send it to referees.

Referee Report

4 major / 4 minor

Summary. Section 3.3 reports that a 2B-parameter Mini-InternVL model fine-tuned on DriveLM-nuScenes with a VQA reformatting and a mixture of general multimodal data achieves CIDEr 0.191 on DriveLM-nuScenes-val, which the authors call an excellent domain adaptation result. The paper describes the Mini-InternVL architecture, a general recipe for reformatting tasks into VQA format, and a table comparing the adapted models with MTMM, Team NVIDIA, and MMFM_AD. The title announces DriVLM, but the body and conclusion consistently refer to 'Mini-InternVL' as the introduced model.

Significance. If substantiated, the result would be notable: a small-scale open MLLM, adapted with a simple data-reformatting recipe, performing competitively on DriveLM would support the viability of low-resource domain adaptation for autonomous driving. However, the manuscript does not provide the evidence needed: no zero-shot baseline, no training hyperparameters, no mixing ratio, no ablations, no code or checkpoints, and no details of the evaluation split or metric computation. The paper also appears to claim credit for Mini-InternVL, an existing model. As submitted, the contribution cannot be independently verified, so the significance of the central claim is currently unestablished.

major comments (4)

[Section 3.2] The domain-adaptation recipe is not specified: the text says 'integrating a balanced proportion of general multimodal data' but never gives the ratio or a procedure for choosing it, and no ablation over this mixture is reported. Because the amount of general data mixed into the DriveLM fine-tuning is a free parameter on which the claimed generalization depends, CIDEr 0.191 is not reproducible and cannot be attributed to the method.
[Section 3.3] The central result is a single table row with no standard deviations, no number of runs, no training hyperparameters, no checkpoint selection rule, and no description of the evaluation protocol (e.g., which metric implementation, prompt template, or generation settings). More importantly, no zero-shot evaluation of the original Mini-InternVL-2B checkpoint on DriveLM-nuScenes-val is reported, so the reader cannot tell whether fine-tuning added anything over the base model's prior knowledge.
[Section 3.3] The comparison in Table 1 does not support the label 'excellent.' The prior systems MTMM, Team NVIDIA, and MMFM_AD are listed without citations or descriptions, metrics are missing for some rows, and there is no statement that all entries use the same validation split, prompt format, and evaluation code. Without this context, the 0.191 CIDEr score is not interpretable as a competitive result.
[Title and Section 4] The manuscript's model identity is inconsistent: the title introduces DriVLM, but the body (Section 3.1) and conclusion say 'we introduce Mini-InternVL,' which is an existing open-source model family. The paper never defines what DriVLM is or how it differs from Mini-InternVL, making it unclear what the actual contribution is. This is a load-bearing issue because the central claim is about the adapted model's performance.

minor comments (4)

[References] References [1] and [14] are the same CLIP citation; [14] should be removed or replaced with the intended source.
[References] Several reference entries contain stray page numbers (e.g., '[7] ... 1, 3, 8' and '[15] ... 2, 3'), apparently citation-manager artifacts.
[Abstract] The abstract is generic and does not mention the experimental setup, the DriveLM dataset, or the reported CIDEr score, making it impossible to judge the paper's contribution from the abstract.
[Section 3.2] Section 3.2 never gives a concrete example of the VQA prompt used for DriveLM, such as the exact formatting of camera labels or the question templates; adding one would materially improve reproducibility.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the reported CIDEr score is an external benchmark evaluation, not a fitted or self-referential quantity.

full rationale

The paper's central quantitative result is empirical: fine-tuning Mini-InternVL on DriveLM-nuScenes 1.1 and testing on DriveLM-nuScenes-version-1.1-val yields CIDEr 0.191 (Section 3.3). This is not a derivation chain, and the CIDEr score is computed with an external evaluation metric on a benchmark split, so it does not reduce by construction to the training data or to any defined quantity. No parameter is fitted to the evaluation set and renamed a prediction, and no load-bearing claim is justified by a self-citation. The main weaknesses are experimental and reproducibility-related: Section 3.2 specifies only that a 'balanced proportion of general multimodal data' is mixed, without giving the ratio; no zero-shot baseline is reported; and Section 3.3's limitation that 'the constraints of the training data and evaluation metrics could impact the potential of larger-scale models' is an honest caveat. These concerns affect whether the result is convincing, but they do not make the argument circular. I find no circular step.

Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

The empirical result is entirely dependent on unreported training choices and for the most part on an undocumented base model.

free parameters (2)

general-to-domain data mixing ratio = unspecified
Section 3.2 says a balanced proportion is integrated, but no exact value or search is reported; the result depends on this choice.
dynamic resolution tile size = 448x448 pixels
Section 3.3 states images are divided into 448x448 tiles, but no ablation is given for this hand-chosen resolution.

assumptions (3)

domain assumption DriveLM-nuScenes 1.1 is a valid and representative benchmark for autonomous driving VQA.
Section 3.3 uses DriveLM-val as the only evaluation without external cross-checks.
domain assumption Mini-InternVL is a suitable base model for the target domain.
Section 3.1 adopts Mini-InternVL without motivating why it transfers to driving beyond generic claims.
ad hoc to paper Full-parameter fine-tuning with a mixture of general and domain data preserves generalization.
This is claimed in Section 3.2 but not tested; it is a stated recipe that the paper does not derive or ablate.

how reviews work

0 comments

Cite this review

Pith. "Pith review of DriVLM: Domain Adaptation of Vision-Language Models in Autonomous Driving." pith.science (2026). https://pith.science/paper/ZQJG4NCS

@misc{pith2026250105081,
  author       = {Pith},
  title        = {Pith review of: DriVLM: Domain Adaptation of Vision-Language Models in Autonomous Driving},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ZQJG4NCS}},
  note         = {Machine review of arXiv:2501.05081}
}

read the original abstract

In recent years, large language models have had a very impressive performance, which largely contributed to the development and application of artificial intelligence, and the parameters and performance of the models are still growing rapidly. In particular, multimodal large language models (MLLM) can combine multiple modalities such as pictures, videos, sounds, texts, etc., and have great potential in various tasks. However, most MLLMs require very high computational resources, which is a major challenge for most researchers and developers. In this paper, we explored the utility of small-scale MLLMs and applied small-scale MLLMs to the field of autonomous driving. We hope that this will advance the application of MLLMs in real-world scenarios.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

18 extracted references · 9 canonical work pages

[1]

Learning transferable visual models f rom natural language supervision

Alec Radford, Jong Wook Kim, Chris Hallacy, A ditya Ramesh, Gabriel Goh, Sandhini Agarwal, Gi rish Sastry, Amanda Askell, Pamela Mishkin, and Jack Clark. Learning transferable visual models f rom natural language supervision. In ICML, 2021

work page 2021
[2]

Bert: Pre- training of deep bidirectio nal transformers for language understanding

Jacob Devlin. Bert: Pre- training of deep bidirectio nal transformers for language understanding. arXi v preprint arXiv:1810.04805, 2018

arXiv 2018
[3]

Qwen 2 technical report

An Yang, Baosong Yang, Binyuan Hui, Bo Zhen g, Bowen Yu, Chang Zhou, Chengpeng Li, Chen gyuan Li, Dayiheng Liu, Fei Huang, et al. Qwen 2 technical report. arXiv preprint arXiv:2407.1067 1, 2024

work page arXiv 2024
[4]

Internlm2 technical report

Zheng Cai, Maosong Cao, Haojiong Chen, Kai C hen, Keyu Chen, Xin Chen, Xun Chen, Zehui Ch en, Zhi Chen, Pei Chu, et al. Internlm2 technical report. arXiv preprint arXiv:2403.17297, 2024

arXiv 2024
[5]

Mmben ch: Is your multi- modal model an all -around play er? arXiv preprint arXiv:2307.06281, 2023

Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo L i, Songyang Zhang, Wangbo Zhao, Yike Yuan, Ji aqi Wang, Conghui He, Ziwei Liu, et al. Mmben ch: Is your multi- modal model an all -around play er? arXiv preprint arXiv:2307.06281, 2023

arXiv 2023
[6]

Chartqa: A bench mark for question answering about charts with vi sual and logical reasoning

Ahmed Masry, Xuan Long Do, Jia Qing Tan, Sh afiq Joty, and Enamul Hoque. Chartqa: A bench mark for question answering about charts with vi sual and logical reasoning. In ACL, pages 2263– 2279, 2022

work page 2022
[7]

Flamingo: a visual language model for few- shot learning

Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcol m Reynolds, et al. Flamingo: a visual language model for few- shot learning. NeurIPS, 35:23716– 2 3736, 2022. 1, 3, 8

work page 2022
[8]

Minigpt -4: Enhancing vi sion-language understanding with advanced large l anguage models

Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt -4: Enhancing vi sion-language understanding with advanced large l anguage models. arXiv preprint arXiv:2304.10592,

arXiv

Show all 18 references

[9]

Visionllm: Large language model is also an open -ended decoder fo r vision -centric tasks

Wenhai Wang, Zhe Chen, Xiaokang Chen, Jianna n Wu, Xizhou Zhu, Gang Zeng, Ping Luo, Tong Lu, Jie Zhou, Yu Qiao, et al. Visionllm: Large language model is also an open -ended decoder fo r vision -centric tasks. NeurIPS, 2023. 1, 3

2023
[10]

Palm -e: An embodied multimodal language model

Danny Driess, Fei Xia, Mehdi SM Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayz aan Wahid, Jonathan Tompson, Quan Vuong, Tia nhe Yu, et al. Palm -e: An embodied multimodal language model. arXiv preprint arXiv:2303.03378,

arXiv
[11]

Sigmoid loss for language i mage pre- training

Xiaohua Zhai, Basil Mustafa, Alexander Kolesnik ov, and Lucas Beyer. Sigmoid loss for language i mage pre- training. In ICCV, pages 11975 –11986, 2023

2023
[12]

Moma: Effici ent early -fusion pre -training with mixture of moda lity-aware experts

Xi Victoria Lin, Akshat Shrivastava, Liang Luo, Srinivasan Iyer, Mike Lewis, Gargi Gosh, Luke Z ettlemoyer, and Armen Aghajanyan. Moma: Effici ent early -fusion pre -training with mixture of moda lity-aware experts. arXiv preprint arXiv:2407.2177 0, 2024

2024
[13]

Minicpm -v: A g pt-4v level mllm on your phone

Yuan Yao, Tianyu Yu, Ao Zhang, Chongyi Wan g, Junbo Cui, Hongji Zhu, Tianchi Cai, Haoyu Li, Weilin Zhao, Zhihui He, et al. Minicpm -v: A g pt-4v level mllm on your phone. arXiv preprint a rXiv:2408.01800, 2024

2024 arXiv
[14]

Learning transferable visual models f rom natural language supervision

Alec Radford, Jong Wook Kim, Chris Hallacy, A ditya Ramesh, Gabriel Goh, Sandhini Agarwal, Gi rish Sastry, Amanda Askell, Pam ela Mishkin, and Jack Clark. Learning transferable visual models f rom natural language supervision. In ICML, 2021

2021
[15]

Llama: Open and effici ent foundation language models

Hugo Touvron, Thibaut Lavril, Gautier Izacard, X avier Martinet, Marie -Anne Lachaux, Timoth ́ee L acroix, Baptiste Rozi`ere, Naman Goyal, Eric Ha mbro, Faisal Azhar, et al. Llama: Open and effici ent foundation language models. arXiv preprint ar Xiv:2302.13971, 2023. 2, 3

2023 arXiv
[16]

Danish, Muzamm al Naseer, Abhijit Das, Salman Khan, and Fahad S

Kartik Kuckreja, Muhammad S. Danish, Muzamm al Naseer, Abhijit Das, Salman Khan, and Fahad S. Khan. Geochat: Grounded large vision -language model for remote sensing. CVPR, 2024

2024
[17]

Llava - med: Training a large language -and-vision assistan t for biomedicine in one day

Chunyuan Li, Cliff Wong, Sheng Zhang, Naoto Usuyama, Haotian Liu, Jianwei Yang, Tristan Na umann, Hoifung Poon, and Jianfeng Gao. Llava - med: Training a large language -and-vision assistan t for biomedicine in one day. arXiv preprint arXi v:2306.00890, 2023

2023 arXiv
[18]

Wong, Zhenguo Li, a nd Hengshuang Zhao

Zhenhua Xu, Yujia Zhang, Enze Xie, Zhen Zhao, Yong Guo, Kwan -Yee K. Wong, Zhenguo Li, a nd Hengshuang Zhao. Drivegpt4: Interpretable end -to-end autonomous driving via large language m odel. arXiv preprint arXiv:2310.01412, 2023

arXiv

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.

[1] [1]

Learning transferable visual models f rom natural language supervision

Alec Radford, Jong Wook Kim, Chris Hallacy, A ditya Ramesh, Gabriel Goh, Sandhini Agarwal, Gi rish Sastry, Amanda Askell, Pamela Mishkin, and Jack Clark. Learning transferable visual models f rom natural language supervision. In ICML, 2021

work page 2021

[2] [2]

Bert: Pre- training of deep bidirectio nal transformers for language understanding

Jacob Devlin. Bert: Pre- training of deep bidirectio nal transformers for language understanding. arXi v preprint arXiv:1810.04805, 2018

arXiv 2018

[3] [3]

Qwen 2 technical report

An Yang, Baosong Yang, Binyuan Hui, Bo Zhen g, Bowen Yu, Chang Zhou, Chengpeng Li, Chen gyuan Li, Dayiheng Liu, Fei Huang, et al. Qwen 2 technical report. arXiv preprint arXiv:2407.1067 1, 2024

work page arXiv 2024

[4] [4]

Internlm2 technical report

Zheng Cai, Maosong Cao, Haojiong Chen, Kai C hen, Keyu Chen, Xin Chen, Xun Chen, Zehui Ch en, Zhi Chen, Pei Chu, et al. Internlm2 technical report. arXiv preprint arXiv:2403.17297, 2024

arXiv 2024

[5] [5]

Mmben ch: Is your multi- modal model an all -around play er? arXiv preprint arXiv:2307.06281, 2023

Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo L i, Songyang Zhang, Wangbo Zhao, Yike Yuan, Ji aqi Wang, Conghui He, Ziwei Liu, et al. Mmben ch: Is your multi- modal model an all -around play er? arXiv preprint arXiv:2307.06281, 2023

arXiv 2023

[6] [6]

Chartqa: A bench mark for question answering about charts with vi sual and logical reasoning

Ahmed Masry, Xuan Long Do, Jia Qing Tan, Sh afiq Joty, and Enamul Hoque. Chartqa: A bench mark for question answering about charts with vi sual and logical reasoning. In ACL, pages 2263– 2279, 2022

work page 2022

[7] [7]

Flamingo: a visual language model for few- shot learning

Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcol m Reynolds, et al. Flamingo: a visual language model for few- shot learning. NeurIPS, 35:23716– 2 3736, 2022. 1, 3, 8

work page 2022

[8] [8]

Minigpt -4: Enhancing vi sion-language understanding with advanced large l anguage models

Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt -4: Enhancing vi sion-language understanding with advanced large l anguage models. arXiv preprint arXiv:2304.10592,

arXiv