Pith. sign in

REVIEW 5 major objections 6 minor 31 references

LaVida Drive: Vision-Text Interaction VLM for Autonomous Driving with Token Selection, Recovery and Enhancement

T0 review · 5 major / 6 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read LaVida Drive claims a 168-fold reduction in visual tokens for autonomous-driving VQA while matching or beating prior benchmarks on most metrics.

desk verdict Promising driving VQA idea with a real empirical hook, but the printed token-selection algorithm cannot be run as written, so the headline result is not reproducible. read the letter →

arxiv 2411.12980 v3 pith:TD5NMUP6 submitted 2024-11-20 cs.CV cs.AI

Siwen Jiao , Yangyi Fang , Baoyun Peng , Wangqun Chen , Bharadwaj Veeravalli This is my paper · ORCID
classification cs.CVcs.AI
keywords visualquestionansweringautonomousdrivingvision-languagemodelsquery-awaretokenselectiontemporalmodelingcompressionhigh-resolutionperceptionspatial-temporalenhancement
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

LaVida Drive tackles a bottleneck in vision-language models for autonomous driving: high-resolution multi-view input is necessary for fine-grained questions, but it floods the language model with tokens. The paper's proposal is to keep high-resolution patches for spatial detail, add a lower-resolution video stream for motion, and then use the user's text query to select only the most relevant visual tokens before they reach the language model. A second module recovers context lost in selection by letting selected tokens attend, through cross-attention, to the full image-encoder and video-encoder outputs. On DriveLM and NuScenes-QA the model reaches the best or second-best scores on most metrics while claiming roughly a 168-fold token compression, which is the pay-off: much cheaper inference without giving away accuracy. The central bet is that semantic similarity between CLIP image tokens and text tokens is a reliable guide to which image regions answer the question.

What carries the argument

The mechanism that carries the argument is the pair of modules around which the architecture is built. Query-aware Token Selection computes a token-wise similarity matrix between the MLP-aligned image patch tokens and the CLIP text tokens of the user's question, normalises it over image tokens with a temperature $\tau$, sums the per-text-token relevance scores, mixes in a weighted token-importance term, and keeps the top-k image tokens; this is what produces the dramatic token reduction. Spatial-Temporal Token Enhancement then applies token-wise cross-attention, $\mathrm{Softmax}(QK^{\top}/\sqrt{d_k})V$, with the selected tokens as queries and the full image-encoder output (spatial restoration) and TimeSformer video-encoder output (temporal enhancement) as keys and values, merging the two enhanced representations through an MLP. That second module is the paper's answer to the objection that dropping tokens destroys spatial context: it lets the language model still access scene-wide and motion information without increasing the token count.

What would settle it

On a held-out set of DriveLM examples with bounding boxes around the objects a question asks about, measure whether the patches containing those objects fall in the top-k selected tokens for the model's best configuration; if they frequently do not, while the model still answers correctly, then the query-aware selection is not the mechanism carrying performance.

Watch

Extended reading notes

Core claim

The paper's central claim is that a VQA model for driving can have both high-resolution spatial understanding and temporal awareness without paying the full token cost, by making token pruning query-dependent and then restoring what pruning removed. Concretely, LaVida Drive processes high-resolution patches with a frozen CLIP image encoder, aligns those patch tokens to the question's text tokens with an MLP and a temperature-scaled cosine-similarity matrix, and keeps only the top-k image tokens ranked by summed relevance to the text. Those surviving tokens are enhanced by token-wise cross-attention that uses the full image-encoder output for spatial restoration and a TimeSformer video-encoder output for temporal enhancement, while the LLM (T5-medium) receives only the compact, enhanced token set plus the query text. In the paper's experiments on DriveLM, this configuration beats the baselines it is compared with on METEOR, ROUGE-L, and CIDEr, and comes second on BLEU-4; on NuScenes-QA it is best on Object, Status, and Comparison categories, second on Exist, all without 3D detector outputs. The 168-fold figure is the overall reduction from the high-resolution patch token count, with ablations showing that the best accuracy comes from splitting that reduction between top-k selection and an MLP compression stage rather than using either alone.

Load-bearing premise

The load-bearing premise is that cosine similarity between CLIP image-patch tokens and the question's text tokens reliably ranks which image regions are needed to answer the question, so that the top-k selection does not throw away the tokens that contain the answer.

Editorial extensions

If this is right

  • If the compression figure holds, high-resolution multi-view driving VQA becomes feasible on a single GPU at inference, since the LLM sees roughly 1/168 of the raw patch tokens.
  • The query dependence of selection means the system can allocate visual tokens per question, which is a step toward running richer perception only when needed.
  • Because the selected tokens are a sparse, query-relevant subset, the model produces a natural visual explanation of its answer (the patches it attended to), which could support debugging of perception failures.
  • The ablation across select/compress ratios implies there is a tunable cost-accuracy frontier: practitioners can dial the reduction while watching the four VQA metrics.
  • The input-type ablation shows the design degrades gracefully when some views or frames are missing, suggesting robustness to sensor dropout.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If CLIP's token-level alignment is imperfect, the top-k selection may discard the very patch containing the answer; a natural stress test is to check selection recall against human-annotated relevant regions, since the paper does not report such a test.
  • The same query-aware selection mechanism could transfer to other vision-language tasks where a text query determines which image regions matter, such as referring expression comprehension or visual grounding, though the paper only evaluates VQA in driving.
  • The enhancement module's reliance on frozen full-image and video encoders means the real computational savings land only inside the language model; the paper does not account for the cost of computing full-resolution image features before selection, so end-to-end savings may be lower than 168-fold.
  • A harder temporal benchmark with fast-moving objects and dense multi-frame queries would test whether the low-resolution video stream can actually supply motion information the high-resolution stream lacks.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. The paper introduces LaVida Drive, a vision-language model framework for visual question answering in autonomous driving. The method uses a frozen CLIP image/text encoder pair plus a TimeSformer video encoder, with two trainable modules: a query-aware token selection module that prunes visual tokens based on cosine similarity between image and text tokens, and a spatial-temporal token enhancement module that uses cross-attention to recover context from support branches. Experiments on DriveLM and NuScenes-QA report improvements over EM-VLM4AD baselines and competitive results against larger models, with a claimed 168-fold token compression. The paper includes component ablations and an input-type ablation.

Significance. If the central claims hold, LaVida Drive would be a useful efficiency-oriented contribution to autonomous-driving VQA: it is plausible that query-aware token pruning plus cross-attention-based context restoration can reduce token counts while keeping or improving accuracy. The paper provides comparisons on standard public benchmarks, a component-wise ablation, and an input-type robustness study, which are strengths. The main significance is undermined, however, by (i) a formally incoherent specification of the token-selection algorithm, (ii) overclaimed 'optimal performance' that is not supported by the tables, and (iii) inconsistent compression-ratio statements between the abstract and the introduction. These issues prevent the reader from reproducing the central mechanism and from assessing the true magnitude of the reported gains.

major comments (5)
  1. [Section 3.2, Eq. (6) and Algorithm 1] The Query-aware Token Selection module is not specified in a computationally coherent way. Equation (6) defines p_i(x) = exp(s(I', T_i)/tau) / sum_{j=1}^N exp(s(I', T_j)/tau), which normalizes over text tokens for a fixed image token. Algorithm 1 then computes S_sum <- sum_{k=1}^K p_i(x); for a fixed image token, summing these normalized values over text tokens yields a constant (approximately 1), so S_sum carries no per-token ranking signal. The subsequent step W <- sum_i I'_i collapses all image-token embeddings into a single d-dimensional vector, and M <- (1-alpha)*S_sum + alpha*W then attempts to add objects of incompatible shapes/meanings. As printed, the pseudocode cannot be executed and the top-k selection is not a well-defined function of the query. The authors must specify whether the softmax should be over image tokens (the usual interpretation) and provide a corrected per-token selection score, or otherwise the reported improvements cannot be traced to the proposed mechanism.
  2. [Abstract and Section 1] The compression-ratio claims are inconsistent. The abstract states 'an impressive 168-fold token compression,' while Section 1 states the method 'reduces visual tokens by 50% to 84%,' which corresponds to a 2x to 6.25x reduction. Table 2 'fixing the overall reduction ratio to 168' suggests that 168x is measured relative to the high-resolution patch token count (49*6*28 tokens), not to the baseline low-resolution token count. The paper should state clearly the reference token count for the 168x claim and reconcile it with the 50-84% statement in Section 1.
  3. [Table 1 and Table 2] The claim of 'optimal performance' in the abstract is not supported by the reported results. On DriveLM, LaVida Drive achieves BLEU-4 of 51.3, which is below DriveLM-Agent's 53.1; on NuScenes-QA, it achieves Exist accuracy of 78.0, which is below NusceneQA-Agent's 84.8. The method is best or second-best on several metrics, but it is not uniformly optimal. The authors should temper the abstract and conclusion to 'competitive or state-of-the-art on most metrics' and discuss the trade-offs where the method is not the best.
  4. [All experimental tables] No error bars, standard deviations, or significance tests are reported for any experiment. Many of the claimed improvements are small (e.g., CIDEr 3.32 vs 3.20, ROUGE-L 73.9 vs 72.0 in Table 1; several +0.5 to +0.6 increments in Table 3), and without variance information it is impossible to judge whether the differences are meaningful or within run-to-run noise. The authors should report means and standard deviations over at least three random seeds, especially for the main comparison and the ablation tables.
  5. [Section 3.2 and Figure 4] The query-aware selection mechanism rests on the assumption that cosine similarity between CLIP image patch tokens and text tokens reliably ranks answer-relevant regions, but the paper provides no quantitative validation of this ranking. The qualitative example in Figure 4 is illustrative only. A concrete test would be to replace the query-aware selection with random selection or a fixed grid selection at the same token budget and compare performance; if the query-aware version does not clearly outperform these baselines, the claim that selection is query-aware is not substantiated.
minor comments (6)
  1. [Section 3.2] Typographical and spacing issues appear throughout, e.g., 'T ext Encoder', 'T op-K', 'T ext', and 'NusceneQA' (inconsistent with 'NuScenes-QA'). Please proofread carefully.
  2. [Table 3] The row label 'High Solution Patch' should read 'High Resolution Patch'. Also, the delta format '+4.0' vs '+0.9' is inconsistent: some deltas are given as absolute points and some as percentage points; please make this uniform.
  3. [Section 4.1] The implementation detail 'each image divided into 4x7 patches of size 224x224' is ambiguous: a 224x224 patch on a typical camera image would imply a very large input resolution (e.g., 896x1568). Please specify the exact input resolution and how the '4x7' grid is derived.
  4. [Section 3.2, Algorithm 1] The algorithm uses 'p_img_i(x)' but never defines x, and the index i in the softmax numerator is not introduced as an image-token index. Please rewrite the notation so that each quantity is defined precisely.
  5. [References] Several references appear to be incomplete or non-verifiable, e.g., [4] 'Journal of Autonomous Systems', [8] 'Proceedings of CVPR, 2024', [23] 'IEEE Transactions on Vehicular Technology', [27] 'IEEE Access', [28] 'IEEE Transactions on Robotics', [29] 'Proceedings of CVPR, 2024', and [31] 'IEEE Transactions on Intelligent Vehicles' lack arXiv identifiers, DOIs, or author-page details that are standard for verification. Please provide complete, verifiable citations.
  6. [Section 4.2] The paper claims improved 'inference efficiency' and 'computational efficiency' but reports no runtime, FLOPs, or latency measurements. Token count is only a proxy for efficiency; please include actual throughput or latency numbers for the final model versus the baselines.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: results are evaluated against external benchmarks and the 168x compression is a fixed experimental condition, not a predicted quantity.

full rationale

I walked the claimed derivation chain and found no step in which a prediction or first-principles result reduces to its own inputs by construction. The central claims are the token-compression factor and the VQA scores on DriveLM and NuScenes-QA; both are measured against external benchmarks using standard metrics (BLEU-4, METEOR, ROUGE-L, CIDEr, and the NuScenes-QA accuracy categories), and neither metric is folded into the definition of the method. The 168-fold compression ratio is presented in Table 2 as a fixed overall reduction condition under which different select/compress splits are ablated; it is a chosen experimental setting, not a quantity derived from the model, so selecting the best row in that table is hyperparameter selection rather than a fitted input being renamed a prediction. The Query-aware Token Selection module uses cosine similarity between CLIP image and text embeddings, which are externally pretrained representations, and the enhancement module uses cross-attention; these are mechanisms with stated assumptions, not definitions of the reported outcomes. I also found no load-bearing self-citation: the cited prior works, including EM-VLM4AD, DriveLM, NuScenes-QA, and CLIP, are external, and no uniqueness theorem or equivalent argument is imported from the authors' own prior work. The noted incoherence in Algorithm 1 and Eq. (6) — the softmax normalization direction, the undefined aggregation over K, and the shape mismatch between S_sum and W — is a reproducibility and specification concern, not a circularity concern, because even a corrected selection rule would not make the benchmark scores true by definition.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

No new physical or metaphysical entities are introduced; the modules are software components built from existing encoders, so there is no graviton problem.

free parameters (5)
  • Top-k selection ratio k = 2 (select ratio), paired with compression ratio 84
    Chosen from Table 2 as the best combination under a fixed 168x reduction; directly determines token count and all downstream results.
  • MLP compression ratio = 84
    Chosen from Table 2; with select ratio 2 gives the overall 168x token reduction used in the abstract and Table 1.
  • Selection weight alpha = not reported
    Learnable parameter in Algorithm 1 blending similarity scores and token weights; final value not shown.
  • Similarity temperature tau = not reported
    Controls softmax sharpness in Eq. 6; no value or schedule reported.
  • Training hyperparameters: learning rate, weight decay, batch size, epochs = 1e-4, 0.05, 4, 12
    Fixed for all training runs; no sensitivity analysis, so results may depend on these choices.
assumptions (5)
  • domain assumption CLIP image and text tokens share a semantic space at token level
    Section 3.2 states "we assume that image and text tokens are mapped into the same semantic space"; the token selection module relies on this to rank visual tokens by cosine similarity to text tokens.
  • domain assumption Frozen CLIP and TimeSformer encoders retain enough task-relevant information
    Section 3.1: all encoders are frozen; the model never adapts encoder features to driving scenes, so it depends on pretrained representations being sufficient.
  • domain assumption TimeSformer video features represent motion needed for temporal enhancement
    Section 3.3: temporal enhancement uses Etemporal as keys and values; if the video encoder does not capture relevant motion, the enhancement cannot add useful temporal context.
  • domain assumption Benchmark metrics and splits are reliable
    Section 4.1: evaluation follows EM-VLM4AD protocols; the paper does not question metric reliability or annotation noise, yet treats score differences as meaningful.
  • standard math Scaled dot-product attention is a valid context fusion mechanism
    Eq. 7 applies standard scaled dot-product attention; no new mathematical content, but the architecture depends on it.

how reviews work

0 comments
Cite this review

Pith. "Pith review of LaVida Drive: Vision-Text Interaction VLM for Autonomous Driving with Token Selection, Recovery and Enhancement." pith.science (2026). https://pith.science/paper/TD5NMUP6

@misc{pith2026241112980,
  author       = {Pith},
  title        = {Pith review of: LaVida Drive: Vision-Text Interaction VLM for Autonomous Driving with Token Selection, Recovery and Enhancement},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/TD5NMUP6}},
  note         = {Machine review of arXiv:2411.12980}
}
read the original abstract

Recent advancements in Visual Language Models (VLMs) have made them crucial for visual question answering (VQA) in autonomous driving, enabling natural human-vehicle interactions. However, existing methods often struggle in dynamic driving environments, as they usually focus on static images or videos and rely on downsampling to manage computational costs. This results in the loss of critical details and the difficulty in effectively integrating spatial and temporal information, undermining fine-grained perception and temporal coherence essential for effective decision-making. To tackle these challenges, we introduce LaVida Drive, a novel and efficient VQA framework for autonomous driving. LaVida Drive seamlessly integrates temporal data while maintaining high-resolution inputs for detailed visual perception. It optimizes spatial processing by retaining high-resolution data for intricate details and using lower-resolution inputs for temporal analysis to focus on motion-related features, thereby boosting computational efficiency. The core of LaVida Drive consists of two modules: the \textit{Query-aware Token Selection} module and the \textit{Spatial-Temporal Token Recovery and Enhancement} module. The former dynamically selects the most relevant visual tokens based on semantic alignment with the input query, reducing the token count from high-resolution spatial input. The latter ensures smooth and coherent interactions between spatial and temporal information, preserving contextual continuity across frames. Extensive experiments on various autonomous driving question-answering benchmarks show that LaVida Drive significantly reduces visual tokens, enhances efficiency, and improves overall performance.

Figures

Figures reproduced from arXiv: 2411.12980 by the authors.

Figure 1
Figure 1. Comparison between LaVida Drive and existing [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Framework of LaVida Drive. High-resolution images are divided into 224 [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. The architecture of the Token-wise Atten [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Lavida Drive Response Example. The areas not covered by the black mask represent the image patches corre [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

31 extracted references · 27 canonical work pages

  1. [1]

    Meteor: An auto- matic metric for mt evaluation with improved correla- tion with human judgments

    Satanjeev Banerjee and Alon Lavie. Meteor: An auto- matic metric for mt evaluation with improved correla- tion with human judgments. In Proceedings of the acl workshop on intrinsic and extrinsic evaluation mea- sures for machine translation and/or summarization , pages 65–72, 2005. 6

    work page 2005

  2. [2]

    Is space-time attention all you need for video under- standing? In Proceedings of the International Confer- ence on Machine Learning (ICML) , 2021

    Gedas Bertasius, Heng Wang, and Lorenzo Torresani. Is space-time attention all you need for video under- standing? In Proceedings of the International Confer- ence on Machine Learning (ICML) , 2021. 6

    work page 2021

  3. [3]

    Driving with llms: Fusing object-level vector modality for explainable au- tonomous driving

    Long Chen, Oleg Sinavski, Jan H¨ unermann, Alice Karnsund, Andrew James Willmott, Danny Birch, Daniel Maund, and Jamie Shotton. Driving with llms: Fusing object-level vector modality for explainable au- tonomous driving. In 2024 IEEE International Con- ference on Robotics and Automation (ICRA) , pages 14093–14100. IEEE, 2024. 1, 3

    work page 2024

  4. [4]

    T. Chen, X. Zhang, and Y. Wang. Contextual ques- tion answering for autonomous driving. Journal of Au- tonomous Systems, 11:98–110, 2023. 3

    work page 2023

  5. [5]

    Y. Chen, L. Xie, and X. Wang. Deep vision-language fusion for autonomous driving planning. In IEEE Transactions on Neural Networks and Learning Sys- tems, pages 4135–4149, 2022. 2

    work page 2022

  6. [6]

    Multi-frame, lightweight & efficient vision- language models for question answering in autonomous driving, 2024

    Akshay Gopalkrishnan, Ross Greer, and Mohan Trivedi. Multi-frame, lightweight & efficient vision- language models for question answering in autonomous driving, 2024. 6

    work page 2024

  7. [7]

    Lan- guage is not all you need: Aligning perception with language models

    Shaohan Huang, Li Dong, Wenhui Wang, Yaru Hao, Saksham Singhal, Shuming Ma, Tengchao Lv, Lei Cui, Owais Khan Mohammed, Barun Patra, et al. Lan- guage is not all you need: Aligning perception with language models. Advances in Neural Information Pro- cessing Systems, 36:72096–72109, 2023. 2

    work page 2023

  8. [8]

    Z. Li, H. Wang, and Y. Xu. Dialogue-based question answering for autonomous vehicles. In Proceedings of CVPR, 2024. 3

    work page 2024

Show all 31 references
  1. [9]

    Vlm2scene: Self-supervised image-text-lidar learning with foundation models for autonomous driving scene understanding

    Guibiao Liao, Jiankun Li, and Xiaoqing Ye. Vlm2scene: Self-supervised image-text-lidar learning with foundation models for autonomous driving scene understanding. In Proceedings of the AAAI Confer- ence on Artificial Intelligence , pages 3351–3359, 2024. 1

    2024

  2. [10]

    Rouge: A package for automatic eval- uation of summaries

    Chin-Yew Lin. Rouge: A package for automatic eval- uation of summaries. In Text summarization branches out, pages 74–81, 2004. 6

    2004

  3. [11]

    Reason2drive: Towards interpretable and chain-based reasoning for autonomous driving

    Ming Nie, Renyuan Peng, Chunwei Wang, Xinyue Cai, Jianhua Han, Hang Xu, and Li Zhang. Reason2drive: Towards interpretable and chain-based reasoning for autonomous driving. In European Conference on Com- puter Vision, pages 292–308. Springer, 2025. 1, 2

    2025

  4. [12]

    Bleu: a method for automatic evaluation of machine translation

    Kishore Papineni, Salim Roukos, Todd Ward, and Wei- Jing Zhu. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguis- tics, pages 311–318, 2002. 6

    2002

  5. [13]

    Nuscenes-qa: A multi-modal vi- sual question answering benchmark for autonomous driving scenario

    Tianwen Qian, Jingjing Chen, Linhai Zhuo, Yang Jiao, and Yu-Gang Jiang. Nuscenes-qa: A multi-modal vi- sual question answering benchmark for autonomous driving scenario. In Proceedings of the AAAI Confer- ence on Artificial Intelligence , pages 4542–4550, 2024. 1, 2

    2024

  6. [14]

    Nuscenes-qa: A multi-modal vi- sual question answering benchmark for autonomous driving scenario, 2024

    Tianwen Qian, Jingjing Chen, Linhai Zhuo, Yang Jiao, and Yu-Gang Jiang. Nuscenes-qa: A multi-modal vi- sual question answering benchmark for autonomous driving scenario, 2024. 6

    2024

  7. [15]

    Learning trans- ferable visual models from natural language supervi- sion

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sas- try, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning trans- ferable visual models from natural language supervi- sion. In Internation...

    2021

  8. [16]

    Learning transferable visual models from natural lan- guage supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sas- try, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural lan- guage supervision. In International conference on ma- chine learni...

    2021

  9. [17]

    Colin Raffel, Noam Shazeer, Adam Roberts, Kather- ine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of trans- fer learning with a unified text-to-text transformer. Journal of Machine Learning Research , 21(140):1–67,

  10. [18]

    Languagempc: Large language models as decision makers for autonomous driving

    Hao Sha, Yao Mu, Yuxuan Jiang, Li Chen, Chenfeng Xu, Ping Luo, Shengbo Eben Li, Masayoshi Tomizuka, Wei Zhan, and Mingyu Ding. Languagempc: Large language models as decision makers for autonomous driving. arXiv preprint arXiv:2310.03026 , 2023. 2

    arXiv

  11. [19]

    Drivelm: Driving with graph visual question answering

    Chonghao Sima, Katrin Renz, Kashyap Chitta, Li Chen, Hanxue Zhang, Chengen Xie, Jens Beißwenger, Ping Luo, Andreas Geiger, and Hongyang Li. Drivelm: Driving with graph visual question answering. arXiv preprint arXiv:2312.14150, 2023. 1, 2, 6

    2023 arXiv

  12. [20]

    Drivevlm: The convergence of autonomous driving and large vision-language models

    Xiaoyu Tian, Junru Gu, Bailin Li, Yicheng Liu, Yang Wang, Zhiyong Zhao, Kun Zhan, Peng Jia, Xianpeng Lang, and Hang Zhao. Drivevlm: The convergence of autonomous driving and large vision-language models. arXiv preprint arXiv:2402.12289 , 2024. 2

    2024 arXiv

  13. [21]

    Cider: Consensus-based image descrip- tion evaluation

    Ramakrishna Vedantam, C Lawrence Zitnick, and Devi Parikh. Cider: Consensus-based image descrip- tion evaluation. In Proceedings of the IEEE confer- ence on computer vision and pattern recognition, pages 4566–4575, 2015. 6

    2015

  14. [22]

    Driving into the future: Multiview visual forecasting and planning with world model for autonomous driving

    Yuqi Wang, Jiawei He, Lue Fan, Hongxin Li, Yuntao Chen, and Zhaoxiang Zhang. Driving into the future: Multiview visual forecasting and planning with world model for autonomous driving. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pat- tern Recognition, page...

    2024

  15. [23]

    Y. Wang, J. Zhang, and Y. Li. Hybrid vision-language models for safe autonomous driving. IEEE Transac- tions on Vehicular Technology, 73(6):3752–3764, 2024. 3

    2024

  16. [24]

    On the road with gpt-4v (ision): Early explorations of visual-language model on autonomous driving

    Licheng Wen, Xuemeng Yang, Daocheng Fu, Xiaofeng Wang, Pinlong Cai, Xin Li, Tao Ma, Yingxuan Li, Lin- ran Xu, Dengke Shang, et al. On the road with gpt-4v (ision): Early explorations of visual-language model on autonomous driving. arXiv preprint arXiv:2311.05332,

    arXiv

  17. [25]

    Drivegpt4: Interpretable end-to-end au- tonomous driving via large language model

    Zhenhua Xu, Yujia Zhang, Enze Xie, Zhen Zhao, Yong Guo, Kwan-Yee K Wong, Zhenguo Li, and Heng- shuang Zhao. Drivegpt4: Interpretable end-to-end au- tonomous driving via large language model. IEEE Robotics and Automation Letters, 2024. 2

    2024

  18. [26]

    Generalized predictive model for autonomous driving

    Jiazhi Yang, Shenyuan Gao, Yihang Qiu, Li Chen, Tianyu Li, Bo Dai, Kashyap Chitta, Penghao Wu, Jia Zeng, Ping Luo, et al. Generalized predictive model for autonomous driving. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogni- tion, pages 14662–14...

    2024

  19. [27]

    X. Yang, W. Li, and L. Zhang. Vision-language models for autonomous driving: A survey. IEEE Access, 10: 67890–67905, 2022. 3

    2022

  20. [28]

    Z. Zeng, L. Xie, and S. Wang. Multimodal vision- language models for autonomous driving. IEEE Trans- actions on Robotics, 39(2):456–470, 2023. 2

    2023

  21. [29]

    Zhang, L

    J. Zhang, L. Liu, and C. Wu. End-to-end vision- language reasoning for autonomous driving. In Pro- ceedings of CVPR, 2024. 2

    2024

  22. [30]

    Vinvl: Revisiting visual representa- tions in vision-language models

    Pengchuan Zhang, Xiujun Li, Xiaowei Hu, Jianwei Yang, Lei Zhang, Lijuan Wang, Yejin Choi, and Jianfeng Gao. Vinvl: Revisiting visual representa- tions in vision-language models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 5579–55...

    2021

  23. [31]

    J. Zhu, M. Li, and W. Hu. Vision-language pretraining for autonomous driving decision making. IEEE Trans- actions on Intelligent Vehicles , 8(3):745–756, 2023. 2

    2023

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.