Pith. sign in

REVIEW 5 major objections 4 minor 3 cited by

OpenEMMA: Open-Source Multimodal Model for End-to-End Autonomous Driving

T0 review · 5 major / 4 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Inserting a chain-of-thought reasoning step into a pretrained vision-language model improves zero-shot trajectory planning on nuScenes.

desk verdict Useful open-source EMMA-style system, but the CoT improvement claim is not supported by the experiments as presented. read the letter →

arxiv 2412.15208 v2 pith:LWOWB2TD submitted 2024-12-19 cs.CV cs.LGcs.RO

Shuo Xing , Chengyuan Qian , Yuping Wang , Hongyuan Hua , Kexin Tian , Yang Zhou , Zhengzhong Tu This is my paper · ORCID
classification cs.CVcs.LGcs.RO
keywords autonomousdrivingmultimodallargelanguagemodelschain-of-thoughtpromptingend-to-endtrajectoryplanningzero-shotnuScenesvalidationspeed-curvatureintegration3Dobjectdetection
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

OpenEMMA's central claim is that a pretrained multimodal language model can plan a vehicle trajectory without any fine-tuning of the language model, provided it is first made to reason about the driving scene. The framework prompts the model to state its intent, describe the scene, list the important objects, and only then output the future speed and curvature that are integrated into a path. On 150 scenes from the nuScenes validation set, the paper reports that this two-stage prompt lowers average trajectory error for two open models and lowers the failure rate for two of the three models tested, with one model trading higher error for far fewer failures. If true, this offers an accessible, open route to EMMA-style end-to-end autonomous driving for researchers who lack proprietary models or large fine-tuning budgets.

What carries the argument

The load-bearing mechanism is a two-stage prompt wrapped around a pretrained MLLM. Stage 1 (Reasoning) forces the model to commit to an intent command, a concise scene description, and the major objects with their locations and relevance; Stage 2 (Predicting) conditions the numerical output on that written reasoning. The output parameterization is speed $s_t$ and curvature $k_t$ for the next $T$ seconds, which are integrated with the cumulative trapezoidal rule into heading $\theta_t$, velocities $v_x(t), v_y(t)$, and positions $(x_t, y_t)$. For 3D object detection, OpenEMMA adds an externally fine-tuned YOLO3D detector, since off-the-shelf MLLMs give poor spatial localization.

What would settle it

Run the same 150-scene nuScenes comparison with a sham reasoning prompt that produces the same structured text (intent, scene, objects) but fills the scene content with generic or blank statements; if L2 error and failure rate stay at the zero-shot baseline level the CoT claim is supported, while matching OpenEMMA's numbers would show the effect is prompt structure rather than reasoning.

Watch

Extended reading notes

Core claim

The paper asserts that adding Chain-of-Thought reasoning before the numerical output is what drives the improvement. OpenEMMA takes the front camera image and five seconds of ego speed and curvature, prompts the MLLM for an intent command, a scene description, and a list of major objects with locations and relevance, then prompts it again to predict speed and curvature for the next five seconds. The trajectory is recovered by trapezoidal integration of heading and velocity. On the nuScenes validation set, the reported numbers show average L2 error falling from 3.24 m to 2.98 m for LLaVA-1.6-Mistral-7B and from 3.00 m to 2.92 m for Llama-3.2-11B-Vision-Instruct, and failure rate falling from 24.00% to 16.11% for Qwen2-VL-7B-Instruct; Qwen's average L2 rises because OpenEMMA now produces trajectories for hard cases the baseline completely failed. The authors interpret these results as evidence that Chain-of-Thought reasoning helps a diverse set of open MLLMs understand and react to complex driving scenes.

Load-bearing premise

The paper attributes the gains to Chain-of-Thought reasoning without running a control that isolates reasoning from prompt length, formatting, or output parsing; if the difference comes from those factors, the central causal claim collapses.

Editorial extensions

If this is right

  • A no-fine-tuning, prompt-only recipe can serve as a reproducible open baseline for MLLM-based planning, bypassing the closed training pipeline of proprietary EMMA.
  • The reported failure-rate drops (for Llama-3.2, 23.92% to 22.00%; for Qwen2-VL, 24.00% to 16.11%) mean the reasoning prompt chiefly prevents large first-second deviations, not just small path errors.
  • Because the intermediate outputs are human-readable intent, scene, and object notes, OpenEMMA yields an interpretable explanation alongside every planned trajectory.
  • The speed-curvature parameterization makes every predicted trajectory dynamically smooth and consistent by construction, so this representation may transfer across different MLLM backbones.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • We would test whether a longer structured prompt with no reasoning content reproduces the gains; if it does, the active ingredient is formatting or output length, not reasoning.
  • If the reasoning stage is genuinely causal, then sampling multiple reasoning chains (self-consistency or tree-of-thought, which the paper names as future work) should reduce failure rates further.
  • The speed/curvature output space is itself a constraint: it biases predictions toward smooth, physically plausible arcs. A variant that outputs waypoints directly but keeps the same reasoning stage would separate the parameterization effect from the reasoning effect.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 4 minor

Summary. This manuscript introduces OpenEMMA, a training-free pipeline that uses pre-trained multimodal large language models (MLLMs) with a chain-of-thought reasoning stage to predict the ego vehicle's speed and curvature, which are then integrated into a future trajectory. A fine-tuned YOLO-style 3D detector is added for object detection. The authors evaluate the planning component on the nuScenes validation set with three open MLLMs and GPT-4o case studies, comparing against a zero-shot baseline. The central claim is that the chain-of-thought reasoning process yields consistent and significant improvements in L2 trajectory error and failure rate.

Significance. If the central claim were supported, OpenEMMA would be a valuable contribution: a computationally efficient, open-source recipe for MLLM-based trajectory planning that requires no fine-tuning of the language model. The paper releases code, uses off-the-shelf open models, and the speed/curvature integration in Section 2.1 is transparent and easy to reproduce. The approach is timely given the interest in EMMA-style end-to-end AD. However, the current experimental evidence is insufficient to establish the claimed benefits, and some of the reported results contradict the paper's own conclusions.

major comments (5)
  1. [Section 3.1, Table 1] The sentence 'OpenEMMA consistently outperforms the zero-shot baseline in both L2 norm error and failure rate, demonstrating the effectiveness of the Chain-of-Thought reasoning process' is not supported by Table 1. For LLaVA-1.6-Mistral-7B, the failure rate under OpenEMMA is 6.12% versus 4.06% for the zero-shot baseline, an increase; for Qwen2-VL-7B-Instruct, the average L2 error increases from 2.46 m to 2.81 m. The text acknowledges the Qwen2-VL L2 regression but not the LLaVA failure-rate regression, and neither is explained with per-scene evidence.
  2. [Section 3.1, baseline] The zero-shot baseline is not specified in enough detail to isolate the contribution of chain-of-thought reasoning. The manuscript only states that the baseline uses historical ego status and the driving scene image 'without incorporating any reasoning process,' and does not report the prompt, output representation (speed/curvature versus direct waypoints), parsing, or formatting. Consequently the comparison in Table 1 varies the reasoning stage and the prompt/output format simultaneously, so the observed improvements cannot be causally attributed to CoT. An ablation that fixes the output format and prompt length is required.
  3. [Section 3.1, statistics] The evaluation uses 150 scenes with no error bars, no variance estimates, and no significance tests, yet the abstract and Section 3.1 use the word 'significant.' The failure-rate metric (L2 > 10 m within the first second) is nonstandard and may be dominated by output-format or parsing failures; the paper should report parse-success rates or use a standard metric such as displacement error at multiple horizons. At minimum, paired per-scene tests (e.g., bootstrap or Wilcoxon) are needed to support the significance claim.
  4. [Section 3.1, Qwen2-VL post hoc explanation] The explanation that OpenEMMA's Qwen2-VL average L2 increases because the model 'successfully generates predictions for many cases where the zero-shot baseline fails' but then 'struggles to produce high-quality trajectories in these challenging scenarios' is not accompanied by any per-scene statistics, such as a breakdown of errors on scenes where the baseline succeeds versus fails, or a difficulty-based stratification. Without such evidence, the explanation is post hoc and does not support the overall claim of effectiveness.
  5. [Section 3.2, object detection] The 3D detection contribution is not evaluated quantitatively. The paper reports only the fine-tuned 2D detector's mAP50 (0.60316) and states that the 3D estimation network uses existing YOLO3D weights; no 3D detection metric (e.g., BEV mAP or 3D bounding box AP) is provided. The claim of 'precise detection of 3D on-road objects' is therefore unsupported.
minor comments (4)
  1. [Throughout] The manuscript uses both 'OpenEMMA' and 'Open-EMMA' inconsistently; please standardize to one form.
  2. [Abstract] The phrase 'We release all the codes' should be corrected to 'We release the code.'
  3. [Section 3.2] The sentence reporting the best mAP50 of 0.60316 at epoch 290 should specify the validation protocol and ideally include a learning curve to demonstrate that the fine-tuning converged.
  4. [Section 4, Related Work] The related work section would be strengthened by a quantitative comparison with prior MLLM-based AD methods such as DriveVLM and LMDrive, rather than only a qualitative summary.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the trajectory output is computed by deterministic kinematic integration of MLLM-generated speed and curvature, with no fitted parameter entering the derivation; the CoT-attribution weakness is an experimental-control concern, not circular reasoning.

full rationale

The paper's only quantitative derivation is the trajectory construction in Section 2.1: the MLLM is prompted to output speed S={s_t} and curvature K={k_t}, and the trajectory is then computed by cumulative trapezoidal integration (theta_t = theta_0 + sum k_i s_i Delta t; x_t = x_0 + sum v_x(i) Delta t; y_t = y_0 + sum v_y(i) Delta t). This is a fixed kinematic forward model, not a fit: no parameter is learned from or tuned to the nuScenes validation trajectories, and the final trajectory is definitionally the integral of the two predicted quantities. That is a reparameterization of the model output, not a circular derivation of the prediction from the ground-truth target. The central empirical claim, that Chain-of-Thought prompting improves on the zero-shot baseline, is not circular either. It is an uncontrolled-comparison and attribution issue: the baseline prompt, output format, and parsing are not fully specified, and Table 1 shows a mixed result (for Qwen2-VL-7B-Instruct, average L2 is worse under OpenEMMA, 2.81 vs 2.46 m). Those facts undermine the strength of the causal claim about CoT, but they do not make the result true by construction. The comparison is not a fitted parameter renamed as a prediction, and the reasoning stage is not defined in terms of the trajectory error it is claimed to improve. The citations to prior work, including EMMA, DriveVLM, and CoT prompting, are used to motivate the prompt design rather than to supply a load-bearing uniqueness theorem or to smuggle in an ansatz. Self-citations such as AutoTrust, PLUM, and EqDrive appear only as related-work context and do not carry the derivation. No known empirical pattern is renamed as a new organization: speed-and-curvature parameterization is a standard kinematic decomposition, and presenting it as human-interpretable driving commands is a framing choice, not a disguised reuse of a known result. Because every computed trajectory value follows from the MLLM's emitted speed/curvature tokens through a stated, fixed integral, and because no fitted parameter or self-citation chain forces the reported improvements, the derivation chain is self-contained. Concerns about baseline comparability, missing error bars, and the nonstandard failure-rate metric belong to experimental rigor and falsifiability, not circularity.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The framework introduces no new fitted constants; it relies on standard numerical integration and off-the-shelf models. The main hidden assumptions are about causal attribution to CoT, representativeness of the 150-scene evaluation, and parseability of MLLM outputs.

assumptions (4)
  • domain assumption The MLLM-generated numeric speed and curvature sequences can be treated as continuous signals and integrated with the trapezoidal rule to produce a valid ego trajectory.
    Section 2.1 Eq. (1)-(6) assumes discrete outputs can be integrated directly; no validation of dynamical feasibility or comparison to alternative integration schemes is provided.
  • domain assumption L2 error to ground truth and the L2>10-within-1s failure rate on 150 nuScenes validation scenes are sufficient and unbiased measures of planning quality.
    Section 3.1 defines metrics and scene count; scene selection is not described and no closed-loop or safety evaluation is included.
  • domain assumption The zero-shot baseline differs from OpenEMMA only by the absence of the reasoning stage, so any performance difference can be attributed to chain-of-thought.
    Section 3.1 Setup describes the baseline but does not specify whether output format, prompt length, and parsing are otherwise identical.
  • domain assumption Off-the-shelf pretrained MLLMs can reliably follow the instruction to output parseable speed and curvature values.
    Section 2.1 Stage 2 depends on parseable numeric outputs; the paper does not report parse success rates or failure modes of the numerical output.

how reviews work

0 comments
Cite this review

Pith. "Pith review of OpenEMMA: Open-Source Multimodal Model for End-to-End Autonomous Driving." pith.science (2026). https://pith.science/paper/LWOWB2TD

@misc{pith2026241215208,
  author       = {Pith},
  title        = {Pith review of: OpenEMMA: Open-Source Multimodal Model for End-to-End Autonomous Driving},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/LWOWB2TD}},
  note         = {Machine review of arXiv:2412.15208}
}
read the original abstract

Since the advent of Multimodal Large Language Models (MLLMs), they have made a significant impact across a wide range of real-world applications, particularly in Autonomous Driving (AD). Their ability to process complex visual data and reason about intricate driving scenarios has paved the way for a new paradigm in end-to-end AD systems. However, the progress of developing end-to-end models for AD has been slow, as existing fine-tuning methods demand substantial resources, including extensive computational power, large-scale datasets, and significant funding. Drawing inspiration from recent advancements in inference computing, we propose OpenEMMA, an open-source end-to-end framework based on MLLMs. By incorporating the Chain-of-Thought reasoning process, OpenEMMA achieves significant improvements compared to the baseline when leveraging a diverse range of MLLMs. Furthermore, OpenEMMA demonstrates effectiveness, generalizability, and robustness across a variety of challenging driving scenarios, offering a more efficient and effective approach to autonomous driving. We release all the codes in https://github.com/taco-group/OpenEMMA.

Figures

Figures reproduced from arXiv: 2412.15208 by the authors.

Figure 1
Figure 1. Illustration of the OpenEMMA framework. • Intent Command: A clear articulation of the ego ve￾hicle’s intended action based on the current scene, such as whether it will continue following the lane to turn left, turn right, or proceed straight. Additionally, it specifies whether the vehicle should maintain its cur￾rent speed, slow down, or accelerate. • Scene Description: A concise description of the driv￾ing scene a… view at source ↗
Figure 2
Figure 2. YOLO 2D detection results. The class-color correspon [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Visualization of OpenEMMA predictions powered by GPT-4o. [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. CMU-Drive and V2V-VLA: Cooperative Multi-agent Unified Driving with Reasoning Benchmark and Vehicle-to-Vehicle Vision-Language-Action Models

    cs.AI 2026-08 conditional novelty 6.0 of 10

    CMU-Drive adds up to 16 connected autonomous vehicles to closed-loop driving scenarios, and V2V-VLA shows that sharing merged occupancy views and communication suggestions improves driving score over a single-agent VL...

  2. ReAL-AD: Towards Human-Like Reasoning in End-to-End Autonomous Driving

    cs.RO 2025-07 conditional novelty 5.0 of 10

    ReAL-AD combines VLM-generated strategy and tactical commands with a two-stage trajectory decoder, cutting open-loop L2 error and collision rate by about a third on nuScenes and Bench2Drive.

  3. Demystifying the Visual Quality Paradox in Multimodal Large Language Models

    cs.CV 2025-06 reject novelty 4.0 of 10

    Multimodal LLM accuracy can improve on visually degraded images, and a lightweight test-time tuning module that modulates input quality yields small accuracy gains on some benchmarks.

Reference graph

Works this paper leans on

69 extracted references · 41 canonical work pages · cited by 3 Pith papers

  1. [1]

    https://github.com/ultralytics/ ultralytics

    ultralytics. https://github.com/ultralytics/ ultralytics. 5

    work page

  2. [2]

    https://github.com/ruhyadi/YOLO3D

    Yolo3d. https://github.com/ruhyadi/YOLO3D . 5

    work page

  3. [3]

    Qwen-vl: A versatile vision-language model for un- derstanding, localization, text reading, and beyond

    Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A versatile vision-language model for un- derstanding, localization, text reading, and beyond. arXiv preprint arXiv:2308.12966, 2023. 5

    arXiv 2023

  4. [4]

    Chauf- feurnet: Learning to drive by imitating the best and synthe- sizing the worst, 2018

    Mayank Bansal, Alex Krizhevsky, and Abhijit Ogale. Chauf- feurnet: Learning to drive by imitating the best and synthe- sizing the worst, 2018. 1

    work page 2018

  5. [5]

    Lan- guage models are few-shot learners

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Sub- biah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakan- tan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Lan- guage models are few-shot learners. Advances in neural in- formation processing systems, 33:1877–1901, 2020. 5

    work page 1901

  6. [6]

    Lang, Sourabh V ora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom

    Holger Caesar, Varun Bankiti, Alex H. Lang, Sourabh V ora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom. nuscenes: A mul- timodal dataset for autonomous driving. arXiv preprint arXiv:1903.11027, 2019. 2, 4, 5

    arXiv 1903

  7. [7]

    Multipath: Multiple probabilistic anchor trajec- tory hypotheses for behavior prediction

    Yuning Chai, Benjamin Sapp, Mayank Bansal, and Dragomir Anguelov. Multipath: Multiple probabilistic anchor trajec- tory hypotheses for behavior prediction. In Leslie Pack Kael- bling, Danica Kragic, and Komei Sugiura, editors, Proceed- ings of the Conference on Robot Learning , volume 100 of Proceedings of Machine Learning Research , pages 86–99. PMLR, 30...

    work page 2020

  8. [8]

    Epro-pnp: Generalized end-to-end prob- abilistic perspective-n-points for monocular object pose es- timation

    Hansheng Chen, Pichao Wang, Fan Wang, Wei Tian, Lu Xiong, and Hao Li. Epro-pnp: Generalized end-to-end prob- abilistic perspective-n-points for monocular object pose es- timation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 2781–2790, June 2022. 3

    work page 2022

Show all 69 references
  1. [9]

    Driving with llms: Fusing object-level vec- tor modality for explainable autonomous driving

    Long Chen, Oleg Sinavski, Jan H ¨unermann, Alice Karnsund, Andrew James Willmott, Danny Birch, Daniel Maund, and Jamie Shotton. Driving with llms: Fusing object-level vec- tor modality for explainable autonomous driving. In 2024 IEEE International Conference on Robotics and Au...

    2024

  2. [10]

    End-to-end autonomous driving: Challenges and frontiers

    Li Chen, Penghao Wu, Kashyap Chitta, Bernhard Jaeger, An- dreas Geiger, and Hongyang Li. End-to-end autonomous driving: Challenges and frontiers. IEEE Transactions on Pat- tern Analysis and Machine Intelligence, 2024. 5

    2024

  3. [11]

    Fleet, and Ge- offrey Hinton

    Ting Chen, Saurabh Saxena, Lala Li, David J. Fleet, and Ge- offrey Hinton. Pix2seq: A language modeling framework for object detection. In International Conference on Learn- ing Representations, 2022. 3

    2022

  4. [12]

    A survey of chain of thought rea- soning: Advances, frontiers and future

    Zheng Chu, Jingchang Chen, Qianglong Chen, Weijiang Yu, Tao He, Haotian Wang, Weihua Peng, Ming Liu, Bing Qin, and Ting Liu. A survey of chain of thought rea- soning: Advances, frontiers and future. arXiv preprint arXiv:2309.15402, 2023. 1

    2023 arXiv

  5. [13]

    Position: Prospective of autonomous driving - multimodal llms, world models, em- bodied intelligence, ai alignment, and mamba

    LLVM-AD Workshop Committee. Position: Prospective of autonomous driving - multimodal llms, world models, em- bodied intelligence, ai alignment, and mamba. In Proceed- ings of the IEEE/CVF Winter Conference on Applications of Computer Vision, 2025. 1

    2025

  6. [14]

    Bert: Pre-training of deep bidirectional transformers for language understanding

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018. 5

    2018 arXiv

  7. [15]

    Trafficgen: Learning to generate diverse and re- alistic traffic scenarios

    Lan Feng, Quanyi Li, Zhenghao Peng, Shuhan Tan, and Bolei Zhou. Trafficgen: Learning to generate diverse and re- alistic traffic scenarios. In 2023 IEEE International Confer- ence on Robotics and Automation (ICRA), pages 3567–3575. IEEE, 2023. 5

    2023

  8. [16]

    Planning-oriented autonomous driv- ing, 2023

    Yihan Hu, Jiazhi Yang, Li Chen, Keyu Li, Chonghao Sima, Xizhou Zhu, Siqi Chai, Senyao Du, Tianwei Lin, Wenhai Wang, Lewei Lu, Xiaosong Jia, Qiang Liu, Jifeng Dai, Yu Qiao, and Hongyang Li. Planning-oriented autonomous driv- ing, 2023. 1

    2023

  9. [17]

    Cramnet: Camera-radar fusion with ray-constrained cross-attention for robust 3d object detec- tion, 2022

    Jyh-Jing Hwang, Henrik Kretzschmar, Joshua Manela, Sean Rafferty, Nicholas Armstrong-Crews, Tiffany Chen, and Dragomir Anguelov. Cramnet: Camera-radar fusion with ray-constrained cross-attention for robust 3d object detec- tion, 2022. 1

    2022

  10. [18]

    Emma: End-to-end multimodal model for autonomous driving

    Jyh-Jing Hwang, Runsheng Xu, Hubert Lin, Wei-Chih Hung, Jingwei Ji, Kristy Choi, Di Huang, Tong He, Paul Covington, Benjamin Sapp, Yin Zhou, James Guo, Dragomir Anguelov, and Mingxing Tan. Emma: End-to-end multimodal model for autonomous driving. arXiv preprint arXiv:2410.23262,

    arXiv

  11. [19]

    Senna: Bridging large vision-language models and end-to-end autonomous driving, 2024

    Bo Jiang, Shaoyu Chen, Bencheng Liao, Xingyu Zhang, Wei Yin, Qian Zhang, Chang Huang, Wenyu Liu, and Xinggang Wang. Senna: Bridging large vision-language models and end-to-end autonomous driving, 2024. 5

    2024

  12. [20]

    Vad: Vectorized scene representation for efficient autonomous driving, 2023

    Bo Jiang, Shaoyu Chen, Qing Xu, Bencheng Liao, Jiajie Chen, Helong Zhou, Qian Zhang, Wenyu Liu, Chang Huang, and Xinggang Wang. Vad: Vectorized scene representation for efficient autonomous driving, 2023. 1

    2023

  13. [21]

    Lang, Sourabh V ora, Holger Caesar, Lubing Zhou, Jiong Yang, and Oscar Beijbom

    Alex H. Lang, Sourabh V ora, Holger Caesar, Lubing Zhou, Jiong Yang, and Oscar Beijbom. Pointpillars: Fast encoders for object detection from point clouds, 2019. 1

    2019

  14. [22]

    Llava-next-interleave: Tackling multi-image, video, and 3d in large multimodal models

    Feng Li, Renrui Zhang, Hao Zhang, Yuanhan Zhang, Bo Li, Wei Li, Zejun Ma, and Chunyuan Li. Llava-next-interleave: Tackling multi-image, video, and 3d in large multimodal models. arXiv preprint arXiv:2407.07895, 2024. 2, 4, 5

    2024 arXiv

  15. [23]

    Light the night: A multi-condition diffusion framework for unpaired low-light enhancement in autonomous driving

    Jinlong Li, Baolu Li, Zhengzhong Tu, Xinyu Liu, Qing Guo, Felix Juefei-Xu, Runsheng Xu, and Hongkai Yu. Light the night: A multi-condition diffusion framework for unpaired low-light enhancement in autonomous driving. In Proceed- ings of the IEEE/CVF Conference on Computer Visi...

    2024

  16. [24]

    Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In In- ternational conference on machine learning , pages 19730– 19742. PMLR, 2023. 5

    2023

  17. [25]

    Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation

    Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In Interna- tional conference on machine learning, pages 12888–12900. PMLR, 2022. 5

    2022

  18. [26]

    Comamba: Real-time co- operative perception unlocked with state space models.arXiv preprint arXiv:2409.10699, 2024

    Jinlong Li, Xinyu Liu, Baolu Li, Runsheng Xu, Jiachen Li, Hongkai Yu, and Zhengzhong Tu. Comamba: Real-time co- operative perception unlocked with state space models.arXiv preprint arXiv:2409.10699, 2024. 1

    2024 arXiv

  19. [27]

    Scenarionet: Open-source platform for large-scale traffic scenario simula- tion and modeling

    Quanyi Li, Zhenghao Mark Peng, Lan Feng, Zhizheng Liu, Chenda Duan, Wenjie Mo, and Bolei Zhou. Scenarionet: Open-source platform for large-scale traffic scenario simula- tion and modeling. Advances in neural information process- ing systems, 36, 2024. 5

    2024

  20. [28]

    Hdmapnet: An online hd map construction and evaluation framework,

    Qi Li, Yue Wang, Yilun Wang, and Hang Zhao. Hdmapnet: An online hd map construction and evaluation framework,

  21. [29]

    Zhiqi Li, Zhiding Yu, Shiyi Lan, Jiahan Li, Jan Kautz, Tong Lu, and Jose M. Alvarez. Is ego status all you need for open- loop end-to-end autonomous driving?, 2024. 1

    2024

  22. [30]

    Visual instruction tuning

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36, 2024. 5

    2024

  23. [31]

    Accurate monocular 3d object detec- tion via color-embedded 3d reconstruction for autonomous driving

    Xinzhu Ma, Zhihui Wang, Haojie Li, Pengbo Zhang, Wanli Ouyang, and Xin Fan. Accurate monocular 3d object detec- tion via color-embedded 3d reconstruction for autonomous driving. In Proceedings of the IEEE/CVF International Con- ference on Computer Vision (ICCV), October 2019. 3

    2019

  24. [32]

    Gpt-driver: Learning to drive with gpt, 2023

    Jiageng Mao, Yuxi Qian, Junjie Ye, Hang Zhao, and Yue Wang. Gpt-driver: Learning to drive with gpt, 2023. 5

    2023

  25. [33]

    Llama 3.2: Revolutionizing edge ai and vision with open, customizable models

    Meta. Llama 3.2: Revolutionizing edge ai and vision with open, customizable models. 2024. 2, 4, 5

    2024

  26. [34]

    3d bounding box estimation using deep learn- ing and geometry

    Arsalan Mousavian, Dragomir Anguelov, John Flynn, and Jana Kosecka. 3d bounding box estimation using deep learn- ing and geometry. 2017. 3

    2017

  27. [35]

    Refaat, and Benjamin Sapp

    Nigamaa Nayakanti, Rami Al-Rfou, Aurick Zhou, Kratarth Goel, Khaled S. Refaat, and Benjamin Sapp. Wayformer: Motion forecasting via simple & efficient attention networks,

  28. [36]

    Hello gpt-4o

    OpenAI. Hello gpt-4o. 2024. 2, 4

    2024

  29. [37]

    Plum: Prompt learning using metaheuristic

    Rui Pan, Shuo Xing, Shizhe Diao, Wenhe Sun, Xiang Liu, Kashun Shum, Renjie Pi, Jipeng Zhang, and Tong Zhang. Plum: Prompt learning using metaheuristic. arXiv preprint arXiv:2311.08364, 2023. 1

    2023 arXiv

  30. [38]

    Learning transferable visual models from natural language supervi- sion

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervi- sion. In International conference on machine learning, ...

    2021

  31. [39]

    Language models are unsu- pervised multitask learners

    Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsu- pervised multitask learners. OpenAI blog, 1(8):9, 2019. 5

    2019

  32. [40]

    Exploring the limits of transfer learning with a unified text-to-text transformer

    Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21(140):1–67, 2020. 5

    2020

  33. [41]

    Yasiru Ranasinghe, Deepti Hegde, and Vishal M. Patel. Monodiff: Monocular 3d object detection and pose estima- tion with diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 10659–10670, June 2024. 3

    2024

  34. [42]

    Code llama: Open foun- dation models for code

    Baptiste Roziere, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, J ´er´emy Rapin, et al. Code llama: Open foun- dation models for code. arXiv preprint arXiv:2308.12950 ,

    arXiv

  35. [43]

    Refaat, Rami Al-Rfou, and Benjamin Sapp

    Ari Seff, Brian Cera, Dian Chen, Mason Ng, Aurick Zhou, Nigamaa Nayakanti, Khaled S. Refaat, Rami Al-Rfou, and Benjamin Sapp. Motionlm: Multi-agent motion forecasting as language modeling, 2023. 1

    2023

  36. [44]

    Lmdrive: Closed-loop end-to-end driving with large language models

    Hao Shao, Yuxuan Hu, Letian Wang, Guanglu Song, Steven L Waslander, Yu Liu, and Hongsheng Li. Lmdrive: Closed-loop end-to-end driving with large language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15120–15130, 2024. 1, 5

    2024

  37. [45]

    Mtr++: Multi-agent motion prediction with symmetric scene modeling and guided intention querying, 2024

    Shaoshuai Shi, Li Jiang, Dengxin Dai, and Bernt Schiele. Mtr++: Multi-agent motion prediction with symmetric scene modeling and guided intention querying, 2024. 1

    2024

  38. [46]

    Drivelm: Driving with graph visual ques- tion answering

    Chonghao Sima, Katrin Renz, Kashyap Chitta, Li Chen, Hanxue Zhang, Chengen Xie, Ping Luo, Andreas Geiger, and Hongyang Li. Drivelm: Driving with graph visual ques- tion answering. arXiv preprint arXiv:2312.14150, 2023. 1, 6

    2023 arXiv

  39. [47]

    Scalability in perception for au- tonomous driving: Waymo open dataset, 2020

    Pei Sun, Henrik Kretzschmar, Xerxes Dotiwalla, Aurelien Chouard, Vijaysai Patnaik, Paul Tsui, James Guo, Yin Zhou, Yuning Chai, Benjamin Caine, Vijay Vasudevan, Wei Han, Jiquan Ngiam, Hang Zhao, Aleksei Timofeev, Scott Ettinger, Maxim Krivokon, Amy Gao, Aditya Joshi, Sheng Zha...

    2020

  40. [48]

    Srinivasan, Jonathan T

    Matthew Tancik, Vincent Casser, Xinchen Yan, Sabeek Prad- han, Ben Mildenhall, Pratul P. Srinivasan, Jonathan T. Bar- ron, and Henrik Kretzschmar. Block-nerf: Scalable large scene neural view synthesis, 2022. 1

    2022

  41. [49]

    Gemini: a family of highly capable multimodal models

    Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805, 2023. 5

    2023 arXiv

  42. [50]

    Qwen2.5: A party of foundation models, September 2024

    Qwen Team. Qwen2.5: A party of foundation models, September 2024. 5

    2024

  43. [51]

    Motion planning for autonomous driving: The state of the art and future perspectives

    Siyu Teng, Xuemin Hu, Peng Deng, Bai Li, Yuchen Li, Yunfeng Ai, Dongsheng Yang, Lingxi Li, Zhe Xu- anyuan, Fenghua Zhu, and Long Chen. Motion planning for autonomous driving: The state of the art and future perspectives. IEEE Transactions on Intelligent Vehicles , 8(6):3692–37...

    2023

  44. [52]

    Drivevlm: The convergence of autonomous driving and large vision-language models

    Xiaoyu Tian, Junru Gu, Bailin Li, Yicheng Liu, Chenxu Hu, Yang Wang, Kun Zhan, Peng Jia, Xianpeng Lang, and Hang Zhao. Drivevlm: The convergence of autonomous driving and large vision-language models. arXiv preprint arXiv:2402.12289, 2024. 6

    2024 arXiv

  45. [53]

    End-to-end model-free reinforcement learning for urban driving using implicit affordances

    Marin Toromanoff, Emilie Wirbel, and Fabien Moutarde. End-to-end model-free reinforcement learning for urban driving using implicit affordances. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 7153–7162, 2020. 5

    2020

  46. [54]

    Llama: Open and efficient foundation language models

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timoth´ee Lacroix, Baptiste Rozi`ere, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023. 5

    2023 arXiv

  47. [55]

    Llama 2: Open foundation and fine-tuned chat models.arXiv preprint arXiv:2307.09288, 2023

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models.arXiv preprint arXiv:2307.09288, 2023. 5

    2023 arXiv

  48. [56]

    Efficient reinforce- ment learning for autonomous driving with parameterized skills and priors

    Letian Wang, Jie Liu, Hao Shao, Wenshuo Wang, Ruobing Chen, Yu Liu, and Steven L Waslander. Efficient reinforce- ment learning for autonomous driving with parameterized skills and priors. arXiv preprint arXiv:2305.04412 , 2023. 5

    2023 arXiv

  49. [57]

    Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution

    Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Yang Fan, Kai Dang, Mengfei Du, Xuancheng Ren, Rui Men, Dayiheng Liu, Chang Zhou, Jingren Zhou, and Jun- yang Lin. Qwen2-vl: Enhancing vision-language model’s ...

    2024 arXiv

  50. [58]

    Self-consistency improves chain of thought reasoning in language models

    Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. arXiv preprint arXiv:2203.11171, 2022. 6

    2022 arXiv

  51. [59]

    Eqdrive: Efficient equivari- ant motion forecasting with multi-modality for autonomous driving

    Yuping Wang and Jier Chen. Eqdrive: Efficient equivari- ant motion forecasting with multi-modality for autonomous driving. In 2023 8th International Conference on Robotics and Automation Engineering (ICRAE) , pages 224–229. IEEE, 2023. 1, 2

    2023

  52. [60]

    Chain-of-thought prompting elicits reasoning in large language models, 2023

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning in large language models, 2023. 1, 6

    2023

  53. [61]

    AutoTrust: Benchmarking Trustworthiness in Large Vision Language Models for Autonomous Driving

    Shuo Xing, Hongyuan Hua, Xiangbo Gao, Shenzhe Zhu, Renjie Li, Kexin Tian, Xiaopeng Li, Heng Huang, Tian- bao Yang, Zhangyang Wang, Yang Zhou, Huaxiu Yao, and Zhengzhong Tu. AutoTrust: Benchmarking Trustworthiness in Large Vision Language Models for Autonomous Driving. arXiv, D...

    2024

  54. [62]

    Qwen2 technical report

    An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jialong Tang, Jialin Wang, Jian Yang, Jianhong Tu, Jian- wei Zhang, Jianxin Ma, Jin Xu, Jingren Zhou, Jinze Bai, Jinzhe...

    2024 arXiv

  55. [63]

    Tree of thoughts: Deliberate problem solving with large lan- guage models

    Shunyu Yao, Dian Yu, Google Deepmind, Jeffrey Zhao, Thomas L Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large lan- guage models. In Advances in Neural Information Process- ing Systems, volume 36, pages 11809–11822, 12 2023. 6

    2023

  56. [64]

    Rag-driver: Gen- eralisable driving explanations with retrieval-augmented in- context learning in multi-modal large language model.arXiv preprint arXiv:2402.10828, 2024

    Jianhao Yuan, Shuyang Sun, Daniel Omeiza, Bo Zhao, Paul Newman, Lars Kunze, and Matthew Gadd. Rag-driver: Gen- eralisable driving explanations with retrieval-augmented in- context learning in multi-modal large language model.arXiv preprint arXiv:2402.10828, 2024. 6

    2024

  57. [65]

    A survey of autonomous driving: Com- mon practices and emerging technologies

    Ekim Yurtsever, Jacob Lambert, Alexander Carballo, and Kazuya Takeda. A survey of autonomous driving: Com- mon practices and emerging technologies. IEEE Access , 8:58443–58469, 2020. 1

    2020

  58. [66]

    Rethinking the open-loop evaluation of end- to-end autonomous driving in nuscenes, 2023

    Jiang-Tian Zhai, Ze Feng, Jinhao Du, Yongqiang Mao, Jiang-Jiang Liu, Zichang Tan, Yifu Zhang, Xiaoqing Ye, and Jingdong Wang. Rethinking the open-loop evaluation of end- to-end autonomous driving in nuscenes, 2023. 1

    2023

  59. [67]

    Monodetr: Depth- guided transformer for monocular 3d object detection

    Renrui Zhang, Han Qiu, Tai Wang, Ziyu Guo, Ziteng Cui, Yu Qiao, Hongsheng Li, and Peng Gao. Monodetr: Depth- guided transformer for monocular 3d object detection. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 9155–9166, October 2023. 3

    2023

  60. [68]

    End-to-end urban driving by imitat- ing a reinforcement learning coach

    Zhejun Zhang, Alexander Liniger, Dengxin Dai, Fisher Yu, and Luc Van Gool. End-to-end urban driving by imitat- ing a reinforcement learning coach. In Proceedings of the IEEE/CVF international conference on computer vision, pages 15222–15232, 2021. 5

    2021

  61. [69]

    Monoedge: Monocular 3d object detection using local per- spectives

    Minghan Zhu, Lingting Ge, Panqu Wang, and Huei Peng. Monoedge: Monocular 3d object detection using local per- spectives. In Proceedings of the IEEE/CVF Winter Con- ference on Applications of Computer Vision (WACV), pages 643–652, January 2023. 3

    2023

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.