Pith. sign in

REVIEW 4 major objections 5 minor 2 cited by

CALMM-Drive: Confidence-Aware Autonomous Driving with Large Multimodal Model

T0 review · 4 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read An LMM that proposes several driving decisions with confidence scores, then lets a diffusion planner generate and score trajectories for each, improves closed-loop driving reliability on long-tail scenarios, the paper argues.

desk verdict The framework idea is genuinely new and the ablations are promising, but the headline SOTA claim over Diffusion-ES is confounded by the un-ablated change in proposal count (128 vs 32), and the paper's own parameter study undermines the reported configuration. read the letter →

arxiv 2412.04209 v2 pith:XJEWJIHX submitted 2024-12-05 cs.RO

Ruoyu Yao , Yubin Wang , Haichao Liu , Rui Yang , Zengqi Peng , Lei Zhu , Jun Ma This is my paper · ORCID
classification cs.RO
keywords autonomousdrivinglargemultimodalmodelconfidenceelicitationdiffusiondecision-makingmotionplanningnuPlanlong-tailscenarios
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper proposes an autonomous-driving method in which a large multimodal model (LMM) offers several plausible high-level decisions—such as accelerating, cruising, or changing lanes—each with a stated confidence, before any trajectory is planned. The claimed payoff is that these confidence-tagged decisions, used to guide diffusion-based trajectory generation and to choose the final trajectory, fix two known failure modes: decisions the planner cannot actually execute, and planners that pick safe-looking but tactically short-sighted trajectories. In nuPlan closed-loop simulations, the method reports the best success-rate and reactive closed-loop scores on the long-tail Test14-Hard benchmark, and it improves over the diffusion-only planner Diffusion-ES on both benchmarks and in both reactive and non-reactive modes. The method thus offers a way to add knowledge-driven reliability to a capable planner without retraining it, provided the reported confidence values carry real information about decision quality.

What carries the argument

The machinery is a hierarchical, confidence-gated selection over decision-conditioned trajectory sets. An LMM first reasons through scene understanding, action selection, and confidence assessment to produce K actions $a^k_t$ and scalar confidences $c^k_t \in [0,1]$; an objective creator then assigns each action a decision-following objective $J^k_f$ (lane and speed adherence) combined with a general objective $J_g$ based on a rule-based Predictive Driver Model scorer, and a gradient-free diffusion optimizer evolves N trajectory proposals for each decision. A confidence-aware selector scores the best proposal for each decision with $S((X^k)^*) = (c^k_t)^{\omega_c} (\tilde J^k_f)^{\tilde\omega_f} (J_g)^{\tilde\omega_g}$ and picks the maximum, so a candidate is penalized for low tactical confidence, poor decision-following, or low general planning quality. The first-stage weights favor following the decision, while the second-stage weights rebalance toward general quality, which is what lets the pipeline keep long-term tactical goals in view without ignoring short-term smoothness and safety.

What would settle it

Run the full pipeline on the same nuPlan benchmarks with the confidence term $c^k_t$ in Eq. (8) replaced by a random permutation of the reported confidences, keeping every other component fixed; if the success-rate and closed-loop scores on Test14-Hard and Test14-Random do not fall materially, then the verbalized confidence is not what carries the reliability gain.

Watch

Extended reading notes

Core claim

The paper's central claim is that treating an LMM's driving suggestions as a set of hypotheses with confidence weights, rather than as a single authoritative command, measurably improves closed-loop driving reliability. The pipeline uses a three-step reasoning procedure—scene understanding, action selection, and confidence assessment—to elicit K candidate decisions with scalar confidences, converts each decision into a decision-following objective, generates N trajectory proposals per decision with a gradient-free diffusion optimizer (Diffusion-ES), and then selects the final trajectory by a product of the confidence score, the decision-following score, and a general driving-quality score. On the long-tail Test14-Hard benchmark it reports an NR-SR of 88.97% versus 84.93% for Diffusion-ES, an R-CLS of 78.13 versus 77.75, and an R-SR of 89.71% versus 87.13%, with success rates above 95% on Test14-Random without any rule-based emergency braking. The ablation studies attribute the gains to the two-stage Top-K reasoning and to including the confidence term in the selector; removing the confidence term lowers NR-CLS by 4.13 and R-CLS by 4.27 on Test14-Random.

Load-bearing premise

The load-bearing premise is that the numbers the model reports as decision confidences genuinely reflect how good each driving decision is; the paper imports this from natural-language calibration results without checking it against driving outcomes, and it runs the language model in a mode where the same question always gets the same answer, so the reported scores are not separately calibrated against driving success.

Editorial extensions

If this is right

  • LMM-empowered driving systems no longer need one-shot decisions: asking for several hypotheses with confidences and arbitrating them through planning is itself a reliability mechanism.
  • On Test14-Hard, the method reports the best NR-SR (88.97%), R-CLS (78.13), and R-SR (89.71%) among the compared planners, with success-rate margins over Diffusion-ES of +4.04 percentage points non-reactive and +2.58 percentage points reactive.
  • On Test14-Random, the method exceeds 95% reactive and non-reactive success rates without rule-based emergency braking, coming within 0.77 and 0.76 percentage points of the best success rates respectively.
  • Ablations imply that both the two-stage Top-K reasoning and the confidence term are load-bearing: single-decision reasoning scores worst on every metric, and dropping the confidence term from the selector costs 4.13 NR-CLS and 4.27 R-CLS.
  • Parameter studies show the architecture still has headroom: K = 5 candidates yields R-CLS 88.32 and R-SR 97.32%, above the reported default configuration.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A testable extension the paper does not run: calibrate the verbalized confidences against actual closed-loop decision quality on a held-out scenario set and re-run the selector; if calibration improves the correlation between $c^k_t$ and plan success, the reported long-tail gains should grow, and if it does not, the current confidence weighting is fragile.
  • Because the LMM is queried at temperature 0.0, the Top-K confidences are a single deterministic readout rather than a sample over forking paths; sampling several responses at higher temperature and aggregating confidences would directly test the paper's stated uncertainty rationale.
  • The same decision-guided generation and confidence-aware selection could wrap any multimodal trajectory proposer, not only diffusion-based ones, so rule-based or learned proposers with stronger local smoothness might combine with the tactical confidence term to close the remaining CLS gap on Test14-Random.
  • The reported sweet spots at K = 5 and a 1 s decision cycle suggest that an adaptive decision cycle—querying the LMM more often in dense interactions and less often on open roads—could capture most of the reliability gain at a fraction of the inference cost.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes CALMM-Drive, an autonomous driving framework that couples an LMM-based decision-maker with a diffusion-based motion planner. The LMM receives a BEV image and textual description, performs structured CoT reasoning (scene understanding, action selection, confidence assessment), and outputs K candidate high-level decisions with verbalized confidence scores. Each decision is mapped to an objective function that guides a gradient-free Diffusion-ES planner to generate N trajectory proposals; a hierarchical scoring stage then selects the final trajectory by combining decision confidence with planning-quality scores. The method is evaluated in the nuPlan closed-loop simulator on Test14-Hard and Test14-Random in both non-reactive and reactive modes, and is compared against rule-based, learning-based, hybrid, and LMM-empowered baselines. The authors report state-of-the-art NR-SR, R-CLS, and R-SR on Test14-Hard, along with ablations of the reasoning stages, the scoring components, key parameters (K, Cd, omega_c), and different foundation models.

Significance. If the reported results are robust, the paper makes a useful contribution: it is among the first to bring verbalized confidence from LMMs into closed-loop motion planning, and the hierarchical scoring idea is a sensible way to combine high-level tactical reasoning with low-level trajectory quality. The manuscript has notable strengths: it includes ablation studies for the reasoning pipeline and the scoring function, a parameter analysis, comparisons across multiple LMMs, and open-source/closed-source baseline handling is mostly explicit. The framework is well motivated and the writing is generally clear. However, the empirical support for the central SOTA claims is weakened by an uncontrolled variable in the Diffusion-ES comparison, parameter tuning on the same benchmark used for final evaluation, a lack of validation for the verbalized confidence scores, and the absence of error bars or significance tests. These issues are addressable but require additional experiments.

major comments (4)
  1. [Section IV-A and Table IV] The comparison with Diffusion-ES is confounded by the number of trajectory proposals N. Section IV-A states that CALMM-Drive's Diffusion-ES component uses 'a proposal number of 128 instead of 32,' while Table IV reports Diffusion-ES as a baseline with its published configuration. All CALMM-Drive ablations (Tables V and VI) and the parameter study (Fig. 6) also use N=128, so no experiment isolates the contribution of the LMM decision/confidence module from the increased proposal budget. The claimed gains over Diffusion-ES (e.g., NR-SR 88.97 vs. 84.93 on Test14-Hard) could be largely due to the larger proposal set, since Diffusion-ES-style evolutionary planning benefits from more proposals. Please add a controlled comparison, such as Diffusion-ES with N=128 and CALMM-Drive with N=32, and explicitly report which N was used for the Diffusion-ES baseline row in Table IV.
  2. [Section IV-E and Fig. 6] The parameter study is performed on the same Test14-Random benchmark used for the headline evaluation, and the reported configuration is not the best configuration the authors found. Fig. 6 shows that K=5 attains R-CLS 88.32 and R-SR 97.32%, and that Cd=1 s attains R-CLS 88.46 and R-SR 96.55%, both clearly higher than the main reported results with K=3 and Cd=2 s (R-CLS 87.11, R-SR 95.79 in Table IV). The manuscript does not explain why the superior configurations are not used in the final evaluation. Since the same benchmark was used for both tuning and final reporting, the reported numbers may reflect selection bias. Please either report the best-performing configuration in the main results, or justify the retention of the inferior configuration, and separate the tuning set from the final evaluation set.
  3. [Eqs. (2e), (8) and Table III] The central mechanism of the framework is the verbalized confidence score c_t^k, but the paper offers no evidence that these scores are informative for driving decisions. The authors cite NLP calibration results [37], [50] without providing any driving-domain calibration analysis, and Table III sets the LMM temperature to 0.0, which removes the sampling stochasticity that motivates the forking-paths discussion in Section III-A2. If the confidences are miscalibrated or arbitrary, the multiplicative term (c_t^k)^{omega_c} in Eq. (8) could add noise to an already competent planner rather than improve it. Please include a calibration study (e.g., reliability diagrams relating confidence to decision/planning quality), an ablation that replaces c_t^k with uniform or random confidences, and a discussion of why temperature 0.0 is consistent with the uncertainty motivation.
  4. [Tables IV-VI and Fig. 6] All quantitative claims are based on single evaluation runs, with no error bars or significance tests. Some of the headline improvements are small in magnitude—for example, R-CLS 78.13 vs. 77.75 for Diffusion-ES on Test14-Hard in Table IV—so the SOTA claims are not robust to run-to-run variability. The manuscript should report multiple independent runs or confidence intervals, or at least state clearly whether the nuPlan evaluation is deterministic and whether the LMM calls introduce nondeterminism.
minor comments (5)
  1. [Section IV-B, finding 1] The phrase 'exceptional capability in handing long-tail' contains a typo: 'handing' should be 'handling'.
  2. [Fig. 5] The abbreviation 'EAV' is used in the agent responses but is not defined in the caption or text; it should be 'ego vehicle' for clarity.
  3. [Section III-A2 and Fig. 5] The action codes such as 'AR', 'CR', and 'DK' used in the textual prompts are only explained in the Fig. 5 example; a compact definition in the main text would improve readability.
  4. [Table IV] The PlanAgent baseline is reported only with NR-CLS and R-CLS, with dash entries for success rates; consider obtaining or clearly stating why the success-rate metrics are unavailable, since Table IV is used for cross-method comparison.
  5. [Section IV-A] The sentence 'with the only difference that we choose a proposal number of 128 instead of 32' is ambiguous about whether the Diffusion-ES baseline row in Table IV was run with 32 proposals in this work or quoted from the original publication; please clarify the exact provenance of the baseline numbers.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation found: the confidence-aware LMM outputs, the diffusion-ES trajectory optimization, and the hierarchical scorer are distinct components, and the headline metrics are measured in nuPlan closed-loop simulation rather than being constructed from the method's own inputs.

full rationale

The paper's central pipeline is not circular by construction. The LMM produces Top-K decisions and verbalized confidences (Eq. 2e) from BEV and text; the diffusion planner optimizes trajectories against objectives in Eqs. (3)-(4); the selector in Eq. (8) combines the LMM confidence with the decision-following and PDM scores. None of these quantities is defined in terms of the final nuPlan metrics (NR-CLS, NR-SR, R-CLS, R-SR), and no parameter is fitted to Test14-Hard and then reported as a prediction of that same benchmark. The self-citations (e.g., [35], [39]) are contextual and not load-bearing. The most significant reviewer concern, that the comparison with Diffusion-ES is confounded because CALMM uses N=128 proposals while the Diffusion-ES baseline uses N=32 (Section IV-A), is a validity or external-correctness issue rather than circularity: the reported improvement does not reduce by definition to an input of the method. Similarly, using Test14-Random for ablations and parameter sweeps, and the lack of a driving-specific calibration check for the LMM's verbalized confidences, are empirical weaknesses, not cases of a prediction being identical to its input. The paper even acknowledges that the LMM is 'without being trained or calibrated' for fine-grained domain characteristics, which is a limitation statement rather than a circular step. Therefore the circularity score is 0.

Assumptions & free parameters 7 free parameters · 4 assumptions · 0 invented entities

The main empirical contributions rely on many hand-set parameters and on the untested assumption that LMM verbalized confidence is meaningful for driving. The diffusion planner and PDM scorer are borrowed, and no code is provided. This is an integration paper rather than a first-principles derivation.

free parameters (7)
  • omega_c (power on decision confidence) = 1.0
    Weight for c_t^k in scorer (8); chosen by hand and explored in Sec. IV-E.
  • K (number of candidate decisions) = 3
    Top-K decisions requested from LMM; Sec. IV-E shows K=5 gives higher scores, but main results use 3.
  • C_d (decision-making cycle) = 2.0 s
    LMM invoked every 2 s; Sec. IV-E shows 1 s performs better.
  • omega_f, omega_g, tilde_omega_f, tilde_omega_g = 5.0, 1.0, 0.3, 1.0
    Relative weights in Eqs. (3) and (8); hand-set in Table III.
  • lambda, gamma, v_bar, d_max = 1.25, 0.75, 2.0 m/s, 5.0 m
    Hand-tuned constants in speed-interval and lane-following objectives, Eqs. (4)-(5).
  • LMM temperature = 0.0
    Sampling temperature set to 0 in Table III, despite the paper's stochasticity motivation.
  • Diffusion-ES proposal count = 128
    Trajectory proposals per decision; differs from Diffusion-ES default of 32 (Sec. IV-A).
assumptions (4)
  • domain assumption nuPlan closed-loop simulation, with log-replay and IDM-reactive modes, is a valid proxy for real-world AV performance.
    All evaluation is simulation-only; the abstract and Sec. V present simulator results as the evidence for the framework's advantages.
  • domain assumption Verbalized confidence from Top-K elicitation is calibrated enough in the driving domain to guide trajectory selection.
    The paper transfers confidence elicitation results from NLP benchmarks [37], [50] to driving without a calibration study.
  • domain assumption The PDM scorer J_g is an appropriate general trajectory-quality objective.
    J_g is adopted directly from [13], [17] and treated as ground truth for low-level quality in Eqs. (3) and (8).
  • domain assumption Diffusion-ES with 10 denoising steps, 2 renoising iterations, and 128 proposals explores the trajectory space well enough.
    The setup is taken from [13] with only the proposal count changed; no convergence or coverage analysis is provided.

how reviews work

0 comments
Cite this review

Pith. "Pith review of CALMM-Drive: Confidence-Aware Autonomous Driving with Large Multimodal Model." pith.science (2026). https://pith.science/paper/XJEWJIHX

@misc{pith2026241204209,
  author       = {Pith},
  title        = {Pith review of: CALMM-Drive: Confidence-Aware Autonomous Driving with Large Multimodal Model},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/XJEWJIHX}},
  note         = {Machine review of arXiv:2412.04209}
}
read the original abstract

Decision-making and motion planning constitute critical components for ensuring the safety and efficiency of autonomous vehicles (AVs). Existing methodologies typically adopt two paradigms: decision then planning or generation then scoring. However, the former architecture often suffers from decision-planning misalignment that incurs risky situations. Meanwhile, the latter struggles to balance short-term operational metrics (e.g., immediate motion smoothness) with long-term tactical goals (e.g., route efficiency), resulting in myopic or overly conservative behaviors. To address these issues, we introduce CALMM-Drive, a novel Confidence-Aware Large Multimodal Model (LMM) empowered Autonomous Driving framework. Our approach integrates driving task-oriented Chain-of-Thought (CoT) reasoning coupled with Top-K confidence elicitation, which facilitates high-level reasoning to generate multiple candidate decisions with their confidence levels. Furthermore, we propose a novel planning module that integrates a diffusion model for trajectory generation and a hierarchical refinement process to find the optimal trajectory. This framework enables the selection over trajectory candidates accounting for both low-level solution quality and high-level tactical confidence, which avoids the risks within one-shot decisions and overcomes the limitations in short-sighted scoring mechanisms. Comprehensive evaluations in nuPlan closed-loop simulation environments demonstrate the competitive performance of CALMM-Drive across both common and long-tail benchmarks, showcasing a significant advancement in the integration of uncertainty in LMM-empowered AVs. The code will be released upon acceptance.

Figures

Figures reproduced from arXiv: 2412.04209 by the authors.

Figure 1
Figure 1. An illustration of decision-making and motion planning paradigms. In [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. The pipeline of CALMM-Drive. It consists of a decision-making module based on an LMM and a motion planning module integrating the process [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. An illustration of different objects presented on the BEV. Annotation [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: An example of forking paths in the generation process of LMMs. [PITH_FULL_IMAGE:figures/full_fig_p004_4.png]
Figure 5
Figure 5. Figure 5: A comparison between our approach and Diffusion-ES in two representative driving scenarios. At the critical steps, compared to the Diffusion-ES [PITH_FULL_IMAGE:figures/full_fig_p009_5.png]
Figure 6
Figure 6. Figure 6: Quantitative effects of the key parameters in our architecture evaluated on Test14-Random under the setting of closed-loop reactive test. [PITH_FULL_IMAGE:figures/full_fig_p011_6.png]
Figure 7
Figure 7. Figure 7: Quantitative performance comparison of CALMM-Drive powered by different foundation models on Test14-Hard and Test14-Random. The subplot [PITH_FULL_IMAGE:figures/full_fig_p012_7.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. VLM-UDMC: VLM-Enhanced Unified Decision-Making and Motion Control for Urban Autonomous Driving

    cs.RO 2025-07 conditional novelty 4.0 of 10

    VLM-UDMC uses a vision-language model to switch safety cost functions in a model predictive controller and a multi-kernel LSTM to predict traffic trajectories, reporting improved urban driving metrics in CARLA and cam...

  2. Chain-of-Thought for Autonomous Driving: A Comprehensive Survey and Future Prospects

    cs.RO 2025-05 conditional novelty 4.0 of 10

    A survey that classifies chain-of-thought methods for autonomous driving into modular, logical, and reflective pipelines, and proposes three evolutionary stages from direct prompting to reinforcement learning.

Reference graph

Works this paper leans on

61 extracted references · 45 canonical work pages · cited by 2 Pith papers

  1. [37]

    Just ask for calibration: Strategies for eliciting calibrated confidence scores from language models fine-tuned with human feedback,

    K. Tian, E. Mitchell, A. Zhou, A. Sharma, R. Rafailov, H. Yao, C. Finn, and C. D. Manning, “Just ask for calibration: Strategies for eliciting calibrated confidence scores from language models fine-tuned with human feedback,” in the Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing , pp. 5433–5442, 2023

    work page 2023

  2. [50]

    Teaching models to express their uncertainty in words,

    S. Lin, J. Hilton, and O. Evans, “Teaching models to express their uncertainty in words,” arXiv preprint arXiv:2205.14334 , 2022

    arXiv 2022

  3. [1]

    A safe hierarchical planning framework for complex driving scenarios based on reinforce- ment learning,

    J. Li, L. Sun, J. Chen, M. Tomizuka, and W. Zhan, “A safe hierarchical planning framework for complex driving scenarios based on reinforce- ment learning,” in the Proceedings of IEEE International Conference on Robotics and Automation , pp. 2660–2666, 2021

    work page 2021

  4. [2]

    Self-learned autonomous driving at unsignalized intersec- tions: A hierarchical reinforced learning approach for feasible decision- making,

    M. Al-Sharman, R. Dempster, M. A. Daoud, M. Nasr, D. Rayside, and W. Melek, “Self-learned autonomous driving at unsignalized intersec- tions: A hierarchical reinforced learning approach for feasible decision- making,” IEEE Transactions on Intelligent Transportation Systems , vol. 24, no. 11, pp. 12345–12356, 2023

    work page 2023

  5. [3]

    LanguageMPC: Large language models as deci- sion makers for autonomous driving,

    H. Sha, Y . Mu, Y . Jiang, L. Chen, C. Xu, P. Luo, S. E. Li, M. Tomizuka, W. Zhan, and M. Ding, “LanguageMPC: Large language models as deci- sion makers for autonomous driving,” arXiv preprint arXiv:2310.03026, 2023

    arXiv 2023

  6. [4]

    A cognition-inspired human-like decision-making method for automated vehicles,

    S. Xie, Y . Yang, M. Fu, and J. Zheng, “A cognition-inspired human-like decision-making method for automated vehicles,” IEEE Transactions on Intelligent Transportation Systems, vol. 25, no. 8, pp. 9852 – 9862, 2024

    work page 2024

  7. [5]

    Hierarchical uncertainty-aware autonomous driving in lane-changing scenarios: Behavior prediction and motion planning,

    R. Yao and X. Sun, “Hierarchical uncertainty-aware autonomous driving in lane-changing scenarios: Behavior prediction and motion planning,” in the Proceedings of 2024 IEEE Intelligent Vehicles Symposium , pp. 715– 721, 2024

    work page 2024

  8. [6]

    Interactive decision-making integrating graph neural networks and model predictive control for autonomous driving,

    K. Yang, S. Li, M. Wang, and X. Tang, “Interactive decision-making integrating graph neural networks and model predictive control for autonomous driving,” IEEE Transactions on Intelligent Transportation Systems, 2025

    work page 2025

Show all 61 references
  1. [7]

    Efficient sampling-based motion planning for on-road autonomous driving,

    L. Ma, J. Xue, K. Kawabata, J. Zhu, C. Ma, and N. Zheng, “Efficient sampling-based motion planning for on-road autonomous driving,” IEEE Transactions on Intelligent Transportation Systems , vol. 16, no. 4, pp. 1961–1976, 2015. 13

    1961

  2. [8]

    End-to-end interpretable neural motion planner,

    W. Zeng, W. Luo, S. Suo, A. Sadat, B. Yang, S. Casas, and R. Urtasun, “End-to-end interpretable neural motion planner,” in the Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8660–8669, 2019

    2019

  3. [9]

    Multi-modal model predictive control through batch non-holonomic trajectory optimization: Application to highway driving,

    V . K. Adajania, A. Sharma, A. Gupta, H. Masnavi, K. M. Krishna, and A. K. Singh, “Multi-modal model predictive control through batch non-holonomic trajectory optimization: Application to highway driving,” IEEE Robotics and Automation Letters , vol. 7, no. 2, pp. 4220–4227, 2022

    2022

  4. [10]

    Conditional predictive behavior planning with inverse reinforcement learning for human-like autonomous driving,

    Z. Huang, H. Liu, J. Wu, and C. Lv, “Conditional predictive behavior planning with inverse reinforcement learning for human-like autonomous driving,” IEEE Transactions on Intelligent Transportation Systems , vol. 24, no. 7, pp. 7244–7258, 2023

    2023

  5. [11]

    Gen-Drive: Enhancing diffusion generative driving policies with reward modeling and reinforcement learning fine-tuning,

    Z. Huang, X. Weng, M. Igl, Y . Chen, Y . Cao, B. Ivanovic, M. Pavone, and C. Lv, “Gen-Drive: Enhancing diffusion generative driving policies with reward modeling and reinforcement learning fine-tuning,” arXiv preprint arXiv:2410.05582, 2024

    arXiv

  6. [12]

    A unified framework integrating decision making and trajectory planning based on spatio-temporal voxels for highway autonomous driving,

    T. Zhang, W. Song, M. Fu, Y . Yang, X. Tian, and M. Wang, “A unified framework integrating decision making and trajectory planning based on spatio-temporal voxels for highway autonomous driving,” IEEE Transactions on Intelligent Transportation Systems, vol. 23, no. 8, pp. 1036...

    2021

  7. [13]

    Diffusion-ES: Gradient-free planning with diffusion for autonomous and instruction-guided driving,

    B. Yang, H. Su, N. Gkanatsios, T.-W. Ke, A. Jain, J. Schneider, and K. Fragkiadaki, “Diffusion-ES: Gradient-free planning with diffusion for autonomous and instruction-guided driving,” in the Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp...

    2024

  8. [14]

    LookOut: Diverse multi-future prediction and planning for self-driving,

    A. Cui, S. Casas, A. Sadat, R. Liao, and R. Urtasun, “LookOut: Diverse multi-future prediction and planning for self-driving,” in the Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 16107–16116, 2021

    2021

  9. [15]

    A critical view of driver behavior models: what do we know, what should we do?,

    J. A. Michon, “A critical view of driver behavior models: what do we know, what should we do?,” in Human Behavior and Traffic Safety , pp. 485–524, Springer, 1985

    1985

  10. [16]

    Optimal trajectory generation for dynamic street scenarios in a frenet frame,

    M. Werling, J. Ziegler, S. Kammel, and S. Thrun, “Optimal trajectory generation for dynamic street scenarios in a frenet frame,” in the Pro- ceedings of IEEE International Conference on Robotics and Automation, pp. 987–993, 2010

    2010

  11. [17]

    Parting with misconceptions about learning-based vehicle motion planning,

    D. Dauner, M. Hallgarten, A. Geiger, and K. Chitta, “Parting with misconceptions about learning-based vehicle motion planning,” in the Proceedings of Conference on Robot Learning , pp. 1268–1281, 2023

    2023

  12. [18]

    ST-P3: End- to-end vision-based autonomous driving via spatial-temporal feature learning,

    S. Hu, L. Chen, P. Wu, H. Li, J. Yan, and D. Tao, “ST-P3: End- to-end vision-based autonomous driving via spatial-temporal feature learning,” in the Proceedings of European Conference on Computer Vision, pp. 533–549, 2022

    2022

  13. [19]

    Rethinking imitation-based planners for autonomous driving,

    J. Cheng, Y . Chen, X. Mei, B. Yang, B. Li, and M. Liu, “Rethinking imitation-based planners for autonomous driving,” in the Proceedings of IEEE International Conference on Robotics and Automation, pp. 14123– 14130, 2024

    2024

  14. [20]

    DriveVLM: The convergence of autonomous driving and large vision-language models,

    X. Tian, J. Gu, B. Li, Y . Liu, Y . Wang, Z. Zhao, K. Zhan, P. Jia, X. Lang, and H. Zhao, “DriveVLM: The convergence of autonomous driving and large vision-language models,” in the Proceedings of Conference on Robot Learning, 2024

    2024

  15. [21]

    Driving with style: Inverse reinforcement learning in general-purpose planning for automated driving,

    S. Rosbach, V . James, S. Großjohann, S. Homoceanu, and S. Roth, “Driving with style: Inverse reinforcement learning in general-purpose planning for automated driving,” in the Proceedings of IEEE/RSJ Inter- national Conference on Intelligent Robots and Systems , pp. 2658–2665, 2019

    2019

  16. [22]

    LMDrive: Closed-loop end-to-end driving with large language models,

    H. Shao, Y . Hu, L. Wang, G. Song, S. L. Waslander, Y . Liu, and H. Li, “LMDrive: Closed-loop end-to-end driving with large language models,” in the Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 15120–15130, 2024

    2024

  17. [23]

    Senna: Bridging large vision-language models and end-to-end autonomous driving,

    B. Jiang, S. Chen, B. Liao, X. Zhang, W. Yin, Q. Zhang, C. Huang, W. Liu, and X. Wang, “Senna: Bridging large vision-language models and end-to-end autonomous driving,” arXiv preprint arXiv:2410.22313 , 2024

    2024 arXiv

  18. [24]

    Do As I Can, Not As I Say: Grounding language in robotic affordances,

    A. Brohan, Y . Chebotar, C. Finn, K. Hausman, A. Herzog, D. Ho, J. Ibarz, A. Irpan, E. Jang, R. Julian, et al. , “Do As I Can, Not As I Say: Grounding language in robotic affordances,” in the Proceedings of Conference on Robot Learning , pp. 287–318, 2023

    2023

  19. [25]

    A survey on multimodal large language models for autonomous driving,

    C. Cui, Y . Ma, X. Cao, W. Ye, Y . Zhou, K. Liang, J. Chen, J. Lu, Z. Yang, K.-D. Liao, et al. , “A survey on multimodal large language models for autonomous driving,” in the Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision , pp. 958–979, 2024

    2024

  20. [26]

    Language models are few-shot learners,

    T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al. , “Language models are few-shot learners,” in the Proceedings of Advances in Neural Information Processing Systems , pp. 1877–1901, 2020

    1901

  21. [27]

    LLaMA: Open and efficient foundation language models,

    H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozi `ere, N. Goyal, E. Hambro, F. Azhar, et al. , “LLaMA: Open and efficient foundation language models,” arXiv preprint arXiv:2302.13971, 2023

    2023 arXiv

  22. [28]

    GPT-4 technical report,

    J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat, et al. , “GPT-4 technical report,” arXiv preprint arXiv:2303.08774 , 2023

    2023 arXiv

  23. [29]

    Visual instruction tuning,

    H. Liu, C. Li, Q. Wu, and Y . J. Lee, “Visual instruction tuning,” in the Proceedings of Advances in Neural Information Processing Systems , pp. 34892–34916, 2023

    2023

  24. [30]

    Improved baselines with visual instruction tuning,

    H. Liu, C. Li, Y . Li, and Y . J. Lee, “Improved baselines with visual instruction tuning,” in the Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pp. 26296–26306, 2024

    2024

  25. [31]

    HE-Drive: Human-like end-to-end driving with vision language models,

    J. Wang, X. Zhang, Z. Xing, S. Gu, X. Guo, Y . Hu, Z. Song, Q. Zhang, X. Long, and W. Yin, “HE-Drive: Human-like end-to-end driving with vision language models,” arXiv preprint arXiv:2410.05051 , 2024

    2024

  26. [32]

    Forking paths in neural text generation,

    E. J. Bigelow, A. Holtzman, H. Tanaka, and T. Ullman, “Forking paths in neural text generation,” in the Proceedings of International Conference on Learning Representations , 2025

    2025

  27. [33]

    To believe or not to believe your LLM: Iterative prompting for estimat- ing epistemic uncertainty,

    Y . Abbasi Yadkori, I. Kuzborskij, A. Gy ¨orgy, and C. Szepesvari, “To believe or not to believe your LLM: Iterative prompting for estimat- ing epistemic uncertainty,” in the Proceedings of Advances in Neural Information Processing Systems , pp. 58077–58117, 2024

    2024

  28. [34]

    Semantic uncertainty: Linguistic invariances for uncertainty estimation in natural language generation,

    L. Kuhn, Y . Gal, and S. Farquhar, “Semantic uncertainty: Linguistic invariances for uncertainty estimation in natural language generation,” in the Proceedings of International Conference on Learning Represen- tations, 2023

    2023

  29. [35]

    Hierarchical prediction uncertainty-aware motion planning for autonomous driving in lane-changing scenarios,

    R. Yao and X. Sun, “Hierarchical prediction uncertainty-aware motion planning for autonomous driving in lane-changing scenarios,” Trans- portation Research Part C: Emerging Technologies, vol. 171, p. 104962, 2025

    2025

  30. [36]

    What uncertainties do we need in Bayesian deep learning for computer vision?,

    A. Kendall and Y . Gal, “What uncertainties do we need in Bayesian deep learning for computer vision?,” in the Proceedings of Advances in Neural Information Processing Systems , 2017

    2017

  31. [38]

    NuPlan: A closed-loop ML- based planning benchmark for autonomous vehicles,

    H. Caesar, J. Kabzan, K. S. Tan, W. K. Fong, E. Wolff, A. Lang, L. Fletcher, O. Beijbom, and S. Omari, “NuPlan: A closed-loop ML- based planning benchmark for autonomous vehicles,” arXiv preprint arXiv:2106.11810, 2021

    2021 arXiv

  32. [39]

    LMMCoDrive: Cooperative driving with large multimodal model,

    H. Liu, R. Yao, Z. Huang, S. Shen, and J. Ma, “LMMCoDrive: Cooperative driving with large multimodal model,” arXiv preprint arXiv:2409.11981, 2024

    2024 arXiv

  33. [40]

    DiLu: A knowledge-driven approach to autonomous driving with large language models,

    L. Wen, D. Fu, X. Li, X. Cai, M. Tao, P. Cai, M. Dou, B. Shi, L. He, and Y . Qiao, “DiLu: A knowledge-driven approach to autonomous driving with large language models,” in the Proceedings of International Conference on Learning Representations , 2024

    2024

  34. [41]

    DrivegGPT4: Interpretable end-to-end autonomous driving via large language model,

    Z. Xu, Y . Zhang, E. Xie, Z. Zhao, Y . Guo, K.-Y . K. Wong, Z. Li, and H. Zhao, “DrivegGPT4: Interpretable end-to-end autonomous driving via large language model,” IEEE Robotics and Automation Letters , vol. 9, no. 10, pp. 8186 – 8193, 2024

    2024

  35. [42]

    PlanAgent: A multi-modal large language agent for closed-loop vehicle motion planning,

    Y . Zheng, Z. Xing, Q. Zhang, B. Jin, P. Li, Y . Zheng, Z. Xia, K. Zhan, X. Lang, Y . Chen, et al. , “PlanAgent: A multi-modal large language agent for closed-loop vehicle motion planning,” arXiv preprint arXiv:2406.01587, 2024

    2024 arXiv

  36. [43]

    R. M. Neal, Bayesian learning for neural networks . Springer Science & Business Media, 2012

    2012

  37. [44]

    Weight uncertainty in neural network,

    C. Blundell, J. Cornebise, K. Kavukcuoglu, and D. Wierstra, “Weight uncertainty in neural network,” in the Proceedings of International Conference on Machine Learning , pp. 1613–1622, 2015

    2015

  38. [45]

    Dropout as a Bayesian approximation: Representing model uncertainty in deep learning,

    Y . Gal and Z. Ghahramani, “Dropout as a Bayesian approximation: Representing model uncertainty in deep learning,” in the Proceedings of International Conference on Machine Learning , pp. 1050–1059, 2016

    2016

  39. [46]

    Simple and scalable predictive uncertainty estimation using deep ensembles,

    B. Lakshminarayanan, A. Pritzel, and C. Blundell, “Simple and scalable predictive uncertainty estimation using deep ensembles,” in the Proceed- ings of Advances in Neural Information Processing Systems , 2017

    2017

  40. [47]

    Look before you leap: An exploratory study of uncertainty mea- surement for large language models,

    Y . Huang, J. Song, Z. Wang, S. Zhao, H. Chen, F. Juefei-Xu, and L. Ma, “Look before you leap: An exploratory study of uncertainty mea- surement for large language models,” arXiv preprint arXiv:2307.10236, 2023. 14

    2023 arXiv

  41. [48]

    A survey of confidence estimation and calibration in large language models,

    J. Geng, F. Cai, Y . Wang, H. Koeppl, P. Nakov, and I. Gurevych, “A survey of confidence estimation and calibration in large language models,” in the Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Langua...

    2024

  42. [49]

    Shifting attention to relevance: Towards the predictive uncertainty quantification of free-form large language models,

    J. Duan, H. Cheng, S. Wang, A. Zavalny, C. Wang, R. Xu, B. Kailkhura, and K. Xu, “Shifting attention to relevance: Towards the predictive uncertainty quantification of free-form large language models,” in the Proceedings of the 62nd Annual Meeting of the Association for Com- p...

    2024

  43. [51]

    Can LLMs express their uncertainty? An empirical evaluation of confidence elicitation in LLMs,

    M. Xiong, Z. Hu, X. Lu, Y . LI, J. Fu, J. He, and B. Hooi, “Can LLMs express their uncertainty? An empirical evaluation of confidence elicitation in LLMs,” in the Proceedings of International Conference on Learning Representations, 2024

    2024

  44. [52]

    Newell, H

    A. Newell, H. A. Simon, et al., Human problem solving . Prentice-hall Englewood Cliffs, NJ, 1972

    1972

  45. [53]

    How do we know that we know? the accessibility model of the feeling of knowing.,

    A. Koriat, “How do we know that we know? the accessibility model of the feeling of knowing.,” Psychological review, vol. 100, no. 4, p. 609, 1993

    1993

  46. [54]

    DriveLM: Driving with graph visual question answering,

    C. Sima, K. Renz, K. Chitta, L. Chen, H. Zhang, C. Xie, J. Beißwenger, P. Luo, A. Geiger, and H. Li, “DriveLM: Driving with graph visual question answering,” in the Proceedings of European Conference on Computer Vision, pp. 256–274, 2024

    2024

  47. [55]

    Denoising diffusion probabilistic models,

    J. Ho, A. Jain, and P. Abbeel, “Denoising diffusion probabilistic models,” in the Proceedings of Advances in Neural Information Processing Systems, pp. 6840–6851, 2020

    2020

  48. [56]

    Generalized force model of traffic dynamics,

    D. Helbing and B. Tilch, “Generalized force model of traffic dynamics,” Physical review E , vol. 58, no. 1, p. 133, 1998

    1998

  49. [57]

    From prediction to planning with goal conditioned lane graph traversals,

    M. Hallgarten, M. Stoll, and A. Zell, “From prediction to planning with goal conditioned lane graph traversals,” in the Proceedings of 2023 IEEE 26th International Conference on Intelligent Transportation Systems , pp. 951–958, 2023

    2023

  50. [58]

    Urban Driver: Learning to drive from real-world demonstrations using policy gradients,

    O. Scheel, L. Bergamini, M. Wolczyk, B. Osi ´nski, and P. Ondruska, “Urban Driver: Learning to drive from real-world demonstrations using policy gradients,” in the Proceedings of Conference on Robot Learning , pp. 718–728, 2022

    2022

  51. [59]

    Diffusion-based planning for autonomous driving with flexible guidance,

    Y . Zheng, R. Liang, K. Zheng, J. Zheng, L. Mao, J. Li, W. Gu, R. Ai, S. E. Li, X. Zhan, et al. , “Diffusion-based planning for autonomous driving with flexible guidance,” in the Proceedings of International Conference on Learning Representations , 2025

    2025

  52. [60]

    GameFormer: Game-theoretic modeling and learning of transformer-based interactive prediction and planning for autonomous driving,

    Z. Huang, H. Liu, and C. Lv, “GameFormer: Game-theoretic modeling and learning of transformer-based interactive prediction and planning for autonomous driving,” in the Proceedings of the IEEE/CVF International Conference on Computer Vision , pp. 3903–3913, 2023

    2023

  53. [61]

    PLUTO: Pushing the limit of imi- tation learning-based planning for autonomous driving,

    J. Cheng, Y . Chen, and Q. Chen, “PLUTO: Pushing the limit of imi- tation learning-based planning for autonomous driving,” arXiv preprint arXiv:2404.14327, 2024

    2024 arXiv

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.