{"id":"20bef28c-7356-4ae6-9215-9baeed215300","arxiv_id":"2412.04209","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":7,"one_line_summary":"CALMM-Drive combines Top-K confident decision elicitation from an LMM with diffusion-based trajectory generation and a two-stage scorer, reporting competitive nuPlan results with higher success rates on long-tail scenarios.","lead":"This paper presents CALMM-Drive, an autonomous driving system that uses a large multimodal model to propose several driving decisions with confidence scores, then plans trajectories for each and selects the best. It reports improved safety for rare, long-tail situations in the nuPlan driving simulator.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"SOTA gains over Diffusion-ES are confounded by proposal count: CALMM uses N=128 while the Diffusion-ES baseline uses N=32, so improved NR-SR/R-SR may not come from confidence-aware LMM selection.","rationale":"Good-faith reading: the paper's actual hypothesis is that verbalized Top-K confidence from an LMM, combined with decision-guided diffusion planning and hierarchical scoring, improves closed-loop robustness. The ablations support that internal mechanism: Table VI shows that adding c_t^k improves R-SR from 93.49 to 95.79 and NR-CLS from 83.05 to 86.70 on Test14-Random, so the confidence signal is not inert within the pipeline. I therefore do not dispute that the components do something. The problem is the external benchmark claim. Section IV-A explicitly reveals that the CALMM planner uses N=128 proposals while the Diffusion-ES baseline uses N=32 as configured in [13]. In diffusion-based test-time optimization, proposal count is a first-order resource: more samples mean better objective coverage and fewer failures in long-tail cases. Since the headline SOTA claims on Test14-Hard are only a few points above Diffusion-ES, the N discrepancy is a plausible alternative explanation. This is more load-bearing than the reader's calibration concern: even perfectly calibrated confidences would not establish the contribution if the baseline was run with fewer proposals. The reader's weakest_assumption correctly notes evaluation fairness problems, but it stops short of identifying N as the concrete confound, so my agreement is partial. The proposed test is cheap and decisive; if the gains vanish, the paper should compare against N=128 Diffusion-ES and/or ablate N before claiming confidence-aware superiority. The verdict remains CONDITIONAL, but the acceptance conditions should explicitly include this controlled comparison.","tokens_in":20449,"tokens_out":5547,"duration_ms":58247,"concrete_test":"Rerun the Diffusion-ES baseline with N=128 (all other settings as in Section IV-A: 10 denoising steps, 2 renoising-denoising iterations, PDM scoring, same LQR tracker) on Test14-Hard and Test14-Random, in both reactive and non-reactive modes. Compare NR-CLS, NR-SR, R-CLS, and R-SR against CALMM-Drive's Table IV numbers and against the N=32 baseline. If the N=128 Diffusion-ES reproduces CALMM's success rates, the reported gains are a proposal-count artifact; if CALMM still exceeds it by the same margins, the LMM confidence module is the active ingredient.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The most load-bearing concern is not confidence calibration but an uncontrolled variable in the headline comparison. In Section IV-A, the authors state that their Diffusion-ES planner follows the efficient version of [13] 'with the only difference that we choose a proposal number of 128 instead of 32.' Table IV lists Diffusion-ES as a separate baseline with its published configuration. CALMM-Drive therefore differs from Diffusion-ES in at least two ways: the LMM decision/confidence module and the number of trajectory proposals N (128 vs 32). The claimed SOTA improvements on Test14-Hard (NR-SR 88.97 vs 84.93; R-SR 89.71 vs 87.13; R-CLS 78.13 vs 77.75) and the 'success rates are always improved over Diffusion-ES' bullet in Section IV-B.3 could be largely or entirely due to the larger proposal set, since more proposals improve exploration in Diffusion-ES-style evolutionary planning, especially in long-tail scenarios. The paper does not ablate N, and all CALMM ablations (Tables V-VI) use N=128, so they cannot separate the LMM's contribution from the increased computational budget. If a N=128 Diffusion-ES matches CALMM, the central novelty claim that confidence-aware LMM decisions drive the gain is unsupported.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes CALMM-Drive, an autonomous driving framework that couples an LMM-based decision-maker with a diffusion-based motion planner. The LMM receives a BEV image and textual description, performs structured CoT reasoning (scene understanding, action selection, confidence assessment), and outputs K candidate high-level decisions with verbalized confidence scores. Each decision is mapped to an objective function that guides a gradient-free Diffusion-ES planner to generate N trajectory proposals; a hierarchical scoring stage then selects the final trajectory by combining decision confidence with planning-quality scores. The method is evaluated in the nuPlan closed-loop simulator on Test14-Hard and Test14-Random in both non-reactive and reactive modes, and is compared against rule-based, learning-based, hybrid, and LMM-empowered baselines. The authors report state-of-the-art NR-SR, R-CLS, and R-SR on Test14-Hard, along with ablations of the reasoning stages, the scoring components, key parameters (K, Cd, omega_c), and different foundation models.","tokens_in":20708,"tokens_out":5394,"duration_ms":52562,"significance":"If the reported results are robust, the paper makes a useful contribution: it is among the first to bring verbalized confidence from LMMs into closed-loop motion planning, and the hierarchical scoring idea is a sensible way to combine high-level tactical reasoning with low-level trajectory quality. The manuscript has notable strengths: it includes ablation studies for the reasoning pipeline and the scoring function, a parameter analysis, comparisons across multiple LMMs, and open-source/closed-source baseline handling is mostly explicit. The framework is well motivated and the writing is generally clear. However, the empirical support for the central SOTA claims is weakened by an uncontrolled variable in the Diffusion-ES comparison, parameter tuning on the same benchmark used for final evaluation, a lack of validation for the verbalized confidence scores, and the absence of error bars or significance tests. These issues are addressable but require additional experiments.","major_comments":[{"comment":"The comparison with Diffusion-ES is confounded by the number of trajectory proposals N. Section IV-A states that CALMM-Drive's Diffusion-ES component uses 'a proposal number of 128 instead of 32,' while Table IV reports Diffusion-ES as a baseline with its published configuration. All CALMM-Drive ablations (Tables V and VI) and the parameter study (Fig. 6) also use N=128, so no experiment isolates the contribution of the LMM decision/confidence module from the increased proposal budget. The claimed gains over Diffusion-ES (e.g., NR-SR 88.97 vs. 84.93 on Test14-Hard) could be largely due to the larger proposal set, since Diffusion-ES-style evolutionary planning benefits from more proposals. Please add a controlled comparison, such as Diffusion-ES with N=128 and CALMM-Drive with N=32, and explicitly report which N was used for the Diffusion-ES baseline row in Table IV.","section":"Section IV-A and Table IV"},{"comment":"The parameter study is performed on the same Test14-Random benchmark used for the headline evaluation, and the reported configuration is not the best configuration the authors found. Fig. 6 shows that K=5 attains R-CLS 88.32 and R-SR 97.32%, and that Cd=1 s attains R-CLS 88.46 and R-SR 96.55%, both clearly higher than the main reported results with K=3 and Cd=2 s (R-CLS 87.11, R-SR 95.79 in Table IV). The manuscript does not explain why the superior configurations are not used in the final evaluation. Since the same benchmark was used for both tuning and final reporting, the reported numbers may reflect selection bias. Please either report the best-performing configuration in the main results, or justify the retention of the inferior configuration, and separate the tuning set from the final evaluation set.","section":"Section IV-E and Fig. 6"},{"comment":"The central mechanism of the framework is the verbalized confidence score c_t^k, but the paper offers no evidence that these scores are informative for driving decisions. The authors cite NLP calibration results [37], [50] without providing any driving-domain calibration analysis, and Table III sets the LMM temperature to 0.0, which removes the sampling stochasticity that motivates the forking-paths discussion in Section III-A2. If the confidences are miscalibrated or arbitrary, the multiplicative term (c_t^k)^{omega_c} in Eq. (8) could add noise to an already competent planner rather than improve it. Please include a calibration study (e.g., reliability diagrams relating confidence to decision/planning quality), an ablation that replaces c_t^k with uniform or random confidences, and a discussion of why temperature 0.0 is consistent with the uncertainty motivation.","section":"Eqs. (2e), (8) and Table III"},{"comment":"All quantitative claims are based on single evaluation runs, with no error bars or significance tests. Some of the headline improvements are small in magnitude—for example, R-CLS 78.13 vs. 77.75 for Diffusion-ES on Test14-Hard in Table IV—so the SOTA claims are not robust to run-to-run variability. The manuscript should report multiple independent runs or confidence intervals, or at least state clearly whether the nuPlan evaluation is deterministic and whether the LMM calls introduce nondeterminism.","section":"Tables IV-VI and Fig. 6"}],"minor_comments":[{"comment":"The phrase 'exceptional capability in handing long-tail' contains a typo: 'handing' should be 'handling'.","section":"Section IV-B, finding 1"},{"comment":"The abbreviation 'EAV' is used in the agent responses but is not defined in the caption or text; it should be 'ego vehicle' for clarity.","section":"Fig. 5"},{"comment":"The action codes such as 'AR', 'CR', and 'DK' used in the textual prompts are only explained in the Fig. 5 example; a compact definition in the main text would improve readability.","section":"Section III-A2 and Fig. 5"},{"comment":"The PlanAgent baseline is reported only with NR-CLS and R-CLS, with dash entries for success rates; consider obtaining or clearly stating why the success-rate metrics are unavailable, since Table IV is used for cross-method comparison.","section":"Table IV"},{"comment":"The sentence 'with the only difference that we choose a proposal number of 128 instead of 32' is ambiguous about whether the Diffusion-ES baseline row in Table IV was run with 32 proposals in this work or quoted from the original publication; please clarify the exact provenance of the baseline numbers.","section":"Section IV-A"}],"recommendation":"major_revision","confidential_remarks":"The paper addresses a timely and relevant topic, and the proposed architecture is well motivated. However, the empirical validation currently does not support the strong SOTA claims because of the confounding proposal-count difference with Diffusion-ES, the parameter tuning on the evaluation benchmark, and the lack of confidence-calibration evidence. These issues are addressable with additional experiments and should be resolved before publication. The parameter-selection reporting in Section IV-E may also raise concerns about cherry-picking; the authors should be asked to either adopt the best-found configuration or provide a clear justification for the configuration they report."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take on CALMM-Drive. The framework idea is sensible: have an LMM propose several driving decisions with verbalized confidences, guide a diffusion planner with each decision, then pick the trajectory using both confidence and planning score. As far as I know, that combination is new, and the paper's ablations on the reasoning stages and the selector components show the pieces matter. I believe the authors when they say the full pipeline beats their own single-decision variant.\n\nThat said, the headline result against Diffusion-ES does not hold up as stated. In Section IV-A they say their Diffusion-ES uses a proposal number of 128 instead of 32. The baseline in Table IV is the published Diffusion-ES with N=32. So the comparison differs by two things at once: the LMM/confidence module and the size of the proposal set. Since more proposals help evolutionary diffusion planning, the +4 NR-SR on Test14-Hard could come from N alone. There is no N=128 Diffusion-ES run, and no ablation on N. That is a load-bearing gap, not a quibble.\n\nThere is also a parameter-selection issue. Figure 6 shows K=5 and C_d=1 s both beat the reported K=3, C_d=2 s on the same Test14-Random benchmark. The authors acknowledge this but still report the worse configuration in Table IV. That suggests the headline numbers are not the result of an honest model-selection procedure. The lack of error bars and the quoting of baselines from other papers are softer problems, but they add up.\n\nThe confidence scores themselves are the other soft spot. They import Top-K elicitation from LLM calibration work without any calibration analysis for driving, and the qualitative examples show confidence values like 0.9 for the selected action, which is exactly what you'd want to see, but it is not evidence. Temperature 0.0 is not necessarily a problem for Top-K elicitation, but it does cut against the forking-paths motivation.\n\nFor whom is this useful? The AV planning and LMM integration crowd. The framework is a reasonable template and the ablations are a good start. But as it stands, the central quantitative claim is not supported. I would send it to review—the idea is worth referee time and the flaws are addressable—but I would not accept it without an N=128 Diffusion-ES ablation, a defensible parameter-selection protocol, and some confidence-calibration evidence.","headline":"The framework idea is genuinely new and the ablations are promising, but the headline SOTA claim over Diffusion-ES is confounded by the un-ablated change in proposal count (128 vs 32), and the paper's own parameter study undermines the reported configuration.","tokens_in":21292,"tokens_out":2450,"would_cite":false,"duration_ms":24605,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"An LMM that proposes several driving decisions with confidence scores, then lets a diffusion planner generate and score trajectories for each, improves closed-loop driving reliability on long-tail scenarios, the paper argues.","keywords":["autonomous driving","large multimodal model","confidence elicitation","diffusion model","decision-making","motion planning","nuPlan","long-tail scenarios"],"falsifier":"Run the full pipeline on the same nuPlan benchmarks with the confidence term $c^k_t$ in Eq. (8) replaced by a random permutation of the reported confidences, keeping every other component fixed; if the success-rate and closed-loop scores on Test14-Hard and Test14-Random do not fall materially, then the verbalized confidence is not what carries the reliability gain.","tokens_in":20204,"feed_emoji":"🚗","tokens_out":12490,"duration_ms":108356,"temperature":0.7,"pith_summary":"This paper proposes an autonomous-driving method in which a large multimodal model (LMM) offers several plausible high-level decisions—such as accelerating, cruising, or changing lanes—each with a stated confidence, before any trajectory is planned. The claimed payoff is that these confidence-tagged decisions, used to guide diffusion-based trajectory generation and to choose the final trajectory, fix two known failure modes: decisions the planner cannot actually execute, and planners that pick safe-looking but tactically short-sighted trajectories. In nuPlan closed-loop simulations, the method reports the best success-rate and reactive closed-loop scores on the long-tail Test14-Hard benchmark, and it improves over the diffusion-only planner Diffusion-ES on both benchmarks and in both reactive and non-reactive modes. The method thus offers a way to add knowledge-driven reliability to a capable planner without retraining it, provided the reported confidence values carry real information about decision quality.","feed_headline":"Confidence-aware AI decisions cut long-tail driving errors by 4 points","feed_subtitle":"A multimodal model that hedges with several plans and confidence levels beats one-shot planners on hard traffic.","key_machinery":"The machinery is a hierarchical, confidence-gated selection over decision-conditioned trajectory sets. An LMM first reasons through scene understanding, action selection, and confidence assessment to produce K actions $a^k_t$ and scalar confidences $c^k_t \\in [0,1]$; an objective creator then assigns each action a decision-following objective $J^k_f$ (lane and speed adherence) combined with a general objective $J_g$ based on a rule-based Predictive Driver Model scorer, and a gradient-free diffusion optimizer evolves N trajectory proposals for each decision. A confidence-aware selector scores the best proposal for each decision with $S((X^k)^*) = (c^k_t)^{\\omega_c} (\\tilde J^k_f)^{\\tilde\\omega_f} (J_g)^{\\tilde\\omega_g}$ and picks the maximum, so a candidate is penalized for low tactical confidence, poor decision-following, or low general planning quality. The first-stage weights favor following the decision, while the second-stage weights rebalance toward general quality, which is what lets the pipeline keep long-term tactical goals in view without ignoring short-term smoothness and safety.","core_discovery":"The paper's central claim is that treating an LMM's driving suggestions as a set of hypotheses with confidence weights, rather than as a single authoritative command, measurably improves closed-loop driving reliability. The pipeline uses a three-step reasoning procedure—scene understanding, action selection, and confidence assessment—to elicit K candidate decisions with scalar confidences, converts each decision into a decision-following objective, generates N trajectory proposals per decision with a gradient-free diffusion optimizer (Diffusion-ES), and then selects the final trajectory by a product of the confidence score, the decision-following score, and a general driving-quality score. On the long-tail Test14-Hard benchmark it reports an NR-SR of 88.97% versus 84.93% for Diffusion-ES, an R-CLS of 78.13 versus 77.75, and an R-SR of 89.71% versus 87.13%, with success rates above 95% on Test14-Random without any rule-based emergency braking. The ablation studies attribute the gains to the two-stage Top-K reasoning and to including the confidence term in the selector; removing the confidence term lowers NR-CLS by 4.13 and R-CLS by 4.27 on Test14-Random.","pith_inferences":["A testable extension the paper does not run: calibrate the verbalized confidences against actual closed-loop decision quality on a held-out scenario set and re-run the selector; if calibration improves the correlation between $c^k_t$ and plan success, the reported long-tail gains should grow, and if it does not, the current confidence weighting is fragile.","Because the LMM is queried at temperature 0.0, the Top-K confidences are a single deterministic readout rather than a sample over forking paths; sampling several responses at higher temperature and aggregating confidences would directly test the paper's stated uncertainty rationale.","The same decision-guided generation and confidence-aware selection could wrap any multimodal trajectory proposer, not only diffusion-based ones, so rule-based or learned proposers with stronger local smoothness might combine with the tactical confidence term to close the remaining CLS gap on Test14-Random.","The reported sweet spots at K = 5 and a 1 s decision cycle suggest that an adaptive decision cycle—querying the LMM more often in dense interactions and less often on open roads—could capture most of the reliability gain at a fraction of the inference cost."],"forward_implications":["LMM-empowered driving systems no longer need one-shot decisions: asking for several hypotheses with confidences and arbitrating them through planning is itself a reliability mechanism.","On Test14-Hard, the method reports the best NR-SR (88.97%), R-CLS (78.13), and R-SR (89.71%) among the compared planners, with success-rate margins over Diffusion-ES of +4.04 percentage points non-reactive and +2.58 percentage points reactive.","On Test14-Random, the method exceeds 95% reactive and non-reactive success rates without rule-based emergency braking, coming within 0.77 and 0.76 percentage points of the best success rates respectively.","Ablations imply that both the two-stage Top-K reasoning and the confidence term are load-bearing: single-decision reasoning scores worst on every metric, and dropping the confidence term from the selector costs 4.13 NR-CLS and 4.27 R-CLS.","Parameter studies show the architecture still has headroom: K = 5 candidates yields R-CLS 88.32 and R-SR 97.32%, above the reported default configuration."],"supporting_citations":[{"why":"Provides the Diffusion-ES gradient-free diffusion planner that generates and evolves the trajectory proposals for each LMM decision; it is the main planning backbone whose success rates the proposed method improves.","marker":"[13]"},{"why":"Supplies the Top-K confidence elicitation prompting method used to obtain multiple candidate decisions with verbalized confidence scores from the LMM.","marker":"[37]"},{"why":"Supplies the PDM rule-based trajectory scorer used as the general objective $J_g$ in both objective creation and confidence-aware selection, and the PDM-Closed baseline.","marker":"[17]"},{"why":"Introduces the nuPlan closed-loop simulation benchmark and the official metrics (NR-CLS, R-CLS, and success rates) on which all comparisons are reported.","marker":"[38]"},{"why":"PlanAgent is the LMM-empowered planning baseline the paper compares against, and its single-decision iterative approach motivates the Top-K single-round reasoning design.","marker":"[42]"},{"why":"Backs the key premise that verbalized confidences can be calibrated to response accuracy, the foundation for treating $c^k_t$ as a decision-quality weight.","marker":"[50]"}],"fun_headline_variants":["Hedged confidence beats one-shot planning in autonomous driving","AI driving: confidence-weighted plans slice long-tail error rates","Multi-plan LMM reduces hard-scenario mistakes in closed-loop tests","Top-K confidence hedges beat single-shot driving decisions"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the numbers the model reports as decision confidences genuinely reflect how good each driving decision is; the paper imports this from natural-language calibration results without checking it against driving outcomes, and it runs the language model in a mode where the same question always gets the same answer, so the reported scores are not separately calibrated against driving success.","fun_headline_variants_meta":{"raw":{"variants":["Hedged confidence beats one-shot planning in autonomous driving","AI driving: confidence-weighted plans slice long-tail error rates","Multi-plan LMM reduces hard-scenario mistakes in closed-loop tests","Top-K confidence hedges beat single-shot driving decisions"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000713,"raw_usage":{"total_tokens":3265,"prompt_tokens":1059,"completion_tokens":2206,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":675,"completion_tokens_details":{"reasoning_tokens":2138}},"tokens_in":675,"tokens_out":2206,"duration_ms":17215,"temperature":1.0,"reasoning_tokens":2138,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T21:38:26.688998+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the full pipeline on the same nuPlan benchmarks with the confidence term $c^k_t$ in Eq. (8) replaced by a random permutation of the reported confidences, keeping every other component fixed; if the success-rate and closed-loop scores on Test14-Hard and Test14-Random do not fall materially, then the verbalized confidence is not what carries the reliability gain.","supporting_citations":[{"cited_title":"Diffusion-ES: Gradient-free planning with diffusion for autonomous and instruction-guided driving,","cited_arxiv_id":null,"evidence_quote":"Provides the Diffusion-ES gradient-free diffusion planner that generates and evolves the trajectory proposals for each LMM decision; it is the main planning backbone whose success rates the proposed method improves."},{"cited_title":"Just ask for calibration: Strategies for eliciting calibrated confidence scores from language models fine-tuned with human feedback,","cited_arxiv_id":null,"evidence_quote":"Supplies the Top-K confidence elicitation prompting method used to obtain multiple candidate decisions with verbalized confidence scores from the LMM."},{"cited_title":"Parting with misconceptions about learning-based vehicle motion planning,","cited_arxiv_id":null,"evidence_quote":"Supplies the PDM rule-based trajectory scorer used as the general objective $J_g$ in both objective creation and confidence-aware selection, and the PDM-Closed baseline."}],"review_version":1}