{"id":"a877deae-befa-4979-882e-e824837b9d4b","arxiv_id":"2412.15208","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Adding a chain-of-thought reasoning step before predicting speed and curvature improves zero-shot trajectory planning of open multimodal LLMs on nuScenes, with code released.","lead":"OpenEMMA is an open-source system that asks off-the-shelf multimodal AI models to reason about driving scenes before predicting the car's future speed and steering, then converts those into a trajectory. On 150 nuScenes driving clips it improves planning over the same models without reasoning, though not consistently for every model.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The CoT-attribution claim is untested: Table 1 shows Qwen2-VL L2 worsens (2.46→2.81), no error bars or ablations are provided, and baseline prompts/parsing are undisclosed; an ablation is required before 'CoT improves planning' can be accepted.","rationale":"The reader's verdict is conditional, and the missing ablation is exactly the right concern. I do not see a reason to change the verdict: the framework is useful and the code release is a concrete contribution, but the headline causal claim is not yet supported. The strongest evidence would be a paired prompt ablation. One nuance beyond the reader's note: Table 1 already contains an internal inconsistency with the 'consistently outperforms' sentence—Qwen2-VL-7B-Instruct L2 worsens—so the paper's own data weaken the central claim even before the causal attribution question. However, this is a correctable experimental-reporting issue rather than a flaw in the method itself, so conditional acceptance with required ablations and prompt disclosure remains the right call.","tokens_in":11526,"tokens_out":4485,"duration_ms":40596,"concrete_test":"Run a controlled ablation on the same 150 nuScenes validation scenes, for LLaVA-1.6-Mistral-7B, Llama-3.2-11B-Vision-Instruct, and Qwen2-VL-7B-Instruct: (A) the exact OpenEMMA CoT prompt; (B) the exact Stage-2 prompt with the reasoning block replaced by a fixed placeholder chain of matched length (e.g., 'The scene is shown in the image. The ego status is provided.'); (C) a zero-shot prompt that uses the same speed/curvature output format and same parsing but omits the reasoning block. Report per-scene paired L2, failure rate, and 95% confidence intervals over at least 3 seeds. If (B) matches (A) within noise, or (C) matches (A), then CoT content is not the causal ingredient and the headline claim should be revised.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central causal claim is stated in §3.1: OpenEMMA 'consistently outperforms the zero-shot baseline in both L2 norm error and failure rate, demonstrating the effectiveness of the Chain-of-Thought reasoning process.' Table 1 does not support that statement. For Qwen2-VL-7B-Instruct, average L2 is worse under OpenEMMA (2.81 m vs 2.46 m), and for Llama-3.2-11B the L2 gain is small (3.00→2.92 m). The baseline is described only as using 'historical ego status and the driving scene image, without incorporating any reasoning process'; its prompt, output representation, and parsing are never given. Thus the comparison changes at least two variables at once—the reasoning stage and the output-format/prompt specification. With 150 scenes and no seeds, error bars, or paired per-scene tests, the word 'significant' in the abstract cannot be verified. The failure-rate metric (L2 > 10 within the first second) is also nonstandard and could be dominated by parse/output-format failures rather than driving competence. The load-bearing assumption is that CoT content, not test-time compute or formatting, causes the improvement; nothing in the paper isolates that factor.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This manuscript introduces OpenEMMA, a training-free pipeline that uses pre-trained multimodal large language models (MLLMs) with a chain-of-thought reasoning stage to predict the ego vehicle's speed and curvature, which are then integrated into a future trajectory. A fine-tuned YOLO-style 3D detector is added for object detection. The authors evaluate the planning component on the nuScenes validation set with three open MLLMs and GPT-4o case studies, comparing against a zero-shot baseline. The central claim is that the chain-of-thought reasoning process yields consistent and significant improvements in L2 trajectory error and failure rate.","tokens_in":11929,"tokens_out":8076,"duration_ms":57197,"significance":"If the central claim were supported, OpenEMMA would be a valuable contribution: a computationally efficient, open-source recipe for MLLM-based trajectory planning that requires no fine-tuning of the language model. The paper releases code, uses off-the-shelf open models, and the speed/curvature integration in Section 2.1 is transparent and easy to reproduce. The approach is timely given the interest in EMMA-style end-to-end AD. However, the current experimental evidence is insufficient to establish the claimed benefits, and some of the reported results contradict the paper's own conclusions.","major_comments":[{"comment":"The sentence 'OpenEMMA consistently outperforms the zero-shot baseline in both L2 norm error and failure rate, demonstrating the effectiveness of the Chain-of-Thought reasoning process' is not supported by Table 1. For LLaVA-1.6-Mistral-7B, the failure rate under OpenEMMA is 6.12% versus 4.06% for the zero-shot baseline, an increase; for Qwen2-VL-7B-Instruct, the average L2 error increases from 2.46 m to 2.81 m. The text acknowledges the Qwen2-VL L2 regression but not the LLaVA failure-rate regression, and neither is explained with per-scene evidence.","section":"Section 3.1, Table 1"},{"comment":"The zero-shot baseline is not specified in enough detail to isolate the contribution of chain-of-thought reasoning. The manuscript only states that the baseline uses historical ego status and the driving scene image 'without incorporating any reasoning process,' and does not report the prompt, output representation (speed/curvature versus direct waypoints), parsing, or formatting. Consequently the comparison in Table 1 varies the reasoning stage and the prompt/output format simultaneously, so the observed improvements cannot be causally attributed to CoT. An ablation that fixes the output format and prompt length is required.","section":"Section 3.1, baseline"},{"comment":"The evaluation uses 150 scenes with no error bars, no variance estimates, and no significance tests, yet the abstract and Section 3.1 use the word 'significant.' The failure-rate metric (L2 > 10 m within the first second) is nonstandard and may be dominated by output-format or parsing failures; the paper should report parse-success rates or use a standard metric such as displacement error at multiple horizons. At minimum, paired per-scene tests (e.g., bootstrap or Wilcoxon) are needed to support the significance claim.","section":"Section 3.1, statistics"},{"comment":"The explanation that OpenEMMA's Qwen2-VL average L2 increases because the model 'successfully generates predictions for many cases where the zero-shot baseline fails' but then 'struggles to produce high-quality trajectories in these challenging scenarios' is not accompanied by any per-scene statistics, such as a breakdown of errors on scenes where the baseline succeeds versus fails, or a difficulty-based stratification. Without such evidence, the explanation is post hoc and does not support the overall claim of effectiveness.","section":"Section 3.1, Qwen2-VL post hoc explanation"},{"comment":"The 3D detection contribution is not evaluated quantitatively. The paper reports only the fine-tuned 2D detector's mAP50 (0.60316) and states that the 3D estimation network uses existing YOLO3D weights; no 3D detection metric (e.g., BEV mAP or 3D bounding box AP) is provided. The claim of 'precise detection of 3D on-road objects' is therefore unsupported.","section":"Section 3.2, object detection"}],"minor_comments":[{"comment":"The manuscript uses both 'OpenEMMA' and 'Open-EMMA' inconsistently; please standardize to one form.","section":"Throughout"},{"comment":"The phrase 'We release all the codes' should be corrected to 'We release the code.'","section":"Abstract"},{"comment":"The sentence reporting the best mAP50 of 0.60316 at epoch 290 should specify the validation protocol and ideally include a learning curve to demonstrate that the fine-tuning converged.","section":"Section 3.2"},{"comment":"The related work section would be strengthened by a quantitative comparison with prior MLLM-based AD methods such as DriveVLM and LMDrive, rather than only a qualitative summary.","section":"Section 4, Related Work"}],"recommendation":"major_revision","confidential_remarks":"The stress-test concern from the reader's report is on point: the central claim of CoT-driven improvement is not supported by the current Table 1, and the baseline is underspecified. The paper's open-source release and clear integration math are strengths, but the experimental evaluation needs substantial strengthening before publication. I recommend major revision rather than rejection because the flaws are fixable with additional experiments and analysis."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"OpenEMMA is a genuinely useful open-source artifact: an EMMA-style MLLM planner with chain-of-thought prompting, speed/curvature intermediate outputs, and an external YOLO3D detector, with code and weights released. For researchers who want a zero-shot MLLM driving baseline without fine-tuning, that is worth having. The integration math is standard and correct, and the speed/curvature representation is a clean, sensible choice.\n\nThe soft spot is the headline claim. Table 1 does not show “significant improvements” across the board: Qwen2-VL’s average L2 error is worse under OpenEMMA (2.81 vs 2.46 m), Llama-3.2’s gain is marginal (3.00 to 2.92 m), and the failure-rate drops, though consistent, rely on a nonstandard metric (L2 > 10 within the first second) that can be driven by parse or format failures rather than driving competence. There are no error bars, no seeds, no paired per-scene tests, and only 150 scenes with no stated selection criterion. More importantly, the baseline prompt and parsing are never specified, so the comparison changes at least two variables at once—the reasoning stage and the output format. The paper attributes the gains to CoT “effectiveness” in §3.1, but nothing isolates that factor. The load-bearing claim is untested.\n\nThis is fixable. An ablation that fixes the output format and toggles only the reasoning stage, plus seeds, full prompt disclosure, and a clearer definition of failure, would make the comparison interpretable. The limitation section is honest about the basic CoT and the external detector, and the authors acknowledge the Qwen anomaly rather than hide it.\n\nWho is this for? People working on MLLM-based planning who want a practical open baseline and a starting point for controlled comparisons. My own verdict on the central claim is skeptical, but the artifact deserves referee time. Send it out with a request for the missing controls.","headline":"Useful open-source EMMA-style system, but the CoT improvement claim is not supported by the experiments as presented.","tokens_in":12334,"tokens_out":2985,"would_cite":true,"duration_ms":27245,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Inserting a chain-of-thought reasoning step into a pretrained vision-language model improves zero-shot trajectory planning on nuScenes.","keywords":["autonomous driving","multimodal large language models","chain-of-thought prompting","end-to-end trajectory planning","zero-shot driving","nuScenes validation","speed-curvature integration","3D object detection"],"falsifier":"Run the same 150-scene nuScenes comparison with a sham reasoning prompt that produces the same structured text (intent, scene, objects) but fills the scene content with generic or blank statements; if L2 error and failure rate stay at the zero-shot baseline level the CoT claim is supported, while matching OpenEMMA's numbers would show the effect is prompt structure rather than reasoning.","tokens_in":11350,"feed_emoji":"🚗","tokens_out":10322,"duration_ms":68413,"temperature":0.7,"pith_summary":"OpenEMMA's central claim is that a pretrained multimodal language model can plan a vehicle trajectory without any fine-tuning of the language model, provided it is first made to reason about the driving scene. The framework prompts the model to state its intent, describe the scene, list the important objects, and only then output the future speed and curvature that are integrated into a path. On 150 scenes from the nuScenes validation set, the paper reports that this two-stage prompt lowers average trajectory error for two open models and lowers the failure rate for two of the three models tested, with one model trading higher error for far fewer failures. If true, this offers an accessible, open route to EMMA-style end-to-end autonomous driving for researchers who lack proprietary models or large fine-tuning budgets.","feed_headline":"Chain-of-thought prompts improve zero-shot driving planners","feed_subtitle":"A prompt-only recipe lowers trajectory error and failure rate for several open models on nuScenes.","key_machinery":"The load-bearing mechanism is a two-stage prompt wrapped around a pretrained MLLM. Stage 1 (Reasoning) forces the model to commit to an intent command, a concise scene description, and the major objects with their locations and relevance; Stage 2 (Predicting) conditions the numerical output on that written reasoning. The output parameterization is speed $s_t$ and curvature $k_t$ for the next $T$ seconds, which are integrated with the cumulative trapezoidal rule into heading $\\theta_t$, velocities $v_x(t), v_y(t)$, and positions $(x_t, y_t)$. For 3D object detection, OpenEMMA adds an externally fine-tuned YOLO3D detector, since off-the-shelf MLLMs give poor spatial localization.","core_discovery":"The paper asserts that adding Chain-of-Thought reasoning before the numerical output is what drives the improvement. OpenEMMA takes the front camera image and five seconds of ego speed and curvature, prompts the MLLM for an intent command, a scene description, and a list of major objects with locations and relevance, then prompts it again to predict speed and curvature for the next five seconds. The trajectory is recovered by trapezoidal integration of heading and velocity. On the nuScenes validation set, the reported numbers show average L2 error falling from 3.24 m to 2.98 m for LLaVA-1.6-Mistral-7B and from 3.00 m to 2.92 m for Llama-3.2-11B-Vision-Instruct, and failure rate falling from 24.00% to 16.11% for Qwen2-VL-7B-Instruct; Qwen's average L2 rises because OpenEMMA now produces trajectories for hard cases the baseline completely failed. The authors interpret these results as evidence that Chain-of-Thought reasoning helps a diverse set of open MLLMs understand and react to complex driving scenes.","pith_inferences":["We would test whether a longer structured prompt with no reasoning content reproduces the gains; if it does, the active ingredient is formatting or output length, not reasoning.","If the reasoning stage is genuinely causal, then sampling multiple reasoning chains (self-consistency or tree-of-thought, which the paper names as future work) should reduce failure rates further.","The speed/curvature output space is itself a constraint: it biases predictions toward smooth, physically plausible arcs. A variant that outputs waypoints directly but keeps the same reasoning stage would separate the parameterization effect from the reasoning effect."],"forward_implications":["A no-fine-tuning, prompt-only recipe can serve as a reproducible open baseline for MLLM-based planning, bypassing the closed training pipeline of proprietary EMMA.","The reported failure-rate drops (for Llama-3.2, 23.92% to 22.00%; for Qwen2-VL, 24.00% to 16.11%) mean the reasoning prompt chiefly prevents large first-second deviations, not just small path errors.","Because the intermediate outputs are human-readable intent, scene, and object notes, OpenEMMA yields an interpretable explanation alongside every planned trajectory.","The speed-curvature parameterization makes every predicted trajectory dynamically smooth and consistent by construction, so this representation may transfer across different MLLM backbones."],"supporting_citations":[{"why":"Defines the EMMA framework—camera plus ego status, driving as visual question answering—that OpenEMMA replicates with open components.","marker":"[18]"},{"why":"Supplies the nuScenes dataset and nuImages data used for the 150-scene validation and for fine-tuning the 2D detector.","marker":"[6]"},{"why":"Introduces Chain-of-Thought prompting, the intervention whose effect on trajectory planning is the paper's central claim.","marker":"[60]"},{"why":"LLaVA-1.6-Mistral-7B, one of the backbone MLLMs in the zero-shot versus OpenEMMA comparison.","marker":"[22]"},{"why":"Llama-3.2-11B-Vision-Instruct, a second backbone whose failure rate drops from 23.92% to 22.00%.","marker":"[33]"},{"why":"Qwen2-VL-7B-Instruct, a third backbone whose failure rate drops from 24.00% to 16.11%.","marker":"[57]"},{"why":"YOLO3D, the monocular 3D detection method integrated into OpenEMMA for object detection.","marker":"[34]"},{"why":"Provides the YOLO11n implementation and pretrained weights that OpenEMMA fine-tunes on nuImages for 2D detection.","marker":"[1]"}],"fun_headline_variants":["OpenEMMA: Chain-of-thought cuts driving errors","OpenEMMA: Open-source chain-of-thought driving","Chain-of-thought prompts boost open-source driving","OpenEMMA: Zero-shot driving with chain-of-thought","OpenEMMA: Chain-of-thought for robust autonomous driving"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper attributes the gains to Chain-of-Thought reasoning without running a control that isolates reasoning from prompt length, formatting, or output parsing; if the difference comes from those factors, the central causal claim collapses.","fun_headline_variants_meta":{"raw":{"variants":["OpenEMMA: Chain-of-thought cuts driving errors","OpenEMMA: Open-source chain-of-thought driving","Chain-of-thought prompts boost open-source driving","OpenEMMA: Zero-shot driving with chain-of-thought","OpenEMMA: Chain-of-thought for robust autonomous driving"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000415,"raw_usage":{"total_tokens":2151,"prompt_tokens":962,"completion_tokens":1189,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":578,"completion_tokens_details":{"reasoning_tokens":1108}},"tokens_in":578,"tokens_out":1189,"duration_ms":12969,"temperature":1.0,"reasoning_tokens":1108,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T11:31:52.231937+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same 150-scene nuScenes comparison with a sham reasoning prompt that produces the same structured text (intent, scene, objects) but fills the scene content with generic or blank statements; if L2 error and failure rate stay at the zero-shot baseline level the CoT claim is supported, while matching OpenEMMA's numbers would show the effect is prompt structure rather than reasoning.","supporting_citations":[{"cited_title":"Chain-of-thought prompting elicits reasoning in large language models, 2023","cited_arxiv_id":null,"evidence_quote":"Introduces Chain-of-Thought prompting, the intervention whose effect on trajectory planning is the paper's central claim."},{"cited_title":"Llama 3.2: Revolutionizing edge ai and vision with open, customizable models","cited_arxiv_id":null,"evidence_quote":"Llama-3.2-11B-Vision-Instruct, a second backbone whose failure rate drops from 23.92% to 22.00%."},{"cited_title":"3d bounding box estimation using deep learn- ing and geometry","cited_arxiv_id":null,"evidence_quote":"YOLO3D, the monocular 3D detection method integrated into OpenEMMA for object detection."},{"cited_title":"https://github.com/ultralytics/ ultralytics","cited_arxiv_id":null,"evidence_quote":"Provides the YOLO11n implementation and pretrained weights that OpenEMMA fine-tunes on nuImages for 2D detection."}],"review_version":1}