{"id":"98eaebcf-38f2-48ea-b84c-0859e1e9bbd7","arxiv_id":"2505.00284","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"LightEMMA evaluates twelve VLMs on nuScenes trajectory prediction with a single image and 3 seconds of history, and reports better ADE than OpenEMMA but no scaling benefit.","lead":"LightEMMA is an open-source framework that asks twelve commercial and open-source vision-language models to predict driving trajectories by describing a scene and outputting speed and curvature values. Across 150 nuScenes scenarios, it finds that larger models and longer reasoning do not improve prediction accuracy, and that all models are far too slow for real-time driving.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 47.6% ADE improvement over OpenEMMA is not controlled: the paper changes input frames, history representation, and prompt template while comparing against published OpenEMMA numbers without a re-run or ablation.","rationale":"The reader's weakest_assumption identifies the core issue: the comparison to OpenEMMA is not controlled. My analysis sharpens this by pointing to the specific uncontrolled variables that the paper itself acknowledges (single-frame vs multi-frame input, speed/curvature vs cartesian history, prompt adjustments) and by noting that the OpenEMMA baseline rows appear to be published numbers rather than re-runs under LightEMMA's protocol. This is the most load-bearing concern because the paper's headline quantitative claim of superiority rests entirely on that 47.6% reduction. If the reduction actually stems from a different prompt or input representation, the claim 'improvement arises from framework design' collapses, although the paper's broader contribution as an open benchmark and its qualitative analyses would retain value. The concrete test directly isolates the causal factor. I agree with the reader's assessment and recommend keeping a CONDITIONAL verdict pending such a controlled comparison or explicit ablations.","tokens_in":10753,"tokens_out":3394,"duration_ms":31927,"concrete_test":"Obtain the OpenEMMA repository and run its original LLaMA-3.2-11B configuration on the same 150 nuScenes scenarios used by LightEMMA, evaluating with LightEMMA's evaluation code (same frame filtering and metrics). Then run LightEMMA while reverting to OpenEMMA's inputs: multi-frame images, cartesian history representation, and the original OpenEMMA prompt template. If ADE remains near 1.53 m when only framework code changes, the attribution holds; if ADE moves toward 2.92 m when inputs/prompts revert, the improvement is caused by input/prompt changes, not by the modular design. Also verify whether the OpenEMMA rows in Table II are re-run results or cited numbers.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim (Section IV.D) that LightEMMA's 47.6% ADE reduction over OpenEMMA 'arises from framework design and integration strategies' is unsupported because the comparison is not controlled. The paper itself lists multiple methodological differences: Section III.B uses only the current frame, whereas OpenEMMA concatenates multiple past frames (as stated: 'rather than concatenating multiple past frames as performed in previous studies [13], [14]'); Section III.C encodes history as (speed, curvature) pairs rather than cartesian coordinates; Section III.D applies 'minor adjustments' to the CoT prompt. Any of these differences, or the new error-handling and frame-filtering procedure, could plausibly explain the improvement. The Table II rows for OpenEMMA show FE and FE Corr as '-' and no re-run is described; the numbers appear to be taken directly from [13], which may use different scenarios, frame filtering, or evaluation code. Without ablations that vary one component at a time, or a re-run of OpenEMMA inside LightEMMA's evaluation harness, the attribution to 'framework design' is a post-hoc label rather than a demonstrated cause.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"LightEMMA is presented as a modular, open-source, zero-shot framework that feeds a single front-view camera image plus a 3-second history of (speed, curvature) pairs to a vision-language model, uses a chain-of-thought prompt to elicit six low-level driving commands, and integrates those commands kinematically to produce a 3-second trajectory. The manuscript evaluates twelve commercial and open-source VLMs on the nuScenes prediction task, reporting inference time, token usage and cost, format-error rates, average/final displacement error, and qualitative failure modes. Its headline quantitative claims are a 47.6% reduction in average ADE for LLaMA-3.2-11B relative to OpenEMMA using the same pretrained weights, a similar reduction for Qwen-2.5-7B, and the observation that advanced or reasoning-enabled model variants do not outperform their basic counterparts.","tokens_in":11163,"tokens_out":5352,"duration_ms":53283,"significance":"If the improvement over OpenEMMA were established under controlled conditions, this would be a useful benchmark contribution: the framework is modular, the code is publicly released, all evaluated models run under a single prompting and evaluation protocol, and the trajectory construction uses textbook kinematic equations with no parameters fitted to nuScenes labels. The paper also provides a broader model survey than prior open-source work and documents cost and latency in a way that is often missing from VLM driving papers. However, the central superiority claim is not yet supported because the OpenEMMA comparison is not controlled, and the reported accuracy numbers are only means without variance or significance information. The benchmark and qualitative insights are plausible and valuable, but the causal attribution of the ADE improvement to framework design requires additional evidence.","major_comments":[{"comment":"The claim that the 47.6% average-ADE reduction over OpenEMMA 'arises from framework design and integration strategies' is not supported by the evidence presented. The OpenEMMA rows appear to be taken from Ref. [13] rather than re-run in the same evaluation harness, and the LightEMMA pipeline differs in at least three documented respects: Section III.B uses only the current frame instead of concatenating multiple past frames, Section III.C encodes driving history as (speed, curvature) pairs rather than cartesian coordinates, and Section III.D applies 'minor adjustments' to the chain-of-thought prompt. Any one of these differences, or the error-parsing and frame-filtering procedure in Section IV.C, could plausibly explain the improvement. Please add ablations that vary one component at a time, or at minimum re-run OpenEMMA inside LightEMMA's evaluation protocol and report both configurations; otherwise the attribution to framework design should be softened to a descriptive comparison rather than a causal claim.","section":"Section IV.D, Table II"},{"comment":"Only mean ADE and FDE values are reported, with no per-scenario distributions, error bars, or statistical tests. The test set is 150 scenarios, VLM outputs are stochastic, and some model pairs are very close (e.g., GPT-4o at 1.07 m versus Claude-3.7-Sonnet at 1.08 m average ADE). To support the model ranking and the claimed 47.6% reduction, please report per-scenario paired errors and a statistical comparison such as a paired test or bootstrap confidence intervals. This is particularly important because the frame-filtering step leaves 3,893 of 3,908 frames, and the effect of filtering on each model's mean is not shown.","section":"Section IV.D, Table II"},{"comment":"The handling of Qwen-2.5-72B is inconsistent with the stated frame-filtering rule. The manuscript says frames are excluded only when any model fails to generate a valid prediction based on the corrected results, but Qwen-2.5-72B is then 'excluded entirely' because of its 62.9% format-error rate (47.5% after correction). This is a post-hoc model-level exclusion rather than the described frame-level filtering, and it changes the evaluated roster from twelve models to eleven. Please include the valid corrected trajectories of the 72B model if possible, justify the exclusion with a principled criterion, or revise the text so that the twelve-model claim is not contradicted by the accuracy table.","section":"Section IV.C, Table II"}],"minor_comments":[{"comment":"The cross-reference to 'Section II-F' appears to point to the example workflow, which actually appears in Section III.F; please correct the reference.","section":"Section III.B"},{"comment":"The heading 'VLM Promoting' should read 'VLM Prompting'.","section":"Section III.D"},{"comment":"The dagger symbols next to the input-token counts for Gemini and LLaMA models are not explained in the table or caption; add a footnote noting that those providers appear to omit image tokens from the reported counts.","section":"Table I"},{"comment":"Add a table note stating that the two OpenEMMA rows are reproduced from Ref. [13] and were not re-run in the present harness, since the '-' entries in the FE and FE Corr columns otherwise hide this asymmetry.","section":"Table II"},{"comment":"Statements such as 'all models consistently fail' in Cases 1 and 2 are made for selected illustrative frames; please qualify them by stating the number of cases observed or labeling them explicitly as qualitative observations rather than quantitative findings.","section":"Section V"},{"comment":"The sentence 'Since all models were evaluated using identical prompts and workflows' should be clarified to mean identical within LightEMMA, because Section III.D notes that the CoT prompt includes adjustments relative to prior studies and the OpenEMMA comparisons are not re-run under the same workflow.","section":"Section IV.C"}],"recommendation":"major_revision","confidential_remarks":"The paper has a useful benchmark contribution and the code release is a strength, but the headline comparison to OpenEMMA is not yet controlled. I would be willing to review a revision that adds ablations or a re-run of OpenEMMA, or that explicitly downgrades the causal claim to a descriptive one. The post-hoc exclusion of Qwen-2.5-72B and the absence of any uncertainty quantification should also be addressed. The fit with cs.RO is appropriate."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know two things about this paper. First, it ships a clean, modular open-source framework for zero-shot VLM driving evaluation on nuScenes, plus a 12-model comparison with cost and token data. Second, its headline number—47.6% ADE improvement over OpenEMMA with the same LLaMA weights—is real as a number, but the causal attribution to \"framework design\" is not actually demonstrated.\n\nThe framework is genuinely useful: modular code, error handling, response logging, and retrospective analysis. The evaluation protocol is uniform across models, and trajectory generation uses textbook kinematics with no fitted parameters. The cost and inference-time table is a nice addition, and the qualitative case studies are illustrative. The finding that larger or reasoning-enabled models do not consistently beat their base counterparts is an empirical observation worth reporting.\n\nThe soft spot is the controlled comparison. The 47.6% figure comes from comparing directly against OpenEMMA's published numbers in Table II, not from a re-run inside LightEMMA's own harness. The paper itself lists at least three differences: single frame instead of multi-frame input, history as speed/curvature pairs instead of cartesian coordinates, and \"minor adjustments\" to the prompt. Any of these could drive the improvement. Without ablations that vary one component at a time, the sentence in Section IV.D claiming the improvement \"arises from framework design\" is an over-claim. Also, means-only reporting without confidence intervals leaves the scaling finding suggestive, not solid.\n\nThis is for researchers who want a working baseline and a model zoo for zero-shot VLM driving. It deserves a serious referee, but the authors should either soften the causal claim or back it with ablations. If they can do that, the paper becomes a solid benchmark contribution. As it stands, it is a useful extension with one unsupported interpretive leap.\n\nI would send it to review, but I would ask for a re-run of OpenEMMA inside their harness and at least a one-component ablation. If they cannot run OpenEMMA, they should at least concede that the comparison is uncontrolled.","headline":"A useful open-source evaluation framework for zero-shot VLM driving, but the headline 47.6% ADE gain over OpenEMMA is an uncontrolled comparison and the causal claim over-reaches.","tokens_in":11451,"tokens_out":2664,"would_cite":true,"duration_ms":26064,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"With identical LLaMA weights, a modular VLM framework halves trajectory error on nuScenes, and bigger reasoning models do not help.","keywords":["LightEMMA","vision-language model","autonomous driving","trajectory prediction","chain-of-thought","nuScenes","zero-shot","open-source framework"],"falsifier":"Take OpenEMMA's LLaMA-3.2-11B deployment and swap in only LightEMMA's exact prompt template and single-frame input, keeping weights, evaluation code, and sampling fixed; if average ADE stays near 2.92 m rather than dropping to about 1.53 m, the framework-design attribution is falsified. Conversely, running LightEMMA's full pipeline with OpenEMMA's multi-frame input and prompt would reveal which component carries the gain.","tokens_in":10613,"feed_emoji":"🚗","tokens_out":4110,"duration_ms":35997,"temperature":0.7,"pith_summary":"LightEMMA is presented as an open-source baseline framework that lets any modern vision-language model drive by mapping a front-view image and the ego vehicle's recent speed-curvature history into a chain-of-thought description and then six low-level control commands. The paper's central quantitative claim is that, with the exact same LLaMA-3.2-11B weights used by OpenEMMA, the framework lowers average displacement error from 2.92 m to 1.53 m, a 47.6% reduction, which the authors attribute to framework design and integration rather than model weights. On 150 nuScenes prediction scenarios, GPT-4o reaches the lowest ADE at 1.07 m, followed closely by Claude-3.7-Sonnet, while larger or reasoning-enabled models such as GPT-5, Gemini-2.5-Pro, and Qwen-2.5-72B do not outperform their smaller or basic counterparts. The authors intend LightEMMA as a practical benchmark for fair VLM comparison, rapid model updates, and failure-mode analysis rather than as a production-ready planner.","feed_headline":"Same LLaMA weights, half the driving error","feed_subtitle":"An open modular VLM framework cuts average ADE from 2.92 m to 1.53 m on nuScenes, and larger reasoning models don't help.","key_machinery":"The load-bearing mechanism is a three-stage Chain-of-Thought prompt that forces the VLM to (1) describe the scene, (2) propose a high-level driving intent given scene and history, and (3) emit six (speed, curvature) pairs, which are then converted to a trajectory by kinematic integration with 0.5 s steps. Inputs are deliberately minimal: a single raw front-view camera frame and six historical speed-curvature pairs sampled over 3 seconds, with no auxiliary perception models. The framework's modular codebase separates initialization, prediction, logging, error handling, and retrospective analysis, which is what the authors say produces the accuracy gain and makes dynamic model swaps possible.","core_discovery":"The central claim is that a well-structured modular VLM framework, without any fine-tuning, can sharply improve open-loop trajectory prediction relative to the previous open-source baseline. Using the identical LLaMA-3.2-11B pretrained weights in both systems, LightEMMA reduces average ADE on nuScenes from 2.92 m to 1.53 m (47.6%), and with Qwen-2.5-7B versus OpenEMMA's Qwen-2.0-7B the reduction is 47.7%. Because the weights are held fixed in the LLaMA comparison, the paper concludes the gain comes from framework design and integration strategies. The paper further claims that advanced or reasoning-enabled VLM variants do not beat their basic counterparts and that GPT-4o achieves the best accuracy with ADE 1.07 m, establishing a substantially stronger open baseline.","pith_inferences":["Attribution of the entire ADE gain to modular framework design is not directly established: the paper does not ablate prompt template, single-frame input, output format, or evaluation filtering, so part of the gain may come from those choices; a controlled ablation would settle it.","The single-frame choice implies the framework ignores explicit temporal dynamics beyond the six-step history, so cases like the right-turn bias in Case 1 might be improved by richer history or explicit map cues, an extension the paper does not explore.","A testable prediction follows from the paper's own failure analysis: adding explicit navigation intent or map-level turn information should remove the systematic left-turn failures described in Case 2, which could be checked by appending a one-line navigation hint to the same prompt.","Open-loop nuScenes ADE may not predict closed-loop safety; the paper's qualitative cases suggest VLM stochasticity can cause abrupt braking or running red lights, so a closed-loop simulator evaluation would be a natural next step."],"forward_implications":["Any current or future VLM that accepts image and text can be plugged into the same evaluation pipeline, so trajectory accuracy, cost, latency, and format reliability become directly comparable across model families.","The LLaMA-3.2-11B result implies that framework choices alone can be worth roughly a factor of two in average displacement error, independent of model generation improvements.","Because stronger and costlier models did not outperform basic ones, the results push the field toward task-specific architecture and prompting design rather than simple parameter scaling for driving.","Single-frame, no-perception-module input keeps the pipeline cheap and fast, supporting use as a continuously updated benchmark where new models can be validated with minimal integration effort."],"supporting_citations":[{"why":"OpenEMMA, the open-source baseline whose LLaMA-3.2-11B and Qwen results LightEMMA compares against; the 47.6% and 47.7% ADE reductions are measured relative to it.","marker":"[13]"},{"why":"EMMA, the original end-to-end multimodal model whose chain-of-thought prompting and language-as-output design LightEMMA follows.","marker":"[11]"},{"why":"nuScenes, the dataset and official prediction metrics (ADE, FDE) used for all trajectory accuracy evaluation.","marker":"[19]"},{"why":"DriveVLM, whose CoT prompt design principles and dual-system ideas inform LightEMMA's prompting and its discussion of deployment acceleration.","marker":"[31]"},{"why":"Gemini model family, the pretrained model behind EMMA and two of the commercial models evaluated.","marker":"[12]"}],"fun_headline_variants":["No fine-tuning, just framework: driving error down 47%","Same weights, half the ADE: LightEMMA's modular win","Bigger reasoning models don't improve driving prediction","LightEMMA: 47% lower ADE without any retraining","Open VLM framework beats fine-tuning: ADE 2.92 to 1.53"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The claim that the trajectory improvement comes from framework design assumes the comparison to OpenEMMA is fully controlled, with identical prompts, input representation, evaluation code, and sampling, yet no component-by-component ablation is reported.","fun_headline_variants_meta":{"raw":{"variants":["No fine-tuning, just framework: driving error down 47%","Same weights, half the ADE: LightEMMA's modular win","Bigger reasoning models don't improve driving prediction","LightEMMA: 47% lower ADE without any retraining","Open VLM framework beats fine-tuning: ADE 2.92 to 1.53"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00087,"raw_usage":{"total_tokens":3754,"prompt_tokens":917,"completion_tokens":2837,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":533,"completion_tokens_details":{"reasoning_tokens":2742}},"tokens_in":533,"tokens_out":2837,"duration_ms":21555,"temperature":1.0,"reasoning_tokens":2742,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T04:45:18.589737+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take OpenEMMA's LLaMA-3.2-11B deployment and swap in only LightEMMA's exact prompt template and single-frame input, keeping weights, evaluation code, and sampling fixed; if average ADE stays near 2.92 m rather than dropping to about 1.53 m, the framework-design attribution is falsified. Conversely, running LightEMMA's full pipeline with OpenEMMA's multi-frame input and prompt would reveal which component carries the gain.","supporting_citations":[{"cited_title":"Openemma: Open-source multimodal model for end-to-end autonomous driving,","cited_arxiv_id":null,"evidence_quote":"OpenEMMA, the open-source baseline whose LLaMA-3.2-11B and Qwen results LightEMMA compares against; the 47.6% and 47.7% ADE reductions are measured relative to it."},{"cited_title":"Emma: End-to-end multimodal model for autonomous driving,","cited_arxiv_id":null,"evidence_quote":"EMMA, the original end-to-end multimodal model whose chain-of-thought prompting and language-as-output design LightEMMA follows."},{"cited_title":"nuScenes: A Multimodal Dataset for Autonomous Driving ,","cited_arxiv_id":null,"evidence_quote":"nuScenes, the dataset and official prediction metrics (ADE, FDE) used for all trajectory accuracy evaluation."},{"cited_title":"Gemini: A family of highly capable multimodal models,","cited_arxiv_id":null,"evidence_quote":"Gemini model family, the pretrained model behind EMMA and two of the commercial models evaluated."}],"review_version":1}